qiime tools import failure despite valid manifest file and fast files

[BonnieG (Bonnie)]

Hello, I am getting the following error in QIIME2-2022-2, running in conda on a Mac. The command is:

qiime tools import
--type 'SampleData[PairedEndSequencesWithQuality]'
--input-path ~/Desktop/archmanifest.csv
--output-path ~/Desktop/AKarchprelim.qza
--input-format PairedEndFastqManifestPhred33

The error I get is:
There was a problem importing /Users/bonniewaring/Desktop/archmanifest.csv:
~/Desktop/archmanifest.csv is not a(n) PairedEndFastqManifestPhred33 file:
File referenced on line 2 could not be found (~/Desktop/ArchTest01_S1_R1_001.fastq).

I have checked the following:
-manifest file is appropriately formatted, with correct column headers. There are no 'empty' rows or columns

• the file mentioned ('referenced on Line 2') has the correct path in the manifest
• the file is not corrupt (I can open and inspect in TextEdit and all looks normal)

What else could be going wrong? Thank you!

[SoilRotifer (Mike Robeson)]

Hi @BonnieG, welcome to !

@BonnieG

File referenced on line 2 could not be found (~/Desktop/ArchTest01_S1_R1_001.fastq)

This error informs us that the file paths in your manifest are relative and not absolute file paths, as outlined here.

Depending on the format of your fastq files you may need to use PairedEndFastqManifestPhred33V2 as your input format.

[BonnieG (Bonnie)]

Thanks so much for the quick response! I am still getting issues and have tested the following:

1. Changed manifest file to have absolute file paths - got the same error.

2. Changed final argument to PairedEndFastqManifestPhred33V2. This time I got a different error:

There was a problem importing /Desktop/archmanifest.csv:

/Desktop/archmanifest.csv is not a(n) PairedEndFastqManifestPhred33V2 file:

Found unrecognized ID column name 'sample-id,absolute-filepath,direction' while searching for header. The first column name in the header defines the ID column, and must be one of these values:

Case-insensitive: 'feature id', 'feature-id', 'featureid', 'id', 'sample id', 'sample-id', 'sampleid'

Case-sensitive: '#OTU ID', '#OTUID', '#Sample ID', '#SampleID', 'sample_name'

NOTE: Metadata files must contain tab-separated values.

There may be more errors present in the metadata file. To get a full report, sample/feature metadata files can be validated with Keemei: https://keemei.qiime2.org

Following this, I installed keemei and attempted to validate the manifest file. Keemei did not identify any problems with the column headers! It did, however, raise an issue with the fact that row names were duplicated (which is normal for forward/reverse reads in a manifest file. I chose the 'QIIME2 metadata' option for validation - there was no option for a manifest file specifically.) This leads me to believe that the original 'PairedEndFastqManifestPhred33' option is the correct one.

Any ideas on what to try next? Thanks so much!

[SoilRotifer (Mike Robeson)]

Hi @BonnieG,

This line in the error message:

@BonnieG

Found unrecognized ID column name 'sample-id,absolute-filepath,direction' while searching for header. The first column name in the header defines the ID column, and must be one of these values:

Tells me that you are using a comma separated file (csv) and not a tab-separated file (tsv):

That is, the commas here:
sample-id,absolute-filepath,direction

should be replace with tab characters:
sample-id absolute-filepath direction

as outlined in the documentation I linked earlier.

[BonnieG (Bonnie)]

Thanks very much and sorry for not catching this myself (tsv instead of csv for PairedEndFastqManifestPhred33V2). The manifest formatting issue appears to be solved, but I still can't import, as seen with the error below. (Note that I did validate the manifest file in Keemei and it found no errors. I've replaced the absolute file paths with ~ in the messages below, but I am using absolute file paths in the manifest document.)

Thank you!

There was a problem importing ~/Desktop/archmanifestforV2.txt:

~/Desktop/archmanifestforV2.tsv is not a(n) PairedEndFastqManifestPhred33V2 file:

Filepath on line 1 and column "forward-absolute-filepath" could not be found (~/Desktop/ArchTest01_S1_R1_001.fastq) for sample "ArchTest01".

[SoilRotifer (Mike Robeson)]

@BonnieG,

Can you confirm that the ArchTest01_S1_R1_001.fastq does indeed exist? Would you be willing to link me (e.g. via Dropbox, etc) the manifest file, and any other files, via a private message?

[BonnieG (Bonnie)]

Sorry for delayed response (I was traveling with no email access). I tried to DM you but got a message saying 'sorry, but you cannot message that user,' would you mind DM'ing me and I'll link the file? Thank you!

[BonnieG (Bonnie)]

Just in case others are looking at this thread, Mike solved the issue - there were quotes in the TSV manifest file that must have originated in the CSV file from which it was generated. Using a raw text editor revealed these; once removed, the manifest file worked!