Issues importing demultiplexed .fastq files

[kris (Kristopher Abdullah)]

Hi, I have been having trouble using an import_manifest.csv to import my data to qiime2. I receive the error "import_manifest.csv is not PairedEndFastqManifestPhred33". Here is the code that I used

qiime tools import --type 'SampleData[PairedEndSequencesWithQuality]' --input-path import_manifest.csv --output-path demux-paired-end.qza --source-format PairedEndFastqManifestPhred33

and the import manifest as viewed from vi

The only difference between myself and my colleagues who didn't have any issues with the same process is that my files are .fastq whereas everyone else seems to have .fastq.gz
Thanks,
Kris

[Nicholas_Bokulich]

Hi @kris,
Could you please attach your actual manifest file here? Thanks!

[kris (Kristopher Abdullah)]

Hi @Nicholas_Bokulich
Sure thing import_manifest.csv (1.6 KB)
Thanks again for the assist.

[Nicholas_Bokulich]

Hi @kris,
Two different things that might be causing issues here:

1. “sample-id” in the first line should not be capitalized
2. it looks like your file contains windows-style line breaks, which could be causing the issue. If fixing “sample-id” does not fix things, try opening up your file in a text editor to replace the line breaks.

I hope that helps!

[kris (Kristopher Abdullah)]

Hi @Nicholas_Bokulich,
That did the trick, thank you once more for the help.

[Cybele_C]

Hello! I'm having the same issue (see screenshot)

I've compared formats with the Importing Data tutorial several times but can't see what is wrong. The manifest file is also attached
ManifestRun2.csv (1.2 KB)

I tried installing the new version, qiime2-2018.8, and this occurred:

[ebolyen (Evan Bolyen)]

Hey @Cybele_C,

Your manifest looks good, but in your second command you just have your first command “quoted”, you’ll notice there’s two: > > on each line, if you get rid of the > the command should work.

[Cybele_C]

Hi Evan - good point about the quote symbols. However, I’m still having issues:
(qiime2-2018.6) Cybeles-MacBook-Pro:~ cybelecollins$ qiime tools import \

> --type ‘SampleData[PairedEndSequencesWithQuality]’ \

> --input-path ManifestRun2.csv \

> --output-path paired-end-demux2.qza \

> --source-format PairedEndFastqManifestPhred33

Usage: qiime tools import [OPTIONS]

Error: Invalid value for “–input-path”: Path “ManifestRun2.csv” does not exist.

(qiime2-2018.6) Cybeles-MacBook-Pro:~ cybelecollins$ qiime tools import \

> --type ‘SampleData[PairedEndSequencesWithQuality]’ \

> --input-path casava-18-paired-end-demultiplexed \

> --input-format CasavaOneEightSingleLanePerSampleDirFmt \

> --output-path demux-paired-end.qza

Error: no such option: --input-format

(qiime2-2018.6) Cybeles-MacBook-Pro:~ cybelecollins$

[Nicholas_Bokulich]

Hi @Cybele_C,
Update to the latest release of QIIME 2 (2018.8) and your importing command should work fine.

I hope that helps!

[Cybele_C]

Hi - I tried the new release (today, and a couple days ago) and got the same error message:

(qiime2-2018.8) Cybeles-MacBook-Pro:~ cybelecollins$ qiime tools import \

> --type ‘SampleData[PairedEndSequencesWithQuality]’ \

> --input-path ManifestRun2.csv \

> --output-path paired-end-demux2.qza \

> --input-format PairedEndFastqManifestPhred33

Usage: qiime tools import [OPTIONS]

Error: Invalid value for “–input-path”: Path “ManifestRun2.csv” does not exist.

Where should the manifest file be? I have tried moving it different places. I also tried the Cassava import method (as in my last post).

Thanks very much

[Nicholas_Bokulich]

Error: Invalid value for “–input-path”: Path “ManifestRun2.csv” does not exist.

That is a different error from what you reported above — in this case, QIIME 2 can’t find the file! It looks like you are running that command from within the wrong directory (in the original error you gave above, it looks like your current working directory was paired-end-sequences. Either change into that directory or use an absolute path for that file to make sure QIIME 2 is looking in the right place.

[Cybele_C]

Thanks. I appreciate the ideas.

A little confused on your response; it was one of the same issues I'd reported before: two failed attempts in one post. One was the same as what I posted last, except that before, I posted the 2018.6 result with a source-format; I had tried 2018.8 with input-format for the later one, and did that again above. Those weren’t the problem, but --input-path. Both had the same results - no file found.

The manifest file is in the paired-end directory, though I copied it to other locations after the error appeared.

Thanks again for the idea about the absolute file path; I will try that.

[Nicholas_Bokulich]

A little confused on your response; it was one of the same issues I’d reported before: two failed attempts in one post.

I am seeing 4 distinct errors reported by you — sorry if I missed this error above and it was indeed duplicated.

1. … not a(n) PairedEndFastqManifestPhred33 file
2. Missing option “–type” (quoting error corrected by @ebolyen’s advice)
3. no such option (–input-format is an argument name in 2018.8 but not 2018.6)
4. Invalid value for “–input-path”: Path “ManifestRun2.csv” does not exist. (at least not in your current working directory)

Try the absolute path and let us know if that solves your issue. Good luck!

[thermokarst]

Hey @Cybele_C, just want to echo @Nicholas_Bokulich’s advice. As well, this is the same “suite” of problems you ran into previously:

Importing raw data - directory issue

Hey there @Cybele_C! I think you have everything you need right there in that error message! Path "Manfest1" does not exist. Based on the screenshot you provided, I agree with that error! I don’t see a file called “Manifest” — but I do see one called “Manifest.csv”. Once you fix that part of your command, you will bump into another problem related to the paths listed in your manifest file: /Users/cybelecollins/qiime2-skate3/paired-end-sequences/CC1_S1_L001_R1_001.fastq.gz /Users/cybel…

Your most recent error is virtually identical to the error you provided in this other thread.

In this case, it looks like the file ManifestRun2.csv doesn’t exist in the folder that your terminal is currently working in. As @Nicholas_Bokulich mentioned, you can change working directories, or specify absolute filepaths. I just want to point out that this is the same “type” of error as before - previously you ran into issues because you left off the end of the file’s name, while this time, you are running into a problem because you are missing the leading part of the file’s name (in this case, the path to the file). To paraphrase @ebolyen - computers are very naive, you have to be exceptionally explicit when telling them what to do, in this case, you need to take care to provide precise filenames.

Thanks for being patient here, I know these kind of general computing errors can be frustrating, but we will get through it together!

[Cybele_C]

I appreciate your patience. It is very hard to see these things when new to command line interface. Thank you so much!