Fastq manifest format error

User Support
,
1

I have the exact same problem and my path is correct. Any help?

> qiime tools import --type SampleData[PairedEndSequencesWithQuality] --input-path BEIZymo-manifest.csv --output-path BEIZymo-demux.qza --source-format PairedEndFastqManifestPhred33
> Traceback (most recent call last):
>   File "/Users/wangj50/miniconda2/envs/qiime2-2017.6/bin/qiime", line 6, in <module>
>     sys.exit(q2cli.__main__.qiime())
>   File "/Users/wangj50/miniconda2/envs/qiime2-2017.6/lib/python3.5/site-packages/click/core.py", line 722, in __call__
>     return self.main(*args, **kwargs)
>   File "/Users/wangj50/miniconda2/envs/qiime2-2017.6/lib/python3.5/site-packages/click/core.py", line 697, in main
>     rv = self.invoke(ctx)
>   File "/Users/wangj50/miniconda2/envs/qiime2-2017.6/lib/python3.5/site-packages/click/core.py", line 1066, in invoke
>     return _process_result(sub_ctx.command.invoke(sub_ctx))
>   File "/Users/wangj50/miniconda2/envs/qiime2-2017.6/lib/python3.5/site-packages/click/core.py", line 1066, in invoke
>     return _process_result(sub_ctx.command.invoke(sub_ctx))
>   File "/Users/wangj50/miniconda2/envs/qiime2-2017.6/lib/python3.5/site-packages/click/core.py", line 895, in invoke
>     return ctx.invoke(self.callback, **ctx.params)
>   File "/Users/wangj50/miniconda2/envs/qiime2-2017.6/lib/python3.5/site-packages/click/core.py", line 535, in invoke
>     return callback(*args, **kwargs)
>   File "/Users/wangj50/miniconda2/envs/qiime2-2017.6/lib/python3.5/site-packages/q2cli/tools.py", line 111, in import_data
>     view_type=source_format)
>   File "/Users/wangj50/miniconda2/envs/qiime2-2017.6/lib/python3.5/site-packages/qiime2/sdk/result.py", line 192, in import_data
>     return cls._from_view(type_, view, view_type, provenance_capture)
>   File "/Users/wangj50/miniconda2/envs/qiime2-2017.6/lib/python3.5/site-packages/qiime2/sdk/result.py", line 217, in _from_view
>     result = transformation(view)
>   File "/Users/wangj50/miniconda2/envs/qiime2-2017.6/lib/python3.5/site-packages/qiime2/core/transform.py", line 57, in transformation
>     self.validate(view)
>   File "/Users/wangj50/miniconda2/envs/qiime2-2017.6/lib/python3.5/site-packages/qiime2/core/transform.py", line 131, in validate
>     view.validate()
>   File "/Users/wangj50/miniconda2/envs/qiime2-2017.6/lib/python3.5/site-packages/qiime2/plugin/model/file_format.py", line 31, in validate
>     % (self.path, self.__class__.__name__))
> ValueError: InPath('BEIZymo-manifest.csv') is not formatted as a PairedEndFastqManifestPhred33 file.

Manifest file:

sample-id,absolute-filepath,direction
Scap_BEIMock,/Users/wangj50/Desktop/16SSCap/2-Data/SCap/16Scap_BEI_Mock_CGGTTGTT-GGAAGGAT__R1_001.fastq.gz,forward
Scap_BEIMock,/Users/wangj50/Desktop/16SSCap/2-Data/SCap/16Scap_BEI_Mock_CGGTTGTT-GGAAGGAT__R2_001.fastq.gz,reverse
Scap_Zymo,/Users/wangj50/Desktop/16SSCap/2-Data/SCap/16Scap_Zymo_Mock_TCCGTATG-CGTGTGTA__R1_001.fastq.gz,forward
Scap_Zymo,/Users/wangj50/Desktop/16SSCap/2-Data/SCap/16Scap_Zymo_Mock_TCCGTATG-CGTGTGTA__R2_001.fastq.gz,reverse

In addition, I wonder what is the difference between PairedEndFastqManifestPhred33 file with those CasavaOneEightSingleLanePerSampleDirFmt.

Thanks!

2

For some reason, changing the file name to 16Scap_BEI_Mock_CGGTTGTT-GGAAGGAT_R1.fastq.gz and others to similar style works.

4

Hey @wangj50,

I suspect the issue is just that those filepaths did not exist, but we’ve had enough issues with underscore characters I just want to double check.

  1. did you change the filepath on disk or just in the manifest?

  2. If you were to run ls on each of those filepaths, do they all come back with that same path (meaning they exist) or do some of them say “no such file or directory” ?

Example:

ls /Users/wangj50/Desktop/16SSCap/2-Data/SCap/16Scap_BEI_Mock_CGGTTGTT-GGAAGGAT__R1_001.fastq.gz

We’ve had a lot of problems with underscores, and we really want to support them, so any help you can provide would be awesome!

6

Hello @ebolyen，

Thanks! The filepaths themselves were legitimate. I changed both the filename (filepath) on disk and in the manifest to make it work. I am not sure if the input for forward has to end with R1 before .fastq.gz and for reverse R2 before .fastq.gz, or the double underscore __ in the file name before R1 messed things up.
Let me know if I could further help.
Thanks!

7

I'm following the traceback in the original post, and there just doesn't seem to be any code that is going to care about the presence or absence of the files, that seems to happen later (and you would have gotten a different error).

What kind of program did you use to write the manifest? Did you use something else the second time? I kind of suspect that if you were to change it back to double-underscore things would still be working.

Ideally with the fastq manifest formats none of that matters because you've set up columns which indicate those details meaning QIIME 2 doesn't need to infer anything from the filenames. (Other import formats will use the filename and expect a particular order, but the manifest is our "escape-hatch" for when your filenames aren't what we expect).

8

Hi Evan,
Thanks! I always used SublimeText3 to write text file and checked the line endings to be unix format. I am not sure what the problem is.
I’ll try to do some test when I have the chance and let you know the results.
Thanks!
Jincheng

1 Like
9

Thanks @wangj50, that would be awesome!

10

This topic was automatically closed 31 days after the last reply. New replies are no longer allowed.