Importing demultiplexed fastq.gz files

Dear Friends
I am newbie to qiime2. Now I am trying to import paired-end and already demultiplexed fastq.gz files from a sequencing company. However, I am having difficulty to import them into qiime2:
this is the command I run
(qiime2-2019.1) sewunet@sewunet-HPC:~/MyMicrobiome$ qiime tools import
–type ‘SampleData[PairedEndSequencesWithQuality]’
–input-path manifest.csv
–output-path paired-end-demux.qza
–input-format PairedEndFastqManifestPhred64
And I’m getting this error
/home/sewunet/miniconda3/envs/qiime2-2019.1/lib/python3.6/site-packages/q2_types/per_sample_sequences/_transformer.py:344: UserWarning: Importing of PHRED 64 data is slow as it is converted internally to PHRED 33. Working with the imported data will not be slower than working with PHRED 33 data.
warnings.warn(_phred64_warning)
Traceback (most recent call last):
File “/home/sewunet/miniconda3/envs/qiime2-2019.1/lib/python3.6/site-packages/q2cli/tools.py”, line 146, in import_data
view_type=input_format)
File “/home/sewunet/miniconda3/envs/qiime2-2019.1/lib/python3.6/site-packages/qiime2/sdk/result.py”, line 240, in import_data
return cls.from_view(type, view, view_type, provenance_capture)
File “/home/sewunet/miniconda3/envs/qiime2-2019.1/lib/python3.6/site-packages/qiime2/sdk/result.py”, line 265, in _from_view
result = transformation(view)
File “/home/sewunet/miniconda3/envs/qiime2-2019.1/lib/python3.6/site-packages/qiime2/core/transform.py”, line 70, in transformation
new_view = transformer(view)
File “/home/sewunet/miniconda3/envs/qiime2-2019.1/lib/python3.6/site-packages/q2_types/per_sample_sequences/_transformer.py”, line 346, in _9
single_end=False)
File “/home/sewunet/miniconda3/envs/qiime2-2019.1/lib/python3.6/site-packages/q2_types/per_sample_sequences/_transformer.py”, line 268, in _fastq_manifest_helper
absolute=True)
File “/home/sewunet/miniconda3/envs/qiime2-2019.1/lib/python3.6/site-packages/q2_types/per_sample_sequences/_transformer.py”, line 158, in _parse_and_validate_manifest
_validate_paired_end_fastq_manifest_directions(manifest)
File “/home/sewunet/miniconda3/envs/qiime2-2019.1/lib/python3.6/site-packages/q2_types/per_sample_sequences/_transformer.py”, line 211, in _validate_paired_end_fastq_manifest_directions
‘“reverse”, but observed: %s’ % direction)
ValueError: Directions can only be “forward” or “reverse”, but observed: forward

An unexpected error has occurred:

Directions can only be “forward” or “reverse”, but observed: forward

See above for debug info.

Thanks

Hi there @sewuneta! If you are trying to import paired-end data, there should be a forward and reverse file pairing for each sample --- are there entries for each file in each sample present in your manifest? Also, unrelated, but do you really have Phred 64 offset data? That is super rare these days...

Thanks a lot. I have paired-end data, 233 forward and 233 reverse sequences for each sample.
Here I have attached my manifest file for the record.manifest.csv (30.6 KB)

Dear Matthew, any suggestions?

I don’t see anything wrong with this manifest - can you try again?

Yes I did Matthew, yet now I’m getting a different error
(qiime2-2019.1) sewunet@sewunet-HPC:~/MyMicrobiome$ qiime tools import --type ‘SampleData[SequencesWithQuality]’ --input-path /home/sewunet/MyMicrobiome/manifest.csv --output-path /home/sewunet/MyMicrobiome/demux.qza --input-format PairedEndFastqManifestPhred33
Traceback (most recent call last):
File “/home/sewunet/miniconda3/envs/qiime2-2019.1/lib/python3.6/site-packages/q2cli/tools.py”, line 146, in import_data
view_type=input_format)
File “/home/sewunet/miniconda3/envs/qiime2-2019.1/lib/python3.6/site-packages/qiime2/sdk/result.py”, line 240, in import_data
return cls.from_view(type, view, view_type, provenance_capture)
File “/home/sewunet/miniconda3/envs/qiime2-2019.1/lib/python3.6/site-packages/qiime2/sdk/result.py”, line 264, in _from_view
recorder=recorder)
File “/home/sewunet/miniconda3/envs/qiime2-2019.1/lib/python3.6/site-packages/qiime2/core/transform.py”, line 59, in make_transformation
(self._view_type, other._view_type))
Exception: No transformation from <class ‘q2_types.per_sample_sequences._format.PairedEndFastqManifestPhred33’> to <class ‘q2_types.per_sample_sequences._format.SingleLanePerSampleSingleEndFastqDirFmt’>

An unexpected error has occurred:

No transformation from <class ‘q2_types.per_sample_sequences._format.PairedEndFastqManifestPhred33’> to <class ‘q2_types.per_sample_sequences._format.SingleLanePerSampleSingleEndFastqDirFmt’>

See above for debug info.

Hi there @sewuneta!

The error is because you changed your --type:

It should be

Update the command and give it a shot! :qiime2:

1 Like

Thanks in advance Matthew.
I did update the command but now I’m getting the earlier error.
I know I’m bothering you with all the nonsense but hope you won’t give up on me.
I spent months working on the tutorials but now I got my own data and couldn’t even import it.
(qiime2-2019.1) sewunet@sewunet-HPC:~/MyMicrobiome$ qiime tools import --type SampleData[PairedEndSequencesWithQuality] --input-path /home/sewunet/MyMicrobiome/manifest.csv --output-path /home/sewunet/MyMicrobiome/demux.qza --input-format PairedEndFastqManifestPhred33
Traceback (most recent call last):
File “/home/sewunet/miniconda3/envs/qiime2-2019.1/lib/python3.6/site-packages/q2cli/tools.py”, line 146, in import_data
view_type=input_format)
File “/home/sewunet/miniconda3/envs/qiime2-2019.1/lib/python3.6/site-packages/qiime2/sdk/result.py”, line 240, in import_data
return cls.from_view(type, view, view_type, provenance_capture)
File “/home/sewunet/miniconda3/envs/qiime2-2019.1/lib/python3.6/site-packages/qiime2/sdk/result.py”, line 265, in _from_view
result = transformation(view)
File “/home/sewunet/miniconda3/envs/qiime2-2019.1/lib/python3.6/site-packages/qiime2/core/transform.py”, line 70, in transformation
new_view = transformer(view)
File “/home/sewunet/miniconda3/envs/qiime2-2019.1/lib/python3.6/site-packages/q2_types/per_sample_sequences/_transformer.py”, line 338, in _8
single_end=False)
File “/home/sewunet/miniconda3/envs/qiime2-2019.1/lib/python3.6/site-packages/q2_types/per_sample_sequences/_transformer.py”, line 268, in _fastq_manifest_helper
absolute=True)
File “/home/sewunet/miniconda3/envs/qiime2-2019.1/lib/python3.6/site-packages/q2_types/per_sample_sequences/_transformer.py”, line 158, in _parse_and_validate_manifest
_validate_paired_end_fastq_manifest_directions(manifest)
File “/home/sewunet/miniconda3/envs/qiime2-2019.1/lib/python3.6/site-packages/q2_types/per_sample_sequences/_transformer.py”, line 211, in _validate_paired_end_fastq_manifest_directions
‘“reverse”, but observed: %s’ % direction)
ValueError: Directions can only be “forward” or “reverse”, but observed: reverse

An unexpected error has occurred:

Directions can only be “forward” or “reverse”, but observed: reverse

See above for debug info.

Hey there @sewuneta! I think I found the problem:

41%20PM

You have a tab after the word "reverse" on line 328 (reverse read for sample s94). Delete that and re-run and you should hopefully be good to go! :crossed_fingers:

2 Likes

Yes Yes Mattew, it worked. It finally worked.
I already imported and checked the quality score.
It is perfect and no words to thank you!!

2 Likes

This topic was automatically closed 31 days after the last reply. New replies are no longer allowed.