How to import raw data on qiime

User Support
1

Hi thanks for join me in this forum. I have a problem to import paired end raw data to qiime2 and I want to use the EMPPairedEndSequences pipeline in the Atacama tutorial, i have 2 fastq.gz forward and reverse fastq for each sample, a list of barcodes reverse and forward in .xlsx file in this format:
Sample_Name I7_Index_ID index I5_Index_ID index2
2740_1 N704 TCCTGAGC S502 CTCTCTAT
2740_1 N704 TCCTGAGC S502 CTCTCTAT

Fastq file are in this format:
88B@CE=@=A581==-;;,,;8,-;,<E,6-8++8C<,;@++++,;E,,6,:,:=++6++88CFF+4+,4,:,4+,++,59B,5,:9,:+,,<,,955@?,>,,,,,,,4,,,+,,,,,8,,+3+5+3,3,::,,5,,,3,,3,,,8,,37,314,*41,,7,,7,+2*,,7+,31++,+,45+2++++,34/33++++/1+++0<+2*+2001)/+1*+20)00-/8++1/)0**(.1))02).)0))))))((-(((),()(
@M05836:3:000000000-BP6KJ:1:1101:18741:1482 1:N:0:16
CCTACGGGAGGCAGCAGTGAGGAATATTGGTCAATGGACGCAAGTCTGAACCAGCCATGCCGCGTGCAGGATGACGGCTCTATGATTTGTAAACTTCTTTTTTACGGGGGTAAACTCTGATACTTTTATCTGTCTTTATGTACCTTTCGTATATGGATCGGCTAACTCCGTTCCAGCTGCCGCGGTTATTCGGAGGTTCCAAGCGTTATCCGGATTTATTTGGTTTATACTGTTCGTATGCGTTTTGTTAAGTTAGTTCTTAAATCCCGTTTCTTAACTCCCGCACTGCCTCTCATT
+
88BBCD=@AF5:)==;==,,;;,6C,CF9,CF@@FFE9C+8+++;EF@@-;CCFE?,6,6CCFFF7@,,:,,6,:,++76C,C<,,<C,<9,,5<,59CEC,>,5++++4@+,,,85,,,8,5,<,<,8@@,,5A,,,,,:7,3,:,3+,,:,,,,,9D7+B>+,3@>,+6,36,33,,363>8*>6,2,,62,,+243;;<92;>+>@++23:C)++++210))/077)2+.9***++1***61))((),.:++300)()((--(..548().)
@M05836:3:000000000-BP6KJ:1:1101:9032:1555 1:N:0:16
CCTACGGGGGGCAGCAGTGGGGAATATTGCACAATGGGGGAAACCCTGATCCAGCGACGCCGCGTGGAGGATGAAGGTTTTCTGTTTTTTAACTCCTGTCTTCGTGGACGTTAATTACCCTTCCCGTTCTTGTTTCCACCTCTTTCTTCTTTCCACCTGCCGCGGTTATACGTACGTTGCAAGCGTTTTCCGGAATTTCTTGGTGTTCTGCGAGCTCATGCGGTTCGGCTTGTTTGTGTTTTAAGCTCTCGGCTCATCCCCCTAACTTCCTTCTTCACTCCCGCTCTTCTGTTGTGCAGTT
ect...

Primers Illumina:
forward: CCTACGGGNGGCWGCAG
reverse: GACTACHVGGGTATCTAATCC

Please can someone help me? I'm new on bioinformatic, Thanks in advance.
Lisa

2

Sounds like you don't have EMP format (EMP format is not demultiplexed, so you have only one forward, one reverse, and one barcode file). See the manifest formats or Casava 1.8 formats in the importing tutorial to see if one of those work for you.

I hope that helps!

3

Thank you for your kind response, I did using the following commands:
qiime tools import
--type 'SampleData[PairedEndSequencesWithQuality]'
--input-path pe-64-manifest
--output-path paired-end-demux.qza
--source-format PairedEndFastqManifestPhred64

the manifest file is the follow:
sample-id,absolute-filepath,direction
24143-19,$PWD/test/24143-19/24143-19-1.fastq.gz,forward
24143-19,$PWD/test/24143-19/24143-19-2.fastq.gz,reverse
24143-14,$PWD/test/24143-14/24143-14-1.fastq.gz,forward
24143-14,$PWD/test/24143-14/24143-14-2.fastq.gz,reverse

obtaining the error message:
/home/qiime2/miniconda/envs/qiime2-2018.4/lib/python3.5/site-packages/q2_types/per_sample_sequences/_transformer.py:344: UserWarning: Importing of PHRED 64 data is slow as it is converted internally to PHRED 33. Working with the imported data will not be slower than working with PHRED 33 data.
warnings.warn(_phred64_warning)
Traceback (most recent call last):
File "/home/qiime2/miniconda/envs/qiime2-2018.4/lib/python3.5/site-packages/q2cli/tools.py", line 116, in import_data
view_type=source_format)
File "/home/qiime2/miniconda/envs/qiime2-2018.4/lib/python3.5/site-packages/qiime2/sdk/result.py", line 218, in import_data
return cls.from_view(type, view, view_type, provenance_capture)
File "/home/qiime2/miniconda/envs/qiime2-2018.4/lib/python3.5/site-packages/qiime2/sdk/result.py", line 243, in _from_view
result = transformation(view)
File "/home/qiime2/miniconda/envs/qiime2-2018.4/lib/python3.5/site-packages/qiime2/core/transform.py", line 70, in transformation
new_view = transformer(view)
File "/home/qiime2/miniconda/envs/qiime2-2018.4/lib/python3.5/site-packages/q2_types/per_sample_sequences/_transformer.py", line 346, in _9
single_end=False)
File "/home/qiime2/miniconda/envs/qiime2-2018.4/lib/python3.5/site-packages/q2_types/per_sample_sequences/_transformer.py", line 268, in _fastq_manifest_helper
absolute=True)
File "/home/qiime2/miniconda/envs/qiime2-2018.4/lib/python3.5/site-packages/q2_types/per_sample_sequences/_transformer.py", line 153, in _parse_and_validate_manifest
'%s' % path)
FileNotFoundError: A path specified in the manifest does not exist or is not accessible: /home/qiime2/Tutorials13-07-2018/TestManifest2/test/24143-19/24143-19-1.fastq.gz

An unexpected error has occurred:

A path specified in the manifest does not exist or is not accessible: /home/qiime2/Tutorials13-07-2018/TestManifest2/test/24143-19/24143-19-1.fastq.gz

See above for debug info.

5

See this post where an identical error was addressed:

Let us know the output of the command listed there and/or if that fixes your problem!

7

Ok, thank you so much It works!