Problems with q2-cutadapt multiplexed paired end sequences without sequence ID lines or quality scores

ropolifroni · August 14, 2018, 10:12pm

Hi everyone;
I have sequences in two files (R1 and R2) that I assume as forward and reverse senses. I followed the q2-cutadapt tutorial in Demultiplexing and Trimming Adapters from Reads with q2-cutadapt.
I tried with:
qiime tools import \

--type MultiplexedPairedEndBarcodeInSequence
--input-path foward.fastq.gz
--input-path reverse.fastq.gz
--output-path multiplexed-seqs.qza
Usage: qiime tools import [OPTIONS]

Error: Option --input-path was specified multiple times in the command.

Then I tried with forward file only as a single end barcode in sequence and I have error again:

$ qiime tools import \

--type MultiplexedSingleEndBarcodeInSequence
--input-path forward.fastq.gz
--output-path multiplexed-seqs.qza

There was a problem importing forward.fastq.gz:

forward.fastq.gz is not a(n) FastqGzFormat file:

Header on line 5 is not FASTQ, records may be misaligned

I checked the file and didn't find nothing. I don't realize where is the mistake.

Could you give me an idea what can I do?

Thank a lot.

Mehrbod_Estaki · August 15, 2018, 10:03pm

Hi @ropolifroni,

The first error you receive during your paired-end import is because you are providing 2 lines of --input-path when the script is expecting one.

See this note at the bottom of the link you posted:

Regarding paired-end reads

The import format for paired-end reads with the barcodes still in the sequence is MultiplexedPairedEndBarcodeInSequence - this format expects two files in a directory ( forward.fastq.gz and reverse.fastq.gz ).

So you need to create a directory that contains your forward.fastq.qz, and reverse.fastq.gz files and refer to this folder's path in your instead in your --input-path parameter.

As for your second error I'm not quite sure what is happening. Could you try running this command and post the output of it:

zcat forward.fastq.gz | head

Finally, keep in mind one another point from that tutorial:

Demultiplexing currently only works if the barcodes are in the forward reads — we plan to support dual-indexing strategies in a future release of QIIME 2.

So if you have barcodes in your forward AND reverse reads (dual indexed) you will not be able to use cutadapt to demultiplex them. You would have to do this outside of qiime2 then import back here. You can do this in qiime1 or some other software of your choice.

Keep us posted!

ropolifroni · August 16, 2018, 3:45pm

Hi Mehrbod Estaki for your advices.
I've created a directory where I copied R1 and R2 sequences. Then I run the script that you said. I run the comand in qiime2 and the result was:

$ zcat forward.fastq.gz | head
NAGTCTCAGTGCAGCCGCCGCGGTAATACGGAGGGTGCAAGCGTTGTTCGGAATTACTGGGCGTAAAGCGCGCGTAGGTGGCTTTTTAAGTCAGATGTGAAAGTCCACGGCTCAACCGTGGAGGTGCATTTGAAACTGGGAAGCTTGAGTATTGGAGGGAGTAGTGGAATTCCCGGTGTAGAGGTGAAATTCGTAGATATCGGGAGGAATACCAGTGGCGAAGGCGACTACCTGGCCAAATACTGACACTGAGGTGCGAAAGCGTGGGGAGCAAACAGGATTAGATACCCGGGTAGTCC
NAGTGTGAGTGCCAGCAGCCGCGGTAATACAGAGGTCTCGAGCGTTAATCGGAGTCACTGGGCTTAAAGGGTGCGTAGGCGGCGCTGCAAGTGCCTTGTGAAATCCCATCGCTCAACGATGGAAATGCTTGGTAGACTGTAGTGCTTGAGGTTGGTAGGGGCTGTCGGAACTCTAGGTGGAGCGGTGAAATGCGTAGATATCTAGAGGAACGCCAAAGGTGAAGACAGACAGCTGGGCCAATTCTGACGCTGAGGCACGAAAGCGTGGGGAGCAAACGGGATTAGATACCCGCGTAGTCC
NAGTGTGAGTGCCAGCAGCCGCGGTAATACAGAGGTCTCGAGCGTTAATCGGAGTCACTGGGCTTAAAGGGTGCGTAGGCGGCGCTATAAGTGCCTTGTGAAATCCCATCGCTCAACGATGGAATTGCTTGGTAGACTGTAGTGCTTGAGGTTGGTAGGGGCTGTCGGAACTCTAGGTGGAGCGGTGAAATGCGTAGATATCTAGAGGAACGCCAAAGGTGAAGACAGACAGCTGGGCCAATTCTGACGCTGAGGCACGAAAGCGTGGGGAGCAAACGGGATTAGATACCCGCGTAGTCCC
GAGTGTGAGTGCCAGCCGCCGCGGTAATACAGAGGTCTCGAGCGTTAATCGGAGTCACTGGGCTTAAAGGGTGCGTAGGCGGCGCTACAAGTGCCTTGTGAAATCCCATCGCTCAACGATGGAATTGCTTGGTAGACTGTAGTGCTTGAGGTTGGTAGGGGCTGTCGGAACTCTAGGTGGAGCGGTGAAATGCGTAGATATCTAGAGGAACGCCAAAGGTGAAGACAGACAGCTGGGCCAATTCTGACGCTGAGGCACGAAAGCGTGGGGAGCAAACGGGATTAGATACCCGCGTAGTCC
GAGTAGTGGTGCCAGCCGCCGCGGTAATACAGAGGGGGCTAGCGTTGTTCGGAATTACTGGGCGTAAAGCGCACGTAGGCGGATCGGAAAGTTGGGGGTGAAATCCCGGGGCTCAACCCCGGAACTGCCTTCAAAACTCCCGGTCTTGAGTTCGAGAGAGGTGAGTGGAATTCCGAGTGTAGAGGTGAAATTCGTAGATATTCGGAGGAACACCAGTGGCGAAGGCGGCTCACTGGCTCGATACTGACGCTGAGGTGCGAAAGTGTGGGGAGCAAACAGGATTAGATACCCGGGTAGTCC
GAGTTCACGTGCCAGCAGCCGCGGTAATACGAACTGCGCAAACGTTATTCGGAATCACTGGGCTTACAGGGTGCGTAGGCGGTCCACCAGGTCAGATGTGAAATCCCACAGCTCAACTGGGGAACTGCATTTGAAACCGGCGGACTTGAGTAGACTAGAGGTACACGGAACTTCAGGTGGAGCGGTGAAATGTGTTGAGATCTGAAGGAACACCAGAGGCGAAAGCGGTGTACTGGAGTCTGTCTGACGCTGAGGCACGAAAGCTAGGGGAGCGAACGGGATTAGATACCCGTGTAGTCC
GAGTCACTGTGCCAGCAGCCGCGGTAATACGGAGGGGGCTAGCGTTGTTCGGAATTACTGGGCGTAAAGCGCACGTAGGCGGCTATCCAAGTTGGGGGTGAAATCCCTGGGCTTAACCCAGGAACTGCCTTCAAAACTGGATGGCTAGAGTCCGAGAGAGGTGAGTGGAATTTCCAGTGTAGAGGTGAAATTCGTAGATATTGGAAAGAACACCAGTGGCGAAGGCGGCTCACTGGCTCGGTACTGACGCTGAGGTGCGACAGCGTGGGGAGCAAACAGGATTAGATACCCGCGTAGTCC
GAGTCACTGTGCCAGCAGCCGCGGTAATACGGAGGGGGCTAGCGTTAATCGGAATTACTGGGCGTAAAGCGCGCGTAGGTGGTTTGTTAAGTTGAATGTGAAAGCCCCGGGCTCAACCTGGGAACTGCATCCAAAACTGGCAAGCTAGAGTATGGCAGAGGGTGGTGGAATTTCCTGTGTAGCGGTGAAATGCGTAGATATAGGAAGGAACACCAGTGGCGAAGGCGACCACCTGGGCTAATACTGACACTGAGGTGCGAAAGCGTGGGGAGCAAACAGGATTAGATACCCCGGTAGTCC
GAGTCAGAGTGCCAGCAGCCGCGGTAATACGGAGGGGGCTAGCGTTGTTCGGAATTACTGGGCGTAAAGCGCACGTAGGCGGATCGGAAAGTTGGGGGTGAAATCCCGGGGCTCAACCCCGGAACTGCCTTCAAAACTCCCGGTCTTGAGTTCGAGAGAGGTGAGTGGAATTCCGAGTGTAGAGGTGAAATTCGTAGATATTCGGAGGAACACCAGTGGCGAAGGCGGCTCACTGGCTCGATACTGACGCTGAGGTGCGAAAGTGTGGGGAGCAAACAGGATTAGATACCCGCGTAGTCCC
GAGTAGTGGTGCCAGCAGCCGCGGTAATACGAAGGGTGCAAGCGTTAATCGGAATTACTGGGCGTAAAGCGCGCGTAGGTGGTTTGTTAAGTTGAATGTGAAAGCCCCGGGCTCAACCTGGGAACTGCATCCAAAACTGGCAAGCTAGAGTATGGCAGAGGGTGGTGGAATTTCCTGTGTAGCGGTGAAATGCGTAGATATAGGAAGGAACACCAGTGGCGAAGGCGACCACCTGGGCTAATACTGACACTGAGGTGCGAAAGCGTGGGGAGCAAACAGGATTAGATACCCCCGTAGTCC

I'll be waiting your answer.
Thanks in advance.

Ro

Mehrbod_Estaki · August 16, 2018, 4:07pm

Hmm, this certainly doesn't look like what I expected. Could you also run the same as above for your other file, the R2, whatever its called now.

ropolifroni · August 16, 2018, 4:20pm

Here is the result:

zcat reverse.fastq.gz | head
zcat reverse.fastq.gz | head
GAGTAGACGTGCCAGCCGCCGCGGTAATACGAAGGGTGCAAGCGTTAATCGGAATTACTGGGCGTAAAGCGCGCGTAGGTGGTTTGTTAAGTTGAATGTGAAAGCCCCGGGCTCAACCTGGGAACTGCATCCATAACTGGCAAGCTAGAGTATGGCAGAGGGTGGTGGAATTTCCTGTGTAGCGGTGAAATGCGTAGATATAGGAAGGAACACCAGTGGCGAAGGCGACCACCTGGGCTAATACTGCCACTGAGGTGCGAAAGCGTGGGGAGCAAACAGGATTAGATATCCGTGTTGTCCC
GAGTAGACGTGCCAGCAGCCGCGGTAATACGAAGGGTGCAAGCGTTACTCGGAATTACTGGGCGTAAAGCGTGCGTAGGTGGTGGTTTAAGTCCGTTGTGAAAGCCCTGGGCTCAACCTGGGAATTGCAGTGGATACTGGATCACTCGAGTGTGGTAGAGGATAGTGGAATTTCCGGTGTAGCAGTGAAATGCGTAGAGATCGGAAGGAACATCTGTGGCGAAGGCGACTATCTGGGCCAACACTGACACCGAGGCACGCAAGCGTGGGGAGCCAACAGGATTAGCAACCCGGTTAGTCAC
GAGTAGTGGTGCCAGCCGCCGCGGTAATACGAAGGGTGCAAGCGTTAATCGGAATTACTGGGCGTAAAGCGCACGTAGGCGGATCGGAAAGTTGGGGGTGAAATCCCGGGGCTCAACCCCGGAACTGCCTTCAAAACTCCCGGTCTTGAGTTCGAGAGAGGTGAGTGGAATTCCGAGTGTAGAGGTGAAATTCGTAGATATTCGGAGGAACACCAGTGGCGAAGGCGGCTCACTGGCTCGATACTGACGCTGAGGTGCGAAAGTGTGGGGAGCAAAGAGGATTAGATACCCCGGGAGTCCC
GAGTAGACGTGCCAGCCGCCGCGGTAATACGTAGGGTGCGAGCGTTGTCCGGAATTACTGGGCGTAAAGAGCTCGTAGGCGGTTTGTCACGTCGTCTGTGAAATCCCTCGGCTTAACCGGGGGCGTGCAGGCGATACGGGCAGACTTGAGTACTACAGGGGAGACTGGAATTCCTGGTGTAGCGGTGAAATGCGCAGATATCAGGAGGAACACCGGTGGCGAAGGTGGGTCTCTGTGTAGTAACTGACGCTGAGGAGGGAAAGCATGGGTAGCGAACGGGATTAGGTATCCCGGTGGTCCC
AGTGCCAGCAGCCGCGGTAATACGAACTGCGCAAACGTTATTCGGAATCACTGGGCTTACAGGGTGCGTAGGCGGTCCACCAGGTCAGATGTGAAATCCCACAGCTCAACTGTGGAACTGCATTTGAAACCGGCGGACTTGAGTAGACTAGAGGTACACGGAACTTCAGGTGGAGCGGTGAAATGTGTTGAGATCTGAAGGAACACCAGAGGCGAAAGCGGTGTACTGGAGTCTGTCTGACGCTGAGGCACGAAAGCTAGGGGAGCGAACGGGATTAGATACCCCTGTAGTACAGAACGGG
GAGTCACTGTGCCAGCAGCCGCGGTAATACGGAGGGGGCTAGCGTTGTTCGGAATTACTGGGCGTAAAGCGCACGTAGGCGGATCGGAAAGTTGGGGGTGAAATCCCGGGGCTCAACCCCGGAACTGCCTTCAAAACTCCCGGTCTTGAGTTCGAGAGAGGTGAGTGGAATTCCGAGGGTAGAGGTGAAATTCGTAGATATTCGGAGGAACACCAGTGGCGAAGGCGGCTCACTGGCTCGATACTGCCGCTGAGGTGCGAAAGTGTGGGGAGCAAACAGGATTAGAGACCCCTGCAGTCCG
GAGTAGTGGTGCCAGCAGCCGCGGTAATACGGAGGGGGCTAGCGTTGTTCGGAATTACCGGGCGTAAAGCGCACGTAGGCGGATCGGAAAGTTGGGGGTGAAATCCCGGGGCTCAACCCCGGAACTGCCTTCAAAACTCCCGGTCTTGAGTTCGAGAGAGGTGAGTGGAATTCCGAGTGTAGAGGTGAACTTCGTAGATATTCGGAGGAACACCAGTGGCGAAGGCGGCTCACTGGCTCGATCCTGCCGCTGAGGTGCGAAAGGGTGGGGGGCAAACCGGCTCAGATACCCTCGTAGGCCC
GAGTCACTGTGCCAGCCGCCGCGGTAATACGGAGGGAGCTAGCGTTGTTCGGAATTACTGGGCGTAAAGCGCACGTAGGCGGCTTTTCAAGTCAGGGGTGAAATCCCGGGGCTCAACCCCGGAACTGCCCTTGAAACTGGATGGCTAGAATACTGGAGAGGTGAGTGGAATTCCGAGTGTAGAGGTGAAATTCGTAGATATTCGGAAGAACACCAGTGGCGAAGGCGACTCACTGGACCGTTATTGACGCTGAGGTGCGAAAGCGTGGGGAGCAAACAGGAATAGATCCCCAGGTAGGCCC
GAGTCAGAGTGCCAGCAGCCGCGGTAATACGGAGGGGGCTAGCGTTGTTCGGAATTACTGGGCGTAAAGCGCACGTAGGCGGCTATCCAAGTTGGGGGTGAAATCCCTGGGCTTAACCCAGGAACTGCCTTCAAAACTGGATGGCTAGAGTCCGAGAGAGGTGAGTGGAATTTCCAGTGTAGAGGTGAAATTCGTAGATATTGGAAAGAACACCAGTGGCGAAGGCGGCTCACTGGCTCGGTACTGACGCTGAGGTGCGACAGCGGGGGGAGCAAACAGGATTAGATACCCGGGTAGTCCC
GAGTCACTGTGCCAGCCGCCGCGGTAATACGAACTGCGCAAACGTTATTCGGAATCACTGGGCTTACAGGGTGCGTAGGCGGTCCACCAGGTCAGATGTGAAATCCCACAGCTCAACTGTGGAACTGCATTTGAAACCGGCGGACTTGAGTAGACTAGAGGTACACGGAACTTCAGGTGGAGCGGTGAAATGTGTTGAGATCTGAAGGAACACCAGAGGCGAAAGCGGTGTACTGGAGTCTGTCTGACGCTGAGGCACGAAAGCTAGGGGAGCGAACGGGATTAGATCCCCCGGTAGTCCC

What do you see wrong?

Mehrbod_Estaki · August 16, 2018, 5:02pm

HI @ropolifroni,
The format in which your files are in are not what we usually see. For example, a typical fastq file would look something like this:

@M02501:63:000000000-BT3BH:1:1101:14311:1491 1:N:0: CGTACTAG+TATCCTCT
GTGCCAGCAGCCGCGGTAACACGATTAACCCAAGTCAATAGAAGCCGGCGTAAAGAGTGTTTTGGATCACCCCCTCCCCAATAAAGCTAAAACTCACCTGAGTTGTAAAAAACTCCAGTTGACACAAAATAGACTACGAAAGTGGCTTTAACATATCTGAACACACAATAGCTAAGACCCAAACTGGGATTAGAAACCCTTGTAGTCC
+
CCCCCGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGCFFGGGGGGGGGGGEFGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGG&lt;FEGGGGGGGGGFGGGGGGGGGGGGGGGGGGGGGGGGFGCDGGGGGDFGGGGGDGFGGGFGFGEGGCFGGGGGFGFGGFFFFFFFGF5FGGFCEG;,5=

Where there are 4 lines associated with each read. 1=header, 2=sequences, 3=filler/blank, 4=quality scores. Your reads seem to be purely sequences without quality scores in 1 line format. Are you sure there are no other files associated with these 2 files you have? Something else that could map these files to individual samples or barcodes that could correspond with these?
Some additional questions: How many samples are meant to be represented in these reads? What sequencing platform are these from? Has there been any other types of pre-processing or quality control done to these reads prior to this point, either by your or the sequencing facility? In particular, can you locate your barcodes in these lines you posted? Anything that could help us demultiplex these...because as it is I can't see a way to do that without that info.

ropolifroni · August 16, 2018, 5:59pm

I have 17 samples that I want to analyze in 3 goups or experiments (ExpA= 4 samples; ExpB= 8 samples and ExpC= 5 samples). Now I'd like to work with ExpA samples (4 samples).
The barcodes of these 4 samples are:
GAGTAGAC
GAGTAGTG
GAGTCACT
GAGTCAGA
I get them from the mapping.file provided by the sequence service.
The platform were came from the sequences is Illumina MiSeq and they are paired, so I have 2 files with the sequences from the 17 samples!!! That is why I want to cut the 4 samples from ExpA.
SAM1-17_S5_L001_R1_001.fastq and SAM1-17_S5_L001_R2_001.fastq are 2.1 Gb each. I get this 2 files from Illumina Basespace.
I extracted the ExpA sequences from the SAM..... R1 and SAM.....R2 files using the barcodes named before.
AHHHH!!! Now I'm realize!!! The extracts aren't in .fastq format!! Are only sequences... Sorry...

How I can get the barcode. gz file from the SAM....R1 and SAM....R2 files, in order to can go ahead?
I tried the Qiime1 comand:
extract_barcodes.py -f inseqs_R1.fastq -r inseqs_R2.fastq -c barcode_paired_end --bc1_len 6 --bc2_len 6 -o processed_seqs

And the results were:

Traceback (most recent call last):
File "/home/rosana/miniconda3/envs/qiime1/bin/extract_barcodes.py", line 4, in
import('pkg_resources').run_script('qiime==1.9.1', 'extract_barcodes.py')
File "/home/rosana/miniconda3/envs/qiime1/lib/python2.7/site-packages/pkg_resources/init.py", line 658, in run_script
self.require(requires)[0].run_script(script_name, ns)
File "/home/rosana/miniconda3/envs/qiime1/lib/python2.7/site-packages/pkg_resources/init.py", line 1438, in run_script
exec(code, namespace, namespace)
File "/home/rosana/miniconda3/envs/qiime1/lib/python2.7/site-packages/qiime-1.9.1-py2.7.egg-info/scripts/extract_barcodes.py", line 12, in
from skbio.util import create_dir
File "/home/rosana/miniconda3/envs/qiime1/lib/python2.7/site-packages/skbio/init.py", line 15, in
import skbio.io
File "/home/rosana/miniconda3/envs/qiime1/lib/python2.7/site-packages/skbio/io/init.py", line 309, in
import_module('skbio.io.clustal')
File "/home/rosana/miniconda3/envs/qiime1/lib/python2.7/importlib/init.py", line 37, in import_module
import(name)
File "/home/rosana/miniconda3/envs/qiime1/lib/python2.7/site-packages/skbio/io/clustal.py", line 123, in
from skbio.alignment import Alignment
File "/home/rosana/miniconda3/envs/qiime1/lib/python2.7/site-packages/skbio/alignment/init.py", line 230, in
from ._alignment import Alignment, SequenceCollection, StockholmAlignment
File "/home/rosana/miniconda3/envs/qiime1/lib/python2.7/site-packages/skbio/alignment/_alignment.py", line 21, in
from skbio.stats.distance import DistanceMatrix
File "/home/rosana/miniconda3/envs/qiime1/lib/python2.7/site-packages/skbio/stats/init.py", line 45, in
from ._subsample import subsample, subsample_counts, isubsample
File "/home/rosana/miniconda3/envs/qiime1/lib/python2.7/site-packages/skbio/stats/_subsample.py", line 22, in
from .__subsample import _subsample_counts_without_replacement
File "init.pxd", line 155, in init skbio.stats.__subsample (skbio/stats/__subsample.c:3964)
ValueError: numpy.dtype has the wrong size, try recompiling

What can I do in order to demultiplex the 17 samples, but could work in 3 independent experiments of 4, 8 and 5 samples each?
I'll appreciate very much your help.

THANKS A LOT

Ro

ropolifroni · August 16, 2018, 7:56pm

I resolved the issue with an external software. Now I have forward.fastq.gz, reverse.fastq.gz and barcodes.fastq.gz files in order to process my data following the Atacama soil tutorial.

Thanks a lot.

Ro

Nicholas_Bokulich · August 16, 2018, 8:29pm

Hi @ropolifroni,
Glad you were able to convert your data into a usable format!

Would you mind posting the steps you used to convert your data? Just in case future users run into the same problems...

Thanks!

ropolifroni · August 16, 2018, 8:44pm

Hi Nicholas,
I used the free software FASTQProcessor (FASTQ Processor) (I not sure that I can share this http
Was pretty easy, but it ran in windows.
But my issue didn't finish, because I get the 3 files (forward.fastq.gz, reverse.fastq.gz and barecodes.fastq.gz) for the 17 samples, and not for only 4 samples from ExpA.
Nonetheless, I ran the next command for the 17 samples:
qiime demux emp-paired
--m-barcodes-file sample-metadata.tsv
--m-barcodes-column BarcodeSequence
--i-seqs emp-paired-end-sequences.qza
--o-per-sample-sequences demux
--p-rev-comp-mapping-barcodes

The terminal give me an error:
Plugin error from demux:

No sequences were mapped to samples. Check that your barcodes are in the correct orientation (see the rev_comp_barcodes and/or rev_comp_mapping_barcodes options).

Debug info has been saved to /tmp/qiime2-q2cli-err-hcc9svrn.log

Here I live the error file content:
Traceback (most recent call last):
File "/home/rosana/miniconda3/envs/qiime2-2018.6/lib/python3.5/site-packages/q2cli/commands.py", line 274, in call
results = action(**arguments)
File "", line 2, in emp_paired
File "/home/rosana/miniconda3/envs/qiime2-2018.6/lib/python3.5/site-packages/qiime2/sdk/action.py", line 232, in bound_callable
output_types, provenance)
File "/home/rosana/miniconda3/envs/qiime2-2018.6/lib/python3.5/site-packages/qiime2/sdk/action.py", line 367, in callable_executor
output_views = self._callable(**view_args)
File "/home/rosana/miniconda3/envs/qiime2-2018.6/lib/python3.5/site-packages/q2_demux/_demux.py", line 378, in emp_paired
raise ValueError('No sequences were mapped to samples. Check that '
ValueError: No sequences were mapped to samples. Check that your barcodes are in the correct orientation (see the rev_comp_barcodes and/or rev_comp_mapping_barcodes options).

Could you tell me what is wrong?

Thanks a lot.

Ro

Nicholas_Bokulich · August 20, 2018, 3:18pm

Hi @ropolifroni,
See this part of the error message:

Make sure your forward and reverse barcodes are also corresponding to the correct read files.

I hope that helps!

system · September 20, 2018, 9:26pm

This topic was automatically closed 31 days after the last reply. New replies are no longer allowed.