Error with demultiplexing the pair-end fastq

Hello, I have 2 fastq files (forward and reverse) from illumina Miseq (V3 reagent kits)

Each file contain 87 samples and the barcode only in the forward sequences (start from 5' end) (some of the barcodes are 5 bp long and some of them are 6 bp long)

I performed demultiplex with qiime cutadapt demux-paired

First, I imported the two file with command below:
(I have already put the forward.fastq.qz and reverse.fastq.qz into the lib1 file)

qiime tools import --type MultiplexedPairedEndBarcodeInSequence \
--input-path /mnt/d/genome/gbs/lib1 \
--output-path /mnt/d/genome/gbs/demux/lib1.qza

Then, I use "lib1.qza" to be demultiplexed

qiime cutadapt demux-paired --i-seqs /mnt/d/genome/gbs/demux/lib1.qza \
--m-forward-barcodes-file /mnt/d/fm_lib1_keyfile.tsv \
--m-forward-barcodes-column Barcode \
--p-error-rate 0.2 \
--o-per-sample-sequences /mnt/d/genome/gbs/demux2/demux.qza \
--o-untrimmed-sequences /mnt/d/genome/gbs/demux3/untrimmed.qza \
--verbose

And here is the error message:

Traceback (most recent call last):
  File "/home/HYC/miniconda3/envs/qiime2-2019.4/lib/python3.6/site-packages/qiime2/plugin/model/file_format.py", line 24, in validate
    self._validate_(level)
  File "/home/HYC/miniconda3/envs/qiime2-2019.4/lib/python3.6/site-packages/q2_types/per_sample_sequences/_format.py", line 279, in _validate_
    self._check_n_records(record_count_map[level])
  File "/home/HYC/miniconda3/envs/qiime2-2019.4/lib/python3.6/site-packages/q2_types/per_sample_sequences/_format.py", line 250, in _check_n_records
    % (i * 4 + 1))
qiime2.core.exceptions.ValidationError: Missing sequence for record beginning on line 5

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/HYC/miniconda3/envs/qiime2-2019.4/lib/python3.6/site-packages/q2cli/commands.py", line 311, in __call__
    results = action(**arguments)
  File "</home/HYC/miniconda3/envs/qiime2-2019.4/lib/python3.6/site-packages/decorator.py:decorator-gen-461>", line 2, in demux_paired
  File "/home/HYC/miniconda3/envs/qiime2-2019.4/lib/python3.6/site-packages/qiime2/sdk/action.py", line 231, in bound_callable
    output_types, provenance)
  File "/home/HYC/miniconda3/envs/qiime2-2019.4/lib/python3.6/site-packages/qiime2/sdk/action.py", line 365, in _callable_executor_
    output_views = self._callable(**view_args)
  File "/home/HYC/miniconda3/envs/qiime2-2019.4/lib/python3.6/site-packages/q2_cutadapt/_demux.py", line 193, in demux_paired
    mux_fmt, batch_size)
  File "/home/HYC/miniconda3/envs/qiime2-2019.4/lib/python3.6/site-packages/q2_cutadapt/_demux.py", line 168, in _demux
    muxed = len(list(per_sample_sequences.sequences.iter_views(FastqGzFormat)))
  File "/home/HYC/miniconda3/envs/qiime2-2019.4/lib/python3.6/site-packages/qiime2/plugin/model/directory_format.py", line 141, in iter_views
    yield fp.relative_to(root), transformation(fp)
  File "/home/HYC/miniconda3/envs/qiime2-2019.4/lib/python3.6/site-packages/qiime2/core/transform.py", line 68, in transformation
    self.validate(view)
  File "/home/HYC/miniconda3/envs/qiime2-2019.4/lib/python3.6/site-packages/qiime2/core/transform.py", line 143, in validate
    view.validate('min')
  File "/home/HYC/miniconda3/envs/qiime2-2019.4/lib/python3.6/site-packages/qiime2/plugin/model/file_format.py", line 29, in validate
    ) from e
qiime2.core.exceptions.ValidationError: /tmp/q2-CasavaOneEightSingleLanePerSampleDirFmt-yo0xkhtx/GBS04_CGCCG_L001_R1_001.fastq.gz is not a(n) FastqGzFormat file:

Missing sequence for record beginning on line 5

Plugin error from cutadapt:

  /tmp/q2-CasavaOneEightSingleLanePerSampleDirFmt-yo0xkhtx/GBS04_CGCCG_L001_R1_001.fastq.gz is not a(n) FastqGzFormat file:

  Missing sequence for record beginning on line 5

See above for debug info.

I'm sure that GBS04 sample is in the forward.fastq.gz file (I used custom R script to check this file)

Below is part of mine forward fastq:

@M02289:58:000000000-CHJ9K:1:1101:19124:1729 1:N:0:1
CAGCTTGCAGCTCCTCGACACGTTTGAGCCACAGCTGCACCTGAGGATCACAGACGTTCAGCTTGTCCTCCTCCAC
+
@[email protected]@,-CCCECE,,@,@,CFF,,,;C,C,,;E,@,CCF,,,,,CC,C,,,;+CFD8,<CE,CEFEEEEEE8C
@M02289:58:000000000-CHJ9K:1:1101:17247:1767 1:N:0:1
TCTGATGCAGCACCGCAGCGCGGAGCGGCACCTCGAGGGCGTGGCCGCGAAGGTGGCGGAGATGGTGAAGAAGGGG
+
[email protected]<[email protected]:[email protected][email protected][email protected][email protected][email protected],,[email protected][email protected][email protected]+++++C,,C+++++B,,C,,,,,,,,++
@M02289:58:000000000-CHJ9K:1:1101:12661:1768 1:N:0:1
ACCGATGCAGCAGTGCCTGGTGGCTATGGCAACGCCAACGCGCTGGCCTCGTGGAACAGCAGCAGCACCGCCAGCC
+
[email protected]@[email protected], C-,C,CCE9,C,,CE,C,,;,[email protected]@++C,[email protected]++8<CC,C,,,,:,,@,,:,,:,CC+:>++C,
@M02289:58:000000000-CHJ9K:1:1101:11012:1773 1:N:0:1
TGGTTCGTTCTTATTACCCTTCTGAATGTCACGCTGATTATTTTGACTTTGAGCGTATCGAGGCTCTTAAACCTGC
+
A--AA-:@,;CF9FF9CCCFFEF9,,C,E<,C,CF,+CF9FEEF,,;CEF,,,[email protected],[email protected],,,+66:CF,,,96C,9
@M02289:58:000000000-CHJ9K:1:1101:15532:1777 1:N:0:1
ATTCAATCATGACTTCGTGATAAAAGATTGAGTGTGAGGTTATAACGCCGAAGCGGTAAAAATTTTAATTTTTGCC

And the reverse one:

@M02289:58:000000000-CHJ9K:1:1101:19124:1729 2:N:0:1
CGGGAGAGCCTGCGGGCCGTCAGGGCGACGGTCAGGGCAGGGGTCACCGTGGAGGAGGACAAGCTGAACGTCTGTT
+
8---,+-,:CE,[email protected][email protected],,,,8++8++8B,,,,8,,[email protected],CC,C,+,,,,,,,:,,,:C,,,:,:BE+C,
@M02289:58:000000000-CHJ9K:1:1101:17247:1767 2:N:0:1
CGTACTTCTCCCCCTTCTTCACCATCTCCGCCACCTTCGCGGCCACGCCCTCGAGGTGCCGCTCCGCGCTGCGGTT
+
[email protected]@[email protected],@[email protected]@:[email protected]@FFF+,,+B,:[email protected]@@C,8+=>+4++B+
@M02289:58:000000000-CHJ9K:1:1101:12661:1768 2:N:0:1
CGGTGAGCCCGCGGGAAGGCAGGCTCAGCGCCACCACTCTGGTTGCGGCCGCCGCACGTCACTCCCTCCCAGCCGC
+
8--8,,-;[email protected]+++++,,;,,+6;@,,;[email protected]+8C,C,:C,,:,,:[email protected]+6+8+::,85A<=?A??,,:4+8
@M02289:58:000000000-CHJ9K:1:1101:11012:1773 2:N:0:1
CGCTGATTATTTTGACTTTGAGCGTATCAAGCCTCAATAGCAGGTTTAATAGCCTCGATACGCTCAAAGTCAAAAT
+
8--8,+CE,EFFG,,6CEF,,,,[email protected],@<,,,6:[email protected],,C,,,,,,;CF,,,,,6;C<,,;,@,[email protected]:+,,,<9,,,,5
@M02289:58:000000000-CHJ9K:1:1101:15532:1777 2:N:0:1
CTAGGTTTTCTTCTTATTATTTTAATCATTTTTTAATTATGTCTTTAAATAAAATTCTTAAACATTTTTAAACTCC 

And here is my barcode:fm_lib1_keyfile.tsv (1.2 KB)

Thanks!

Hi there @11113! We have seen this issue before in the past, this happens because cutadapt will keep 0-length sequences post trimming and/or demuxing. In 2019.4 we added a new minimum_length parameter to trim-single and trim-paired, which sets the min length to 1 by default (instead of 0). We didn’t implement this flag on the demux methods in that plugin though (although, we are just wrapping up the addition for inclusion in the 2019.7 release). In the meantime you could run cutadapt on its own (setting -m 1 when you do that).

Relevant Github issue: https://github.com/qiime2/q2-cutadapt/issues/17

1 Like

This topic was automatically closed 31 days after the last reply. New replies are no longer allowed.