Error when trying to demultiplex with cutadapt

lineroager · March 30, 2020, 3:51pm

I am running QIIME2 2020.2 on a Mac using Miniconda.

The data I am handling consists of multiplexed paired-end sequences with barcodes in-line with sequences. I have two fastq.gz files with data, one with forward reads, one with reverse reads. I do not have a fastq file for the barcode sequences, but I have a metadata file in tsv format containing barcode sequences.

I am then following this tutorial, using the q2-cutadapt plugin.

I have successfully imported my data, and then I try to run the demultiplexing step like this:
qiime cutadapt demux-paired
--i-seqs multiplexed-seqs.qza
--m-forward-barcodes-file Metadata.tsv
--m-reverse-barcodes-file Metadata.tsv
--m-forward-barcodes-column Forward
--m-reverse-barcodes-column Reverse
--p-error-rate 0
--o-per-sample-sequences demultiplexed-seqs.qza
--o-untrimmed-sequences untrimmed.qza
--verbose

Unfortunately, I get the following error:
Plugin error from cutadapt:

Command '['cutadapt', '--front', 'file:/var/folders/33/kgh2579x0x9fvzpgs9z9c86c0000gr/T/tmpe7kza4xr', '--error-rate', '0.1', '--minimum-length', '1', '-o', '/var/folders/33/kgh2579x0x9fvzpgs9z9c86c0000gr/T/q2-CasavaOneEightSingleLanePerSampleDirFmt-amtz7r_q/{name}.1.fastq.gz', '--untrimmed-output', '/var/folders/33/kgh2579x0x9fvzpgs9z9c86c0000gr/T/q2-MultiplexedPairedEndBarcodeInSequenceDirFmt-7ygb7_ux/forward.fastq.gz', '--pair-adapters', '-G', 'file:/var/folders/33/kgh2579x0x9fvzpgs9z9c86c0000gr/T/tmpnyg9ou9b', '-p', '/var/folders/33/kgh2579x0x9fvzpgs9z9c86c0000gr/T/q2-CasavaOneEightSingleLanePerSampleDirFmt-amtz7r_q/{name}.2.fastq.gz', '--untrimmed-paired-output', '/var/folders/33/kgh2579x0x9fvzpgs9z9c86c0000gr/T/q2-MultiplexedPairedEndBarcodeInSequenceDirFmt-7ygb7_ux/reverse.fastq.gz', '/var/folders/33/kgh2579x0x9fvzpgs9z9c86c0000gr/T/qiime2-archive-qnqhr4fd/31231443-ebcc-4a3b-a16e-75d3b340321d/data/forward.fastq.gz', '/var/folders/33/kgh2579x0x9fvzpgs9z9c86c0000gr/T/qiime2-archive-qnqhr4fd/31231443-ebcc-4a3b-a16e-75d3b340321d/data/reverse.fastq.gz']' returned non-zero exit status 1.

See above for debug info.

I cannot really make sense of this error. What does it mean? What do I need to change?
I have also tried to change the error-rate to 0.1 with the same result.

Thank you in advance!
Line

thermokarst · March 30, 2020, 4:51pm

There should be a lot more to that --verbose error log (cutadapt is pretty chatty) - please copy and paste the entire output. Thanks!

lineroager · March 31, 2020, 12:19pm

Oh, okay, sorry about that!

This is the entire output from running the command I pasted in my previous post:

Running external command line application. This may print messages to stdout and/or stderr.
The command being run is below. This command cannot be manually re-run as it will depend on temporary files that no longer exist.

Command: cutadapt --front file:/var/folders/33/kgh2579x0x9fvzpgs9z9c86c0000gr/T/tmpe7kza4xr --error-rate 0.1 --minimum-length 1 -o /var/folders/33/kgh2579x0x9fvzpgs9z9c86c0000gr/T/q2-CasavaOneEightSingleLanePerSampleDirFmt-amtz7r_q/{name}.1.fastq.gz --untrimmed-output /var/folders/33/kgh2579x0x9fvzpgs9z9c86c0000gr/T/q2-MultiplexedPairedEndBarcodeInSequenceDirFmt-7ygb7_ux/forward.fastq.gz --pair-adapters -G file:/var/folders/33/kgh2579x0x9fvzpgs9z9c86c0000gr/T/tmpnyg9ou9b -p /var/folders/33/kgh2579x0x9fvzpgs9z9c86c0000gr/T/q2-CasavaOneEightSingleLanePerSampleDirFmt-amtz7r_q/{name}.2.fastq.gz --untrimmed-paired-output /var/folders/33/kgh2579x0x9fvzpgs9z9c86c0000gr/T/q2-MultiplexedPairedEndBarcodeInSequenceDirFmt-7ygb7_ux/reverse.fastq.gz /var/folders/33/kgh2579x0x9fvzpgs9z9c86c0000gr/T/qiime2-archive-qnqhr4fd/31231443-ebcc-4a3b-a16e-75d3b340321d/data/forward.fastq.gz /var/folders/33/kgh2579x0x9fvzpgs9z9c86c0000gr/T/qiime2-archive-qnqhr4fd/31231443-ebcc-4a3b-a16e-75d3b340321d/data/reverse.fastq.gz

This is cutadapt 2.8 with Python 3.6.7
Command line parameters: --front file:/var/folders/33/kgh2579x0x9fvzpgs9z9c86c0000gr/T/tmpe7kza4xr --error-rate 0.1 --minimum-length 1 -o /var/folders/33/kgh2579x0x9fvzpgs9z9c86c0000gr/T/q2-CasavaOneEightSingleLanePerSampleDirFmt-amtz7r_q/{name}.1.fastq.gz --untrimmed-output /var/folders/33/kgh2579x0x9fvzpgs9z9c86c0000gr/T/q2-MultiplexedPairedEndBarcodeInSequenceDirFmt-7ygb7_ux/forward.fastq.gz --pair-adapters -G file:/var/folders/33/kgh2579x0x9fvzpgs9z9c86c0000gr/T/tmpnyg9ou9b -p /var/folders/33/kgh2579x0x9fvzpgs9z9c86c0000gr/T/q2-CasavaOneEightSingleLanePerSampleDirFmt-amtz7r_q/{name}.2.fastq.gz --untrimmed-paired-output /var/folders/33/kgh2579x0x9fvzpgs9z9c86c0000gr/T/q2-MultiplexedPairedEndBarcodeInSequenceDirFmt-7ygb7_ux/reverse.fastq.gz /var/folders/33/kgh2579x0x9fvzpgs9z9c86c0000gr/T/qiime2-archive-qnqhr4fd/31231443-ebcc-4a3b-a16e-75d3b340321d/data/forward.fastq.gz /var/folders/33/kgh2579x0x9fvzpgs9z9c86c0000gr/T/qiime2-archive-qnqhr4fd/31231443-ebcc-4a3b-a16e-75d3b340321d/data/reverse.fastq.gz
Processing reads on 1 core in paired-end mode ...
cutadapt: error: Error in sequence file at unknown line: Reads are improperly paired. There are more reads in file 1 than in file 2.
Traceback (most recent call last):
  File "/Users/username/opt/miniconda3/envs/qiime2-2020.2/lib/python3.6/site-packages/q2cli/commands.py", line 328, in __call__
    results = action(**arguments)
  File "</Users/username/opt/miniconda3/envs/qiime2-2020.2/lib/python3.6/site-packages/decorator.py:decorator-gen-465>", line 2, in demux_paired
  File "/Users/username/opt/miniconda3/envs/qiime2-2020.2/lib/python3.6/site-packages/qiime2/sdk/action.py", line 245, in bound_callable
    output_types, provenance)
  File "/Users/username/opt/miniconda3/envs/qiime2-2020.2/lib/python3.6/site-packages/qiime2/sdk/action.py", line 390, in _callable_executor_
    output_views = self._callable(**view_args)
  File "/Users/username/opt/miniconda3/envs/qiime2-2020.2/lib/python3.6/site-packages/q2_cutadapt/_demux.py", line 198, in demux_paired
    mux_fmt, batch_size, minimum_length)
  File "/Users/username/opt/miniconda3/envs/qiime2-2020.2/lib/python3.6/site-packages/q2_cutadapt/_demux.py", line 162, in _demux
    run_command(cmd)
  File "/Users/username/opt/miniconda3/envs/qiime2-2020.2/lib/python3.6/site-packages/q2_cutadapt/_demux.py", line 36, in run_command
    subprocess.run(cmd, check=True)
  File "/Users/username/opt/miniconda3/envs/qiime2-2020.2/lib/python3.6/subprocess.py", line 418, in run
    output=stdout, stderr=stderr)
subprocess.CalledProcessError: Command '['cutadapt', '--front', 'file:/var/folders/33/kgh2579x0x9fvzpgs9z9c86c0000gr/T/tmpe7kza4xr', '--error-rate', '0.1', '--minimum-length', '1', '-o', '/var/folders/33/kgh2579x0x9fvzpgs9z9c86c0000gr/T/q2-CasavaOneEightSingleLanePerSampleDirFmt-amtz7r_q/{name}.1.fastq.gz', '--untrimmed-output', '/var/folders/33/kgh2579x0x9fvzpgs9z9c86c0000gr/T/q2-MultiplexedPairedEndBarcodeInSequenceDirFmt-7ygb7_ux/forward.fastq.gz', '--pair-adapters', '-G', 'file:/var/folders/33/kgh2579x0x9fvzpgs9z9c86c0000gr/T/tmpnyg9ou9b', '-p', '/var/folders/33/kgh2579x0x9fvzpgs9z9c86c0000gr/T/q2-CasavaOneEightSingleLanePerSampleDirFmt-amtz7r_q/{name}.2.fastq.gz', '--untrimmed-paired-output', '/var/folders/33/kgh2579x0x9fvzpgs9z9c86c0000gr/T/q2-MultiplexedPairedEndBarcodeInSequenceDirFmt-7ygb7_ux/reverse.fastq.gz', '/var/folders/33/kgh2579x0x9fvzpgs9z9c86c0000gr/T/qiime2-archive-qnqhr4fd/31231443-ebcc-4a3b-a16e-75d3b340321d/data/forward.fastq.gz', '/var/folders/33/kgh2579x0x9fvzpgs9z9c86c0000gr/T/qiime2-archive-qnqhr4fd/31231443-ebcc-4a3b-a16e-75d3b340321d/data/reverse.fastq.gz']' returned non-zero exit status 1.

Plugin error from cutadapt:

  Command '['cutadapt', '--front', 'file:/var/folders/33/kgh2579x0x9fvzpgs9z9c86c0000gr/T/tmpe7kza4xr', '--error-rate', '0.1', '--minimum-length', '1', '-o', '/var/folders/33/kgh2579x0x9fvzpgs9z9c86c0000gr/T/q2-CasavaOneEightSingleLanePerSampleDirFmt-amtz7r_q/{name}.1.fastq.gz', '--untrimmed-output', '/var/folders/33/kgh2579x0x9fvzpgs9z9c86c0000gr/T/q2-MultiplexedPairedEndBarcodeInSequenceDirFmt-7ygb7_ux/forward.fastq.gz', '--pair-adapters', '-G', 'file:/var/folders/33/kgh2579x0x9fvzpgs9z9c86c0000gr/T/tmpnyg9ou9b', '-p', '/var/folders/33/kgh2579x0x9fvzpgs9z9c86c0000gr/T/q2-CasavaOneEightSingleLanePerSampleDirFmt-amtz7r_q/{name}.2.fastq.gz', '--untrimmed-paired-output', '/var/folders/33/kgh2579x0x9fvzpgs9z9c86c0000gr/T/q2-MultiplexedPairedEndBarcodeInSequenceDirFmt-7ygb7_ux/reverse.fastq.gz', '/var/folders/33/kgh2579x0x9fvzpgs9z9c86c0000gr/T/qiime2-archive-qnqhr4fd/31231443-ebcc-4a3b-a16e-75d3b340321d/data/forward.fastq.gz', '/var/folders/33/kgh2579x0x9fvzpgs9z9c86c0000gr/T/qiime2-archive-qnqhr4fd/31231443-ebcc-4a3b-a16e-75d3b340321d/data/reverse.fastq.gz']' returned non-zero exit status 1.

See above for debug info.

I hope this is better

thermokarst · March 31, 2020, 2:15pm

No worries, thanks for sending along this info!

It certainly is - the error message is right near the top:

So the complaint here is that your input muxed seqs aren't the same length between number of fwd reads and number of reverse reads. We often see this happen when file transfers from sequencing center or data servers fail - the fwd reads are usually transferred first and completed, but the rev reads will be cut short due to some network failure. Double-check that your files are fully transferred, reimport, and try again!

:qiime2:

lineroager · April 1, 2020, 1:03pm

This was very helpful information - thank you so much!

I was now able to demultiplex my data and can now proceed with my analysis. Thank you so much for your help!