Plugin error from cutadapt

Technical Support
1

Greetings,
I keep getting the same error report. At this point, I have done the following with the data:

qiime tools import --type MultiplexedPairedEndBarcodeInSequence --input-path MultiplexedPairedEndBarcodeInSequence --output-path MultiplexedPairedEndBarcodeInSequence.qza

Within the input, I have forward.fastq.gz and reverse.fastq.gz files.
The output produced MultiplexedPairedEndBarcodeInSequence.qza

I get the error for the next command:
qiime cutadapt demux-paired --i-seqs MultiplexedPairedEndBarcodeInSequence.qza --m-forward-barcodes-file MilkNSwab_Metadata.tsv --m-forward-barcodes-column BarcodeSequence --o-per-sample-sequences PairedEndSequencesWithQuality --o-untrimmed-sequences MultiplexedPairedEndBarcodeInSequenceUntrimmed

The error is:
Plugin error from cutadapt:

Command ‘[‘cutadapt’, ‘–front’, ‘file:/tmp/tmp16tlnvc7’, ‘–error-rate’, ‘0.1’, ‘-o’, ‘/tmp/q2-CasavaOneEightSingleLanePerSampleDirFmt-jsr621nq/{name}.1.fastq.gz’, ‘–untrimmed-output’, ‘/tmp/q2-MultiplexedPairedEndBarcodeInSequenceDirFmt-z6_1rgue/forward.fastq.gz’, ‘-p’, ‘/tmp/q2-CasavaOneEightSingleLanePerSampleDirFmt-jsr621nq/{name}.2.fastq.gz’, ‘–untrimmed-paired-output’, ‘/tmp/q2-MultiplexedPairedEndBarcodeInSequenceDirFmt-z6_1rgue/reverse.fastq.gz’, ‘/tmp/qiime2-archive-ly2htst8/65d2cb65-c12e-4337-9f64-5aad6ba7987d/data/forward.fastq.gz’, ‘/tmp/qiime2-archive-ly2htst8/65d2cb65-c12e-4337-9f64-5aad6ba7987d/data/reverse.fastq.gz’]’ returned non-zero exit status 1.

Do you have any idea what I might be missing?
Thanks,
Mike

2

Hi @MMski!

Looks like you missed some required items listed in our topic template:

  • What version of QIIME 2 are you running?
  • What is the complete error message? This means either attaching the error log, or running the command with the --verbose flag.

Thanks! :qiime2:

3

I’m running QIIME2 2019.1.

Here’s what --verbose returns:

Plugin error from cutadapt:

Command ‘[‘cutadapt’, ‘–front’, ‘file:/tmp/tmp16tlnvc7’, ‘–error-rate’, ‘0.1’, ‘-o’, ‘/tmp/q2-CasavaOneEightSingleLanePerSampleDirFmt-jsr621nq/{name}.1.fastq.gz’, ‘–untrimmed-output’, ‘/tmp/q2-MultiplexedPairedEndBarcodeInSequenceDirFmt-z6_1rgue/forward.fastq.gz’, ‘-p’, ‘/tmp/q2-CasavaOneEightSingleLanePerSampleDirFmt-jsr621nq/{name}.2.fastq.gz’, ‘–untrimmed-paired-output’, ‘/tmp/q2-MultiplexedPairedEndBarcodeInSequenceDirFmt-z6_1rgue/reverse.fastq.gz’, ‘/tmp/qiime2-archive-ly2htst8/65d2cb65-c12e-4337-9f64-5aad6ba7987d/data/forward.fastq.gz’, ‘/tmp/qiime2-archive-ly2htst8/65d2cb65-c12e-4337-9f64-5aad6ba7987d/data/reverse.fastq.gz’]’ returned non-zero exit status 1.

4

Sorry…
Verbose…

Running external command line application. This may print messages to stdout and/or stderr.
The command being run is below. This command cannot be manually re-run as it will depend on temporary files that no longer exist.

Command: cutadapt --front file:/tmp/tmpwun0_xy0 --error-rate 0.1 -o /tmp/q2-CasavaOneEightSingleLanePerSampleDirFmt-vjmx9ib5/{name}.1.fastq.gz --untrimmed-output /tmp/q2-MultiplexedPairedEndBarcodeInSequenceDirFmt-qegwaj3r/forward.fastq.gz -p /tmp/q2-CasavaOneEightSingleLanePerSampleDirFmt-vjmx9ib5/{name}.2.fastq.gz --untrimmed-paired-output /tmp/q2-MultiplexedPairedEndBarcodeInSequenceDirFmt-qegwaj3r/reverse.fastq.gz /tmp/qiime2-archive-70k6ct6n/65d2cb65-c12e-4337-9f64-5aad6ba7987d/data/forward.fastq.gz /tmp/qiime2-archive-70k6ct6n/65d2cb65-c12e-4337-9f64-5aad6ba7987d/data/reverse.fastq.gz

This is cutadapt 1.18 with Python 3.6.7
Command line parameters: --front file:/tmp/tmpwun0_xy0 --error-rate 0.1 -o /tmp/q2-CasavaOneEightSingleLanePerSampleDirFmt-vjmx9ib5/{name}.1.fastq.gz --untrimmed-output /tmp/q2-MultiplexedPairedEndBarcodeInSequenceDirFmt-qegwaj3r/forward.fastq.gz -p /tmp/q2-CasavaOneEightSingleLanePerSampleDirFmt-vjmx9ib5/{name}.2.fastq.gz --untrimmed-paired-output /tmp/q2-MultiplexedPairedEndBarcodeInSequenceDirFmt-qegwaj3r/reverse.fastq.gz /tmp/qiime2-archive-70k6ct6n/65d2cb65-c12e-4337-9f64-5aad6ba7987d/data/forward.fastq.gz /tmp/qiime2-archive-70k6ct6n/65d2cb65-c12e-4337-9f64-5aad6ba7987d/data/reverse.fastq.gz
Processing reads on 1 core in paired-end legacy mode …
Traceback (most recent call last):
File “/home/qiime2/miniconda/envs/qiime2-2019.1/lib/python3.6/site-packages/xopen.py”, line 243, in _open_gz
File “/home/qiime2/miniconda/envs/qiime2-2019.1/lib/python3.6/site-packages/xopen.py”, line 79, in init
OSError: [Errno 24] Too many open files: ‘/dev/null’

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File “/home/qiime2/miniconda/envs/qiime2-2019.1/bin/cutadapt”, line 11, in
File “/home/qiime2/miniconda/envs/qiime2-2019.1/lib/python3.6/site-packages/cutadapt/main.py”, line 798, in main
File “/home/qiime2/miniconda/envs/qiime2-2019.1/lib/python3.6/site-packages/cutadapt/pipeline.py”, line 188, in run
File “/home/qiime2/miniconda/envs/qiime2-2019.1/lib/python3.6/site-packages/cutadapt/pipeline.py”, line 329, in process_reads
File “/home/qiime2/miniconda/envs/qiime2-2019.1/lib/python3.6/site-packages/cutadapt/filters.py”, line 299, in call
File “/home/qiime2/miniconda/envs/qiime2-2019.1/lib/python3.6/site-packages/cutadapt/filters.py”, line 250, in call
File “/home/qiime2/miniconda/envs/qiime2-2019.1/lib/python3.6/site-packages/cutadapt/seqio.py”, line 695, in open
File “/home/qiime2/miniconda/envs/qiime2-2019.1/lib/python3.6/site-packages/cutadapt/seqio.py”, line 793, in _seqopen1
File “/home/qiime2/miniconda/envs/qiime2-2019.1/lib/python3.6/site-packages/cutadapt/seqio.py”, line 468, in init
File “/home/qiime2/miniconda/envs/qiime2-2019.1/lib/python3.6/site-packages/xopen.py”, line 294, in xopen
File “/home/qiime2/miniconda/envs/qiime2-2019.1/lib/python3.6/site-packages/xopen.py”, line 245, in _open_gz
File “/home/qiime2/miniconda/envs/qiime2-2019.1/lib/python3.6/gzip.py”, line 53, in open
File “/home/qiime2/miniconda/envs/qiime2-2019.1/lib/python3.6/gzip.py”, line 163, in init
OSError: [Errno 24] Too many open files: ‘/tmp/q2-CasavaOneEightSingleLanePerSampleDirFmt-vjmx9ib5/748.2.fastq.gz’
Traceback (most recent call last):
File “/home/qiime2/miniconda/envs/qiime2-2019.1/lib/python3.6/site-packages/q2cli/commands.py”, line 274, in call
results = action(**arguments)
File “</home/qiime2/miniconda/envs/qiime2-2019.1/lib/python3.6/site-packages/decorator.py:decorator-gen-452>”, line 2, in demux_paired
File “/home/qiime2/miniconda/envs/qiime2-2019.1/lib/python3.6/site-packages/qiime2/sdk/action.py”, line 231, in bound_callable
output_types, provenance)
File “/home/qiime2/miniconda/envs/qiime2-2019.1/lib/python3.6/site-packages/qiime2/sdk/action.py”, line 365, in callable_executor
output_views = self._callable(**view_args)
File “/home/qiime2/miniconda/envs/qiime2-2019.1/lib/python3.6/site-packages/q2_cutadapt/_demux.py”, line 136, in demux_paired
return _demux(seqs, forward_barcodes, error_rate, untrimmed)
File “/home/qiime2/miniconda/envs/qiime2-2019.1/lib/python3.6/site-packages/q2_cutadapt/_demux.py”, line 109, in _demux
run_command(cmd)
File “/home/qiime2/miniconda/envs/qiime2-2019.1/lib/python3.6/site-packages/q2_cutadapt/_demux.py”, line 33, in run_command
subprocess.run(cmd, check=True)
File “/home/qiime2/miniconda/envs/qiime2-2019.1/lib/python3.6/subprocess.py”, line 418, in run
output=stdout, stderr=stderr)
subprocess.CalledProcessError: Command ‘[‘cutadapt’, ‘–front’, ‘file:/tmp/tmpwun0_xy0’, ‘–error-rate’, ‘0.1’, ‘-o’, ‘/tmp/q2-CasavaOneEightSingleLanePerSampleDirFmt-vjmx9ib5/{name}.1.fastq.gz’, ‘–untrimmed-output’, ‘/tmp/q2-MultiplexedPairedEndBarcodeInSequenceDirFmt-qegwaj3r/forward.fastq.gz’, ‘-p’, ‘/tmp/q2-CasavaOneEightSingleLanePerSampleDirFmt-vjmx9ib5/{name}.2.fastq.gz’, ‘–untrimmed-paired-output’, ‘/tmp/q2-MultiplexedPairedEndBarcodeInSequenceDirFmt-qegwaj3r/reverse.fastq.gz’, ‘/tmp/qiime2-archive-70k6ct6n/65d2cb65-c12e-4337-9f64-5aad6ba7987d/data/forward.fastq.gz’, ‘/tmp/qiime2-archive-70k6ct6n/65d2cb65-c12e-4337-9f64-5aad6ba7987d/data/reverse.fastq.gz’]’ returned non-zero exit status 1.

1 Like
5

Ah, that is an existing bug. You could try splitting your input metadata into 2-4 subsets, demux each subset, then export each demuxed set and reimport together. Sorry, that is a bit of a pain!
:qiime2:

6

Matthew, thank you for this information. I split the metadata file in half. So far the first half has been successful. Running the second half now. When these are completed, are there instructions for reimporting files together? Thanks, Mike

7

Yep! https://docs.qiime2.org/2019.1/tutorials/importing/

8

Matthew,
After splitting the metatadata files for the demux-paired, to restore the demux data as one I put the two .qza files (PairedEndSequencesWithQuality) into a a folder named “PairedEndDemuxSeq” and tried the following code to import the two files:

qiime tools import --type SampleData[PairedEndSeuqnecesWithQuality] --input-path PairedEndDemuxSeq --output-path CombinedPairedEndSequencesWithQuality.qza

I received the following message:
There was a problem importing PairedEndDemuxSeq:

Missing one or more files for SingleLanePerSamplePairedEndFastqDirFmt: ‘.+_.+_L[0-9][0-9][0-9]_R[12]_001\.fastq\.gz’

Am I missing something?
At this point, I have two files:
MilkPairedEndSequencesWithQuality.qza
SwabPairedEndSequencesWithQuality.qza
These represent demultiplexed paired end data from what is now two metadata files (milk, swab). I also have the complete metadata file.

Questions:

  1. After splitting the metadata file and demultiplexing, should I rejoin these two .qza files? If so, how do I do this?
  2. After splitting the metadata file and demultiplexing, should I not rejoin these two .qza files and instead proceed to trimming and denoising, and then join representative sequences? If so, how do I do join the representative sequences?
  3. At one point can I bring together the data from the two metadata files representing data from the same MiSeq run?

I guess I’m just trying to get my data back to one qza file after working around that bug. Thank you for any information.

Mike

10

Ah, you need to export first.

qiime tools export --input-path MilkPairedEndSequencesWithQuality.qza --output-path exported
qiime tools export --input-path SwabPairedEndSequencesWithQuality.qza --output-path exported
rm exported/MANIFEST exported/metadata.yml
qiime tools import --type 'SampleData[PairedEndSequencesWithQuality]' --input-format CasavaOneEightSingleLanePerSampleDirFmt --input-path exported --output-path CombinedPairedEndSequencesWithQuality.qza

The other option is to proceed to quality-control steps with the two separate SampleData[PairedEndSequencesWithQuality] inputs. This will produce two FeatureTable[Frequency] and two FeatureData[Sequence] outputs, they can be merged following examples listed in the FMT Tutorial.

12

Thanks… merging files at FeatureTable and FeatureData as with the FMT Tutorial seems to be working fine.

1 Like