Plug-in error from cutadat

[emescioglu (Esra Mescioglu)]

Hello!

I’m trying to use cutadapt to demultiplex pair-end reads.

Step 1: I have imported my data (worked):

#16S_rawdata is a folder that contains two files (forward.fastq.gz and reverse.fastq.gz)

qiime tools import
–type MultiplexedPairedEndBarcodeInSequence --input-path 16S_rawdata --output-path …/16S_processing/multiplexed-seqs.qza

Step 2 (doesn’t work):
qiime cutadapt demux-paired
–i-seqs …/16S_processing/multiplexed-seqs.qza
–m-forward-barcodes-file …/MappingFiles/Mesocosm_16S_Metadata.tsv
–m-forward-barcodes-column BarcodeSequence
–p-error-rate 0
–o-per-sample-sequences …/16S_processing/demultiplexed-seqs.qza
–o-untrimmed-sequences …/16S_processing/untrimmed.qza
–verbose

I got a very long error:

Traceback (most recent call last):
File “/Users/esramescioglu/miniconda3/envs/qiime2-2018.2/lib/python3.5/site-packages/xopen-0.3.2-py3.5.egg/xopen.py”, line 241, in xopen
File “/Users/esramescioglu/miniconda3/envs/qiime2-2018.2/lib/python3.5/site-packages/xopen-0.3.2-py3.5.egg/xopen.py”, line 66, in init
OSError: [Errno 24] Too many open files: ‘/dev/null’

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File “/Users/esramescioglu/miniconda3/envs/qiime2-2018.2/bin/cutadapt”, line 11, in
File “/Users/esramescioglu/miniconda3/envs/qiime2-2018.2/lib/python3.5/site-packages/cutadapt/main.py”, line 742, in main
File “/Users/esramescioglu/miniconda3/envs/qiime2-2018.2/lib/python3.5/site-packages/cutadapt/pipeline.py”, line 173, in run
File “/Users/esramescioglu/miniconda3/envs/qiime2-2018.2/lib/python3.5/site-packages/cutadapt/pipeline.py”, line 292, in process_reads
File “/Users/esramescioglu/miniconda3/envs/qiime2-2018.2/lib/python3.5/site-packages/cutadapt/filters.py”, line 268, in call
File “/Users/esramescioglu/miniconda3/envs/qiime2-2018.2/lib/python3.5/site-packages/cutadapt/filters.py”, line 223, in write
File “/Users/esramescioglu/miniconda3/envs/qiime2-2018.2/lib/python3.5/site-packages/cutadapt/seqio.py”, line 698, in open
File “/Users/esramescioglu/miniconda3/envs/qiime2-2018.2/lib/python3.5/site-packages/cutadapt/seqio.py”, line 796, in _seqopen1
File “/Users/esramescioglu/miniconda3/envs/qiime2-2018.2/lib/python3.5/site-packages/cutadapt/seqio.py”, line 471, in init
File “/Users/esramescioglu/miniconda3/envs/qiime2-2018.2/lib/python3.5/site-packages/xopen-0.3.2-py3.5.egg/xopen.py”, line 243, in xopen
File “/Users/esramescioglu/miniconda3/envs/qiime2-2018.2/lib/python3.5/gzip.py”, line 53, in open
File “/Users/esramescioglu/miniconda3/envs/qiime2-2018.2/lib/python3.5/gzip.py”, line 163, in init
OSError: [Errno 24] Too many open files: ‘/var/folders/gc/t0lly6tx217bq0p81wrgmtp80000gn/T/q2-CasavaOneEightSingleLanePerSampleDirFmt-nxwgnl34/T3E1.2.fastq.gz’

• 1. Plugin error from cutadapt:
• 2. Command ‘[‘cutadapt’, ‘–front’, ‘file:/var/folders/gc/t0lly6tx217bq0p81wrgmtp80000gn/T/tmpz6j_472e’, ‘–error-rate’, ‘0.0’, ‘-o’, ‘/var/folders/gc/t0lly6tx217bq0p81wrgmtp80000gn/T/q2-CasavaOneEightSingleLanePerSampleDirFmt-nxwgnl34/{name}.1.fastq.gz’, ‘–untrimmed-output’, ‘/var/folders/gc/t0lly6tx217bq0p81wrgmtp80000gn/T/q2-MultiplexedPairedEndBarcodeInSequenceDirFmt-8yognpbv/forward.fastq.gz’, ‘-p’, ‘/var/folders/gc/t0lly6tx217bq0p81wrgmtp80000gn/T/q2-CasavaOneEightSingleLanePerSampleDirFmt-nxwgnl34/{name}.2.fastq.gz’, ‘–untrimmed-paired-output’, ‘/var/folders/gc/t0lly6tx217bq0p81wrgmtp80000gn/T/q2-MultiplexedPairedEndBarcodeInSequenceDirFmt-8yognpbv/reverse.fastq.gz’, ‘/var/folders/gc/t0lly6tx217bq0p81wrgmtp80000gn/T/qiime2-archive-zl664b22/b3090139-ed26-4224-842c-0de04b3f0ebe/data/forward.fastq.gz’, ‘/var/folders/gc/t0lly6tx217bq0p81wrgmtp80000gn/T/qiime2-archive-zl664b22/b3090139-ed26-4224-842c-0de04b3f0ebe/data/reverse.fastq.gz’]’ returned non-zero exit status 1

I’m not sure if this has something to do with cutadapt, or that I simply have “too many files open” and my computer is unhappy with me… I saw a similar error that someone else had posted about, but there wasn’t an answer as to why he was seeing the message.

Let me know if this is clear or if I should provide any other information to make it so

Thanks!
Esra

[thermokarst]

Yikes! @emescioglu, I think you found a bug . This certainly isn't your fault, in fact, something similar happened when we originally wrote the demux emp-single and demux emp-paired functionality - this is apparently the nature of demuxing a bunch of samples all at once. I have opened a bug report to track the fix for this, and we will update this thread when a fix lands. In the meantime, maybe you can try demuxing your reads in smaller chunks? For example, split your metadata file into two to four subdivisions, then demux using those. You would need to export and reimport the result demux files to get them all back into one dataset, but, this could get you moving in the right direction.

@emescioglu

I saw a similar error that someone else had posted about, but there wasn’t an answer as to why he was seeing the message.

Yours is the first report I have seen - would you mind linking us to the thread you had in mind?

@emescioglu

Let me know if this is clear

Crystal clear - thanks, this made our job easier!

Thanks!

[emescioglu (Esra Mescioglu)]

Hey!

I googled around a bit and found that my computer (mac) wasn’t allowed to run as many files as it needed to for cutadapt. I found advice to run the command below to change the limit of files open, and then it worked

$ ulimit -n 2048

Here is the link to the previous post

Thanks for the fast reply!

Esra

[thermokarst]

Hi @emescioglu! Thanks for sharing, but a word of caution - adjusting the ulimit like that can lead to some unexpected (and often undesirable) consequences. Those limits exist for a reason, so please exercise caution. Thanks!

[emescioglu (Esra Mescioglu)]

eeeekster, I will change it back ! Thanks