QIIME 2 demux emp using all available file descriptors

Thank you @gregcaporaso!

I have started running one MiSeq run on my Macbook pro to make sure the code is running ok before I use the university’s cluster for my 6 runs.

I succesfully imported my fastq.gz files, however when demultiplexing I got the following error:

OSError: [Errno 24] Too many open files: ‘/var/folders/sn/92jxwrcd3qn6xp_y7c5rfdkc0000gn/T/q2-SingleLanePerSampleSingleEndFastqDirFmt-kff_ph7v/1025Cplus18LR_233_L001_R1_001.fastq.gz’

Any thoughts on how to work around it?

Thank you!

In case you want to see my code:

(qiime2) dhcp-morris-2243:Ganda ErikaGanda$ qiime tools import --type RawSequences --input-path QIIME2mmct/ --output-path raw-sequences1.qza

(qiime2) dhcp-morris-2243:Ganda ErikaGanda$ qiime demux emp --i-seqs raw-sequences1.qza --m-barcodes-file sample-metadata1.tsv --m-barcodes-category BarcodeSequence --o-per-sample-sequences demux

Traceback (most recent call last):
File “/Users/ErikaGanda/miniconda3/envs/qiime2/bin/qiime”, line 6, in
sys.exit(q2cli.main.qiime())
File “/Users/ErikaGanda/miniconda3/envs/qiime2/lib/python3.5/site-packages/click/core.py”, line 716, in call
return self.main(*args, **kwargs)
File “/Users/ErikaGanda/miniconda3/envs/qiime2/lib/python3.5/site-packages/click/core.py”, line 696, in main
rv = self.invoke(ctx)
File “/Users/ErikaGanda/miniconda3/envs/qiime2/lib/python3.5/site-packages/click/core.py”, line 1060, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File “/Users/ErikaGanda/miniconda3/envs/qiime2/lib/python3.5/site-packages/click/core.py”, line 1060, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File “/Users/ErikaGanda/miniconda3/envs/qiime2/lib/python3.5/site-packages/click/core.py”, line 889, in invoke
return ctx.invoke(self.callback, **ctx.params)
File “/Users/ErikaGanda/miniconda3/envs/qiime2/lib/python3.5/site-packages/click/core.py”, line 534, in invoke
return callback(*args, **kwargs)
File “/Users/ErikaGanda/miniconda3/envs/qiime2/lib/python3.5/site-packages/q2cli-0.0.6-py3.5.egg/q2cli/commands.py”, line 210, in call
File “”, line 2, in emp
File “/Users/ErikaGanda/miniconda3/envs/qiime2/lib/python3.5/site-packages/qiime-2.0.6-py3.5.egg/qiime/core/callable.py”, line 227, in callable_wrapper
output_types, provenance)
File “/Users/ErikaGanda/miniconda3/envs/qiime2/lib/python3.5/site-packages/qiime-2.0.6-py3.5.egg/qiime/core/callable.py”, line 328, in callable_executor
output_views = callable(**view_args)
File “/Users/ErikaGanda/miniconda3/envs/qiime2/lib/python3.5/site-packages/q2_demux-0.0.6-py3.5.egg/q2_demux/_demux.py”, line 196, in emp
File “/Users/ErikaGanda/miniconda3/envs/qiime2/lib/python3.5/gzip.py”, line 53, in open
binary_file = GzipFile(filename, gz_mode, compresslevel)
File “/Users/ErikaGanda/miniconda3/envs/qiime2/lib/python3.5/gzip.py”, line 163, in init
fileobj = self.myfileobj = builtins.open(filename, mode or ‘rb’)
OSError: [Errno 24] Too many open files: ‘/var/folders/sn/92jxwrcd3qn6xp_y7c5rfdkc0000gn/T/q2-SingleLanePerSampleSingleEndFastqDirFmt-lwvz6hdj/1025Cplus18LR_233_L001_R1_001.fastq.gz’

Hi @ErikaGanda! The error message (OSError: [Errno 24] Too many open files) indicates that your Macbook ran out of available file descriptors (a finite resource). You should be able to increase that by running the following commands:

a) verify current setting for available file descriptors:

$ ulimit -a

On my system I see:

-t: cpu time (seconds)              unlimited
-f: file size (blocks)              unlimited
-d: data seg size (kbytes)          unlimited
-s: stack size (kbytes)             8192
-c: core file size (blocks)         0
-v: address space (kbytes)          unlimited
-l: locked-in-memory size (kbytes)  unlimited
-u: processes                       709
-n: file descriptors                256

So it looks like I have 256 file descriptors (according to the very last line in the output above)

b) bump up the available file descriptors (I will quadruple mine, you might need to experiment with this):

$ ulimit -Sn 1024

c) verify that the setting was correctly updated:

$ ulimit -a
-t: cpu time (seconds)              unlimited
-f: file size (blocks)              unlimited
-d: data seg size (kbytes)          unlimited
-s: stack size (kbytes)             8192
-c: core file size (blocks)         0
-v: address space (kbytes)          unlimited
-l: locked-in-memory size (kbytes)  unlimited
-u: processes                       709
-n: file descriptors                1024

Hope that helps! If not, you know where to find us!

Thank you so much @thermokarst!!! =)

1 Like

Opened an issue here.

3 posts were split to a new topic: Metadata category parsing error