cluster-features-open-reference Memory / disk space error

Hi, when I run the following command I have an error:

qiime vsearch cluster-features-open-reference
–i-table 6.1_uchime-ref-out/table-nonchimeric-wo-borderline.qza
–i-sequences 6.1_uchime-ref-out/rep-seqs-nonchimeric-wo-borderline.qza
–i-reference-sequences $reference_seqs
–p-perc-identity 0.85
–o-clustered-table 6.2_table-op_ref-85.qza
–o-clustered-sequences 6.2_rep-seqs-op_ref-85.qza
–o-new-reference-sequences 6.2_new-ref-seqs-op_ref-85.qza
–p-threads 24

I have a memory error ( 32 core and 126Go Mem ).
Here is the traceback:

Running external co_mmand line application. This may print messages to stdout and/or stderr.

The command being run is below. This command cannot be manually re-run as it will depend on temporary files that no longer exist.

Command: vsearch --usearch_global /tmp/tmp9bmzjk7j --id 0.85 --db /tmp/qiime2-archive-ew75tn5v/9f4b9a95-e8c4-45be-a980-6faaa9b857c7/data/dna-sequences.fasta --uc /tmp/tmpqsl93h8o --strand plus --qmask none --notmatched /tmp/tmp72o2x5lw --threads 24

vsearch v2.7.0_linux_x86_64, 125.9GB RAM, 32 cores

Reading fiand subsamplingle /tmp/qiime2-archive-ew75tn5v/9f4b9a95-e8c4-45be-a980-6faaa9b857c7/data/dna-sequences.fasta 100%
521145303 nt in 369953 seqs, min 900, max 2961, avg 1409
Masking 100%
Counting k-mers 100%
Creating k-mer index 100%
Searching 100%
Matching query sequences: 844922 of 2092876 (40.37%)
Running external command line application. This may print messages to stdout and/or stderr.
The command being run is below. This command cannot be manually re-run as it will depend on temporary files that no longer exist.

Command: vsearch --sortbysize /tmp/tmp72o2x5lw --xsize --output /tmp/q2-DNAFASTAFormat-anta19n4

vsearch v2.7.0_linux_x86_64, 125.9GB RAM, 32 cores

Reading file /tmp/tmp72o2x5lw 100%
1747135600 nt in 1247954 seqs, min 1400, max 1400, avg 1400
Getting sizes 100%
Sorting 100%
Median abundance: 1
Writing output 100%
Running external command line application. This may print messages to stdout and/or stderr.
The command being run is below. This command cannot be manually re-run as it will depend on temporary files that no longer exist.

Command: vsearch --cluster_size /tmp/tmpa7j30vmp --id 0.85 --centroids /tmp/q2-DNAFASTAFormat-xiom9wn4 --uc /tmp/tmpwz5to5f2 --qmask none --xsize --threads 24

vsearch v2.7.0_linux_x86_64, 125.9GB RAM, 32 cores

Reading file /tmp/tmpa7j30vmp 100%
1747135600 nt in 1247954 seqs, min 1400, max 1400, avg 1400
Sorting by abundance 100%
Counting k-mers 100%
Clustering 100%
Sorting clusters 100%
Writing clusters 100%
Clusters: 512147 Size min 1, max 55355, avg 2.4
Singletons: 482081, 38.6% of seqs, 94.1% of clusters
Traceback (most recent call last):
File “/home/bioinfo/anaconda3/envs/qiime2-2019.7/lib/python3.6/site-packages/q2cli/commands.py”, line 327, in call
results = action(**arguments)
File “</home/bioinfo/anaconda3/envs/qiime2-2019.7/lib/python3.6/site-packages/decorator.py:decorator-gen-126>”, line 2, in cluster_features_open_reference
File “/home/bioinfo/anaconda3/envs/qiime2-2019.7/lib/python3.6/site-packages/qiime2/sdk/action.py”, line 240, in bound_callable
output_types, provenance)
File “/home/bioinfo/anaconda3/envs/qiime2-2019.7/lib/python3.6/site-packages/qiime2/sdk/action.py”, line 502, in callable_executor
prov = provenance.fork(name, output)
File “/home/bioinfo/anaconda3/envs/qiime2-2019.7/lib/python3.6/site-packages/qiime2/core/archive/provenance.py”, line 438, in fork
forked.add_ancestor(alias)
File “/home/bioinfo/anaconda3/envs/qiime2-2019.7/lib/python3.6/site-packages/qiime2/core/archive/provenance.py”, line 167, in add_ancestor
shutil.copytree(str(grandcestor), str(destination))
File “/home/bioinfo/anaconda3/envs/qiime2-2019.7/lib/python3.6/shutil.py”, line 359, in copytree
raise Error(errors)
shutil.Error: [(’/tmp/qiime2-archive-7k562zzb/94346e54-6bf0-4227-95a8-533257fdabcc/provenance/artifacts/1635fd96-bf73-4ae8-bcf4-a392a2f1d920/action/metadata.tsv’, ‘/tmp/qiime2-provenance-mi86hqxk/artifacts/1635fd96-bf73-4ae8-bcf4-a392a2f1d920/action/metadata.tsv’, ‘[Errno 28] No space left on device’), (’/tmp/qiime2-archive-7k562zzb/94346e54-6bf0-4227-95a8-533257fdabcc/provenance/artifacts/1635fd96-bf73-4ae8-bcf4-a392a2f1d920/action/action.yaml’, ‘/tmp/qiime2-provenance-mi86hqxk/artifacts/1635fd96-bf73-4ae8-bcf4-a392a2f1d920/action/action.yaml’, ‘[Errno 28] No space left on device’), (’/tmp/qiime2-archive-7k562zzb/94346e54-6bf0-4227-95a8-533257fdabcc/provenance/artifacts/1635fd96-bf73-4ae8-bcf4-a392a2f1d920/metadata.yaml’, ‘/tmp/qiime2-provenance-mi86hqxk/artifacts/1635fd96-bf73-4ae8-bcf4-a392a2f1d920/metadata.yaml’, ‘[Errno 28] No space left on device’), (’/tmp/qiime2-archive-7k562zzb/94346e54-6bf0-4227-95a8-533257fdabcc/provenance/artifacts/1635fd96-bf73-4ae8-bcf4-a392a2f1d920/citations.bib’, ‘/tmp/qiime2-provenance-mi86hqxk/artifacts/1635fd96-bf73-4ae8-bcf4-a392a2f1d920/citations.bib’, ‘[Errno 28] No space left on device’)]_

Good morning @nbargues,

Welcome to the forums! :qiime2:

Thanks for posting your full error message. Here’s the part I noticed:

[Errno 28] No space left on device’)

The good news is that this is a disk space error, and not a memory error. (Memory errors would require you get a bigger machine, but for disk space, you can just remove unneeded files!)

See if you can free up some space in /tmp/ and let us know if the commands works!
Colin

hi @colinbrislawn ,
thanks for the response. That’s what I thought so I assign a new tmp_dir with more volume and restart the command. I keep you posted if the command works or not.

Cheers

1 Like