Plugin error from dada2: An error was encountered while running DADA2 in R (return code 1) [removing chimeras?]

a_staubus · March 1, 2024, 2:51am

Hello. I am consistently getting the following error when I run denoising with dada2:

====================================================
Plugin error from dada2:
An error was encountered while running DADA2 in R (return code 1), please inspect stdout and stderr to learn more.

Debug info has been saved to /tmp/qiime2-q2cli-err-4if6ev_9.log

====================================================

Here is the debug log:

====================================================
Running external command line application(s). This may print messages to stdout and/or stderr.
The command(s) being run are below. These commands cannot be manually re-run as they will depend on temporary files that no longer exist.

Command: run_dada.R --input_directory /tmp/tmpirn9gach/forward --input_directory_reverse /tmp/tmpirn9gach/reverse --output_path /tmp/tmpirn9gach/output.tsv.biom --output_track /tmp/tmpirn9gach/track.tsv --filtered_directory /tmp/tmpirn9gach/filt_f --filtered_directory_reverse /tmp/tmpirn9gach/filt_r --truncation_length 180 --truncation_length_reverse 180 --trim_left 0 --trim_left_reverse 0 --max_expected_errors 4 --max_expected_errors_reverse 4 --truncation_quality_score 2 --min_overlap 20 --pooling_method pseudo --chimera_method pooled --min_parental_fold 1.0 --allow_one_off False --num_threads 0 --learn_min_reads 1000000

Warning message:
package ‘optparse’ was built under R version 4.2.3
R version 4.2.2 (2022-10-31)
Loading required package: Rcpp
DADA2: 1.26.0 / Rcpp: 1.0.12 / RcppParallel: 5.1.6
2) Filtering ........................................
3) Learning Error Rates
301163580 total bases in 1673131 reads from 2 samples will be used for learning the error rates.
301163580 total bases in 1673131 reads from 2 samples will be used for learning the error rates.
3) Denoise samples ........................................
Pseudo-pool step ........................................
........................................
5) Remove chimeras (method = pooled)
Error in mcfork() :
unable to fork, possible reason: Cannot allocate memory
Calls: removeBimeraDenovo ... isBimeraDenovo -> mclapply -> lapply -> FUN -> mcfork
6: mcfork()
5: FUN(X[[i]], ...)
4: lapply(seq_len(cores), inner.do)
3: mclapply(mc.indices, loopFun, unqs.loop = unqs.int, allowOneOff = allowOneOff,
minFoldParentOverAbundance = minFoldParentOverAbundance,
minParentAbundance = minParentAbundance, minOneOffParentDistance = minOneOffParentDistance,
maxShift = maxShift, mc.cores = mc.cores)
2: isBimeraDenovo(unqs[[i]], ..., verbose = verbose)
1: removeBimeraDenovo(seqtab, method = chimeraMethod, minFoldParentOverAbundance = minParentFold,
allowOneOff = allowOneOff, multithread = multithread)

Traceback (most recent call last):
File "/home/aos2/miniconda3/envs/qiime2-amplicon-2024.2/lib/python3.8/site-packages/q2_dada2/_denoise.py", line 350, in denoise_paired
run_commands([cmd])
File "/home/aos2/miniconda3/envs/qiime2-amplicon-2024.2/lib/python3.8/site-packages/q2_dada2/_denoise.py", line 37, in run_commands
subprocess.run(cmd, check=True)
File "/home/aos2/miniconda3/envs/qiime2-amplicon-2024.2/lib/python3.8/subprocess.py", line 516, in run
raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['run_dada.R', '--input_directory', '/tmp/tmpirn9gach/forward', '--input_directory_reverse', '/tmp/tmpirn9gach/reverse', '--output_path', '/tmp/tmpirn9gach/output.tsv.biom', '--output_track', '/tmp/tmpirn9gach/track.tsv', '--filtered_directory', '/tmp/tmpirn9gach/filt_f', '--filtered_directory_reverse', '/tmp/tmpirn9gach/filt_r', '--truncation_length', '180', '--truncation_length_reverse', '180', '--trim_left', '0', '--trim_left_reverse', '0', '--max_expected_errors', '4', '--max_expected_errors_reverse', '4', '--truncation_quality_score', '2', '--min_overlap', '20', '--pooling_method', 'pseudo', '--chimera_method', 'pooled', '--min_parental_fold', '1.0', '--allow_one_off', 'False', '--num_threads', '0', '--learn_min_reads', '1000000']' returned non-zero exit status 1.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/home/aos2/miniconda3/envs/qiime2-amplicon-2024.2/lib/python3.8/site-packages/q2cli/commands.py", line 520, in call
results = self._execute_action(
File "/home/aos2/miniconda3/envs/qiime2-amplicon-2024.2/lib/python3.8/site-packages/q2cli/commands.py", line 581, in _execute_action
results = action(**arguments)
File "", line 2, in denoise_paired
File "/home/aos2/miniconda3/envs/qiime2-amplicon-2024.2/lib/python3.8/site-packages/qiime2/sdk/action.py", line 342, in bound_callable
outputs = self.callable_executor(
File "/home/aos2/miniconda3/envs/qiime2-amplicon-2024.2/lib/python3.8/site-packages/qiime2/sdk/action.py", line 566, in callable_executor
output_views = self._callable(**view_args)
File "/home/aos2/miniconda3/envs/qiime2-amplicon-2024.2/lib/python3.8/site-packages/q2_dada2/_denoise.py", line 363, in denoise_paired
raise Exception("An error was encountered while running DADA2"
Exception: An error was encountered while running DADA2 in R (return code 1), please inspect stdout and stderr to learn more.

I am using qiime2-2024.2. I removed the primers with cutadapt prior to importing the reads. To import the reads I used:
qiime tools import
--type 'SampleData[PairedEndSequencesWithQuality]'
--input-path qiime2/Round4_manifest.tsv
--output-path qiime2/Round4_sequences.tsv
--input-format PairedEndFastqManifestPhred33V2

I've attached my manifest
Round4_manifest.tsv (7.3 KB)

The sequences were already demultiplexed prior to importing.

I've also attached my summary visualization
Round4_sequences.qzv (311.6 KB)

I would greatly appreciate any help with this! I've found several threads on similar topics to this already, but I don't think any apply to my exact case. But if I'm wrong I'd be grateful if you could point me towards an existing thread. Thank you!

Oddant1 · March 1, 2024, 5:27pm

Hello @a_staubus, the root error appears to be:

Error in mcfork() :
unable to fork, possible reason: Cannot allocate memory
Calls: removeBimeraDenovo ... isBimeraDenovo -> mclapply -> lapply -> FUN -> mcfork

which suggests you may be running out of memory. Can you please run the following commands and post the results here?

First:

free -h

Then:

/usr/bin/time -v <your failing qiime command>

This will tell me how much RAM you have available and how much RAM your QIIME 2 command is using.

a_staubus · March 1, 2024, 6:07pm

Thank you for your help with this!

$ free -h
total used free shared buff/cache available
Mem: 7.5Gi 3.1Gi 1.9Gi 50Mi 2.5Gi 4.1Gi
Swap: 0B 0B 0B

$ /usr/bin/time -v qiime dada2 denoise-paired --i-demultiplexed-seqs qiime2/Round4_sequences.qza --p-trunc-len-f 180 --p-trunc-len-r 180 --p-max-ee-f 4 --p-max-ee-r 4 --p-min-overlap 20 --p-n-threads 0 --p-pooling-method 'pseudo' --p-chimera-method 'pooled' --o-table qiime2/Round4_table.qza --o-representative-sequences qiime2/Round4_rep_seqs.qza --o-denoising-stats qiime2/Round4_denoising_stats.qza --verbose &

Plugin error from dada2:

An error was encountered while running DADA2 in R (return code 1), please inspect stdout and stderr to learn more.

Debug info has been saved to /tmp/qiime2-q2cli-err-ggiliee7.log

Command exited with non-zero status 1

Command being timed: "qiime dada2 denoise-paired --i-demultiplexed-seqs qiime2/Round4_sequences.qza --p-trunc-len-f 180 --p-trunc-len-r 180 --p-max-ee-f 4 --p-max-ee-r 4 --p-min-overlap 20 --p-n-threads 0 --p-pooling-method pseudo --p-chimera-method pooled --o-table qiime2/Round4_table.qza --o-representative-sequences qiime2/Round4_rep_seqs.qza --o-denoising-stats qiime2/Round4_denoising_stats.qza"

User time (seconds): 51.55

System time (seconds): 51.36

Percent of CPU this job got: 11%

Elapsed (wall clock) time (h:mm:ss or m:ss): 14:34.35

Average shared text size (kbytes): 0

Average unshared data size (kbytes): 0

Average stack size (kbytes): 0

Average total size (kbytes): 0

Maximum resident set size (kbytes): 1053024

Average resident set size (kbytes): 0

Major (requiring I/O) page faults: 8914

Minor (reclaiming a frame) page faults: 796855

Voluntary context switches: 11136

Involuntary context switches: 6912

Swaps: 0

File system inputs: 6108790

File system outputs: 20840

Socket messages sent: 0

Socket messages received: 0

Signals delivered: 0

Page size (bytes): 4096

Exit status: 1

Oddant1 · March 1, 2024, 6:21pm

So I suspect you are running out of memory. It says there that at the time you ran free -h you had ~8gigs total memory and ~4gigs available.

In your other command you have:

Maximum resident set size (kbytes): 1053024

Which means the memory usage of the process peaked at that many kilobytes which is ~1gig; however, R will need to fork the process n times where n is the value of the --p-n-threads parameter. When R tries to fork the process, it needs to copy a non-trivial amount of that memory for every fork. It most likely asked for the memory it needed to create the forks in one request, but it wasn't able to acquire it hence the low peak memory usage. The memory it asked for but didn't receive (because it didn't exist) isn't reflected in that number.

You passed --p-n-threads 0 which means the process attempted to create as many forks as your computer has CPUs. Based on your free command, if this number was 4 or more I would expect to see this error, and if this number was 8 or more I'd expect to see this error even if your command had access to literally all of the RAM your computer has (which it never will).

I would recommend rerunning the command with --p-n-threads 1 which is the default value, so you can just not pass the argument. This will use as little memory as possible because it won't create any forks.

In general, if you run a command with multiple threads and it fails try again with one thread.

a_staubus · March 1, 2024, 6:23pm

Thanks for taking time to respond to this and for the advice! I'll give that a try update here if it works

a_staubus · March 2, 2024, 9:14pm

Okay I reran it as suggested and got a slightly different error

Running external command line application(s). This may print messages to stdout and/or stderr.
The command(s) being run are below. These commands cannot be manually re-run as they will depend on temporary files that no longer exist.

Command: run_dada.R --input_directory /tmp/tmp89lcidx0/forward --input_directory_reverse /tmp/tmp89lcidx0/reverse --output_path /tmp/tmp89lcidx0/output.tsv.biom --output_track /tmp/tmp89lcidx0/track.tsv --filtered_directory /tmp/tmp89lcidx0/filt_f --filtered_directory_reverse /tmp/tmp89lcidx0/filt_r --truncation_length 180 --truncation_length_reverse 180 --trim_left 0 --trim_left_reverse 0 --max_expected_errors 4 --max_expected_errors_reverse 4 --truncation_quality_score 2 --min_overlap 20 --pooling_method pseudo --chimera_method pooled --min_parental_fold 1.0 --allow_one_off False --num_threads 0 --learn_min_reads 1000000

Warning message:
package ‘optparse’ was built under R version 4.2.3
R version 4.2.2 (2022-10-31)
Loading required package: Rcpp
DADA2: 1.26.0 / Rcpp: 1.0.12 / RcppParallel: 5.1.6
2) Filtering Error in names(answer) <- names1 : attempt to set an attribute on NULL
4: mcmapply(fastqPairedFilter, mapply(c, fwd, rev, SIMPLIFY = FALSE),
mapply(c, filt, filt.rev, SIMPLIFY = FALSE), MoreArgs = list(truncQ = truncQ,
truncLen = truncLen, trimLeft = trimLeft, trimRight = trimRight,
maxLen = maxLen, minLen = minLen, maxN = maxN, minQ = minQ,
maxEE = maxEE, rm.phix = rm.phix, rm.lowcomplex = rm.lowcomplex,
orient.fwd = orient.fwd, matchIDs = matchIDs, id.sep = id.sep,
id.field = id.field, n = n, OMP = OMP, qualityType = qualityType,
compress = compress, verbose = verbose), mc.cores = ncores,
mc.silent = TRUE)
3: filterAndTrim(unfilts, filts, unfiltsR, filtsR, truncLen = c(truncLen,
truncLenR), trimLeft = c(trimLeft, trimLeftR), maxEE = c(maxEE,
maxEER), truncQ = truncQ, rm.phix = TRUE, multithread = multithread)
2: withCallingHandlers(expr, warning = function(w) if (inherits(w,
classes)) tryInvokeRestart("muffleWarning"))
1: suppressWarnings(filterAndTrim(unfilts, filts, unfiltsR, filtsR,
truncLen = c(truncLen, truncLenR), trimLeft = c(trimLeft,
trimLeftR), maxEE = c(maxEE, maxEER), truncQ = truncQ,
rm.phix = TRUE, multithread = multithread))
Traceback (most recent call last):
File "/home/aos2/miniconda3/envs/qiime2-amplicon-2024.2/lib/python3.8/site-packages/q2_dada2/_denoise.py", line 350, in denoise_paired
run_commands([cmd])
File "/home/aos2/miniconda3/envs/qiime2-amplicon-2024.2/lib/python3.8/site-packages/q2_dada2/_denoise.py", line 37, in run_commands
subprocess.run(cmd, check=True)
File "/home/aos2/miniconda3/envs/qiime2-amplicon-2024.2/lib/python3.8/subprocess.py", line 516, in run
raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['run_dada.R', '--input_directory', '/tmp/tmp89lcidx0/forward', '--input_directory_reverse', '/tmp/tmp89lcidx0/reverse', '--output_path', '/tmp/tmp89lcidx0/output.tsv.biom', '--output_track', '/tmp/tmp89lcidx0/track.tsv', '--filtered_directory', '/tmp/tmp89lcidx0/filt_f', '--filtered_directory_reverse', '/tmp/tmp89lcidx0/filt_r', '--truncation_length', '180', '--truncation_length_reverse', '180', '--trim_left', '0', '--trim_left_reverse', '0', '--max_expected_errors', '4', '--max_expected_errors_reverse', '4', '--truncation_quality_score', '2', '--min_overlap', '20', '--pooling_method', 'pseudo', '--chimera_method', 'pooled', '--min_parental_fold', '1.0', '--allow_one_off', 'False', '--num_threads', '0', '--learn_min_reads', '1000000']' returned non-zero exit status 1.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/home/aos2/miniconda3/envs/qiime2-amplicon-2024.2/lib/python3.8/site-packages/q2cli/commands.py", line 520, in call
results = self._execute_action(
File "/home/aos2/miniconda3/envs/qiime2-amplicon-2024.2/lib/python3.8/site-packages/q2cli/commands.py", line 581, in _execute_action
results = action(**arguments)
File "", line 2, in denoise_paired
File "/home/aos2/miniconda3/envs/qiime2-amplicon-2024.2/lib/python3.8/site-packages/qiime2/sdk/action.py", line 342, in bound_callable
outputs = self.callable_executor(
File "/home/aos2/miniconda3/envs/qiime2-amplicon-2024.2/lib/python3.8/site-packages/qiime2/sdk/action.py", line 566, in callable_executor
output_views = self._callable(**view_args)
File "/home/aos2/miniconda3/envs/qiime2-amplicon-2024.2/lib/python3.8/site-packages/q2_dada2/_denoise.py", line 363, in denoise_paired
raise Exception("An error was encountered while running DADA2"
Exception: An error was encountered while running DADA2 in R (return code 1), please inspect stdout and stderr to learn more.

Oddant1 · March 4, 2024, 6:41pm

Looking into other occurrences of that error message, it looks as though that error is also related to being out of memory. I know it looks nothing like an out of memory error, but they often manifest in very strange ways especially in R.

If you rerun your command with usr/bin/time -v again I can see how much memory it's using now with only 1 thread. 8gigs of RAM is enough for normal computer use, but it isn't a lot for this sort of work. If you have access to a machine with more RAM or even to a cluster run by your institution I would suggest using those if at all possible.

a_staubus · March 5, 2024, 10:08pm

I set up a VM through my university with 32 gb of RAM. This allowed me to run dada2, even with n-threads=0. Thanks for your suggestion!

system · April 6, 2024, 4:48am

This topic was automatically closed 31 days after the last reply. New replies are no longer allowed.