DADA2 denoising not working (First return error 9 then 1 after increasing memory to 16 GB)

[susan.pyakurel]

• Did not find the exact answer to this on the forum; new to QIIME2, apologies.
• QIIME2-2023.5, running in Conda through university's central Linux-based system

• Apply DADA2 denoising

qiime dada2 denoise-paired --i-demultiplexed-seqs paired-end-demux.qza --p-trunc-len-f 280 --p-trunc-len-r 220 --p-n-threads 24 --o-representative-sequences rep-seqs-dada.qza --o-table table-dada2.qza --o-denoising-stats dada-stats.qza --verbose

R version 4.2.3 (2023-03-15)
Loading required package: Rcpp
DADA2: 1.26.0 / Rcpp: 1.0.10 / RcppParallel: 5.1.6
2) Filtering Error in names(answer) <- names1 :
'names' attribute [49] must be the same length as the vector [32]
4: mcmapply(fastqPairedFilter, mapply(c, fwd, rev, SIMPLIFY = FALSE),
mapply(c, filt, filt.rev, SIMPLIFY = FALSE), MoreArgs = list(truncQ = truncQ,
truncLen = truncLen, trimLeft = trimLeft, trimRight = trimRight,
maxLen = maxLen, minLen = minLen, maxN = maxN, minQ = minQ,
maxEE = maxEE, rm.phix = rm.phix, rm.lowcomplex = rm.lowcomplex,
orient.fwd = orient.fwd, matchIDs = matchIDs, id.sep = id.sep,
id.field = id.field, n = n, OMP = OMP, qualityType = qualityType,
compress = compress, verbose = verbose), mc.cores = ncores,
mc.silent = TRUE)
3: filterAndTrim(unfilts, filts, unfiltsR, filtsR, truncLen = c(truncLen,
truncLenR), trimLeft = c(trimLeft, trimLeftR), maxEE = c(maxEE,
maxEER), truncQ = truncQ, rm.phix = TRUE, multithread = multithread)
2: withCallingHandlers(expr, warning = function(w) if (inherits(w,
classes)) tryInvokeRestart("muffleWarning"))
1: suppressWarnings(filterAndTrim(unfilts, filts, unfiltsR, filtsR,
truncLen = c(truncLen, truncLenR), trimLeft = c(trimLeft,
trimLeftR), maxEE = c(maxEE, maxEER), truncQ = truncQ,
rm.phix = TRUE, multithread = multithread))
Traceback (most recent call last):
File "/home/susan.pyakurel/miniconda3/envs/qiime2-2023.5/lib/python3.8/site-packages/q2_dada2/_denoise.py", line 326, in denoise_paired
run_commands([cmd])
File "/home/susan.pyakurel/miniconda3/envs/qiime2-2023.5/lib/python3.8/site-packages/q2_dada2/_denoise.py", line 36, in run_commands
subprocess.run(cmd, check=True)
File "/home/susan.pyakurel/miniconda3/envs/qiime2-2023.5/lib/python3.8/subprocess.py", line 516, in run
raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['run_dada.R', '--input_directory', '/tmp/tmp8fykuqns/forward', '--input_directory_reverse', '/tmp/tmp8fykuqns/reverse', '--output_path', '/tmp/tmp8fykuqns/output.tsv.biom', '--output_track', '/tmp/tmp8fykuqns/track.tsv', '--filtered_directory', '/tmp/tmp8fykuqns/filt_f', '--filtered_directory_reverse', '/tmp/tmp8fykuqns/filt_r', '--truncation_length', '280', '--truncation_length_reverse', '220', '--trim_left', '0', '--trim_left_reverse', '0', '--max_expected_errors', '2.0', '--max_expected_errors_reverse', '2.0', '--truncation_quality_score', '2', '--min_overlap', '12', '--pooling_method', 'independent', '--chimera_method', 'consensus', '--min_parental_fold', '1.0', '--allow_one_off', 'False', '--num_threads', '24', '--learn_min_reads', '1000000']' returned non-zero exit status 1.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/home/susan.pyakurel/miniconda3/envs/qiime2-2023.5/lib/python3.8/site-packages/q2cli/commands.py", line 468, in call
results = action(**arguments)
File "", line 2, in denoise_paired
File "/home/susan.pyakurel/miniconda3/envs/qiime2-2023.5/lib/python3.8/site-packages/qiime2/sdk/action.py", line 274, in bound_callable
outputs = self.callable_executor(
File "/home/susan.pyakurel/miniconda3/envs/qiime2-2023.5/lib/python3.8/site-packages/qiime2/sdk/action.py", line 509, in callable_executor
output_views = self._callable(**view_args)
File "/home/susan.pyakurel/miniconda3/envs/qiime2-2023.5/lib/python3.8/site-packages/q2_dada2/_denoise.py", line 339, in denoise_paired
raise Exception("An error was encountered while running DADA2"
Exception: An error was encountered while running DADA2 in R (return code 1), please inspect stdout and stderr to learn more.
See above for debug info.
Running external command line application(s). This may print messages to stdout and/or stderr.
The command(s) being run are below. These commands cannot be manually re-run as they will depend on temporary files that no longer exist.

Command: run_dada.R --input_directory /tmp/tmp8fykuqns/forward --input_directory_reverse /tmp/tmp8fykuqns/reverse --output_path /tmp/tmp8fykuqns/output.tsv.biom --output_track /tmp/tmp8fykuqns/track.tsv --filtered_directory /tmp/tmp8fykuqns/filt_f --filtered_directory_reverse /tmp/tmp8fykuqns/filt_r --truncation_length 280 --truncation_length_reverse 220 --trim_left 0 --trim_left_reverse 0 --max_expected_errors 2.0 --max_expected_errors_reverse 2.0 --truncation_quality_score 2 --min_overlap 12 --pooling_method independent --chimera_method consensus --min_parental_fold 1.0 --allow_one_off False --num_threads 24 --learn_min_reads 1000000

slurmstepd: error: Detected 8 oom-kill event(s) in StepId=22089232.batch. Some of your processes may have been killed by the cgroup out-of-memory handler.

Thank you for your help!

[Oddant1]

Hello @susan.pyakurel. How large is your data? I think it is likely that error at the end there is correct and you are running out of RAM here. larger data will use more RAM, but so will more threads. If you have access to more than 16GB of RAM then you can increase the amount you're requesting, or you can reduce the number of threads you are creating which will reduce RAM usage.

[susan.pyakurel]

Seems like memory indeed was the issue. Thanks so much; I increased the memory to 32G and it worked perfectly.

[susan.pyakurel]

And I had sequence information from approximately 110 samples; don't know if it is considered a large dataset or small.

[susan.pyakurel]

Can you please briefly let me know the concept of threads you mentioned?

[5cr34m (Nikonov_Sergey)]

from here

.... there are a lot of factors at play here but I have heard recommendations on the forum of dedicating at least 8GB of ram. I think that’s a good safe start, though I have been able to handle some small datasets with 6GB as well. Never tried with any less…
...
That depends again on how much of your system you dedicate to these tasks. If you set your system to fully focus on these tasks using the --p-n-thread 0 option (uses all available cores) then I wouldn’t run anything else. Given that you only have 4 cores to start with, I wouldn’t run much else on the computer personally.