Error while denoising with DADA2

Technical Support
1

Hi everyone
I am reposting an older message that was embedded on an older closed discussion and perhaps missed. I am denoising with DADA2 eleven samples paired end. I imported the sequences using the Casava1.8 format. Here is the summarized demux file:
18S-demux.qzv (310.1 KB)
18S-demux.qzv (310.1 KB)

I denoised using the following script:
qiime dada2 denoise-paired
--i-demultiplexed-seqs paired-end-demux.qza
--p-trunc-len-f 150
--p-trunc-len-r 150
--p-n-threads 0
--o-table 18S-table.qza
--o-representative-sequences 18S-rep-seqs.qza
--o-denoising-stats 18S-stats.qza
--verbose
It took almost three days to run on an alienware this is my system confog:

System Manufacturer Alienware
System Model Alienware m18 R2
System Type x64-based PC
System SKU 0C9C
Processor Intel(R) Core(TM) i9-14900HX, 2200 Mhz, 24 Core(s), 32 Logical Processor(s)
NVIDIA RTX Geoforce 4060 and 64 Gigs of RAM and 2 TB memory

and this is the error:

"DADA2 error 113024

a.R --input_directory /tmp/tmpdi3t8yal/forward --input_directory_reverse /tmp/tmpdi3t8yal/reverse --output_path /tmp/tmpdi3t8yal/output.tsv.biom --output_track /tmp/tmpdi3t8yal/track.tsv --filtered_directory /tmp/tmpdi3t8yal/filt_f --filtered_directory_reverse /tmp/tmpdi3t8yal/filt_r --truncation_length 150 --truncation_length_reverse 150 --trim_left 0 --trim_left_reverse 0 --max_expected_errors 2.0 --max_expected_errors_reverse 2.0 --truncation_quality_score 2 --min_overlap 12 --pooling_method independent --chimera_method consensus --min_parental_fold 1.0 --allow_one_off False --num_threads 0 --learn_min_reads 1000000

R version 4.3.3 (2024-02-29)
Loading required package: Rcpp
DADA2: 1.30.0 / Rcpp: 1.0.13.1 / RcppParallel: 5.1.9
2) Filtering ...........
3) Learning Error Rates
407029650 total bases in 2713531 reads from 1 samples will be used for learning the error rates.
407029650 total bases in 2713531 reads from 1 samples will be used for learning the error rates.
3) Denoise samples .....Traceback (most recent call last):
File "/home/unix/miniconda3/envs/qiime2-amplicon-2024.10/lib/python3.10/site-packages/q2_dada2/_denoise.py", line 353, in denoise_paired
run_commands([cmd])
File "/home/unix/miniconda3/envs/qiime2-amplicon-2024.10/lib/python3.10/site-packages/q2_dada2/_denoise.py", line 38, in run_commands
subprocess.run(cmd, check=True)
File "/home/unix/miniconda3/envs/qiime2-amplicon-2024.10/lib/python3.10/subprocess.py", line 526, in run
raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['run_dada.R', '--input_directory', '/tmp/tmpdi3t8yal/forward', '--input_directory_reverse', '/tmp/tmpdi3t8yal/reverse', '--output_path', '/tmp/tmpdi3t8yal/output.tsv.biom', '--output_track', '/tmp/tmpdi3t8yal/track.tsv', '--filtered_directory', '/tmp/tmpdi3t8yal/filt_f', '--filtered_directory_reverse', '/tmp/tmpdi3t8yal/filt_r', '--truncation_length', '150', '--truncation_length_reverse', '150', '--trim_left', '0', '--trim_left_reverse', '0', '--max_expected_errors', '2.0', '--max_expected_errors_reverse', '2.0', '--truncation_quality_score', '2', '--min_overlap', '12', '--pooling_method', 'independent', '--chimera_method', 'consensus', '--min_parental_fold', '1.0', '--allow_one_off', 'False', '--num_threads', '0', '--learn_min_reads', '1000000']' died with <Signals.SIGKILL: 9>.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/home/unix/miniconda3/envs/qiime2-amplicon-2024.10/lib/python3.10/site-packages/q2cli/commands.py", line 530, in call
results = self._execute_action(
File "/home/unix/miniconda3/envs/qiime2-amplicon-2024.10/lib/python3.10/site-packages/q2cli/commands.py", line 602, in _execute_action
results = action(**arguments)
File "", line 2, in denoise_paired
File "/home/unix/miniconda3/envs/qiime2-amplicon-2024.10/lib/python3.10/site-packages/qiime2/sdk/action.py", line 299, in bound_callable
outputs = self.callable_executor(
File "/home/unix/miniconda3/envs/qiime2-amplicon-2024.10/lib/python3.10/site-packages/qiime2/sdk/action.py", line 570, in callable_executor
output_views = self._callable(**view_args)
File "/home/unix/miniconda3/envs/qiime2-amplicon-2024.10/lib/python3.10/site-packages/q2_dada2/_denoise.py", line 366, in denoise_paired
raise Exception("An error was encountered while running DADA2"
Exception: An error was encountered while running DADA2 in R (return code -9), please inspect stdout and stderr to learn more.

Plugin error from dada2:

An error was encountered while running DADA2 in R (return code -9), please inspect stdout and stderr to learn more.

See above for debug info."

I have also tried to denoise deblur by first merging with vsearch:

qiime vsearch merge-pairs
--i-demultiplexed-seqs 18S-demux-paired-end.qza
--fastq_qmax 45
--o-merged-sequences 18S-demux-joined.qza
--o-unmerged-sequences 18S-demux-unjoined.qza
--verbose
and I got this error:

" (qiime2-amplicon-2024.10) unix@ZL071124132317:/mnt/c/Users/nietof/Documents$ qiime vsearch merge-pairs
--i-demultiplexed-seqs 18S-demux-paired-end.qza
--o-merged-sequences 18S-demux-joined.qza
--o-unmerged-sequences 18S-demux-unjoined.qza
--verbose"

and here the error I received this time:
"vsearch v2.22.1_linux_x86_64, 15.5GB RAM, 32 cores

Merging reads 0%

Fatal error: FASTQ quality value (43) above qmax (41)
By default, quality values range from 0 to 41.
To allow higher quality values, please use the option --fastq_qmax 43
Traceback (most recent call last):
File "/home/unix/miniconda3/envs/qiime2-amplicon-2024.10/lib/python3.10/site-packages/q2cli/commands.py", line 530, in call
results = self._execute_action(
File "/home/unix/miniconda3/envs/qiime2-amplicon-2024.10/lib/python3.10/site-packages/q2cli/commands.py", line 602, in _execute_action
results = action(**arguments)
File "", line 2, in merge_pairs
File "/home/unix/miniconda3/envs/qiime2-amplicon-2024.10/lib/python3.10/site-packages/qiime2/sdk/action.py", line 299, in bound_callable
outputs = self.callable_executor(
File "/home/unix/miniconda3/envs/qiime2-amplicon-2024.10/lib/python3.10/site-packages/qiime2/sdk/action.py", line 570, in callable_executor
output_views = self._callable(**view_args)
File "/home/unix/miniconda3/envs/qiime2-amplicon-2024.10/lib/python3.10/site-packages/q2_vsearch/_merge_pairs.py", line 52, in merge_pairs
_, merged, unmerged = _merge_pairs_w_command_output(
File "/home/unix/miniconda3/envs/qiime2-amplicon-2024.10/lib/python3.10/site-packages/q2_vsearch/_merge_pairs.py", line 164, in _merge_pairs_w_command_output
run_command(cmd)
File "/home/unix/miniconda3/envs/qiime2-amplicon-2024.10/lib/python3.10/site-packages/q2_vsearch/_cluster_features.py", line 33, in run_command
subprocess.run(cmd, check=True)
File "/home/unix/miniconda3/envs/qiime2-amplicon-2024.10/lib/python3.10/subprocess.py", line 526, in run
raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['vsearch', '--fastq_mergepairs', '/tmp/qiime2/unix/data/b6bb8dbf-0ccc-475e-ae4b-eef230f85f72/data/NIETO1_S1_L001_R1_001.fastq.gz', '--reverse', '/tmp/qiime2/unix/data/b6bb8dbf-0ccc-475e-ae4b-eef230f85f72/data/NIETO1_S1_L001_R2_001.fastq.gz', '--fastqout', '/tmp/qiime2/unix/processes/204501-1733078781.87@unix/tmp/q2-OutPath-78iwd0lm/NIETO1_0_L001_R1_001.fastq', '--fastqout_notmerged_fwd', '/tmp/qiime2/unix/processes/204501-1733078781.87@unix/tmp/q2-OutPath-v0231mpo/NIETO1_0_L001_R1_001.fastq', '--fastqout_notmerged_rev', '/tmp/qiime2/unix/processes/204501-1733078781.87@unix/tmp/q2-OutPath-v0231mpo/NIETO1_0_L001_R2_001.fastq', '--fastq_ascii', '33', '--fastq_minlen', '1', '--fastq_minovlen', '10', '--fastq_maxdiffs', '10', '--fastq_qmin', '0', '--fastq_qminout', '0', '--fastq_qmax', '41', '--fastq_qmaxout', '41', '--fasta_width', '0', '--threads', '1']' returned non-zero exit status 1.

Plugin error from vsearch:

Command '['vsearch', '--fastq_mergepairs', '/tmp/qiime2/unix/data/b6bb8dbf-0ccc-475e-ae4b-eef230f85f72/data/NIETO1_S1_L001_R1_001.fastq.gz', '--reverse', '/tmp/qiime2/unix/data/b6bb8dbf-0ccc-475e-ae4b-eef230f85f72/data/NIETO1_S1_L001_R2_001.fastq.gz', '--fastqout', '/tmp/qiime2/unix/processes/204501-1733078781.87@unix/tmp/q2-OutPath-78iwd0lm/NIETO1_0_L001_R1_001.fastq', '--fastqout_notmerged_fwd', '/tmp/qiime2/unix/processes/204501-1733078781.87@unix/tmp/q2-OutPath-v0231mpo/NIETO1_0_L001_R1_001.fastq', '--fastqout_notmerged_rev', '/tmp/qiime2/unix/processes/204501-1733078781.87@unix/tmp/q2-OutPath-v0231mpo/NIETO1_0_L001_R2_001.fastq', '--fastq_ascii', '33', '--fastq_minlen', '1', '--fastq_minovlen', '10', '--fastq_maxdiffs', '10', '--fastq_qmin', '0', '--fastq_qminout', '0', '--fastq_qmax', '41', '--fastq_qmaxout', '41', '--fasta_width', '0', '--threads', '1']' returned non-zero exit status 1.

See above for debug info."

Any help would be greatly appreciated.
Fernando

2

Hi @nietof,
Sorry if we missed your earlier message - thanks for following up!

This is almost certainly a result of running out of memory. I wonder if it might work to re-run the DADA2 step with fewer threads - for example passing --p-n-threads 4. That still might not work, in which case I think the next step would be to try a system with more memory (let me know if you need suggestions for how to achieve that).

3

HI @gregcaporaso
Yes, i am surprised the alienware can't handle it, but the sequence files have a huge number of reads in them.
Yes, I would appreciate guidance on how to go about using a system with more memory.
Thank you
Fernando

4

@nietof, A few thoughts:

  • Do you have access to an institution HPC system that you could run this on?
  • Another option that might work would be https://cancer.usegalaxy.org.
  • I've been working with the team that runs the ACES and FASTER systems in the ACCESS network - you can apply for compute time on these machines (see here), and they have large memory nodes where this should run without a problem. They also have QIIME 2 2024.10 installed (amplicon, metagenome, and pathogenome distributions), so you should be able to do everything you need to. I'm in the process of testing this myself, and then plan to put some more specific instructions together on how to use QIIME 2 on these systems, but that'll probably take me a few works weeks to get together.

Hope one of these options helps!