DADA2 Error: An error was encountered while running DADA2 in R (return code 1), please inspect stdout and stderr to learn more

[Naimur_Rahman1 (Naimur Rahman)]

qiime dada2 denoise-paired
--i-demultiplexed-seqs inputs/demux-Sabrina.qza
--p-trim-left-f 19
--p-trim-left-r 20
--p-max-ee-f 3
--p-max-ee-r 3
--p-trunc-q 5
--p-trunc-len-f 140
--p-trunc-len-r 140
--p-n-threads 4
--o-representative-sequences dada2/rep-seqs_Sabrina_dada2.qza
--o-table dada2/table_Sabrina_dada2.qza
--o-denoising-stats dada2/stats_Sabrina_dada2.qza
--verbose
Running external command line application(s). This may print messages to stdout and/or stderr.
The command(s) being run are below. These commands cannot be manually re-run as they will depend on temporary files that no longer exist.

Command: run_dada.R --input_directory /tmp/tmpw1qqmoco/forward --input_directory_reverse /tmp/tmpw1qqmoco/reverse --output_path /tmp/tmpw1qqmoco/output.tsv.biom --output_track /tmp/tmpw1qqmoco/track.tsv --filtered_directory /tmp/tmpw1qqmoco/filt_f --filtered_directory_reverse /tmp/tmpw1qqmoco/filt_r --truncation_length 140 --truncation_length_reverse 140 --trim_left 19 --trim_left_reverse 20 --max_expected_errors 3 --max_expected_errors_reverse 3 --truncation_quality_score 5 --min_overlap 12 --pooling_method independent --chimera_method consensus --min_parental_fold 1.0 --allow_one_off False --num_threads 4 --learn_min_reads 1000000

R version 4.3.3 (2024-02-29)
Loading required package: Rcpp
DADA2: 1.30.0 / Rcpp: 1.0.12 / RcppParallel: 5.1.6
2) Filtering ....
3) Learning Error Rates
46535632 total bases in 384592 reads from 4 samples will be used for learning the error rates.
46151040 total bases in 384592 reads from 4 samples will be used for learning the error rates.
3) Denoise samples ....
....
5) Remove chimeras (method = consensus)
Error in isBimeraDenovoTable(unqs[[i]], ..., verbose = verbose) :
Input must be a valid sequence table.
Calls: removeBimeraDenovo -> isBimeraDenovoTable
3: stop("Input must be a valid sequence table.")
2: isBimeraDenovoTable(unqs[[i]], ..., verbose = verbose)
1: removeBimeraDenovo(seqtab, method = chimeraMethod, minFoldParentOverAbundance = minParentFold,
allowOneOff = allowOneOff, multithread = multithread)
Traceback (most recent call last):
File "/home/naimur/miniconda3/envs/qiime2-amplicon-2024.5/lib/python3.9/site-packages/q2_dada2/_denoise.py", line 350, in denoise_paired
run_commands([cmd])
File "/home/naimur/miniconda3/envs/qiime2-amplicon-2024.5/lib/python3.9/site-packages/q2_dada2/_denoise.py", line 37, in run_commands
subprocess.run(cmd, check=True)
File "/home/naimur/miniconda3/envs/qiime2-amplicon-2024.5/lib/python3.9/subprocess.py", line 528, in run
raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['run_dada.R', '--input_directory', '/tmp/tmpw1qqmoco/forward', '--input_directory_reverse', '/tmp/tmpw1qqmoco/reverse', '--output_path', '/tmp/tmpw1qqmoco/output.tsv.biom', '--output_track', '/tmp/tmpw1qqmoco/track.tsv', '--filtered_directory', '/tmp/tmpw1qqmoco/filt_f', '--filtered_directory_reverse', '/tmp/tmpw1qqmoco/filt_r', '--truncation_length', '140', '--truncation_length_reverse', '140', '--trim_left', '19', '--trim_left_reverse', '20', '--max_expected_errors', '3', '--max_expected_errors_reverse', '3', '--truncation_quality_score', '5', '--min_overlap', '12', '--pooling_method', 'independent', '--chimera_method', 'consensus', '--min_parental_fold', '1.0', '--allow_one_off', 'False', '--num_threads', '4', '--learn_min_reads', '1000000']' returned non-zero exit status 1.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/home/naimur/miniconda3/envs/qiime2-amplicon-2024.5/lib/python3.9/site-packages/q2cli/commands.py", line 520, in call
results = self._execute_action(
File "/home/naimur/miniconda3/envs/qiime2-amplicon-2024.5/lib/python3.9/site-packages/q2cli/commands.py", line 581, in _execute_action
results = action(**arguments)
File "", line 2, in denoise_paired
File "/home/naimur/miniconda3/envs/qiime2-amplicon-2024.5/lib/python3.9/site-packages/qiime2/sdk/action.py", line 342, in bound_callable
outputs = self.callable_executor(
File "/home/naimur/miniconda3/envs/qiime2-amplicon-2024.5/lib/python3.9/site-packages/qiime2/sdk/action.py", line 576, in callable_executor
output_views = self._callable(**view_args)
File "/home/naimur/miniconda3/envs/qiime2-amplicon-2024.5/lib/python3.9/site-packages/q2_dada2/_denoise.py", line 363, in denoise_paired
raise Exception("An error was encountered while running DADA2"
Exception: An error was encountered while running DADA2 in R (return code 1), please inspect stdout and stderr to learn more.

Plugin error from dada2:

An error was encountered while running DADA2 in R (return code 1), please inspect stdout and stderr to learn more.

See above for debug info.

please help me, i've tried various things. But failed to solve the issue

[salias (Sergio Alías-Segura)]

Hi there,

Welcome to the :qiime2: forum!

The (most) interesting part of your error message is:

@Naimur_Rahman1

Error in isBimeraDenovoTable(unqs[[i]], ..., verbose = verbose) :
Input must be a valid sequence table.

I looked for that error in the forum and I found some posts.

Click for links

1. Q2-dada2: Error in isBimeraDenovoTable(unqs[[i]], ..., verbose = verbose) : Input must be a valid sequence table
2. running dada2,error:Error in isBimeraDenovoTable(unqs[[i]], ..., verbose = verbose) : Input must be a valid sequence table.
3. Error in isBimeraDenovoTable(unqs[[i]], ..., verbose = verbose) : Input must be a valid sequence table.
4. DADA2 fails to run due to 'Error in isBimeraDenovoTable'

All of them agree the problem is that DADA2 is not merging your paired end reads so you end up with an empty table prior to the chimera removal step. They say that the problem here could be one of the following:

1. Your forward and reverse reads could be the same. Looking at your demux.qzv I don't think this is the problem¹, but just in case: how did you import your sequences? Via manifest file? Could you check you didn't put forward paths in the reverse paths column?
2. Your truncation parameters could be too aggressive for paired ends to overlap. I'm also skeptical about this one because 140 looks sensible to me. What is your expected amplicon size?

Cheers,

Sergio

--

¹ Anyway, the shape of the plots looks somewhat strange to me. I don't have enough experience to tell what is going on there but they look different from the typical plot shape.

[Naimur_Rahman1 (Naimur Rahman)]

I imported the the sequences via Manifest file(screenshot attached). As i am new with this analysis, so i have no idea about expected amplicon size. i have tried again without truncating the sequence, still failed.Here is the output-
qiime dada2 denoise-paired
--i-demultiplexed-seqs inputs/demux-Sabrina.qza
--p-trim-left-f 19
--p-trim-left-r 20
--p-max-ee-f 3
--p-max-ee-r 3
--p-trunc-q 5
--p-trunc-len-f 0
--p-trunc-len-r 0
--p-n-threads 4
--o-representative-sequences dada2/rep-seqs_Sabrina_dada2.qza
--o-table dada2/table_Sabrina_dada2.qza
--o-denoising-stats dada2/stats_Sabrina_dada2.qza
--verbose
Running external command line application(s). This may print messages to stdout and/or stderr.
The command(s) being run are below. These commands cannot be manually re-run as they will depend on temporary files that no longer exist.

Command: run_dada.R --input_directory /tmp/tmpjyc6i7lt/forward --input_directory_reverse /tmp/tmpjyc6i7lt/reverse --output_path /tmp/tmpjyc6i7lt/output.tsv.biom --output_track /tmp/tmpjyc6i7lt/track.tsv --filtered_directory /tmp/tmpjyc6i7lt/filt_f --filtered_directory_reverse /tmp/tmpjyc6i7lt/filt_r --truncation_length 0 --truncation_length_reverse 0 --trim_left 19 --trim_left_reverse 20 --max_expected_errors 3 --max_expected_errors_reverse 3 --truncation_quality_score 5 --min_overlap 12 --pooling_method independent --chimera_method consensus --min_parental_fold 1.0 --allow_one_off False --num_threads 4 --learn_min_reads 1000000

R version 4.3.3 (2024-02-29)
Loading required package: Rcpp
DADA2: 1.30.0 / Rcpp: 1.0.12 / RcppParallel: 5.1.6
2) Filtering ....
3) Learning Error Rates
48335975 total bases in 367000 reads from 4 samples will be used for learning the error rates.
47600627 total bases in 367000 reads from 4 samples will be used for learning the error rates.
3) Denoise samples ....
....
5) Remove chimeras (method = consensus)
Error in isBimeraDenovoTable(unqs[[i]], ..., verbose = verbose) :
Input must be a valid sequence table.
Calls: removeBimeraDenovo -> isBimeraDenovoTable
3: stop("Input must be a valid sequence table.")
2: isBimeraDenovoTable(unqs[[i]], ..., verbose = verbose)
1: removeBimeraDenovo(seqtab, method = chimeraMethod, minFoldParentOverAbundance = minParentFold,
allowOneOff = allowOneOff, multithread = multithread)
Traceback (most recent call last):
File "/home/naimur/miniconda3/envs/qiime2-amplicon-2024.5/lib/python3.9/site-packages/q2_dada2/_denoise.py", line 350, in denoise_paired
run_commands([cmd])
File "/home/naimur/miniconda3/envs/qiime2-amplicon-2024.5/lib/python3.9/site-packages/q2_dada2/_denoise.py", line 37, in run_commands
subprocess.run(cmd, check=True)
File "/home/naimur/miniconda3/envs/qiime2-amplicon-2024.5/lib/python3.9/subprocess.py", line 528, in run
raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['run_dada.R', '--input_directory', '/tmp/tmpjyc6i7lt/forward', '--input_directory_reverse', '/tmp/tmpjyc6i7lt/reverse', '--output_path', '/tmp/tmpjyc6i7lt/output.tsv.biom', '--output_track', '/tmp/tmpjyc6i7lt/track.tsv', '--filtered_directory', '/tmp/tmpjyc6i7lt/filt_f', '--filtered_directory_reverse', '/tmp/tmpjyc6i7lt/filt_r', '--truncation_length', '0', '--truncation_length_reverse', '0', '--trim_left', '19', '--trim_left_reverse', '20', '--max_expected_errors', '3', '--max_expected_errors_reverse', '3', '--truncation_quality_score', '5', '--min_overlap', '12', '--pooling_method', 'independent', '--chimera_method', 'consensus', '--min_parental_fold', '1.0', '--allow_one_off', 'False', '--num_threads', '4', '--learn_min_reads', '1000000']' returned non-zero exit status 1.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/home/naimur/miniconda3/envs/qiime2-amplicon-2024.5/lib/python3.9/site-packages/q2cli/commands.py", line 520, in call
results = self._execute_action(
File "/home/naimur/miniconda3/envs/qiime2-amplicon-2024.5/lib/python3.9/site-packages/q2cli/commands.py", line 581, in _execute_action
results = action(**arguments)
File "", line 2, in denoise_paired
File "/home/naimur/miniconda3/envs/qiime2-amplicon-2024.5/lib/python3.9/site-packages/qiime2/sdk/action.py", line 342, in bound_callable
outputs = self.callable_executor(
File "/home/naimur/miniconda3/envs/qiime2-amplicon-2024.5/lib/python3.9/site-packages/qiime2/sdk/action.py", line 576, in callable_executor
output_views = self._callable(**view_args)
File "/home/naimur/miniconda3/envs/qiime2-amplicon-2024.5/lib/python3.9/site-packages/q2_dada2/_denoise.py", line 363, in denoise_paired
raise Exception("An error was encountered while running DADA2"
Exception: An error was encountered while running DADA2 in R (return code 1), please inspect stdout and stderr to learn more.

Plugin error from dada2:

An error was encountered while running DADA2 in R (return code 1), please inspect stdout and stderr to learn more.

See above for debug info.

[salias (Sergio Alías-Segura)]

@Naimur_Rahman1

As i am new with this analysis, so i have no idea about expected amplicon size.

Maybe knowing which primers were used we can pull the thread. I suppose you know them since you are trimming them by length.

@Naimur_Rahman1

I imported the the sequences via Manifest file(screenshot attached).

The manifest file, at least if we trust the filenames, is correct. Just to make sure, could you move to your FASTQ folder, run md5sum *.fastq and check if the output is different for forward and reverse files?

@Naimur_Rahman1

i have tried again without truncating the sequence, still failed.

Do you perform any quality filtering/trimming prior to DADA2? Either before or after importing to QIIME 2? I keep thinking that the demux.qzv plots look strange, but I don't know exactly why.

[Naimur_Rahman1 (Naimur Rahman)]

1. Forward primer was 5'GTGCCAGCMGCCGCGGTAA-3' and reverse primer was 5'GGACTACHVGGGTWTCTAAT- 3'.
2. Screenshot of FASTQ folder attached.
3. I did not perform any quality filtering/trimming prior to DADA2. this is the raw data.
   
   

[salias (Sergio Alías-Segura)]

@Naimur_Rahman1

Forward primer was 5'GTGCCAGCMGCCGCGGTAA-3' and reverse primer was 5'GGACTACHVGGGTWTCTAAT- 3'.

These are the 16S V4 primers F515/R806 from Caporaso et al., 2011. From the paper Methods section:

For reference, this primer pair amplifies the region 533–786 in the Escherichia coli strain 83972 sequence

That is 253 bp. Your sequence pairs should be long enough to merge. So length is not a problem here.

@Naimur_Rahman1

Screenshot of FASTQ folder attached.

Forward and reverse are different files, as expected. Problem is not here either.

@Naimur_Rahman1

I did not perform any quality filtering/trimming prior to DADA2. this is the raw data.

I should have noted earlier (sorry!) that you are using bigger maxEE and truncQ parameters than the default value, 2. These parameters work in the following way: individual reads are truncated at the first base with a quality score lower than or equal to truncQ (5 in your case). Then, all reads with an equal or higher number of expected errors than maxEE (3 in your case) are removed.

I don't know the reason you had for changing the defaults, but maybe that high truncQ value is the one that is causing trouble. Try to run your original DADA2 command with --p-max-ee-f 2 --p-max-ee-r 2 --p-trunc-q 2 (or, since these are defaults, you can directly omit them). I think truncQ value is too agressive and it is quite likely the reason behind the complete removal of your sequences.

If doing that does not solve the issue, maybe the best option would be to run DADA2 only in forward reads with qiime dada2 denoise-single and see if we keep sequences. But we will cross that bridge when (if) we get to it.

[Naimur_Rahman1 (Naimur Rahman)]

Changing the papmeters to default does not work either.

qiime dada2 denoise-paired
--i-demultiplexed-seqs inputs/demux-Sabrina.qza
--p-trim-left-f 19
--p-trim-left-r 20
--p-max-ee-f 2
--p-max-ee-r 2
--p-trunc-q 2
--p-trunc-len-f 0
--p-trunc-len-r 0
--p-n-threads 4
--o-representative-sequences dada2/rep-seqs_Sabrina_dada2.qza
--o-table dada2/table_Sabrina_dada2.qza
--o-denoising-stats dada2/stats_Sabrina_dada2.qza
--verbose
Running external command line application(s). This may print messages to stdout and/or stderr.
The command(s) being run are below. These commands cannot be manually re-run as they will depend on temporary files that no longer exist.

Command: run_dada.R --input_directory /tmp/tmp8odors2z/forward --input_directory_reverse /tmp/tmp8odors2z/reverse --output_path /tmp/tmp8odors2z/output.tsv.biom --output_track /tmp/tmp8odors2z/track.tsv --filtered_directory /tmp/tmp8odors2z/filt_f --filtered_directory_reverse /tmp/tmp8odors2z/filt_r --truncation_length 0 --truncation_length_reverse 0 --trim_left 19 --trim_left_reverse 20 --max_expected_errors 2 --max_expected_errors_reverse 2 --truncation_quality_score 2 --min_overlap 12 --pooling_method independent --chimera_method consensus --min_parental_fold 1.0 --allow_one_off False --num_threads 4 --learn_min_reads 1000000

R version 4.3.3 (2024-02-29)
Loading required package: Rcpp
DADA2: 1.30.0 / Rcpp: 1.0.12 / RcppParallel: 5.1.6
2) Filtering ....
3) Learning Error Rates
35830663 total bases in 272140 reads from 4 samples will be used for learning the error rates.
35232608 total bases in 272140 reads from 4 samples will be used for learning the error rates.
3) Denoise samples ....
....
5) Remove chimeras (method = consensus)
Error in isBimeraDenovoTable(unqs[[i]], ..., verbose = verbose) :
Input must be a valid sequence table.
Calls: removeBimeraDenovo -> isBimeraDenovoTable
3: stop("Input must be a valid sequence table.")
2: isBimeraDenovoTable(unqs[[i]], ..., verbose = verbose)
1: removeBimeraDenovo(seqtab, method = chimeraMethod, minFoldParentOverAbundance = minParentFold,
allowOneOff = allowOneOff, multithread = multithread)
Traceback (most recent call last):
File "/home/naimur/miniconda3/envs/qiime2-amplicon-2024.5/lib/python3.9/site-packages/q2_dada2/_denoise.py", line 350, in denoise_paired
run_commands([cmd])
File "/home/naimur/miniconda3/envs/qiime2-amplicon-2024.5/lib/python3.9/site-packages/q2_dada2/_denoise.py", line 37, in run_commands
subprocess.run(cmd, check=True)
File "/home/naimur/miniconda3/envs/qiime2-amplicon-2024.5/lib/python3.9/subprocess.py", line 528, in run
raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['run_dada.R', '--input_directory', '/tmp/tmp8odors2z/forward', '--input_directory_reverse', '/tmp/tmp8odors2z/reverse', '--output_path', '/tmp/tmp8odors2z/output.tsv.biom', '--output_track', '/tmp/tmp8odors2z/track.tsv', '--filtered_directory', '/tmp/tmp8odors2z/filt_f', '--filtered_directory_reverse', '/tmp/tmp8odors2z/filt_r', '--truncation_length', '0', '--truncation_length_reverse', '0', '--trim_left', '19', '--trim_left_reverse', '20', '--max_expected_errors', '2', '--max_expected_errors_reverse', '2', '--truncation_quality_score', '2', '--min_overlap', '12', '--pooling_method', 'independent', '--chimera_method', 'consensus', '--min_parental_fold', '1.0', '--allow_one_off', 'False', '--num_threads', '4', '--learn_min_reads', '1000000']' returned non-zero exit status 1.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/home/naimur/miniconda3/envs/qiime2-amplicon-2024.5/lib/python3.9/site-packages/q2cli/commands.py", line 520, in call
results = self._execute_action(
File "/home/naimur/miniconda3/envs/qiime2-amplicon-2024.5/lib/python3.9/site-packages/q2cli/commands.py", line 581, in _execute_action
results = action(**arguments)
File "", line 2, in denoise_paired
File "/home/naimur/miniconda3/envs/qiime2-amplicon-2024.5/lib/python3.9/site-packages/qiime2/sdk/action.py", line 342, in bound_callable
outputs = self.callable_executor(
File "/home/naimur/miniconda3/envs/qiime2-amplicon-2024.5/lib/python3.9/site-packages/qiime2/sdk/action.py", line 576, in callable_executor
output_views = self._callable(**view_args)
File "/home/naimur/miniconda3/envs/qiime2-amplicon-2024.5/lib/python3.9/site-packages/q2_dada2/_denoise.py", line 363, in denoise_paired
raise Exception("An error was encountered while running DADA2"
Exception: An error was encountered while running DADA2 in R (return code 1), please inspect stdout and stderr to learn more.

Plugin error from dada2:

An error was encountered while running DADA2 in R (return code 1), please inspect stdout and stderr to learn more.

See above for debug info.

[salias (Sergio Alías-Segura)]

It looks like the quality of your sequences does not allow DADA2 to merge them. In this case, I would go only with forward reads. If you want, you can try importing only forward reads and run qiime dada2 denoise-single on them.

[Mike_Stevenson]

Hi, can I ask why the —p-trunc-q parameter is set to 2 by default? Is this not very low? For example, if most of my sequences are on average Q20+, should I increase the value above 2 in order to keep higher quality sequences? Or is there an advantage to using this lower default value? Apologies, I’m very new to all of this. Many thanks in advance!

[salias (Sergio Alías-Segura)]

Hi @Mike_Stevenson ,

There is a really interesting discussion about that in this issue of the DADA2 GitHub repository.

[colinbrislawn (Colin J Brislawn)]

@Mike_Stevenson

if most of my sequences are on average Q20+

I used to use average q-scores too, until I read this:
https://www.drive5.com/usearch/manual/avgq.html

should I increase the value above 2 in order to keep higher quality sequences?

You could, but DADA2 has an internal denoiser so it matters a lot less for this particular program.

[Mike_Stevenson]

Hi Colin. Instead of using average sequence quality scores from FastQC, is there another metric you would recommend for selecting one sequencing run over another for the same samples? Initially I had used MultiQC to visually determine the “best run”, however recently I swapped to using the average sequence quality from FastQC for my batches. I then put the runs in order according to this metric and chose which runs to use for downstream analysis based on this. If there is a more logical/better methodology for choosing one run over another, would you be able to recommend it? Many thanks!

[colinbrislawn (Colin J Brislawn)]

Hi Mike,

We appear to be on a Robert Edgar theme lately. He recommends Expected Error as a replacement for average q-score:
https://www.drive5.com/usearch/manual/exp_errs.html

So you could get pre-run stats, then choose the run with the lowest Expected Error.
https://docs.qiime2.org/2024.5/plugins/available/vsearch/fastq-stats/

If both runs are of okay quality, you could use both of them as technical replicates.