Error in filtering using DADA2

User Support
#1

Hello, all

I am using qiime2 to analyze ITS2 amplicon sequencing data, and now saw immediate errors when using the following command,

qiime dada2 denoise-paired --i-demultiplexed-seqs paired-end-demux-ITS-paired-end.qza --p-trunc-len-f 265 --p-trim-left-f 19 --p-trunc-len-r 250 --p-trim-left-r 20 --p-n-threads 16 --p-max-ee 4 --o-table ITS-table-both_forward_reverse_max_ee_4 --o-representative-sequences ITS-seqs_both_forward_reverse_max_ee_4 --verbose

Errors are

Running external command line application(s). This may print messages to stdout and/or stderr.
The command(s) being run are below. These commands cannot be manually re-run as they will depend on temporary files that no longer exist.

Command: run_dada_paired.R /tmp/tmpyf29ch6o/forward /tmp/tmpyf29ch6o/reverse /tmp/tmpyf29ch6o/output.tsv.biom /tmp/tmpyf29ch6o/filt_f /tmp/tmpyf29ch6o/filt_r 265 250 19 20 4.0 2 consensus 1.0 16 1000000

R version 3.4.1 (2017-06-30)
Loading required package: Rcpp
DADA2 R package version: 1.6.0

  1. Filtering Error in filterAndTrim(unfiltsF, filtsF, unfiltsR, filtsR, truncLen = c(truncLenF, :
    These are the errors (up to 5) encountered in individual cores…
    Error in (function (fn, fout, maxN = c(0, 0), truncQ = c(2, 2), truncLen = c(0, :
    Mismatched forward and reverse sequence files: 34413, 28920.
    Error in (function (fn, fout, maxN = c(0, 0), truncQ = c(2, 2), truncLen = c(0, :
    Mismatched forward and reverse sequence files: 34413, 28920.
    Error in (function (fn, fout, maxN = c(0, 0), truncQ = c(2, 2), truncLen = c(0, :
    Mismatched forward and reverse sequence files: 34413, 28920.
    Error in (function (fn, fout, maxN = c(0, 0), truncQ = c(2, 2), truncLen = c(0, :
    Mismatched forward and reverse sequence files: 34413, 28920.
    Error in (function (fn, fout, maxN = c(0, 0), truncQ = c(2, 2), truncLen = c(0, :
    Mismatched forward and reverse sequence files: 34413, 28920.
    Execution halted
    Traceback (most recent call last):
    File “/opt/miniconda3/envs/qiime2-2017.12/lib/python3.5/site-packages/q2_dada2/_denoise.py”, line 179, in denoise_paired
    run_commands([cmd])
    File “/opt/miniconda3/envs/qiime2-2017.12/lib/python3.5/site-packages/q2_dada2/_denoise.py”, line 35, in run_commands
    subprocess.run(cmd, check=True)
    File “/opt/miniconda3/envs/qiime2-2017.12/lib/python3.5/subprocess.py”, line 398, in run
    output=stdout, stderr=stderr)
    subprocess.CalledProcessError: Command ‘[‘run_dada_paired.R’, ‘/tmp/tmpyf29ch6o/forward’, ‘/tmp/tmpyf29ch6o/reverse’, ‘/tmp/tmpyf29ch6o/output.tsv.biom’, ‘/tmp/tmpyf29ch6o/filt_f’, ‘/tmp/tmpyf29ch6o/filt_r’, ‘265’, ‘250’, ‘19’, ‘20’, ‘4.0’, ‘2’, ‘consensus’, ‘1.0’, ‘16’, ‘1000000’]’ returned non-zero exit status 1

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File “/opt/miniconda3/envs/qiime2-2017.12/lib/python3.5/site-packages/q2cli/commands.py”, line 224, in call
results = action(**arguments)
File “”, line 2, in denoise_paired
File “/opt/miniconda3/envs/qiime2-2017.12/lib/python3.5/site-packages/qiime2/sdk/action.py”, line 228, in bound_callable
output_types, provenance)
File “/opt/miniconda3/envs/qiime2-2017.12/lib/python3.5/site-packages/qiime2/sdk/action.py”, line 363, in callable_executor
output_views = self._callable(**view_args)
File “/opt/miniconda3/envs/qiime2-2017.12/lib/python3.5/site-packages/q2_dada2/_denoise.py”, line 194, in denoise_paired
" and stderr to learn more." % e.returncode)
Exception: An error was encountered while running DADA2 in R (return code 1), please inspect stdout and stderr to learn more.

Plugin error from dada2:

An error was encountered while running DADA2 in R (return code 1), please inspect stdout and stderr to learn more.

See above for debug info.

And also, for analyzing ITS2 data, what is the recommended overlap length for doing the end merging? Any tutorial on this now?

Cheers

DADA2 Error (return code 1)
Error in dada2 denoised-paired
#2
#3

Hi @hongwei2017! Fortunately, this error has nothing to do with the fact that these are ITS reads. Check out this error message:

Apparently something is wrong with at least one of your samples - the number of reads in the reverse sequences are significantly less than in the forward sequences. These values should be identical (one reverse read for each forward read).

You can run qiime tools validate paired-end-demux-ITS-paired-end.qza to get a detailed validation report, although you probably won’t learn anything new. You should double check your source data - did you import the correct file? Also, did you use a manifest format for import? If so, double-check that each forward/reverse pair belongs to the same sample - it is easy to make typos here!

I believe DADA2 will need an overlap of at least nts (this isn’t specific to any particular amplicon, just in general). We don’t have an ITS tutorial just yet, but stay tuned! Feel free to ask any additional ITS-related questions in a new thread. Thanks! :t_rex:

#4
#5

Hi @thermokarst

I didn’t solve the problem. I checked my manifest file, it was OK, doesn’t see any typo mistakes there. My pipeline worked well for other data sets, but really no idea why it stopped working on these new samples.

Errors
R version 3.4.1 (2017-06-30)
Loading required package: Rcpp
DADA2 R package version: 1.6.0

  1. Filtering Error in filterAndTrim(unfiltsF, filtsF, unfiltsR, filtsR, truncLen = c(truncLenF, :
    These are the errors (up to 5) encountered in individual cores…
    Error in (function (fn, fout, maxN = c(0, 0), truncQ = c(2, 2), truncLen = c(0, :
    Mismatched forward and reverse sequence files: 34413, 28920.
    Error in (function (fn, fout, maxN = c(0, 0), truncQ = c(2, 2), truncLen = c(0, :
    Mismatched forward and reverse sequence files: 34413, 28920.
    Error in (function (fn, fout, maxN = c(0, 0), truncQ = c(2, 2), truncLen = c(0, :
    Mismatched forward and reverse sequence files: 34413, 28920.
    Error in (function (fn, fout, maxN = c(0, 0), truncQ = c(2, 2), truncLen = c(0, :
    Mismatched forward and reverse sequence files: 34413, 28920.
    Error in (function (fn, fout, maxN = c(0, 0), truncQ = c(2, 2), truncLen = c(0, :
    Mismatched forward and reverse sequence files: 34413, 28920.
    Execution halted
    Traceback (most recent call last):
    File “/opt/miniconda3/envs/qiime2-2017.12/lib/python3.5/site-packages/q2_dada2/_denoise.py”, line 179, in denoise_paired
    run_commands([cmd])
    File “/opt/miniconda3/envs/qiime2-2017.12/lib/python3.5/site-packages/q2_dada2/_denoise.py”, line 35, in run_commands
    subprocess.run(cmd, check=True)
    File “/opt/miniconda3/envs/qiime2-2017.12/lib/python3.5/subprocess.py”, line 398, in run
    output=stdout, stderr=stderr)
    subprocess.CalledProcessError: Command ‘[‘run_dada_paired.R’, ‘/tmp/tmpjhr8nw1r/forward’, ‘/tmp/tmpjhr8nw1r/reverse’, ‘/tmp/tmpjhr8nw1r/output.tsv.biom’, ‘/tmp/tmpjhr8nw1r/filt_f’, ‘/tmp/tmpjhr8nw1r/filt_r’, ‘240’, ‘230’, ‘19’, ‘20’, ‘4.0’, ‘2’, ‘consensus’, ‘1.0’, ‘16’, ‘1000000’]’ returned non-zero exit status 1

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File “/opt/miniconda3/envs/qiime2-2017.12/lib/python3.5/site-packages/q2cli/commands.py”, line 224, in call
results = action(**arguments)
File “”, line 2, in denoise_paired
File “/opt/miniconda3/envs/qiime2-2017.12/lib/python3.5/site-packages/qiime2/sdk/action.py”, line 228, in bound_callable
output_types, provenance)
File “/opt/miniconda3/envs/qiime2-2017.12/lib/python3.5/site-packages/qiime2/sdk/action.py”, line 363, in callable_executor
output_views = self._callable(**view_args)
File “/opt/miniconda3/envs/qiime2-2017.12/lib/python3.5/site-packages/q2_dada2/_denoise.py”, line 194, in denoise_paired
" and stderr to learn more." % e.returncode)
Exception: An error was encountered while running DADA2 in R (return code 1), please inspect stdout and stderr to learn more.

Cheers

#6
#7

Hi @hongwei2017, please double-check my post from earlier:

At least one of your samples has a corrupt forward/reverse pair. These files should have the exact same number of sequences in them, but for some reason yours don’t. This isn’t something caused by QIIME 2, or something you can fix with QIIME 2 - you will need to go back to your source data to figure out what is going on. My suggestion about checking your manifest is just one of many places where things could have gone wrong. Maybe your files didn’t download completely when you received them from the sequencing center, for example. Keep us posted. Thanks! :t_rex:

#8
#9

Hi @thermokarst

I have double checked my raw sequence data, it seems like the reads number in forward and reverse is not equal. This should have happened to at least one sample, but don’t know how to find it/them. Is there anything I can do to make the both end sequences even before going for the denoising step? I heard from others that we may do this using Qiime1, but I have no experience with it and not sure what I can do here:).

Thanks!

#10
#11

Yep, I have pointed that out now several times in this thread.

There isn’t anything in QIIME 2 for this, but you can try to run the following to help diagnose:

qiime tools export paired-end-demux-ITS-paired-end.qza --output-dir debugging
cd debugging
for f in *.fastq.gz; do r=$(( $(gunzip -c $f | wc -l | tr -d '[:space:]') / 4 )); echo $r $f; done

Theoretically you should see output that looks similar to this:

1046 YUN3428.1_30_L001_R1_001.fastq.gz
1046 YUN3428.1_30_L001_R2_001.fastq.gz
1338 YUN3428.2_45_L001_R1_001.fastq.gz
1338 YUN3428.2_45_L001_R2_001.fastq.gz
954 YUN3428.3_13_L001_R1_001.fastq.gz
954 YUN3428.3_13_L001_R2_001.fastq.gz
897 YUN3533.1.1_37_L001_R1_001.fastq.gz
897 YUN3533.1.1_37_L001_R2_001.fastq.gz
861 YUN3533.1.2_51_L001_R1_001.fastq.gz
861 YUN3533.1.2_51_L001_R2_001.fastq.gz

Every two lines should correspond to a single sample (forward and reverse), the two number should be identical (these are the number of raw reads in the file) - you are looking for the pair that isn’t matched. Keep us posted! :t_rex:

Mismatched forward and reverse sequence files
#12
#13

Hi @thermokarst

Using this command I found the sample that has unmatched reads number in F and R files. Now denoising has passed through Filtering step…

Thanks a lot with this help.

Cheers

#14

This topic was automatically closed 31 days after the last reply. New replies are no longer allowed.