dada2 error: filtered all reads

Technical Support
1

I also have problem with another dada2 run for 730 paired-end forward reads as follows

qiime dada2 denoise-single --i-demultiplexed-seqs pe_f-demux.qza --output-dir dada2_f/ --p-trunc-len 280 --p-trim-left 20 --p-n-threads 16

Plugin error from dada2:

  An error was encountered while running DADA2 in R (return code 1), please inspect stdout and stderr to learn more.

Debug info has been saved to /tmp/qiime2-q2cli-err-n7gkm2ku.log
(qiime2-2019.4) ashutosh@fisher:~/data/$ less  /tmp/qiime2-q2cli-err-n7gkm2ku.log
Running external command line application(s). This may print messages to stdout and/or stderr.
The command(s) being run are below. These commands cannot be manually re-run as they will depend on temporary files that no longer exist.

Command: run_dada_single.R /tmp/qiime2-archive-ao0pdm85/d57de86e-9a19-4c2d-8b94-1c9855e7180e/data /tmp/tmpd9noj5qg/output.tsv.biom /tmp/tmpd9noj5qg/track.tsv /tmp/tmpd9noj5qg 280 20 2.0 2 Inf consensus 1.0 16 1000000 NULL 16

R version 3.5.1 (2018-07-02)
Loading required package: Rcpp
DADA2: 1.10.0 / Rcpp: 1.0.1 / RcppParallel: 4.4.2
1) Filtering The filter removed all reads: /tmp/tmpd9noj5qg/SRX1617839_75_L001_R1_001.fastq.gz not written.
The filter removed all reads: /tmp/tmpd9noj5qg/SRX1617905_91_L001_R1_001.fastq.gz not written.
The filter removed all reads: /tmp/tmpd9noj5qg/SRX1617843_78_L001_R1_001.fastq.gz not written.
The filter removed all reads: /tmp/tmpd9noj5qg/SRX1617954_94_L001_R1_001.fastq.gz not written.
The filter removed all reads: /tmp/tmpd9noj5qg/SRX1617841_76_L001_R1_001.fastq.gz not written.

and the end of the log file is as follows

The filter removed all reads: /tmp/tmpd9noj5qg/SRX3660940_352_L001_R1_001.fastq.gz not written.
The filter removed all reads: /tmp/tmpd9noj5qg/SRX3660956_368_L001_R1_001.fastq.gz not written.
The filter removed all reads: /tmp/tmpd9noj5qg/SRX5453580_656_L001_R1_001.fastq.gz not written.
Error in filterAndTrim(unfilts, filts, truncLen = truncLen, trimLeft = trimLeft,  :
  These are the errors (up to 5) encountered in individual cores...
Error in writeFastq(fq, fout, "a", compress = compress) :
  failed to write record 18184
Error in writeFastq(fq, fout, "a", compress = compress) :
  failed to write record 18184
Error in writeFastq(fq, fout, "a", compress = compress) :
  failed to write record 18184
Error in writeFastq(fq, fout, "a", compress = compress) :
  failed to write record 18184
Error in writeFastq(fq, fout, "a", compress = compress) :
  failed to write record 18184
Execution halted
Traceback (most recent call last):
  File "/home/ashutosh/anaconda2/envs/qiime2-2019.4/lib/python3.6/site-packages/q2_dada2/_denoise.py", line 152, in _denoise_single
    run_commands([cmd])
  File "/home/ashutosh/anaconda2/envs/qiime2-2019.4/lib/python3.6/site-packages/q2_dada2/_denoise.py", line 36, in run_commands
    subprocess.run(cmd, check=True)
  File "/home/ashutosh/anaconda2/envs/qiime2-2019.4/lib/python3.6/subprocess.py", line 418, in run
    output=stdout, stderr=stderr)
subprocess.CalledProcessError: Command '['run_dada_single.R', '/tmp/qiime2-archive-ao0pdm85/d57de86e-9a19-4c2d-8b94-1c9855e7180e/data', '/tmp/tmpd9noj5qg/output.tsv.biom', '/tmp/tmpd9noj5qg/track.tsv', '/tmp/tmpd9noj5qg', '280', '20', '2.0', '2', 'Inf', 'consensus', '1.0', '16', '1000000', 'NULL', '16']' returned non-zero exit status 1.

During handling of the above exception, another exception occurred:

Can you tell me what is going on

Thanks In advance

Ashutosh

2

Hello Ashutosh,

Thanks for posting that full error message. Here's the core part of the error message:

1) Filtering The filter removed all reads: /tmp/tmpd9noj5qg/SRX1617839_75_L001_R1_001.fastq.gz not written.
The filter removed all reads: /tmp/tmpd9noj5qg/SRX1617905_91_L001_R1_001.fastq.gz not written.
The filter removed all reads: /tmp/tmpd9noj5qg/SRX1617843_78_L001_R1_001.fastq.gz not written.
The filter removed all reads: /tmp/tmpd9noj5qg/SRX1617954_94_L001_R1_001.fastq.gz not written.
The filter removed all reads: /tmp/tmpd9noj5qg/SRX1617841_76_L001_R1_001.fastq.gz not written.

So because the filtered removed all reads, those samples are totally gone.

Do avoid this error, can you rerun this script without those 4 samples and see if the error goes away?

Also, I wanted to ask you about this:

730 paired-end forward reads

Did you mean 730 samples, or a total of 730 reads from all samples? 730 is huge number of samples, but a small number of reads. I just wanted to check in.

Colin

3

hi @colinbrislawn
Thanks for the reply
Sorry, my bad, it is 730 samples, to avoid the error I ran dada2 separately for different run/ project’s data, 540 samples worked except 190 samples data from a single bioproject did not work, came with this error

Plugin error from dada2:
An error was encountered while running DADA2 in R (return code 1), please inspect stdout and stderr to learn more.
Debug info has been saved to /home/ashutosh/kiera_data/data/temp/qiime2-q2cli-err-pyh5ahoa.log

Running external command line application(s). This may print messages to stdout and/or stderr.
The command(s) being run are below. These commands cannot be manually re-run as they will depend on temporary files that no longer exist.

Command: run_dada_paired.R /home/ashutosh/kiera_data/data/temp/tmpxcsyp3h0/forward /home/ashutosh/kiera_data/data/temp/tmpxcsyp3h0/reverse /home/ashutosh/kiera_data/data/temp/tmpxcsyp3h0/output.tsv.biom /home/ashutosh/kiera_data/data/temp/tmpxcsyp3h0/track.tsv /home/ashutosh/kiera_data/data/temp/tmpxcsyp3h0/filt_f /home/ashutosh/kiera_data/data/temp/tmpxcsyp3h0/filt_r 290 250 15 15 2.0 2 consensus 1.0 16 1000000

R version 3.5.1 (2018-07-02)
Loading required package: Rcpp
DADA2: 1.10.0 / Rcpp: 1.0.1 / RcppParallel: 4.4.2
1) Filtering Error in filterAndTrim(unfiltsF, filtsF, unfiltsR, filtsR, truncLen = c(truncLenF,  :
  These are the errors (up to 5) encountered in individual cores...
Error in (function (fn, fout, maxN = c(0, 0), truncQ = c(2, 2), truncLen = c(0,  :
  Mismatched forward and reverse sequence files: 61405, 93519.
Error in (function (fn, fout, maxN = c(0, 0), truncQ = c(2, 2), truncLen = c(0,  :
  Mismatched forward and reverse sequence files: 61405, 93519.
Error in (function (fn, fout, maxN = c(0, 0), truncQ = c(2, 2), truncLen = c(0,  :
  Mismatched forward and reverse sequence files: 61405, 93519.
Error in (function (fn, fout, maxN = c(0, 0), truncQ = c(2, 2), truncLen = c(0,  :
  Mismatched forward and reverse sequence files: 61405, 93519.
Error in (function (fn, fout, maxN = c(0, 0), truncQ = c(2, 2), truncLen = c(0,  :
  Mismatched forward and reverse sequence files: 61405, 93519.
Execution halted
Traceback (most recent call last):
  File "/home/ashutosh/anaconda2/envs/qiime2-2019.4/lib/python3.6/site-packages/q2_dada2/_denoise.py", line 231, in denoise_paired
run_commands([cmd])
  File "/home/ashutosh/anaconda2/envs/qiime2-2019.4/lib/python3.6/site-packages/q2_dada2/_denoise.py", line 36, in run_commands
subprocess.run(cmd, check=True)
  File "/home/ashutosh/anaconda2/envs/qiime2-2019.4/lib/python3.6/subprocess.py", line 418, in run
output=stdout, stderr=stderr)
subprocess.CalledProcessError: Command '['run_dada_paired.R', '/home/ashutosh/kiera_data/data/temp/tmpxcsyp3h0/forward', '/home/ashutosh/kiera_data/data/temp/tmpxcsyp3h0/reverse', '/home/ashutosh/kiera_data/data/temp/tmpxcsyp3h0/output.tsv.biom', '/home/ashutosh/kiera_data/data/temp/tmpxcsyp3h0/track.tsv', '/home/ashutosh/kiera_data/data/temp/tmpxcsyp3h0/filt_f', '/home/ashutosh/kiera_data/data/temp/tmpxcsyp3h0/filt_r', '290', '250', '15', '15', '2.0', '2', 'consensus', '1.0', '16', '1000000']' returned non-zero exit status 1.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/ashutosh/anaconda2/envs/qiime2-2019.4/lib/python3.6/site-packages/q2cli/commands.py", line 311, in __call__
results = action(**arguments)
  File "</home/ashutosh/anaconda2/envs/qiime2-2019.4/lib/python3.6/site-packages/decorator.py:decorator-gen-451>", line 2, in denoise_paired
  File "/home/ashutosh/anaconda2/envs/qiime2-2019.4/lib/python3.6/site-packages/qiime2/sdk/action.py", line 231, in bound_callable
output_types, provenance)
  File "/home/ashutosh/anaconda2/envs/qiime2-2019.4/lib/python3.6/site-packages/qiime2/sdk/action.py", line 365, in _callable_executor_
output_views = self._callable(**view_args)
  File "/home/ashutosh/anaconda2/envs/qiime2-2019.4/lib/python3.6/site-packages/q2_dada2/_denoise.py", line 246, in denoise_paired
" and stderr to learn more." % e.returncode)
Exception: An error was encountered while running DADA2 in R (return code 1), please inspect stdout and stderr to learn more.

Any clue, should I go with only forward reads for these samples?

Thanks in advanced

Ashutosh

4

Good morning Ashutosh,

Great!

For this new error, it looks like the forward file is missing reads compared to the reverse file. That could be due to the primer filtering.

You could continue using only forward reads... but it would be best to get filtering working and use ALL your data. I'm not sure the best way to get this up with DADA2. Maybe a qiime developer can offer a suggestion?

Colin