Dada2 Error Return Code 1, Paired-End Seqs

[MSullivan]

Hi, I have been running dada2 on paired-end sequences, and had it prematurely terminate. I ran it again with the verbose command and here is the output:

R version 3.4.1 (2017-06-30)
Loading required package: Rcpp
DADA2 R package version: 1.6.0

1. Filtering …

2. Learning Error Rates
   2a) Forward Reads
   Initializing error rates to maximum possible estimate.
   Sample 1 - 111645 reads in 64913 unique sequences.
   Sample 2 - 104707 reads in 58972 unique sequences.
   Sample 3 - 117878 reads in 68120 unique sequences.
   Sample 4 - 92369 reads in 57417 unique sequences.
   Sample 5 - 106093 reads in 62690 unique sequences.
   Sample 6 - 91907 reads in 46850 unique sequences.
   Sample 7 - 94549 reads in 51322 unique sequences.
   Sample 8 - 95859 reads in 57717 unique sequences.
   Sample 9 - 88629 reads in 48246 unique sequences.
   Sample 10 - 89586 reads in 54495 unique sequences.
   Sample 11 - 2311 reads in 371 unique sequences.
   Sample 12 - 116696 reads in 58206 unique sequences.
   selfConsist step 2
   selfConsist step 3
   selfConsist step 4
   selfConsist step 5
   selfConsist step 6
   selfConsist step 7
   selfConsist step 8
   selfConsist step 9
   selfConsist step 10
   Self-consistency loop terminated before convergence.
   2b) Reverse Reads
   Initializing error rates to maximum possible estimate.
   Sample 1 - 111645 reads in 72821 unique sequences.
   Sample 2 - 104707 reads in 66766 unique sequences.
   Sample 3 - 117878 reads in 76935 unique sequences.
   Sample 4 - 92369 reads in 64675 unique sequences.
   Sample 5 - 106093 reads in 71034 unique sequences.
   Sample 6 - 91907 reads in 54980 unique sequences.
   Sample 7 - 94549 reads in 59063 unique sequences.
   Sample 8 - 95859 reads in 64916 unique sequences.
   Sample 9 - 88629 reads in 55434 unique sequences.
   Sample 10 - 89586 reads in 60773 unique sequences.
   Sample 11 - 2311 reads in 618 unique sequences.
   Sample 12 - 116696 reads in 69672 unique sequences.
   selfConsist step 2
   selfConsist step 3
   selfConsist step 4
   selfConsist step 5
   selfConsist step 6
   selfConsist step 7
   selfConsist step 8
   Convergence after 8 rounds.

3. Denoise remaining samples …Error in open.connection(con, “rb”) : cannot open the connection
   Calls: derepFastq … FastqStreamer -> FastqStreamer -> open -> open.connection
   In addition: Warning message:
   In open.connection(con, “rb”) :
   cannot open file ‘/var/folders/9f/8pzx758x42z3yb1fxfk774vc0000gr/T/tmpvpikxey3/filt_f/MO.BAC.9.F.1_S820_L002_R1_001.fastq.gz’: No such file or directory
   Execution halted
   Traceback (most recent call last):
   File “/Users/madsensullivan/miniconda2/envs/qiime2-2017.12/lib/python3.5/site-packages/q2_dada2/_denoise.py”, line 179, in denoise_paired
   run_commands([cmd])
   File “/Users/madsensullivan/miniconda2/envs/qiime2-2017.12/lib/python3.5/site-packages/q2_dada2/_denoise.py”, line 35, in run_commands
   subprocess.run(cmd, check=True)
   File “/Users/madsensullivan/miniconda2/envs/qiime2-2017.12/lib/python3.5/subprocess.py”, line 398, in run
   output=stdout, stderr=stderr)
   subprocess.CalledProcessError: Command ‘[‘run_dada_paired.R’, ‘/var/folders/9f/8pzx758x42z3yb1fxfk774vc0000gr/T/tmpvpikxey3/forward’, ‘/var/folders/9f/8pzx758x42z3yb1fxfk774vc0000gr/T/tmpvpikxey3/reverse’, ‘/var/folders/9f/8pzx758x42z3yb1fxfk774vc0000gr/T/tmpvpikxey3/output.tsv.biom’, ‘/var/folders/9f/8pzx758x42z3yb1fxfk774vc0000gr/T/tmpvpikxey3/filt_f’, ‘/var/folders/9f/8pzx758x42z3yb1fxfk774vc0000gr/T/tmpvpikxey3/filt_r’, ‘235’, ‘235’, ‘0’, ‘0’, ‘2.0’, ‘2’, ‘consensus’, ‘1.0’, ‘1’, ‘1000000’]’ returned non-zero exit status 1

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File “/Users/madsensullivan/miniconda2/envs/qiime2-2017.12/lib/python3.5/site-packages/q2cli/commands.py”, line 224, in call
results = action(**arguments)
File “”, line 2, in denoise_paired
File “/Users/madsensullivan/miniconda2/envs/qiime2-2017.12/lib/python3.5/site-packages/qiime2/sdk/action.py”, line 228, in bound_callable
output_types, provenance)
File “/Users/madsensullivan/miniconda2/envs/qiime2-2017.12/lib/python3.5/site-packages/qiime2/sdk/action.py”, line 363, in callable_executor
output_views = self._callable(**view_args)
File “/Users/madsensullivan/miniconda2/envs/qiime2-2017.12/lib/python3.5/site-packages/q2_dada2/_denoise.py”, line 194, in denoise_paired
" and stderr to learn more." % e.returncode)
Exception: An error was encountered while running DADA2 in R (return code 1), please inspect stdout and stderr to learn more.

Plugin error from dada2:

An error was encountered while running DADA2 in R (return code 1), please inspect stdout and stderr to learn more.

See above for debug info.

[MSullivan]

Also, when I first ran it without the --verbose command, it said the debug info had been stored, but when I went to the filepath the file didn’t exist. Here was the filepath it had saved the debug info to: /var/folders/9f/8pzx758x42z3yb1fxfk774vc0000gr/T/qiime2-q2cli-err-zrwezvys.log
Not sure if that helps or not! Thanks for your help, I appreciate it!

[MSullivan]

Additionally, this took me about 5 days to run, in case that helps

[jairideout (Jai Ram Rideout)]

Hi @MSullivan! It looks like you're running QIIME 2 on a Mac and you ran into a problem where macOS cleaned out the system's temporary directory while DADA2 was running (this is why the error message is trying to open a temporary file that no longer exists). We have an open issue tracking this macOS problem (check it out for more details).

There are a couple of workarounds you can try; the easiest may be to set a new TMPDIR environment variable. See @ebolyen's post for instructions:

Error involving temporary file (?) in de-noise DADA2 step

Hey @adrian, It looks like you ran into a limitation of OS X. We’ve seen this before and what is happening is your computer is automatically deleting data from its temporary directory that is 3 days old. The solution is to run the command with a different temporary directory set: # Make a directory to use for temporary storage mkdir qiime2-tmp/ # Set that directory as our temporary directory for this session # (this variable only lasts for as long as the terminal exists) export TMPDIR="$PWD/…

Additionally, check out @ebolyen's second post about running DADA2 with multiple threads, which can speed up the denoising process quite a bit:

Error involving temporary file (?) in de-noise DADA2 step

Also, to speed things up, you can use --p-n-threads <some number> which will run some of the steps in parallel. (You can set it to 0 to use every CPU).

[MSullivan]

Oh, gotcha. Thanks @jairideout! I appreciate the help and will try making a new temporary environment and running multiple threads. I’ll update this as I figure it out.

[MSullivan]

Running multiple threads fixed the issue for me, the file came out just fine! Thanks for the help @jairideout!