Dada2 "No such file or directory" and "Error in open.connection" message

[ebolyen (Evan Bolyen)]

@Bing

I can send my data to you again for the testing. Please let me know and thanks!!!

Thanks for the offer! I think I'm going to try and reproduce with a smaller dataset (since it looks like a hardware problem). I haven't had a chance to really get started on this yet, but I'll let you know how it goes.

[Bing (Jinbing Bai)]

Hi,
I cannot solve that error in my iMAC so I changed to Virtual Machine in windows. Just got the following error:

qiime2@qiime2core2017-4:~/ShareWindows/vm$ qiime dada2 denoise-paired --i-demultiplexed-seqs demux-paired-end.qza --p-trim-left-f 0 --p-trunc-len-f 270 --p-trim-left-r 0 --p-trunc-len-r 215 --o-table bing-dada2-table.qza --o-representative-sequences bing-data2-rep-seqs.qza --verbose

Running external command line application(s). This may print messages to stdout and/or stderr.
The command(s) being run are below. These commands cannot be manually re-run as they will depend on temporary files that no longer exist.

Command: run_dada_paired.R /tmp/tmpftxl8adr/forward /tmp/tmpftxl8adr/reverse /tmp/tmpftxl8adr/output.tsv.biom /tmp/tmpftxl8adr/filt_f /tmp/tmpftxl8adr/filt_r 270 215 0 0 2.0 2 pooled 1.0 1 1000000

R version 3.3.1 (2016-06-21) 
Loading required package: Rcpp
Warning messages:
1: multiple methods tables found for ‘arbind’ 
2: multiple methods tables found for ‘acbind’ 
3: replacing previous import ‘IRanges::arbind’ by ‘SummarizedExperiment::arbind’ when loading ‘GenomicAlignments’ 
4: replacing previous import ‘IRanges::acbind’ by ‘SummarizedExperiment::acbind’ when loading ‘GenomicAlignments’ 
5: multiple methods tables found for ‘left’ 
6: multiple methods tables found for ‘right’ 
DADA2 R package version: 1.1.7 
1) Filtering ..............Traceback (most recent call last):
  File "/miniconda3/lib/python3.5/site-packages/q2cli/commands.py", line 218, in __call__
    results = action(**arguments)
  File "<decorator-gen-225>", line 2, in denoise_paired
  File "/miniconda3/lib/python3.5/site-packages/qiime2/sdk/action.py", line 171, in callable_wrapper
    output_types, provenance)
  File "/miniconda3/lib/python3.5/site-packages/qiime2/sdk/action.py", line 248, in _callable_executor_
    output_views = callable(**view_args)
  File "/miniconda3/lib/python3.5/site-packages/q2_dada2/_denoise.py", line 163, in denoise_paired
    run_commands([cmd])
  File "/miniconda3/lib/python3.5/site-packages/q2_dada2/_plot.py", line 26, in run_commands
    subprocess.run(cmd, check=True)
  File "/miniconda3/lib/python3.5/subprocess.py", line 398, in run
    output=stdout, stderr=stderr)
subprocess.CalledProcessError: Command '['run_dada_paired.R', '/tmp/tmpftxl8adr/forward', '/tmp/tmpftxl8adr/reverse', '/tmp/tmpftxl8adr/output.tsv.biom', '/tmp/tmpftxl8adr/filt_f', '/tmp/tmpftxl8adr/filt_r', '270', '215', '0', '0', '2.0', '2', 'pooled', '1.0', '1', '1000000']' returned non-zero exit status -9

Plugin error from dada2:

  Command '['run_dada_paired.R', '/tmp/tmpftxl8adr/forward',
  '/tmp/tmpftxl8adr/reverse', '/tmp/tmpftxl8adr/output.tsv.biom',
  '/tmp/tmpftxl8adr/filt_f', '/tmp/tmpftxl8adr/filt_r', '270', '215',
  '0', '0', '2.0', '2', 'pooled', '1.0', '1', '1000000']' returned non-
  zero exit status -9

See above for debug info.

Please help me and thanks!

[ebolyen (Evan Bolyen)]

I can't say I've seen -9 as an exit code before, but I wonder if it's not a response from SIGKILL. There's not much else in that output, so I can't really tell what, if anything, went wrong there. Out of curiosity, is ~/ShareWindows a mounted directory from your host OS? Does anything change when you work out of $HOME (.qza's and all)?

Related-ly, I went back through this entire thread and I completely missed how you had re-run on your iMac's internal drive. So I don't think it has anything to do with multiple devices (which is the bug we're sorting out atm). I'm kind of back to thinking this is a memory situation (for the iMac).

[ebolyen (Evan Bolyen)]

Also, I've been unable to reproduce this issue with a USB-drive locally , so that's also part of my pivot back towards this being memory-related.

[Bing (Jinbing Bai)]

@ebolyen [quote="ebolyen, post:43, topic:412"]
Out of curiosity, is ~/ShareWindows a mounted directory from your host OS? Does anything change when you work out of $HOME (.qza's and all)?
[/quote]

Yes, it is a mounted directory from the drive in the windows desktop. The problem keeps happening in my iMac so I decide to give it a try in the windows' desktop!!! I have not tried the $home yet!

@ebolyen

Also, I've been unable to reproduce this issue with a USB-drive locally , so that's also part of my pivot back towards this being memory-related.

I feel frustrated about this problem. It got successful once and then suddenly not working any more!! Hopefully the memory extension will solve the problem~!!!

When you run a dataset, what is the minimum memory required for the 10GB dataset?

Thanks,
Bing

[ebolyen (Evan Bolyen)]

@Bing

I feel frustrated about this problem. It got successful once and then suddenly not working any more!! Hopefully the memory extension will solve the problem~!!!

That's pretty much how I feel too. These are some very strange issues we've seen in this thread. (Even still, this is probably my favorite thread on the forum so far, thanks for all of the help!)

@Bing

When you run a dataset, what is the minimum memory required for the 10GB dataset?

I'm not sure anyone actually knows a concrete answer for that.

[Bing (Jinbing Bai)]

@ebolyen

Thank you for the patient and continuous support. I finally noticed that memory is an issue in this case. I have shifted all my analysis to AWS and it works fast and no error at all!!

Thanks,
Bing

[jairideout (Jai Ram Rideout)]

Awesome, thanks for letting us know what the solution was, and for all your patience with debugging! This forum topic will be really useful to others that might have the issue in the future.

[elolimy (Ahmed Elolimy)]

Gentlemen,

I am using the latest Qiime version 2020.2 and I have been getting the following error after the denoising step:

===============
Plugin error from dada2:

An error was encountered while running DADA2 in R (return code 1), please inspect stdout and stderr to learn more.

Debug info has been saved to /var/folders/0_/kqg94rwj0hn532c1t4jlgtlh1__xx4/T/qiime2-q2cli-err-iua5kczp.log

I gave the following command

===================
qiime dada2 denoise-paired \

--i-demultiplexed-seqs demux.qza
--p-trim-left-f 5
--p-trim-left-r 5
--p-trunc-len-f 240
--p-trunc-len-r 240
--o-table table.qza
--o-representative-sequences rep-seqs.qza
--o-denoising-stats denoising-stats.qza
======================

I have used the following code to open the log file mentioned above "cat /var/folders/0_/kqg94rwj0hn532c1t4jlgtlh1__xx4/T/qiime2-q2cli-err-iua5kczp.log"

And I got the following log file

=======================

Running external command line application(s). This may print messages to stdout and/or stderr.
The command(s) being run are below. These commands cannot be manually re-run as they will depend on temporary files that no longer exist.

Command: run_dada_paired.R /var/folders/0_/kqg94rwj0hn532c1t4jlgtlh1__xx4/T/tmpmryfu4tc/forward /var/folders/0_/kqg94rwj0hn532c1t4jlgtlh1__xx4/T/tmpmryfu4tc/reverse /var/folders/0_/kqg94rwj0hn532c1t4jlgtlh1__xx4/T/tmpmryfu4tc/output.tsv.biom /var/folders/0_/kqg94rwj0hn532c1t4jlgtlh1__xx4/T/tmpmryfu4tc/track.tsv /var/folders/0_/kqg94rwj0hn532c1t4jlgtlh1__xx4/T/tmpmryfu4tc/filt_f /var/folders/0_/kqg94rwj0hn532c1t4jlgtlh1__xx4/T/tmpmryfu4tc/filt_r 240 240 5 5 2.0 2.0 2 consensus 1.0 1 1000000

R version 3.5.1 (2018-07-02)
Loading required package: Rcpp
DADA2: 1.10.0 / Rcpp: 1.0.3 / RcppParallel: 4.4.4

1. Filtering ................................................................................................................................................
2. Learning Error Rates
   315475515 total bases in 1342449 reads from 3 samples will be used for learning the error rates.
   315475515 total bases in 1342449 reads from 3 samples will be used for learning the error rates.
3. Denoise remaining samples .................................................................Error in open.connection(con, "rb") : cannot open the connection
   Calls: derepFastq ... FastqStreamer -> FastqStreamer -> open -> open.connection
   In addition: Warning message:
   In open.connection(con, "rb") :
   cannot open file '/var/folders/0_/kqg94rwj0hn532c1t4jlgtlh1__xx4/T/tmpmryfu4tc/filt_f/28_11_L001_R1_001.fastq.gz': No such file or directory
   Execution halted
   Traceback (most recent call last):
   File "/Users/elolimyahmed/miniconda3/envs/qiime2-2020.2/lib/python3.6/site-packages/q2_dada2/denoise.py", line 257, in denoise_paired
   run_commands([cmd])
   File "/Users/elolimyahmed/miniconda3/envs/qiime2-2020.2/lib/python3.6/site-packages/q2_dada2/denoise.py", line 36, in run_commands
   subprocess.run(cmd, check=True)
   File "/Users/elolimyahmed/miniconda3/envs/qiime2-2020.2/lib/python3.6/subprocess.py", line 418, in run
   output=stdout, stderr=stderr)
   subprocess.CalledProcessError: Command '['run_dada_paired.R', '/var/folders/0/kqg94rwj0hn532c1t4jlgtlh1__xx4/T/tmpmryfu4tc/forward', '/var/folders/0/kqg94rwj0hn532c1t4jlgtlh1__xx4/T/tmpmryfu4tc/reverse', '/var/folders/0_/kqg94rwj0hn532c1t4jlgtlh1__xx4/T/tmpmryfu4tc/output.tsv.biom', '/var/folders/0_/kqg94rwj0hn532c1t4jlgtlh1__xx4/T/tmpmryfu4tc/track.tsv', '/var/folders/0_/kqg94rwj0hn532c1t4jlgtlh1__xx4/T/tmpmryfu4tc/filt_f', '/var/folders/0_/kqg94rwj0hn532c1t4jlgtlh1__xx4/T/tmpmryfu4tc/filt_r', '240', '240', '5', '5', '2.0', '2.0', '2', 'consensus', '1.0', '1', '1000000']' returned non-zero exit status 1.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/Users/elolimyahmed/miniconda3/envs/qiime2-2020.2/lib/python3.6/site-packages/q2cli/commands.py", line 328, in call
results = action(**arguments)
File "</Users/elolimyahmed/miniconda3/envs/qiime2-2020.2/lib/python3.6/site-packages/decorator.py:decorator-gen-455>", line 2, in denoise_paired
File "/Users/elolimyahmed/miniconda3/envs/qiime2-2020.2/lib/python3.6/site-packages/qiime2/sdk/action.py", line 245, in bound_callable
output_types, provenance)
File "/Users/elolimyahmed/miniconda3/envs/qiime2-2020.2/lib/python3.6/site-packages/qiime2/sdk/action.py", line 390, in callable_executor
output_views = self._callable(**view_args)
File "/Users/elolimyahmed/miniconda3/envs/qiime2-2020.2/lib/python3.6/site-packages/q2_dada2/_denoise.py", line 272, in denoise_paired
" and stderr to learn more." % e.returncode)
Exception: An error was encountered while running DADA2 in R (return code 1), please inspect stdout and stderr to learn more.

ANY HELP IS APPRECIATED

[andrewsanchez]

Welcome to the forum, @elolimy!

While I do happen to be a gentleman, it's worth noting that the :qiime2: community takes pride in its diversity!

This is a bit of an odd error which has, fortunately for us, come up on the forum before. Please review the existing discussions (links below). I suspect you will find everything you need there. But if not, don't hesitate to open a new topic with details about how your problem differs from those that have already been solved.

Firstly, the topic I have merged your post into: Dada2 "No such file or directory" and "Error in open.connection" message - #47 by Bing

And for (much) more context:

An error was encountered while running DADA2 in R (return code -9), please inspect stdout and stderr to learn more.

Dear everyone, I am trying using qiime2 to handle my microbiome sequences. But i was stuck into dada2 for merging data and denoising. Here is my screenshot and debug log file. I really need helps, crying... [image] debug-return_error_code_9.txt (2.6 KB)

DADA2 out of memory error

Hi thermokarst, Yes, I moved forward but not too far, just one step beyond . I tried to run dada2 with the options below since I work with 515F/806R primer combination and I found out in the forum that the options below should be fine. However I got an error message (please see the attached files which are screenshots of my computer). I attach as well the quality plots. qiime dada2 denoise-paired --i-demultiplexed-seqs demux.qza --o-table table --o-representative-sequences rep-seqs…

Thanks for your question, @elolimy!

Best,
Andrew