Dada2 "No such file or directory" and "Error in open.connection" message

An off-topic reply has been split into a new topic: DADA2 Duplicate Sample IDs

Please keep replies on-topic in the future.

Here is the results:
`Last login: Sun May 7 23:45:33 on console
Jinbings-iMac:~ jinbingbai$ df -h
Filesystem Size Used Avail Capacity iused ifree %iused Mounted on
/dev/disk0s2 931Gi 287Gi 644Gi 31% 2138066 4292829213 0% /
devfs 182Ki 182Ki 0Bi 100% 630 0 100% /dev
map -hosts 0Bi 0Bi 0Bi 100% 0 0 100% /net
map auto_home 0Bi 0Bi 0Bi 100% 0 0 100% /home
/dev/disk1s1 1.5Ti 59Gi 1.4Ti 4% 786 4294966493 0% /Volumes/LaCie
/dev/disk1s2 373Gi 145Mi 372Gi 1% 0 0 100% /Volumes/LACIE SHARE

Jinbings-iMac:~ jinbingbai$ mount
/dev/disk0s2 on / (hfs, local, journaled)
devfs on /dev (devfs, local, nobrowse)
map -hosts on /net (autofs, nosuid, automounted, nobrowse)
map auto_home on /home (autofs, automounted, nobrowse)
/dev/disk1s1 on /Volumes/LaCie (hfs, local, nodev, nosuid, journaled, noowners)
/dev/disk1s2 on /Volumes/LACIE SHARE (msdos, local, nodev, nosuid, noowners)
Jinbings-iMac:~ jinbingbai$ `

Here is the results: Jinbings-iMac:~ jinbingbai$ source activate qiime2-2017.4
(qiime2-2017.4) Jinbings-iMac:~ jinbingbai$ python -c "import tempfile; print(tempfile.gettempdir())"
/var/folders/sd/vlrr9d916gg5qj3yzzf3srfw0000gn/T

So do I need to fix anything?

Thanks

Thanks for that info @Bing! Which volumes were you using (/Volumes/LaCie or /Volumes/LACIE SHARE) in your analyses that were failing?

@jairideout
I use /Volumes/LaCie

Thanks

Thanks @Bing! Please let us know how things go when you have a chance to try a different USB drive.

(The underlying issue may be related to this bug).

Hello @jairideout

I tried another USB 500GB external drive, and also tried the internal drive of my iMac with 700GB available. Both methods failed with the same issue as reported before. The interesting thing is that the missing file for each time is different. I finally gave up trying it. What I did is that I reinstall my iMac to see how that will go!!

Is there anything I can do to handle this issue if it happens again and again?

Thanks,
Bing

2 Likes

We're still not quite sure why this is happening, but I'm going to try replicating this in the next week or two (since you can reliably reproduce with different hardware).

We'll keep you posted as we learn more!

2 Likes

I can send my data to you again for the testing. Please let me know and thanks!!!

Thanks for the offer! I think I'm going to try and reproduce with a smaller dataset (since it looks like a hardware problem). I haven't had a chance to really get started on this yet, but I'll let you know how it goes.

Hi,
I cannot solve that error in my iMAC so I changed to Virtual Machine in windows. Just got the following error:

qiime2@qiime2core2017-4:~/ShareWindows/vm$ qiime dada2 denoise-paired --i-demultiplexed-seqs demux-paired-end.qza --p-trim-left-f 0 --p-trunc-len-f 270 --p-trim-left-r 0 --p-trunc-len-r 215 --o-table bing-dada2-table.qza --o-representative-sequences bing-data2-rep-seqs.qza --verbose

Running external command line application(s). This may print messages to stdout and/or stderr.
The command(s) being run are below. These commands cannot be manually re-run as they will depend on temporary files that no longer exist.

Command: run_dada_paired.R /tmp/tmpftxl8adr/forward /tmp/tmpftxl8adr/reverse /tmp/tmpftxl8adr/output.tsv.biom /tmp/tmpftxl8adr/filt_f /tmp/tmpftxl8adr/filt_r 270 215 0 0 2.0 2 pooled 1.0 1 1000000

R version 3.3.1 (2016-06-21) 
Loading required package: Rcpp
Warning messages:
1: multiple methods tables found for ‘arbind’ 
2: multiple methods tables found for ‘acbind’ 
3: replacing previous import ‘IRanges::arbind’ by ‘SummarizedExperiment::arbind’ when loading ‘GenomicAlignments’ 
4: replacing previous import ‘IRanges::acbind’ by ‘SummarizedExperiment::acbind’ when loading ‘GenomicAlignments’ 
5: multiple methods tables found for ‘left’ 
6: multiple methods tables found for ‘right’ 
DADA2 R package version: 1.1.7 
1) Filtering ..............Traceback (most recent call last):
  File "/miniconda3/lib/python3.5/site-packages/q2cli/commands.py", line 218, in __call__
    results = action(**arguments)
  File "<decorator-gen-225>", line 2, in denoise_paired
  File "/miniconda3/lib/python3.5/site-packages/qiime2/sdk/action.py", line 171, in callable_wrapper
    output_types, provenance)
  File "/miniconda3/lib/python3.5/site-packages/qiime2/sdk/action.py", line 248, in _callable_executor_
    output_views = callable(**view_args)
  File "/miniconda3/lib/python3.5/site-packages/q2_dada2/_denoise.py", line 163, in denoise_paired
    run_commands([cmd])
  File "/miniconda3/lib/python3.5/site-packages/q2_dada2/_plot.py", line 26, in run_commands
    subprocess.run(cmd, check=True)
  File "/miniconda3/lib/python3.5/subprocess.py", line 398, in run
    output=stdout, stderr=stderr)
subprocess.CalledProcessError: Command '['run_dada_paired.R', '/tmp/tmpftxl8adr/forward', '/tmp/tmpftxl8adr/reverse', '/tmp/tmpftxl8adr/output.tsv.biom', '/tmp/tmpftxl8adr/filt_f', '/tmp/tmpftxl8adr/filt_r', '270', '215', '0', '0', '2.0', '2', 'pooled', '1.0', '1', '1000000']' returned non-zero exit status -9

Plugin error from dada2:

  Command '['run_dada_paired.R', '/tmp/tmpftxl8adr/forward',
  '/tmp/tmpftxl8adr/reverse', '/tmp/tmpftxl8adr/output.tsv.biom',
  '/tmp/tmpftxl8adr/filt_f', '/tmp/tmpftxl8adr/filt_r', '270', '215',
  '0', '0', '2.0', '2', 'pooled', '1.0', '1', '1000000']' returned non-
  zero exit status -9

See above for debug info.

Please help me and thanks!

I can’t say I’ve seen -9 as an exit code before, but I wonder if it’s not a response from SIGKILL. There’s not much else in that output, so I can’t really tell what, if anything, went wrong there. Out of curiosity, is ~/ShareWindows a mounted directory from your host OS? Does anything change when you work out of $HOME (.qza's and all)?

Related-ly, I went back through this entire thread and I completely missed how you had re-run on your iMac’s internal drive. So I don’t think it has anything to do with multiple devices (which is the bug we’re sorting out atm). I’m kind of back to thinking this is a memory situation (for the iMac).

1 Like

Also, I’ve been unable to reproduce this issue with a USB-drive locally :frowning2:, so that’s also part of my pivot back towards this being memory-related.

@ebolyen [quote="ebolyen, post:43, topic:412"]
Out of curiosity, is ~/ShareWindows a mounted directory from your host OS? Does anything change when you work out of $HOME (.qza's and all)?
[/quote]

Yes, it is a mounted directory from the drive in the windows desktop. The problem keeps happening in my iMac so I decide to give it a try in the windows' desktop!!! I have not tried the $home yet!

I feel frustrated about this problem. It got successful once and then suddenly not working any more!! Hopefully the memory extension will solve the problem~!!!

When you run a dataset, what is the minimum memory required for the 10GB dataset?

Thanks,
Bing

That's pretty much how I feel too. These are some very strange issues we've seen in this thread. (Even still, this is probably my favorite thread on the forum so far, thanks for all of the help!)

I'm not sure anyone actually knows a concrete answer for that.

2 Likes

@ebolyen

Thank you for the patient and continuous support. I finally noticed that memory is an issue in this case. I have shifted all my analysis to AWS and it works fast and no error at all!!

Thanks,
Bing

5 Likes

Awesome, thanks for letting us know what the solution was, and for all your patience with debugging! This forum topic will be really useful to others that might have the issue in the future.

1 Like

Gentlemen,

I am using the latest Qiime version 2020.2 and I have been getting the following error after the denoising step:

===============
Plugin error from dada2:

An error was encountered while running DADA2 in R (return code 1), please inspect stdout and stderr to learn more.

Debug info has been saved to /var/folders/0_/kqg94rwj0hn532c1t4jlgtlh1__xx4/T/qiime2-q2cli-err-iua5kczp.log

I gave the following command

===================
qiime dada2 denoise-paired \

--i-demultiplexed-seqs demux.qza
--p-trim-left-f 5
--p-trim-left-r 5
--p-trunc-len-f 240
--p-trunc-len-r 240
--o-table table.qza
--o-representative-sequences rep-seqs.qza
--o-denoising-stats denoising-stats.qza
======================

I have used the following code to open the log file mentioned above "cat /var/folders/0_/kqg94rwj0hn532c1t4jlgtlh1__xx4/T/qiime2-q2cli-err-iua5kczp.log"

And I got the following log file

=======================

Running external command line application(s). This may print messages to stdout and/or stderr.
The command(s) being run are below. These commands cannot be manually re-run as they will depend on temporary files that no longer exist.

Command: run_dada_paired.R /var/folders/0_/kqg94rwj0hn532c1t4jlgtlh1__xx4/T/tmpmryfu4tc/forward /var/folders/0_/kqg94rwj0hn532c1t4jlgtlh1__xx4/T/tmpmryfu4tc/reverse /var/folders/0_/kqg94rwj0hn532c1t4jlgtlh1__xx4/T/tmpmryfu4tc/output.tsv.biom /var/folders/0_/kqg94rwj0hn532c1t4jlgtlh1__xx4/T/tmpmryfu4tc/track.tsv /var/folders/0_/kqg94rwj0hn532c1t4jlgtlh1__xx4/T/tmpmryfu4tc/filt_f /var/folders/0_/kqg94rwj0hn532c1t4jlgtlh1__xx4/T/tmpmryfu4tc/filt_r 240 240 5 5 2.0 2.0 2 consensus 1.0 1 1000000

R version 3.5.1 (2018-07-02)
Loading required package: Rcpp
DADA2: 1.10.0 / Rcpp: 1.0.3 / RcppParallel: 4.4.4

  1. Filtering ................................................................................................................................................
  2. Learning Error Rates
    315475515 total bases in 1342449 reads from 3 samples will be used for learning the error rates.
    315475515 total bases in 1342449 reads from 3 samples will be used for learning the error rates.
  3. Denoise remaining samples .................................................................Error in open.connection(con, "rb") : cannot open the connection
    Calls: derepFastq ... FastqStreamer -> FastqStreamer -> open -> open.connection
    In addition: Warning message:
    In open.connection(con, "rb") :
    cannot open file '/var/folders/0_/kqg94rwj0hn532c1t4jlgtlh1__xx4/T/tmpmryfu4tc/filt_f/28_11_L001_R1_001.fastq.gz': No such file or directory
    Execution halted
    Traceback (most recent call last):
    File "/Users/elolimyahmed/miniconda3/envs/qiime2-2020.2/lib/python3.6/site-packages/q2_dada2/denoise.py", line 257, in denoise_paired
    run_commands([cmd])
    File "/Users/elolimyahmed/miniconda3/envs/qiime2-2020.2/lib/python3.6/site-packages/q2_dada2/denoise.py", line 36, in run_commands
    subprocess.run(cmd, check=True)
    File "/Users/elolimyahmed/miniconda3/envs/qiime2-2020.2/lib/python3.6/subprocess.py", line 418, in run
    output=stdout, stderr=stderr)
    subprocess.CalledProcessError: Command '['run_dada_paired.R', '/var/folders/0
    /kqg94rwj0hn532c1t4jlgtlh1__xx4/T/tmpmryfu4tc/forward', '/var/folders/0
    /kqg94rwj0hn532c1t4jlgtlh1__xx4/T/tmpmryfu4tc/reverse', '/var/folders/0_/kqg94rwj0hn532c1t4jlgtlh1__xx4/T/tmpmryfu4tc/output.tsv.biom', '/var/folders/0_/kqg94rwj0hn532c1t4jlgtlh1__xx4/T/tmpmryfu4tc/track.tsv', '/var/folders/0_/kqg94rwj0hn532c1t4jlgtlh1__xx4/T/tmpmryfu4tc/filt_f', '/var/folders/0_/kqg94rwj0hn532c1t4jlgtlh1__xx4/T/tmpmryfu4tc/filt_r', '240', '240', '5', '5', '2.0', '2.0', '2', 'consensus', '1.0', '1', '1000000']' returned non-zero exit status 1.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/Users/elolimyahmed/miniconda3/envs/qiime2-2020.2/lib/python3.6/site-packages/q2cli/commands.py", line 328, in call
results = action(**arguments)
File "</Users/elolimyahmed/miniconda3/envs/qiime2-2020.2/lib/python3.6/site-packages/decorator.py:decorator-gen-455>", line 2, in denoise_paired
File "/Users/elolimyahmed/miniconda3/envs/qiime2-2020.2/lib/python3.6/site-packages/qiime2/sdk/action.py", line 245, in bound_callable
output_types, provenance)
File "/Users/elolimyahmed/miniconda3/envs/qiime2-2020.2/lib/python3.6/site-packages/qiime2/sdk/action.py", line 390, in callable_executor
output_views = self._callable(**view_args)
File "/Users/elolimyahmed/miniconda3/envs/qiime2-2020.2/lib/python3.6/site-packages/q2_dada2/_denoise.py", line 272, in denoise_paired
" and stderr to learn more." % e.returncode)
Exception: An error was encountered while running DADA2 in R (return code 1), please inspect stdout and stderr to learn more.

ANY HELP IS APPRECIATED :slight_smile:

Welcome to the forum, @elolimy!

While I do happen to be a gentleman, it's worth noting that the :qiime2: community takes pride in its diversity! :grinning: :woman_scientist: :man_scientist: :rainbow:

This is a bit of an odd error which has, fortunately for us, come up on the forum before. Please review the existing discussions (links below). I suspect you will find everything you need there. But if not, don't hesitate to open a new topic with details about how your problem differs from those that have already been solved.

Firstly, the topic I have merged your post into: Dada2 "No such file or directory" and "Error in open.connection" message - #47 by Bing

And for (much) more context:

Thanks for your question, @elolimy!

Best,
Andrew

1 Like

This topic was automatically closed after 19 days. New replies are no longer allowed.