Dada2 "No such file or directory" and "Error in open.connection" message

User Support
27

Thanks for the update @Bing!

Hmm, that could very well be the problem. Where are you running the analysis? Are you inside the directory that is mounted from your USB drive?

What should be happening is QIIME 2 will read the .qza file and unzip it to your system's temporary directory. Out of curiosity what does this command do inside your environment?

python -c "import tempfile; print(tempfile.gettempdir())"

Does that happen to be your USB drive or is it some crazy path like:
/var/folder/w5/t9wf37ss1cg6s6lhtt8s33gw0000gn/T (which is what I would expect)?

I suppose another situation could be that when QIIME 2 tries to unzip from your USB, some kind of disk-failure/fault occurs that Python's zip library isn't catching.

Thanks again for the update!

28

@ebolyen

Hmm, that could very well be the problem. Where are you running the analysis? Are you inside the directory that is mounted from your USB drive?

I created a folder in my USB drive which is only used for the data analysis. I put all the .fastq in a folder based on the tutorial, and then all the analysis is running in that folder. Does that mean I need to create directory use mkdir?

Out of curiosity what does this command do inside your environment?
    python -c "import tempfile; print(tempfile.gettempdir())"

Do I need to run this command in qiime2 environment or terminal window?

Does that happen to be your USB drive or is it some crazy path like:
/var/folder/w5/t9wf37ss1cg6s6lhtt8s33gw0000gn/T (which is what I would expect)?

This is exactly what it showed in the errors. If that is the case, what are your suggestions to avoid this?

Thanks,
Bing

29

Nope that should work fine, mkdir doesn't do anything special.

Good point. That answers the question then, you don't need to run anything. QIIME 2 is working off of your system drive which probably means that the issue happens from some kind of semi-corrupted unzip. (Unless your system drive is the unstable one and we're just getting lucky).

So I think we can probably point this at hardware failure and something in either the framework or Python 3's standard library that isn't handling the failure with an exception and instead keeps running resulting in "missing files". We're having another kind of error right now related to the zip files, so maybe there is something non-standard happening.

Since you don't have this issue when working on your system's hard-drive I would stick to that. (You may also want to look into backing up your current USB drive, it may be failing).

P.S. Totally unrelated, but on this forum you can just highlight text, while you have the reply editor open, you can quote things (it back-links and everything)!

30

@Bing, is it possible for you to try out this same analysis on a different USB drive? This would let us rule out hardware failure if it succeeds. If you have time/resources to do this we’d really appreciate it! We’d like to track down the source of the issue since it has affected multiple users.

Also, what is the output from running these commands (with your USB drive plugged in)?

df -h
mount

Thanks!

31

I will try this in another USB drive soon and will tell the results! Also I will try the command after I am back to school!

3 Likes
32

An off-topic reply has been split into a new topic: DADA2 Duplicate Sample IDs

Please keep replies on-topic in the future.

33

Here is the results:
`Last login: Sun May 7 23:45:33 on console
Jinbings-iMac:~ jinbingbai$ df -h
Filesystem Size Used Avail Capacity iused ifree %iused Mounted on
/dev/disk0s2 931Gi 287Gi 644Gi 31% 2138066 4292829213 0% /
devfs 182Ki 182Ki 0Bi 100% 630 0 100% /dev
map -hosts 0Bi 0Bi 0Bi 100% 0 0 100% /net
map auto_home 0Bi 0Bi 0Bi 100% 0 0 100% /home
/dev/disk1s1 1.5Ti 59Gi 1.4Ti 4% 786 4294966493 0% /Volumes/LaCie
/dev/disk1s2 373Gi 145Mi 372Gi 1% 0 0 100% /Volumes/LACIE SHARE

Jinbings-iMac:~ jinbingbai$ mount
/dev/disk0s2 on / (hfs, local, journaled)
devfs on /dev (devfs, local, nobrowse)
map -hosts on /net (autofs, nosuid, automounted, nobrowse)
map auto_home on /home (autofs, automounted, nobrowse)
/dev/disk1s1 on /Volumes/LaCie (hfs, local, nodev, nosuid, journaled, noowners)
/dev/disk1s2 on /Volumes/LACIE SHARE (msdos, local, nodev, nosuid, noowners)
Jinbings-iMac:~ jinbingbai$ `

34

Here is the results: Jinbings-iMac:~ jinbingbai$ source activate qiime2-2017.4
(qiime2-2017.4) Jinbings-iMac:~ jinbingbai$ python -c "import tempfile; print(tempfile.gettempdir())"
/var/folders/sd/vlrr9d916gg5qj3yzzf3srfw0000gn/T

So do I need to fix anything?

Thanks

35

Thanks for that info @Bing! Which volumes were you using (/Volumes/LaCie or /Volumes/LACIE SHARE) in your analyses that were failing?

36

@jairideout
I use /Volumes/LaCie

Thanks

37

Thanks @Bing! Please let us know how things go when you have a chance to try a different USB drive.

(The underlying issue may be related to this bug).

38

Hello @jairideout

I tried another USB 500GB external drive, and also tried the internal drive of my iMac with 700GB available. Both methods failed with the same issue as reported before. The interesting thing is that the missing file for each time is different. I finally gave up trying it. What I did is that I reinstall my iMac to see how that will go!!

Is there anything I can do to handle this issue if it happens again and again?

Thanks,
Bing

2 Likes
39

We're still not quite sure why this is happening, but I'm going to try replicating this in the next week or two (since you can reliably reproduce with different hardware).

We'll keep you posted as we learn more!

2 Likes
40

I can send my data to you again for the testing. Please let me know and thanks!!!

41

Thanks for the offer! I think I'm going to try and reproduce with a smaller dataset (since it looks like a hardware problem). I haven't had a chance to really get started on this yet, but I'll let you know how it goes.

42

Hi,
I cannot solve that error in my iMAC so I changed to Virtual Machine in windows. Just got the following error:

qiime2@qiime2core2017-4:~/ShareWindows/vm$ qiime dada2 denoise-paired --i-demultiplexed-seqs demux-paired-end.qza --p-trim-left-f 0 --p-trunc-len-f 270 --p-trim-left-r 0 --p-trunc-len-r 215 --o-table bing-dada2-table.qza --o-representative-sequences bing-data2-rep-seqs.qza --verbose

Running external command line application(s). This may print messages to stdout and/or stderr.
The command(s) being run are below. These commands cannot be manually re-run as they will depend on temporary files that no longer exist.

Command: run_dada_paired.R /tmp/tmpftxl8adr/forward /tmp/tmpftxl8adr/reverse /tmp/tmpftxl8adr/output.tsv.biom /tmp/tmpftxl8adr/filt_f /tmp/tmpftxl8adr/filt_r 270 215 0 0 2.0 2 pooled 1.0 1 1000000

R version 3.3.1 (2016-06-21) 
Loading required package: Rcpp
Warning messages:
1: multiple methods tables found for ‘arbind’ 
2: multiple methods tables found for ‘acbind’ 
3: replacing previous import ‘IRanges::arbind’ by ‘SummarizedExperiment::arbind’ when loading ‘GenomicAlignments’ 
4: replacing previous import ‘IRanges::acbind’ by ‘SummarizedExperiment::acbind’ when loading ‘GenomicAlignments’ 
5: multiple methods tables found for ‘left’ 
6: multiple methods tables found for ‘right’ 
DADA2 R package version: 1.1.7 
1) Filtering ..............Traceback (most recent call last):
  File "/miniconda3/lib/python3.5/site-packages/q2cli/commands.py", line 218, in __call__
    results = action(**arguments)
  File "<decorator-gen-225>", line 2, in denoise_paired
  File "/miniconda3/lib/python3.5/site-packages/qiime2/sdk/action.py", line 171, in callable_wrapper
    output_types, provenance)
  File "/miniconda3/lib/python3.5/site-packages/qiime2/sdk/action.py", line 248, in _callable_executor_
    output_views = callable(**view_args)
  File "/miniconda3/lib/python3.5/site-packages/q2_dada2/_denoise.py", line 163, in denoise_paired
    run_commands([cmd])
  File "/miniconda3/lib/python3.5/site-packages/q2_dada2/_plot.py", line 26, in run_commands
    subprocess.run(cmd, check=True)
  File "/miniconda3/lib/python3.5/subprocess.py", line 398, in run
    output=stdout, stderr=stderr)
subprocess.CalledProcessError: Command '['run_dada_paired.R', '/tmp/tmpftxl8adr/forward', '/tmp/tmpftxl8adr/reverse', '/tmp/tmpftxl8adr/output.tsv.biom', '/tmp/tmpftxl8adr/filt_f', '/tmp/tmpftxl8adr/filt_r', '270', '215', '0', '0', '2.0', '2', 'pooled', '1.0', '1', '1000000']' returned non-zero exit status -9

Plugin error from dada2:

  Command '['run_dada_paired.R', '/tmp/tmpftxl8adr/forward',
  '/tmp/tmpftxl8adr/reverse', '/tmp/tmpftxl8adr/output.tsv.biom',
  '/tmp/tmpftxl8adr/filt_f', '/tmp/tmpftxl8adr/filt_r', '270', '215',
  '0', '0', '2.0', '2', 'pooled', '1.0', '1', '1000000']' returned non-
  zero exit status -9

See above for debug info.

Please help me and thanks!

43

I can’t say I’ve seen -9 as an exit code before, but I wonder if it’s not a response from SIGKILL. There’s not much else in that output, so I can’t really tell what, if anything, went wrong there. Out of curiosity, is ~/ShareWindows a mounted directory from your host OS? Does anything change when you work out of $HOME (.qza's and all)?

Related-ly, I went back through this entire thread and I completely missed how you had re-run on your iMac’s internal drive. So I don’t think it has anything to do with multiple devices (which is the bug we’re sorting out atm). I’m kind of back to thinking this is a memory situation (for the iMac).

1 Like
44

Also, I’ve been unable to reproduce this issue with a USB-drive locally :frowning2:, so that’s also part of my pivot back towards this being memory-related.

45

@ebolyen [quote="ebolyen, post:43, topic:412"]
Out of curiosity, is ~/ShareWindows a mounted directory from your host OS? Does anything change when you work out of $HOME (.qza's and all)?
[/quote]

Yes, it is a mounted directory from the drive in the windows desktop. The problem keeps happening in my iMac so I decide to give it a try in the windows' desktop!!! I have not tried the $home yet!

I feel frustrated about this problem. It got successful once and then suddenly not working any more!! Hopefully the memory extension will solve the problem~!!!

When you run a dataset, what is the minimum memory required for the 10GB dataset?

Thanks,
Bing

46

That's pretty much how I feel too. These are some very strange issues we've seen in this thread. (Even still, this is probably my favorite thread on the forum so far, thanks for all of the help!)

I'm not sure anyone actually knows a concrete answer for that.

2 Likes