Just in case I miss anything. I attached all the information of my own iMac information below. I actually stored all my fastq. gz and .qza or qzv files in an 2TB external drive LACIE. Please see the pics below. If that is the issue, please let me know what do I need to do to run the data.
We’ve got the datasets and have been trying to reproduce the issues, but so far very little luck. I’m currently trying to use a bisection method with ulimit -v under the current assumption that memory is somehow playing a role here.
@Bing’s dataset is still running, so no real info on that yet.
This is going to take a while (and it didn’t help that I initially thought that -v took bytes instead of kilobytes). I’ll keep you posted as to any new info. @thermokarst was able to create an unhandled malloc warning on the atacama tutorial data with a smallish EC2 instance, but it finished anyways and the results looked sane.
We can’t seem to reproduce these issues. We’ve limited memory in a lot of ways via ulimit -v (which also limits paging) and while we’ve seen several unique errors from R, nothing really matches the issue.
It is also possible there are transient issues with storage causing this, but we don’t have a way to reproduce that either.
I guess the important thing is there doesn’t seem to be anything wrong with the structure of the .qza input data at this time.
I rerun our data again with the dada2, same error happened again! Then I tried the method that @apzlo has used to move all the files in the same folder. Finally it works. I am just curious whether it is because I run the data analysis using an external USB rather than in local drive of my computer?
Hmm, that could very well be the problem. Where are you running the analysis? Are you inside the directory that is mounted from your USB drive?
I created a folder in my USB drive which is only used for the data analysis. I put all the .fastq in a folder based on the tutorial, and then all the analysis is running in that folder. Does that mean I need to create directory use mkdir?
Out of curiosity what does this command do inside your environment?
python -c "import tempfile; print(tempfile.gettempdir())"
Do I need to run this command in qiime2 environment or terminal window?
Does that happen to be your USB drive or is it some crazy path like:
/var/folder/w5/t9wf37ss1cg6s6lhtt8s33gw0000gn/T (which is what I would expect)?
This is exactly what it showed in the errors. If that is the case, what are your suggestions to avoid this?
Nope that should work fine, mkdir doesn’t do anything special.
Good point. That answers the question then, you don’t need to run anything. QIIME 2 is working off of your system drive which probably means that the issue happens from some kind of semi-corrupted unzip. (Unless your system drive is the unstable one and we’re just getting lucky).
So I think we can probably point this at hardware failure and something in either the framework or Python 3’s standard library that isn’t handling the failure with an exception and instead keeps running resulting in “missing files”. We’re having another kind of error right now related to the zip files, so maybe there is something non-standard happening.
Since you don’t have this issue when working on your system’s hard-drive I would stick to that. (You may also want to look into backing up your current USB drive, it may be failing).
P.S. Totally unrelated, but on this forum you can just highlight text, while you have the reply editor open, you can quote things (it back-links and everything)!
@Bing, is it possible for you to try out this same analysis on a different USB drive? This would let us rule out hardware failure if it succeeds. If you have time/resources to do this we’d really appreciate it! We’d like to track down the source of the issue since it has affected multiple users.
Also, what is the output from running these commands (with your USB drive plugged in)?
I tried another USB 500GB external drive, and also tried the internal drive of my iMac with 700GB available. Both methods failed with the same issue as reported before. The interesting thing is that the missing file for each time is different. I finally gave up trying it. What I did is that I reinstall my iMac to see how that will go!!
Is there anything I can do to handle this issue if it happens again and again?