I am currently learning to use Qiime2 installed on a virtual machine, and I keep coming up against an impassable memory error associated with clustering and assigning taxonomy using the feature-classifier. I am using a laptop with 8gb of memory, and have probed the wiki and this forum for what memory requirements are necessary. I have tried all the tricks proposed to past question-askers with this same issue: increasing the memory allocated to Vbox, reducing the reads-per-batch, etc, to no avail.
My boss would like me to purchase a new computer so that I can run these analyses. I don't want to purchase a new machine without verifying the hardware requirements for running feature-classifier. The wiki claims that 32 gb should be sufficient, but I have seen folks here claim they have 32 gb memory and still run into memory-related errors. Are these errors usually associated with a virtual machine due to sharing memory with the native OS? Should I purchase 64gb if I still plan on using a virtual machine?
Oh yeah, that's probably not enough RAM for q2-feature-classifier.
Unfortunately there are no concrete "rules" here - it depends! It depends on how big your dataset is, and it depends on the size of the reference database(s) you want to use.
For the pre-trained full-length SILVA classifiers that we distribute at docs.qiime2.org we use 64 GB of RAM, for reference.
We actually don't have an official QIIME 2 wiki, perhaps you're reading a community resource? Or something related to QIIME 1?
You won't be able to dedicate all of the memory to your VM, usually you need to keep 4-8GB held back for the host OS, so you still might not be able to train something like the SILVA classifier on that particular setup. If you installed linux on the machine, then you would avoid the VM "penalty".
A common solution here is to offload the resource-intensive steps to somewhere like your university's HPC, or renting an AWS machine for a few hrs/days for the big steps like training feature classifiers. Most everything else is much lighter on the necessary resources, so you can continue the analysis on the 8GB machine you already have. Just some food for thought!
Yowza! I tried to use the pre-trained Silva classifier you mentioned, and sadly I was coming up with an error that indicated I was using an incompatible version of some plugin. Other questions found on the forum said that I really ought to be training my own classifier. I tried to follow the steps of RESCRIPt, but again, memory error.
My apologies, I meant the user documentation. On this overview: https://docs.qiime2.org/2021.8/tutorials/overview/, the section on "Taxonomy classification and taxonomic analyses" speaks to how memory intensive feature classification can be, but that 4-32gb of RAM is required... depending.
Sadly, I am not affiliated with a university. We have several colleagues there, but due to concerns about the privacy of our clients' data, my boss has elected not to use the university cluster. Is there any tutorials about how to use an AWS instance? I found the installation guide and set up a Qiime2 instance some time ago, but couldn't figure out why scp wouldn't work to transfer files to it. I did find an old Qiime1 AWS tutorial, and a key pair is referenced, so I may have to do some digging, but that is a very good suggestion.
Well then you're in luck! That's an easy problem to solve - simply download the classifier that corresponds with the version of QIIME 2 that you're using!
Navigate to the "data resources" section of the docs:
Hmm, there isn't any trick here, just scp <src> <dst> - I just double checked, there are no special flags or configuration things to take care of. You can find an older demo post here:
Something to keep in mind: QIIME 1 is not the same thing as QIIME 2 - they are two completely different projects. I think you'll have a hard time trying to make a QIIME 1 tutorial make any sense in QIIME 2.
Thanks so much for all the help, @thermokarst. I am so impressed with all the community resources available. I started using an FTP client to transfer files to the AWS instance since the graphical interface is a little easier for me to use.
The project manager still uses Qiime1, and he found an online seminar that we both signed up for on how to use Qiime2. With enough tutorials, hopefully we can begin to switch over to Qiime2. I really appreciate your advice. I think we can get AWS to work for us.
