I'm still trouble shooting some things for our laboratory. One thing that I was also tasked with was getting QIIME2 to run on the HPC at the medical center.
We run a Linux cluster of approximately 70 nodes. We've been trying to get it working in a pipeline so we can streamline our analysis and run DADA2 from these nodes.
I found a couple of threads in the past, but I was wondering what strategies that you all have used to run it on HPC/parallelization. Thank you. Again, greatly appreciate your input.
We have some long-term goals to support direct integration with various task schedulers, etc, but not quite sure when something like that would happen. Otherwise, I tend to submit q2cli-based QIIME 2 commands as torque jobs on our local cluster. Some actions in QIIME 2 expose parameters like n_threads or n_cores - I will allocate nodes accordingly, when applicable. Sorry I don't have a more slick answer for you, but for now, we basically treat QIIME 2 like any other package on our HPC.
FWIW - we have generally found that analyses that utilize ASVs over OTUs have a significantly smaller computational burden (obviously there are plenty of examples where that isn't the case, but, broadly speaking...).
Thanks, that's greatly appreciated. I was reading on the DADA command for pushing for more threads I think it was --p-n-threads. If helpful, I can put my commands on here. I have been submitting QIIME2 as batch commands through their SGE scheduler(?). The problem is that when issues arise I cannot see errors, but this isn't an issue with QIIME2 it is an issue with the SGE scheduler.
As an aside, most of the DADA2 and denoise takes less than 12 hours with the HPC (compared to almost running some over a whole day. I do however, run multiple threads option with DADA and that may have something to do with the speed at which it runs.
:qiime2: