Hi! I've started using a HPC system with two compute nodes, under OpenMPI-slurm environment. After some tweaks with seemingly relevant parameters (--p-threads from the QIIME2 command, --nodes, --ntasks from slurm sbatch command), I noticed it always uses one node for the 'classify-consensus-vsearch' command. So it looks likes this:
Regarding this issue, I think the following post from 2019 is most relevant:
So, in short, I'm looking for ways to use both of the compute nodes for a single command, but my understanding is that it is currently not supported in QIIME2-2022.8. Or is there something I am missing? I will appreciate any available information on this. Thank you!
@Oddant1 Thank you very much for the information! I will look forward to the date
@colinbrislawn Thank you a lot for the enlightment! As this is only metataxonomic analysis, I think I will divide the dataset in two and give each node separate works to do. After I gather all data, nonetheless, I will have to try one single phylogeny analysis, but that's another problem for later
With my 12 samples (~80–90k reads/sample) of 16S V1–V9 amplicon data using Oxford Nanopore R9.4.1, it took only 3–4 hours to run (which is a huge improvement from the duration of several days using the previous 16 cores workstation). But due to a metadata error I have to run them again, so I will try again by dividing the job in half and with some PacBio HiFi reads too that I just got my hands on.