HPC multi-node support for vsearch?

[hoohugokim]

Hi! I've started using a HPC system with two compute nodes, under OpenMPI-slurm environment. After some tweaks with seemingly relevant parameters (--p-threads from the QIIME2 command, --nodes, --ntasks from slurm sbatch command), I noticed it always uses one node for the 'classify-consensus-vsearch' command. So it looks likes this:

Regarding this issue, I think the following post from 2019 is most relevant:

Threads vs Nodes

That’s a great question @jwdebelius! This has to do with how parallelism is achieved inside an operating system. There are a few predominant strategies, two of the most common being processes and threads. A process is hopefully the more intuitive concept, it is how an operating system multi-tasks. So you would have a process for your browser (at least one anyhow), a process for your console, a process for your solitaire game, etc. Each process lives in it’s own virtual bubble of memory and r…

So, in short, I'm looking for ways to use both of the compute nodes for a single command, but my understanding is that it is currently not supported in QIIME2-2022.8. Or is there something I am missing? I will appreciate any available information on this. Thank you!

[colinbrislawn (Colin J Brislawn)]

@hoohugokim

So, in short, I'm looking for ways to use both of the compute nodes for a single command, but my understanding is that it is currently not supported in QIIME2-2022.8.

I think this is correct.

And the underly issue is that it's not supported in VSEARCH. (Vsearch uses internal threading, and not a message passing system like MPI to share data across compute notes.)

Most alignment processes are trivializable parallel using a map-reduce model, so one way forward is to chunk up your input data and run this several times using several jobs on several nodes.

You are successfully saturateing the CPU of a single node, and you have lots of core, so I think you are off to a great start! How long does this step take when using one node?

[Oddant1]

Hello @hoohugokim, we do not currently support executing across multiple nodes on an HPC cluster. We are working on a way of doing this, but we don't have a release date for it yet.

[hoohugokim]

@Oddant1 Thank you very much for the information! I will look forward to the date

@colinbrislawn Thank you a lot for the enlightment! As this is only metataxonomic analysis, I think I will divide the dataset in two and give each node separate works to do. After I gather all data, nonetheless, I will have to try one single phylogeny analysis, but that's another problem for later

With my 12 samples (~80–90k reads/sample) of 16S V1–V9 amplicon data using Oxford Nanopore R9.4.1, it took only 3–4 hours to run (which is a huge improvement from the duration of several days using the previous 16 cores workstation). But due to a metadata error I have to run them again, so I will try again by dividing the job in half and with some PacBio HiFi reads too that I just got my hands on.