parallelize qiime rescript evaluate-fit-classifier on hpc

[brittair (Rebecca)]

qiime2-amplicon-2023.9, miniconda3, macM2

command:
qiime rescript evaluate-fit-classifier
--i-sequences V1_V2_27f_338r/silva-138.1-ssu-nr99-seqs-27f-338r-derep.qza
--i-taxonomy V1_V2_27f_338r/silva-138.1-ssu-nr99-tax-27f-338r-derep.qza
--p-n-jobs -2
--o-classifier V1_V2_27f_338r/silva-138.1-ssu-nr99-27f-338r-classifier.qza
--o-observed-taxonomy V1_V2_27f_338r/silva-138.1-ssu-nr99-27f-338r_predicted_taxonomy.qza
--o-evaluation V1_V2_27f_338r/silva-138.1-ssu-nr99-27f-338r_classifier_eval.qzv

question: I read that this step requires a lot of memory and and time (e.g., v3-4 takes >35 hours with 12 cores according to Step 6: Taxonomy assignment - shenjean/diversity GitHub Wiki). I've read the community plugin support post from 2021 (Is there a way to parallelize evaluate-fit-classifier?), but other than flagging --p-n-jobs I'm not sure how to best configure this on an hpc.
Any advice about how to parallelize this on an hpc to speed things up (with fairly detailed explanations/instructions for a newbie)? I've visited the help documentation and see options for --parallel-config and --parallel, but am not experienced enough to know how to make a parallel configuration and set reasonable parameters, or how many cpus, tasks, time, etc. should be allocated for the job. Or perhaps I am doomed to pay the price of 35 hours. Any experienced advice will be much appreciated.

[Nicholas_Bokulich]

Hi @brittair ,

@brittair

Or perhaps I am doomed to pay the price of 35 hours

Unfortunately, yes. Because as mentioned on that other topic:

@Nicholas_Bokulich

This is a pipeline with different steps in a series, some paralellizable, others not. Unfortunately, one of the most time-consuming steps — fitting the classifier — is not parallelizable. So when you check top this step is most likely still running.

But the good news is that you only need to fit the classifier once (hopefully), and then you can re-use it in your environment many times, no need to repeat this step.

You can also just fit the classifier with q2-feature-classifier if you do not want to perform the evaluation steps. This will save you some hours of additional time. That's the step that is parallelized by the --p-n-jobs option, though, so you can leverage your hpc to speed up that step.

Sorry I can't give a more satisfying answer!

[brittair (Rebecca)]

I suspected that might be the case from the 2021 post, but wanted to ask just in case things have changed. Thank you kindly for the clarification!