I'm attempting to run qiime feature-classifier classify-consensus-blast in parallel and the documentation stipulates that I can execute this in parallel using the option --parallel. This flag uses my default parallel configuration - is this set somehow by QIIME2 already? How does one format a file specified by --parallel-config?
Hello @neocallimastix, parallelization of pipelines was added in QIIME 2 2023.5, and is currently documented here. That documentation describes how the parallelization works and what the default configuration looks like.
Note that while all pipelines do support parallelization (the pipeline should complete execution and not error), not all of them will benefit from it at this point. We just added generic parallel support for pipelines and have it functioning as a proof of concept with speedups for some existing pipelines, but some will need to be rewritten to benefit from this new feature.
Unfortunately, the pipeline you are trying to parallelize will not get any benefit from it at this time. You can still do it, and it shouldn't break anything, but you will most likely not see any speed up.
1 Like
This topic was automatically closed 31 days after the last reply. New replies are no longer allowed.