QIIME2 using too many nodes

Technical Support
1

Hi,

Similar post was found here: 'qiime diversity alpha-rarefaction' consume too much cpus and run weird slow but it was closed so I cannot reply.

I started noticing that the newest versions of QIIME2 are using too many nodes, specifically when we run this commands:

qiime diversity alpha-rarefaction
--i-table table.qza
--i-phylogeny rooted-tree.qza
--p-max-depth 15000
--p-steps 50
--m-metadata-file metadata.tsv
--o-visualization alpha-rarefaction.qzv

When I check system resources it showed that at that step, qiime2 was using close to 200 nodes out of a total of 256.

Thanks in advance!!

2

Good afternoon,

Welcome to the fourms! :qiime2:

Can you tell us more about your computer and how you are running this script, and any other clues you can provide?

Are you running this on a SLURM queue, or a torq queue, or a single computer?

When you say nodes, do you mean cores, or threads, whole compute nodes?

3

Thank you for your reply and for welcoming me to the QIIME2 forum! Apologies for the misleading terminology; I used the same one I found in the last post just in case. Our server has basically 128 cores and 256 threads, and QIIME2 is using close to 200 threads when we run the command I posted above.

We have a couple of QIIME2 installations (with conda) in our system and I am 100% certain we have seen this issue with version 2021.8 and I will need to verify with my colleagues who run version 2022.2.

As far as I know we don't use slurm or torq queues. The script is run exactly as posted above, just with nohup so that it can run in the background as we connect to the server through our personal computers.

5

Got it!

This sounds like a known issue with the backend of the alpha-rarefaction plugin. See
https://forum.qiime2.org/t/qiime2-2022-2-slow-alpha-rarefaction/22893/2

I wanted to check in on how many nodes were being used, as that could be a new issue. But fortunately it sounds like this existing one.

1 Like
6

This topic was automatically closed 31 days after the last reply. New replies are no longer allowed.