q2cli very slow to display plugin information after upgrading to 2023.9

I have been using qiime version 2020.8.0 via the q2cli for several years now. I finally got around to upgrading to the latest amplicon distribution version 2023.9.2. Both of these versions were installed via anaconda3 v2018.

I immediately noticed that any command that requires retrieving or generating the cached plugin info takes orders of magnitude longer than it had in my 2020.8.0 install.

In 2020.8.0:
calling time qiime

qiime  0.28s user 0.16s system 54% cpu 0.800 total

calling time qiime diversity

qiime diversity  0.83s user 4.61s system 483% cpu 1.125 total

In qiime 2023.9.2:
calling time qiime

qiime  1.11s user 2.35s system 80% cpu 4.319 total

calling time qiime diversity

qiime diversity  9.03s user 6.09s system 30% cpu 49.878 total

I uninstalled and reinstalled the new environment several times, to no change. I also asked my HPC to install an environment in an administrator location, and this had the same problem. This is quite harmful to my workflow, I'm used to being able to quickly reference various --help texts on the fly, but it now takes nearly a minute for each one. Has there been so much change in the 3 years between these versions that this is just the way it is now?

Thanks in advance,
Adam

I should point out that this extra time applies to all the actual commands too. E.g. if I have a simple syntax error in some command it still takes nearly a minute to output that error. Thanks again and let me know what additional info I can provide!

Hi @Adam_Cantor,

Welcome to the :qiime2: forum!

I timed these commands on my end (I'm using a Mac, not an HPC - just for reference) and I'm getting similar numbers to you for those commands. It is somewhat expected that 2023.9 will be slower than 2020.8, just due to the increase in the number of plugins and associated external dependencies that exist in our newest environments. You can take a look at our dependency lists for 2020.8 and 2023.9 below, if you're curious (the number has almost doubled between those two releases):

2020.8 environment file
2023.9 environment file

That being said, if this increase in run time is a real concern for you, there are a couple of suggestions I have offhand:

  • You can continue using an older version of QIIME 2, just keeping in mind that you may not have access to new plugins/functionality. It's kind of a balance of what new features/methods/etc you need access to in your analysis.
  • You can install the QIIME 2 Tiny Distribution and then just install the particular plugins you need for your analysis within that environment as you go (i.e. if you just needed dada2, feature-table, and diversity).

Hope this helps! Cheers :lizard:

Hi Liz,

Thanks very much for the welcome. The forum has been an invaluable resource for me and I appreciate all the work put into it.

That all makes sense, thanks for the explanation! I probably will take your suggestion and make multiple smaller environments for specific tasks, rather than the single large one.

Best,
Adam