I have been using qiime version 2020.8.0 via the q2cli for several years now. I finally got around to upgrading to the latest amplicon distribution version 2023.9.2. Both of these versions were installed via anaconda3 v2018.
I immediately noticed that any command that requires retrieving or generating the cached plugin info takes orders of magnitude longer than it had in my 2020.8.0 install.
In 2020.8.0:
calling time qiime
qiime 0.28s user 0.16s system 54% cpu 0.800 total
calling time qiime diversity
qiime diversity 0.83s user 4.61s system 483% cpu 1.125 total
In qiime 2023.9.2:
calling time qiime
qiime 1.11s user 2.35s system 80% cpu 4.319 total
calling time qiime diversity
qiime diversity 9.03s user 6.09s system 30% cpu 49.878 total
I uninstalled and reinstalled the new environment several times, to no change. I also asked my HPC to install an environment in an administrator location, and this had the same problem. This is quite harmful to my workflow, I'm used to being able to quickly reference various --help texts on the fly, but it now takes nearly a minute for each one. Has there been so much change in the 3 years between these versions that this is just the way it is now?
I should point out that this extra time applies to all the actual commands too. E.g. if I have a simple syntax error in some command it still takes nearly a minute to output that error. Thanks again and let me know what additional info I can provide!
I timed these commands on my end (I'm using a Mac, not an HPC - just for reference) and I'm getting similar numbers to you for those commands. It is somewhat expected that 2023.9 will be slower than 2020.8, just due to the increase in the number of plugins and associated external dependencies that exist in our newest environments. You can take a look at our dependency lists for 2020.8 and 2023.9 below, if you're curious (the number has almost doubled between those two releases):
That being said, if this increase in run time is a real concern for you, there are a couple of suggestions I have offhand:
You can continue using an older version of QIIME 2, just keeping in mind that you may not have access to new plugins/functionality. It's kind of a balance of what new features/methods/etc you need access to in your analysis.
You can install the QIIME 2 Tiny Distribution and then just install the particular plugins you need for your analysis within that environment as you go (i.e. if you just needed dada2, feature-table, and diversity).
Thanks very much for the welcome. The forum has been an invaluable resource for me and I appreciate all the work put into it.
That all makes sense, thanks for the explanation! I probably will take your suggestion and make multiple smaller environments for specific tasks, rather than the single large one.
forum!