Hi Everyone!
@thermokarst and I have been working on upgrading QIIME 2 to Python 3.6 (previously Python 3.5) for this upcoming 2019.1 release (due out shortly).
Why are things changing?
In an effort to stay up to date with the latest versions of numpy, scipy, scikit-learn, pandas, and matplotlib (all fundamental parts of the scientific Python stack), we discovered Python 3.5 builds were no longer supported on conda-forge.
In retrospect this makes a lot of sense:
Python 3.6 was released on December 23, 2016
We have clearly delayed upgrading our Python version for quite a bit longer than is sensible.
Since this was the first release of the year and we had hit a wall with our dependencies, it made sense to take some time to address this in our build infrastructure.
What's new?
QIIME 2's build pipeline now uses conda-build 3's build-variants. This means we'll be able to adopt new versions of Python quickly, and handle compile-time and run-time linking more effectively than before. This fits in line with the recent progress on conda-forge and bioconda which are also in the process of migrating to conda-build 3 (conda-forge has more or less finished its migration).
Additionally, we were able to pay down some technical debt in our conda channel order, making it much simpler with the long-term goal of migrating all QIIME 2 packages to bioconda (instead of a release-specific QIIME 2 channel).
Before this change, the channel order looked like this:
channels:
- qiime2/label/r2018.11
- qiime2
- conda-forge
- defaults
- bioconda
- biocore
After this change:
channels:
- qiime2/label/r2019.1
- conda-forge
- bioconda
- defaults
With the exception of the release specific QIIME 2 channel (qiime2/label/r2019.1), this matches the bioconda channel order.
Do I have to update my plugin?
Maybe. If you are doing the "right" thing, you will have a conda package for your plugin with all of your dependencies similarly sourced from other packages.
If your plugin is built as a "noarch" package (which isn't well supported on bioconda yet), you shouldn't need to do anything. Otherwise you may need to rebuild your package using Python 3.6 to remain compatible with QIIME 2 into the new year. This process should only take ~10 minutes, but let us know if you run into more difficulty than expected!
If you do not currently use a conda package to distribute your plugin, now is a great time to start! Otherwise, you shouldn't need to change anything as 3.5 and 3.6 are extremely similar with only a few breaking changes.
I don't have time to update my plugin right this moment, can people still use QIIME 2 2018.11 with my plugin?
Sure! We're happy to continue supporting 2018.11 for the next release or two as we understand changing the version of Python can be a large change (though in this case we hope it's very straight-forward update in general).
What about Python 3.7?
We initially started this as a jump from 3.5 to 3.7, however bioconda was waiting on conda-forge to finish their migration to a new libc version which delayed adoption. This means that bioconda currently only supports 3.5 and 3.6. This should change in the next few months as they are similarly impacted by the lack of 3.5 builds on conda-forge. We can expect that bioconda will sync up with conda-forge soon in their supported build variants (if you'd like to assist, I'm sure they would appreciate it).
Python 3.7 does still have some attractive language features so it is likely we will jump to 3.7 once it is available in bioconda. We may be able to support Python 3.7 builds as soon as 2019.4, but that timeline is pretty uncertain at this point.