We started cutting this release last week and ran into several issues with anaconda.org (we weren't able to upload many of the conda packages that we built). I have reached out to the anaconda.org support channels, but in the meantime @andrewsanchez and I are going to make some "structural" changes to the release system that will remove anaconda.org as a bottleneck.
Busywork Background
The QIIME 2 release system is almost completely automated, using a home-grown system called Busywork (GitHub - qiime2/busywork). Every time a new commit is created on the master branch of any of the ~30 or so QIIME 2 packages, Busywork does several things:
- builds two new conda packages for the new commit (one for mac os, one for linux)
- uploads the built packages to an internal QIIME 2 server where "testing" packages are stored
- runs the unit tests for the two conda packages (this makes sure the plugin is sound, in isolation)
- runs the integration tests for all of the conda packages in the entire ecosystem (this makes sure the plugin is sound, as part of the greater ecosystem)
- after a package makes it through integration testing, two things happen:
a. the "testing" conda packages get promoted to "staging" and are uploaded to qiime2-staging | Anaconda.org
b. a new "staging" environment is created: GitHub - qiime2/environment-files - finally, if the version-string for the packages that passed through step 5 are "release" packages (that is to say, they don't contain a
.devanywhere in the version identifier), the "staging" conda packages get promoted to "release" and are uploaded to qiime2 | Anaconda.org, and a new "release" environment file is created
Something noteworthy about this process is that the conda packages built in step 1 are the same packages released in step 6 - the only thing that changes about them is where they are stored - the location confers meaning, basically.
Incident
While step 5 above was running, we started experiencing failures during the upload process to qiime2-staging | Anaconda.org (e.g. this job). Packages are uploaded to anaconda.org via the anconda cli tool. Basically, the upload process appears to completely fail on our end:
# exhibit 1: manually uploading q2-longitudinal
$ anaconda -vvvv --show-traceback upload \
-u qiime2-staging \
-l r2020.11-test \
osx-64/q2-longitudinal-2020.11.0-py36_0.tar.bz2 \
linux-64/q2-longitudinal-2020.11.0-py36_0.tar.bz2
Using Anaconda API: https://api.anaconda.org
[DEBUG] Found login token: /Users/matthew/Library/Application Support/binstar/https%3A%2F%2Fapi.anaconda.org.token
Using "qiime2-staging" as upload username
Processing 'osx-64/q2-longitudinal-2020.11.0-py36_0.tar.bz2'
Detecting file type...
[DEBUG] Testing if conda package ..
[DEBUG] This is a conda package
File type is "conda"
Extracting conda package attributes for upload
Creating package "q2-longitudinal"
Creating release "2020.11.0"
Uploading file "qiime2-staging/q2-longitudinal/2020.11.0/osx-64/q2-longitudinal-2020.11.0-py36_0.tar.bz2"
# exhibit 2: exhibit 1 kinda _looks_ like it ran successfully, but
# checking the exit code reveals something to the contrary
$ echo $?
1
My main concern at this point was that somehow the packages that Busywork built were corrupt, or otherwise broken, so I attempted the same exercise as above, but uploading to my personal anaconda.org account:
# exhibit 3: uploading to my personal anaconda.org account
$ anaconda -vvvv --show-traceback upload \
-u thermokarst \
-l r2020.11-test \
osx-64/q2-longitudinal-2020.11.0-py36_0.tar.bz2 \
linux-64/q2-longitudinal-2020.11.0-py36_0.tar.bz2
Using Anaconda API: https://api.anaconda.org
[DEBUG] Found login token: /Users/matthew/Library/Application Support/binstar/https%3A%2F%2Fapi.anaconda.org.token
Using "thermokarst" as upload username
Processing 'osx-64/q2-longitudinal-2020.11.0-py36_0.tar.bz2'
Detecting file type...
[DEBUG] Testing if conda package ..
[DEBUG] This is a conda package
File type is "conda"
Extracting conda package attributes for upload
Creating package "q2-longitudinal"
Creating release "2020.11.0"
Uploading file "thermokarst/q2-longitudinal/2020.11.0/osx-64/q2-longitudinal-2020.11.0-py36_0.tar.bz2"
uploaded 7821 of 7821Kb: 100.00% ETA: 0.0 minutes
Upload complete
Processing 'linux-64/q2-longitudinal-2020.11.0-py36_0.tar.bz2'
Detecting file type...
[DEBUG] Testing if conda package ..
[DEBUG] This is a conda package
File type is "conda"
Extracting conda package attributes for upload
Creating package "q2-longitudinal"
Creating release "2020.11.0"
Uploading file "thermokarst/q2-longitudinal/2020.11.0/linux-64/q2-longitudinal-2020.11.0-py36_0.tar.bz2"
uploaded 7846 of 7846Kb: 100.00% ETA: 0.0 minutes
Upload complete
conda package located at:
https://anaconda.org/thermokarst/q2-longitudinal
conda package located at:
https://anaconda.org/thermokarst/q2-longitudinal
# exhibit 4: everything uploaded as expected
$ echo $?
0
So at this point I don't really have any idea what is going on, but from my perspective, it looks like there might be something broken on the anaconda.org side of things, but I can't say for sure.
Proposed Resolution
So, one reason that the lack of "staging" files is problematic is because the Busywork system pulls the packages from "staging" in order to push them up to "release." If the staging files are missing, as configured, Busywork won't be able to pull down the staging files, which means it won't have files to upload to the release channel...
Some potential solutions that were discussed at Q2HQ
- manually upload the built packages to the release channel
- create a new anaconda.org staging channel
- rewire Busywork to pull the testing packages for release, instead of pulling from staging
- probably some others that I'm not remembering at the moment
Ultimately, we decided to move forward with this plan:
As part of our ongoing Library enhancements, Library is gradually replacing major elements of Busywork, with the goal of exposing all of this great automation and testing to all QIIME 2 plugins. At that time we expect that there will be no distinction between the "Core Distribution" and "Community Plugins" - the goal is that all plugins stand on their own. One element that was slated for replacement in Q1ish of 2021 was setting up a conda staging channel on our new conda package server (https://packages.qiime2.org), basically moving this functionality out of Busywork, and into Library. Nothing would change (yet) from the developer perspective, we would essentially be laying the foundation for subsequent Library-related changes in the near future, while "unsticking" our current situation.
Since those goals align pretty well with our current needs, @andrewsanchez and I are going to postpone the release by 1-4ish days to work on setting up and migrating the staging package machinery from Busywork to Library.
This will allow us to move forward, without depending on any specific resolution or action from the anaconda.org team, and will tick off a todo-item on the Library checklist. We will update y'all here when we know more - at this point we are still expecting to release 2020.11 this week - all the work is done and the packages are built, they just need a place to upload to!
:qiime2: 