[Preview] QIIME 2 2020.11 development changelog

Developer Discussion
,
1

QIIME 2 2020.11 has been released!
Please see the official changelog for more details.

Original content of post

:exclamation: Important :exclamation:

The following is an early developer preview of the changes expected in 2020.11

This post is a live-document which will be updated throughout our development cycle. Any links will in this topic will be broken until the release is officially published. When we are ready for release, we’ll copy this changelog and create a new post in the Announcements category.

Important Developer Information

Dates (please keep an :eye: on this post, these :calendar: might change):

  • PRs must be submitted by: November 6, 2020
  • PRs must be merged by: November 18, 2020
  • Repo Freeze and Package Building: November 23, 2020
  • Release Day: November 30, 2020 (postponed due to packaging issues)

Developer Project Board: https://github.com/orgs/qiime2/projects/22

:exclamation: BREAKING CHANGES

Please update scripts, workflows, etc. as appropriate. Stuck? Reach out here on the forum for help. Want to test this prior to the next release of QIIME 2? Check out the “development environment” instructions here:

https://dev.qiime2.org/latest/quickstart/

Here are the highlights of the release:

2

We started cutting this release last week and ran into several issues with anaconda.org (we weren't able to upload many of the conda packages that we built). I have reached out to the anaconda.org support channels, but in the meantime @andrewsanchez and I are going to make some "structural" changes to the release system that will remove anaconda.org as a bottleneck.

Busywork Background

The QIIME 2 release system is almost completely automated, using a home-grown system called Busywork (GitHub - qiime2/busywork). Every time a new commit is created on the master branch of any of the ~30 or so QIIME 2 packages, Busywork does several things:

  1. builds two new conda packages for the new commit (one for mac os, one for linux)
  2. uploads the built packages to an internal QIIME 2 server where "testing" packages are stored
  3. runs the unit tests for the two conda packages (this makes sure the plugin is sound, in isolation)
  4. runs the integration tests for all of the conda packages in the entire ecosystem (this makes sure the plugin is sound, as part of the greater ecosystem)
  5. after a package makes it through integration testing, two things happen:
    a. the "testing" conda packages get promoted to "staging" and are uploaded to qiime2-staging :: Anaconda.org
    b. a new "staging" environment is created: GitHub - qiime2/environment-files
  6. finally, if the version-string for the packages that passed through step 5 are "release" packages (that is to say, they don't contain a .dev anywhere in the version identifier), the "staging" conda packages get promoted to "release" and are uploaded to qiime2 :: Anaconda.org, and a new "release" environment file is created

Something noteworthy about this process is that the conda packages built in step 1 are the same packages released in step 6 - the only thing that changes about them is where they are stored - the location confers meaning, basically.

Incident

While step 5 above was running, we started experiencing failures during the upload process to qiime2-staging :: Anaconda.org (e.g. this job). Packages are uploaded to anaconda.org via the anconda cli tool. Basically, the upload process appears to completely fail on our end:

# exhibit 1: manually uploading q2-longitudinal
$ anaconda -vvvv --show-traceback upload \
  -u qiime2-staging \
  -l r2020.11-test \
  osx-64/q2-longitudinal-2020.11.0-py36_0.tar.bz2 \
  linux-64/q2-longitudinal-2020.11.0-py36_0.tar.bz2
Using Anaconda API: https://api.anaconda.org
[DEBUG] Found login token: /Users/matthew/Library/Application Support/binstar/https%3A%2F%2Fapi.anaconda.org.token
Using "qiime2-staging" as upload username
Processing 'osx-64/q2-longitudinal-2020.11.0-py36_0.tar.bz2'
Detecting file type...
[DEBUG] Testing if conda package ..
[DEBUG] This is a conda package
File type is "conda"
Extracting conda package attributes for upload
Creating package "q2-longitudinal"
Creating release "2020.11.0"
Uploading file "qiime2-staging/q2-longitudinal/2020.11.0/osx-64/q2-longitudinal-2020.11.0-py36_0.tar.bz2"

# exhibit 2: exhibit 1 kinda _looks_ like it ran successfully, but
# checking the exit code reveals something to the contrary
$ echo $?
1

My main concern at this point was that somehow the packages that Busywork built were corrupt, or otherwise broken, so I attempted the same exercise as above, but uploading to my personal anaconda.org account:

# exhibit 3: uploading to my personal anaconda.org account
$ anaconda -vvvv --show-traceback upload \
  -u thermokarst \
  -l r2020.11-test \
  osx-64/q2-longitudinal-2020.11.0-py36_0.tar.bz2 \
  linux-64/q2-longitudinal-2020.11.0-py36_0.tar.bz2
Using Anaconda API: https://api.anaconda.org
[DEBUG] Found login token: /Users/matthew/Library/Application Support/binstar/https%3A%2F%2Fapi.anaconda.org.token
Using "thermokarst" as upload username
Processing 'osx-64/q2-longitudinal-2020.11.0-py36_0.tar.bz2'
Detecting file type...
[DEBUG] Testing if conda package ..
[DEBUG] This is a conda package
File type is "conda"
Extracting conda package attributes for upload
Creating package "q2-longitudinal"
Creating release "2020.11.0"
Uploading file "thermokarst/q2-longitudinal/2020.11.0/osx-64/q2-longitudinal-2020.11.0-py36_0.tar.bz2"
 uploaded 7821 of 7821Kb: 100.00% ETA: 0.0 minutes
Upload complete

Processing 'linux-64/q2-longitudinal-2020.11.0-py36_0.tar.bz2'
Detecting file type...
[DEBUG] Testing if conda package ..
[DEBUG] This is a conda package
File type is "conda"
Extracting conda package attributes for upload
Creating package "q2-longitudinal"
Creating release "2020.11.0"
Uploading file "thermokarst/q2-longitudinal/2020.11.0/linux-64/q2-longitudinal-2020.11.0-py36_0.tar.bz2"
 uploaded 7846 of 7846Kb: 100.00% ETA: 0.0 minutes
Upload complete

conda package located at:
https://anaconda.org/thermokarst/q2-longitudinal

conda package located at:
https://anaconda.org/thermokarst/q2-longitudinal

# exhibit 4: everything uploaded as expected
$ echo $?
0

So at this point I don't really have any idea what is going on, but from my perspective, it looks like there might be something broken on the anaconda.org side of things, but I can't say for sure.

Proposed Resolution

So, one reason that the lack of "staging" files is problematic is because the Busywork system pulls the packages from "staging" in order to push them up to "release." If the staging files are missing, as configured, Busywork won't be able to pull down the staging files, which means it won't have files to upload to the release channel...

Some potential solutions that were discussed at Q2HQ

  • manually upload the built packages to the release channel
  • create a new anaconda.org staging channel
  • rewire Busywork to pull the testing packages for release, instead of pulling from staging
  • probably some others that I'm not remembering at the moment

Ultimately, we decided to move forward with this plan:

As part of our ongoing Library enhancements, Library is gradually replacing major elements of Busywork, with the goal of exposing all of this great automation and testing to all QIIME 2 plugins. At that time we expect that there will be no distinction between the "Core Distribution" and "Community Plugins" - the goal is that all plugins stand on their own. One element that was slated for replacement in Q1ish of 2021 was setting up a conda staging channel on our new conda package server (https://packages.qiime2.org), basically moving this functionality out of Busywork, and into Library. Nothing would change (yet) from the developer perspective, we would essentially be laying the foundation for subsequent Library-related changes in the near future, while "unsticking" our current situation.

Since those goals align pretty well with our current needs, @andrewsanchez and I are going to postpone the release by 1-4ish days to work on setting up and migrating the staging package machinery from Busywork to Library.

This will allow us to move forward, without depending on any specific resolution or action from the anaconda.org team, and will tick off a todo-item on the Library checklist. We will update y'all here when we know more - at this point we are still expecting to release 2020.11 this week - all the work is done and the packages are built, they just need a place to upload to!

:qiime2: :t_rex:

6 Likes
3

Quick status update: we sorted out our migration of the "staging" channel from anaconda.org -> packages.qiime.org - yay! (preview: https://packages.qiime2.org/qiime2/staging/2020.11/)

Unfortunately, along the way we discovered some packaging issues that crept into our development cycle a few weeks back, basically there are some issues with how q2-demux and q2-diversity packages were built:

Screen Shot 2020-12-04 at 2.22.23 PM
Screen Shot 2020-12-04 at 2.22.23 PM522×635 42.4 KB

The .dirty business on the version identifiers for demux and diversity indicate that somehow during the package building step of those plugins, some of the plugin's source code was modified in some way. As best I can tell, this is only impacting those two plugins, and only on darwin hosts.

I haven't yet started digging into this problem, but it is safe to say it is wholly unrelated to the anaconda.org issues that I reported above.

Anyway, the current plan is to re-enter the 2020.11 development cycle: I just started the process of updating all the the QIIME 2 packages to 2020.11.1.dev0 (note the .1.dev0, as opposed to .0.dev0), that way we have a clean-slate for this release, and can start debugging these issues.

Sorry about the hold up! More soon, after we get a chance to dig into these .dirty oddities.

:t_rex:

4 Likes
4

Okay, I think that is all sorted out - preview docs can be found here:

https://docs.qiime2.org/2020.11/

env files: Natively installing QIIME 2 — QIIME 2 2020.11.1 documentation

I still need to kick the tires a bit, so to speak, but we should be all set for making a public announcement today!

:qiime2:

4 Likes
6

So ultimately, the .dirty version-strings were caused by npm - several of our JS-visualizations use npm behind the scenes to download JS dependencies and render “compiled” JS for us. That build process was mutating one of the source files, and alphabetizing the list of JS dependencies. versioneer, the versioning tool we use for python, noticed the edits to the file, and marked the package version as “dirty”. Mystery solved!

5 Likes