[Preview] QIIME 2 2021.4 development changelog

Developer Discussion
,
1

QIIME 2 2021.4 has been released!
Please see the official changelog for more details.

Original content of post

:exclamation: Important :exclamation:

The following is an early developer preview of the changes expected in 2021.4

This post is a live-document which will be updated throughout our development cycle. Any links will in this topic will be broken until the release is officially published. When we are ready for release, we’ll copy this changelog and create a new post in the Announcements category.

Important Developer Information

Dates (please keep an :eye: on this post, these :calendar: might change):

  • PRs must be submitted by: April 9, 2021
  • PRs must be merged by: April 13, 2021
  • Repo Freeze and Package Building: April 15, 2021 - April 16, 2021 (these dates are flexible)
  • Classifier Training: April 19, 2021 - April 20, 2021
  • Release Day: April 21, 2021

Developer Project Board: https://github.com/orgs/qiime2/projects/26

:exclamation: BREAKING CHANGES

Please update scripts, workflows, etc. as appropriate. Stuck? Reach out here on the forum for help. Want to test this prior to the next release of QIIME 2? Check out the "development environment" instructions here:

https://dev.qiime2.org/latest/quickstart/

Here are the highlights of the release:

  • New Interface: q2galaxy :milky_way: by @Oddant1 and @ebolyen
    • This is an alpha release of an interface which automatically generates tool descriptions for Galaxy
      • This means that as plugins are created and evolve, we can automatically generate complete Galaxy tools in a few seconds, rather than developing them by hand as is typically done.
      • This interface is tested with a new plugin named q2-mystery-stew, which produces 553 integration tests.
    • For more information on how to get started, check out the README, or watch this video on setting up Docker on Windows (please see below, at the bottom of this list item).
      • Please note that this is still very experimental, if you run into issues, do let us know!
    • As it is an alpha release, there are still some limitations:
      • No importing (yet! - exporting is available however)
      • The conda environment is not pre-installed on docker, so the first QIIME 2 command will take some time as it is installed. Jobs run afterwards will start immediately.
      • Visualizations have to be viewed manually in q2view unless your galaxy deployment is configured with HTTPS and CORS headers (you may need a Galaxy sys-admin to help you).
      • Semantic properties are not quite working yet, this may impact some actions which use them as the history-selector will not recognize some valid inputs.
      • TypeMap does not enforce any constraints in the UI at this time. (The framework will of course error as usual if any constraint is violated.)
      • Only TSV metadata is supported, there is not yet a mechanism to view artifacts as metadata or to convert them to a tabular format (this will be available soon).

https://www.youtube.com/watch?v=pjVJ0ncBRsM

  • New plugin: q2-mystery-stew :shallow_pan_of_food: by @Oddant1 and @ebolyen
    • This plugin is designed to test interfaces by generating a large number of actions with different inputs and parameters. It currently has 553 tests and was instrumental in testing q2galaxy’s generated tools.
    • Future work will include connecting this plugin with other interfaces such as q2cli and the Artifact API.
  • New version of Python (3.8) :snake:
    • All QIIME 2 packages in the Core Distribution now are built using Python 3.8! :partying_face:
    • Not all Community plugins have had a chance to update their conda packages to work with Python 3.8, so we will continue supporting QIIME 2 2021.2 until the 2021.8 release (currently scheduled for late August 2021) :calendar:
    • For some examples of interesting things that can be done in this version, check out this older post by @ebolyen :necktie:
  • QIIME 2 Framework
    • @thermokarst and @ebolyen fixed an issue in the framework that prevented users from generating data in certain conditions, most notably on some networked filesystems, such as BeeGFS, (common in HPC environments). The issue would crop up when attempting to move files (behind the scenes) before zipping them up into Artifacts - we solved this by falling back to a filesystem copy operation (which might be a little bit slower, but should work as expected in those off cases). :t_rex:
  • docs
    • @andrewsanchez updated the Read Joining tutorial to reflect the latest and greatest visualizations available! :woman_genie:
    • @BenKaehler added new weighted Naive Bayes classifiers will be published shortly after this release - please watch this topic thread for updates when they are made available! :milky_way:
  • q2cli
    • @lizgehret fixed a :bug: affecting Zsh users that prevented tab auto-completion from being enabled by default.
  • q2-dada2
    • @Oddant1 added a new parameter to denoise-paired that allows users of this method to control that minimum length for forward/reverse overlap. The default value of 12 remains unchanged from previous versions of this method. :face_with_hand_over_mouth:
  • q2-phylogeny
    • @SoilRotifer updated several IQ-TREE 2 and RAxML related tests, to support a new version of RAxML (2). :racing_car:
  • q2-types
    • @SoilRotifer refactored DNASequence :dna: to only accept DNA, and not RNA, sequences.
    • @SoilRotifer added the RNASequence and AlignedRNASequence types. :alien:
    • @SoilRotifer added RNAFASTAFormat and other related RNA formats. :derelict_house:
    • This means future and existing plugins can be updated to handle RNA sequence data directly. :diamond_shape_with_a_dot_inside:
  • q2-sample-classifier
    • @jwchen added a link to download the raw data of RFE plots. :butterfly:
    • @adamova updated SVC and SVC_linear to return calibrated probability values (as described here) and to adjust the SVC_linear classifier to be defined by SVC(kernel='linear') instead of LinearSVC(). :bowling:
    • This update also includes a verification for all classifiers to output predicted probabilities in a range from 0 to 1. :100:
  • q2-diversity
    • procrustes-analysis: GH User @adhdilmore added a ProcrustesStatistics output that stores the disparity (m^2), a permutation-based p-value, and the number of permutations for a procrustes fit. :peacock:
    • procrustes-analysis: @yoshiki added a --p-permutations parameter to select the number of permutations to compute the Procrustes analysis’ p-value. :pretzel:
  • q2-emperor
    • procrustes-plot: GH User @adhdilmore added a way to view ProcrustesStatistics values in the top left of a Procrustes plot in Emperor (p and m^2 values). :chart_with_upwards_trend:
    • @yoshiki updated to depend on the newest Emperor :penguin: release (1.0.3).
  • view.qiime2.org
    • @lizgehret updated the gallery links to reflect the newest release version (2021.4). :game_die:
  • q2-taxa
    • @Keegan-Evans changed barplot :bar_chart: so that providing sample metadata is now optional. The visualization will display the number of columns of sample metadata provided or 0 if no sample metadata is provided. :garlic:
    • @fedarko created an improvement for barplot :bar_chart: so that at all levels, the color scheme displays with the same color topmost. :broccoli:
    • @wasade made an update that avoids a dense representation of the FeatureTable when performing a taxonomy collapse. :curly_loop:
  • q2-longitudinal
    • @thermokarst fixed a :bug: where the matplotlib canvas would occasionally fail to clear, leading to downstream errors with things like setting axis labels. :lollipop:
2

The admin team is meeting in a week or two to discuss the specific release timing - stay tuned.

3

The release dates have been added up above - please let me know if you have any questions!

4

A big change coming up in this release is that we have moved from Python 3.6 to Python 3.8! The rationale here is that Python 3.6 EOL is slated for the end of 2021, and, we were starting to fall behind on other Python-related dependencies because of this (some of the major conda channels out there appear to be dropping their support for Python 3.6, or at least winding down). We have moved our minimum support version up to Python 3.8 - and we hope to add support for 3.9 later this year (:crossed_fingers:).

In the meantime, if you want to test out your current plugin in Python 3.8, you can get your hands on a 3.8 development environment here:

macOS

wget https://raw.githubusercontent.com/qiime2/environment-files/master/latest/staging/qiime2-latest-py38-osx-conda.yml
conda env create -n qiime2-dev --file qiime2-latest-py38-osx-conda.yml
rm qiime2-latest-py38-osx-conda.yml

linux

wget https://raw.githubusercontent.com/qiime2/environment-files/master/latest/staging/qiime2-latest-py38-linux-conda.yml
conda env create -n qiime2-dev --file qiime2-latest-py38-linux-conda.yml
rm qiime2-latest-py38-linux-conda.yml

We will most likely continue to support QIIME 2 2021.2 until our August 2021 QIIME 2 release (2021.8), that way we give plugin devs enough time to deal with moving over. For what it's worth though, the upgrade path for us was relatively painless, and there aren't any specific items that we think plugin devs should be aware of.

cc list (apologies if I miss anyone, just want to loop in folks I think are working on plugins right now):

@gibsramen @adamova @misialq @Nicholas_Bokulich @jwdebelius @BenKaehler @ChrisKeefe @Keegan-Evans @gregcaporaso @ebolyen @gmdouglas @Adam_Rivers @Oddant1

8 Likes
9

A quick release update for those interested - the release is almost all set - we will hopefully post it first thing tomorrow morning (Pacific Time). This is a little bit later than we had previously advertised, but not by much. Thanks!

4 Likes