[Preview] QIIME 2 2019.7 development changelog

Developer Discussion
,
1

QIIME 2 2019.7 has been released!
Please see the official changelog for more details.

Original content of post

:exclamation: Important :exclamation:

The following is an early developer preview of the changes expected in 2019.7 This post is a live-document which will be updated throughout our development cycle. Any links in this topic will be broken until the release is officially published. When we are ready for release, we’ll copy this changelog and create a new post in the Announcements category.

Important Developer Dates (please keep an :eye: on this post, these :calendar: might change):

  • PRs must be submitted by: July 22, 2019
  • PRs must be merged by: July 26, 2019
  • Repo Freeze and Package Building: July 29, 2019
  • Release Day: July 30, 2019

Here’s the highlights of the release:

  • QIIME 2 Framework
    • GH user @asoback updated the framework source code's README file to provide a bit more context for the casual reader.
    • @Oddant1 made it so artifacts now validate the entire file on import, which should hopefully help prevent down-stream data issues. This will add a little bit of additional time to importing, but, we think it is worth the wait! :timer_clock:
    • @Oddant1 added support for action deprecation in plugins! Using deprecated action will raise a warning indicating that the action is deprecated and will be removed in a future release of that plugin.
    • @Nicholas_Bokulich added the definitive QIIME 2 citation (Nature Biotech, 2019) to the plugin citations, as well as to the README file.
    • @thermokarst added a new is_union helper to support union input types. Also fixed q2cli to support composite primitives
    • @thermokarst pinned pandas=0.24.2 for the 2019.7 release.
  • docs
    • @thermokarst updated many of the metadata column names in a handful of tutorials, to use a new naming convention (e.g. BarcodeSequence is now barcode-sequence). :barber:
    • @thermokarst updated the SQL-where clause quoting convention in the tutorials to use the [column-name] syntax, which we think is a little easier to read! :eyeglasses:
    • @jwdebelius wrote a tutorial introducing a new dataset based on a mice study of Parkinson's disease. :mouse2:
    • @thermokarst added the ability to capture command streams, which is useful for tutorial writing efforts. Neat! :meat_on_bone:
    • @thermokarst finally got around to adding the workshop deployment script to the main code base in this project, a feature useful for workshop instructors. :woman_teacher:
    • New pre-trained feature classifiers are available now. These pretrained feature classifiers reflect the recent update to scikit-learn 0.21.2. :ledger:
    • @Nicholas_Bokulich added the definitive QIIME 2 citation (Nature Biotech, 2019) to the "Citing QIIME 2" page.
    • @Nicholas_Bokulich added descriptions of two new features to the sample-classifier tutorial: ROC curves generated as part of the classify-samples pipeline and confusion-matrix visualizer; and the Probabilities output from classify-samples, regress-samples, predict-classification, and predict-regression.
  • dev-docs
    • @cduvallet added a new tutorial describing how to update plugins.
  • library
    • @David-Rod Updated copyright in page to reflect current year.
  • q2cli
    • @Oddant1 added proper reporting of "out of space on drive" and "file path does not exist" errors.
    • @David-Rod Made changes to the help text for qiime tools view. The change should now show users a valid command suggestion for extracting a Visualization.
    • @Oddant1 added the ability to customize command line text formatting.
    • @thermokarst added support for a new warning formatting text class and the new deprecated plugin attribute.
    • @ebolyen fixed a bug to enable tab-completion on root-installed environments. :broom:
  • q2templates
    • @David-Rod Resized tables, adjusted text alignment, and centered table in browser window.
  • q2view
    • @David-Rod Renamed Peek tab to Details and moved the provenance-recorded citations panel from the Provenance tab to the newly renamed Details tab.
    • @David-Rod Refactored React code in Gallery file to help keep things clean for future developers! :spider_web:
  • q2-feature-classifier
    • @Nicholas_Bokulich added a new pipeline: classify-hybrid-vsearch-sklearn. This pipeline performs taxonomic classification of sequences in multiple steps: 1) [optional] sequences are pre-filtered against a subsampled reference database to remove non-target sequences; 2) sequences are classified with QIIME 2's vsearch+LCA taxonomy classifier, using the new search-exact parameter to find exact matches followed by consensus taxonomy assignment; 3) any sequence not classified by the exact match classifier are classified with QIIME 2's classify-sklearn method, using a pre-trained supervised-learning taxonomy classifier. :exploding_head:
    • @Nicholas_Bokulich added two new parameters to the classify-consensus-vsearch method: search-exact, and top-hits-only. search-exact will search a reference database for exact matches to the query sequences; only these matches are returned as hits. The top-hits-only parameter causes only reference database matches that are tied for the top percentage identity to be recorded as hits. For both search-exact and top-hits-only it is highly possible that multiple exact matches or top hits (ties) will be found, so consensus taxonomy classification is still performed by q2-feature-classifier. :tophat:
    • Updated to scikit-learn v0.21.2! :classical_building:
    • @BenKaehler added joblib as a dependency (following deprecation of vendored code in sklearn).
    • @thermokarst updated the tests to use BIOMV210 format. :spider_web:
  • q2-cutadapt
    • @Oddant1 added a minimum_length parameter to the two demux methods in this plugin! :straight_ruler:
  • q2-demux
    • @Oddant1 added a filter-samples method for separating data from a single demux run using sample metadata.
  • q2-dada2
    • GH user @angrybee exposed individual forward and reverse maxEE parameters for denoise-paired. :honeybee:
  • q2-deblur
    • @Oddant1 added the ability for q2-deblur to parse sequence identifiers that include semicolons.
  • q2-diversity
    • @gwarmstrong Updated to a newer version of the UniFrac package. This includes a few bug fixes and an optimized implementation of Faith's phylogenetic diversity.
    • @gwarmstrong added a new method qiime diversity alpha-phylogenetic-alt which uses an optimized implementation of Faith's Phylogenetic diversity. This method has a smaller memory footprint and much faster execution time for large datasets :fire: (check out some of the benchmarks here). The results produced by alpha-phylogentic and alpha-phylogenetic-alt are otherwise identical.
    • @gwarmstrong updated beta-phylogenetic to remove an extraneous error check. Previously, phylogenies were being loaded as skbio.TreeNode objects to confirm that the phylogeny represented the entire feature table, but the latest version of unifrac performs the same check. This will speed up beta phylogenetic methods in QIIME 2. :racing_car:
    • @Nicholas_Bokulich updated the BetaRarefactionTests to use scipy.special.comb instead of scipy.misc.comb, which was deprecated in scipy v1.0.0 and removed in the latest version of scipy.
    • @Stefan fixed a typo in the action description of adonis.
    • @antgonza fixed the view type for adonis to be int instead of str.
  • q2-taxa
    • @fedarko added a slider to adjust the bar widths in the barplot visualization! :bar_chart: :left_right_arrow:
    • @Oddant1 added a new error message for when the metadata is missing any Sample IDs that are present in the FeatureTable[Frequency].
    • @fedarko fixed a bug where the color scheme in the barplot would get reset when changing the taxonomic level! :bug: :art:
  • q2-sample-classifier
    • @cduvallet & @Oddant1 collaborated to allow split_tables to work with all kinds of FeatureTables! :potable_water:
    • @David-Rod Removed styling rule for tables generated by templates.
    • @Nicholas_Bokulich fixed the heatmap metadata descriptions in the help documentation.
    • @Nicholas_Bokulich updated test data and tests for recursive feature extraction.
    • @Nicholas_Bokulich added joblib as a dependency (following deprecation of vendored code in sklearn).
    • @Nicholas_Bokulich added a Probabilities output to predict-classification, classify-samples, and classify-samples-ncv. These methods and pipeline now output predicted class probabilities for each test sample.
    • @Nicholas_Bokulich fixed a minor bug that caused feature importance scores to be read as non-numeric values.
    • @Nicholas_Bokulich cleaned up the code base to prevent deprecation errors related to pandas, sklearn, and some other dependencies. Some test data was updated to reflect expected changes caused by the updates (e.g., SVM estimators have been updated to use gamma=scale by default, which impacts test results).
    • @Nicholas_Bokulich added a new feature to the visualization generated as part of the classify-samples pipeline and confusion-matrix visualizer: Receiving Operator Characteristic (ROC) plots and area under the curve (AUC). See the q2-sample-classifier tutorial for more details. :chart_with_upwards_trend:
  • q2-composition
    • GH user @Jiung-Wen fixed a bug in the ANCOM visualizer that prevented users from getting their hands on the backing data from the volcano plot. Thanks! :bug: :handshake:
  • q2-gneiss
    • :exclamation::exclamation::exclamation: Several visualizers have been deprecated and will be removed in a future release of q2-gneiss (we anticipate they will be removed in 2019.10) :exclamation::exclamation::exclamation::
      • balance-taxonomy
      • dendrogram-heatmap
      • ols-regression
      • lme-regression
    • @Oddant1 improved the error message produced when attempting to use categorical metadata columns where all values are numerical when running the balance_taxonomy visualizer. :sunglasses:
    • @Nicholas_Bokulich added tight layout formatting to q2-gneiss visualizations. This prevents plots with long labels from being truncated. :haircut_woman:
  • q2-longitudinal
    • @Nicholas_Bokulich updated the test data to accommodate changes to statsmodels=0.10.0 that resulted in very slight changes to MixedLM results when multiple random effects are used. Linear mixed effects models that used < 2 random effects were not impacted.
    • @Nicholas_Bokulich fixed broken DOIs in citation list.
  • q2-types
  • https://qiime2.org

Happy QIIME-ing! :sun_with_face:

5 Likes
2

A friendly reminder that PRs for the upcoming 2019.7 release must be submitted by: July 22, 2019. If you have something cooking right now please let us know, that way we can get it on our radar. Thanks!

3

@thermokarst I've been slammed recently (as I'm sure you all have been as well :slight_smile: ) but I will try to find some time over the weekend to polish up the q2-taxa barplot a little bit (e.g. making bar widths persist even when the chart is re-rendered due to something like the taxonomic level change).

I'll try to have it in by the end of the weekend (and of course this isn't a super high priority fix, so I completely understand if it won't fit in before the next release -- don't want to stress you all out with more work right now).

3 Likes