[Preview] QIIME 2 2019.10 development changelog

Developer Discussion
,
1

QIIME 2 2019.10 has been released!
Please see the official changelog for more details.

Original content of post

:exclamation: Important :exclamation:

The following is an early developer preview of the changes expected in 2019.10 This post is a live-document which will be updated throughout our development cycle. Any links will in this topic will be broken until the release is officially published. When we are ready for release, we’ll copy this changelog and create a new post in the Announcements category.

Important Developer Information

Dates (please keep an :eye: on this post, these :calendar: might change):

  • PRs must be submitted by: October 18, 2019
  • PRs must be merged by: October 25, 2019
  • Repo Freeze and Package Building: October 31, 2019
  • Release Day: November 1, 2019

Developer Project Board: 2019.10 · GitHub

Here’s the highlights of the release:

  • QIIME 2 Framework
    • @Oddant1 added a view method to the TextFileFormat and BinaryFileFormat classes, which allows for easier type conversion (especially for developers!). :jack_o_lantern:
    • @Nicholas_Bokulich updated the citation info for the canonical QIIME 2 citation (Bolyen et al., 2019 Nature Biotechnol) :clinking_glasses:
    • @David-Rod upgraded our core distribution to use the latest (and greatest) version of Pandas :panda_face:!
    • @Oddant1 implemented a highly sought after feature - Metadata constructed via the Artifact API now gets all of its whitespace stripped off :ghost:! This is really useful for the dozens of transformers that allow for various semantic types to be "viewed" as Metadata! :hotsprings:
    • @ebolyen fixed a bug that prevented MetadataColumnvalues from being decoded properly - this was mostly an issue for q2studio. :raccoon:
    • @thermokarst made some changes to a framework utility that uses the networkx package --- QIIME 2 is now compatible with the latest release of this tool! :wrench:
  • docs
    • @Nicholas_Bokulich updated the tutorials to reflect changes to the feature-table heatmap API (primarily the parameter rename from metadata to sample-metadata). :file_cabinet:
    • @SoilRotifer added the tutorial "Phylogenetic inference with q2-phylogeny". :evergreen_tree: This is the latest version of his Community Tutorial of the same name :trackball:
    • @BenKaehler added a nifty new section to the Parkinson's Mice Tutorial that demonstrates how to create a bespoke classifier. :ferris_wheel:
    • @ChrisKeefe found a pesky typo in the Parkinson's Mice Tutorial dataset :mouse:, but fortunately those days are behind us!
    • @thermokarst created a new "SEPP Refs" section of the Data Resources page on the QIIME 2 Docs - these handy little databases are for use with q2-fragment-insertion! :framed_picture:
    • @gregcaporaso re-wrote the User Glossary of QIIME 2 terms - if you have ever puzzled over our definitions of words like "Artifact" or "Pipeline" - this is the page for you! :purple_heart:
  • Library
    • @ebolyen, chief drop-shadow design guru for the QIIME 2 team did what he does best: beefed up the drop shadow on the Library! :auto_rickshaw:
  • q2cli
    • @David-Rod fixed a bug :bug: that caused help text generation to slow down significantly! For the average user this wasn't much of a problem, but, if you're in the business of building QIIME 2 docs, this slowdown was significant. :sweat_smile:
    • @Oddant1 fixed a bug that prevented exporting to a local path. We think Rumpelstiltskin was involved - but that's neither here nor there - @Oddant1 got us sorted out! :man_cartwheeling:
    • @ebolyen fixed a bug that caused things to explode :exploding_head: when saving files into a non-existent directory!
    • @Oddant1 improved the error message presented when an invalid primitive value is identified --- previously the error message was pretty opaque, now it actually tells you what is wrong! :mobile_phone_off:
  • q2-feature-classifier
    • @Oddant1 exposed new n_jobs and batch_size parameters in the extract_reads method allowing for parallelization of the method. :moneybag:
    • @Oddant1 changed extract-reads so it will now ensure that min-len is greater than trunc-len minus trim-left before executing. :grapes:
  • q2-sample-classifier
    • @Oddant1 added the heatmap visualizer to the party :partying_face: known as the classify-samples pipeline.
    • @Nicholas_Bokulich fixed a bug in confusion-matrix that caused ROC plots to fail on unstratified, imbalanced data. :call_me_hand:
    • @Nicholas_Bokulich added cividis to the list of available colormaps. :chair: :rainbow:
    • @Oddant1 added the ability to adjust color scale of the confusion-matrix heatmap. :butter:
    • @David-Rod made some updates to the transformers defined in this plugin to allow it to work with the latest versions of Pandas :panda_face:!
    • @Nicholas_Bokulich added a new drop_all_unique parameter to the metatable action - this prevents metadata with all unique columns from ruining the party :partying_face:
    • @ChrisKeefe fixed a bug in the heatmap visualizations produced in this plugin that caused some cells to be trimmed, which looked pretty weird! :ice_cube:
  • q2-feature-table
    • @Nicholas_Bokulich added cividis to the list of available colormaps for heatmap. :rainbow:
    • @Nicholas_Bokulich updated heatmap to accept both sample-metadata and feature-metadata, enabling sample/feature labeling along each axis. :european_castle:
  • q2-longitudinal
    • @Nicholas_Bokulich added cividis to the list of available colormaps. Wow, @Nicholas_Bokulich was really excited about this colormap! :garlic:
    • @Nicholas_Bokulich made the plot-feature-volatility visualization a lot more performant by building-in a simple "filter" system - by default only the top 100 most important features will be displayed. There are a few other mechanisms for filtering importances - check out the docs here! :astonished:
  • q2-gneiss
    • @Oddant1 created a new ignore_missing_samples parameter in the gradient_clustering method that, well, allows for any missing samples in the metadata to be, ahem ignored. :woman_mage:
    • @ebolyen made some updates to the test suite defined in this plugin to allow it to work with the latest versions of Pandas :panda_face:!
  • q2-types
    • @David-Rod removed white space and comment support from TaxonomyFormat and TSVTaxonomyFormat. These elements generally caused more problems than they solved, so we decided to part ways with those aspects of the formats.
    • @Oddant1 made it so DNAFASTAFormat disallows duplicate records, disallows whitespace at the start of an ID, and a few other housekeeping elements. :world_map:
    • @David-Rod made some updates to the tests defined in this plugin to allow it to work with the latest versions of Pandas :panda_face:!
    • @David-Rod beefed up :cow2: the validation for TSVTaxonomyFormat - users will be presented with much clearer error messages whenever there is a problem importing or creating one of these files!
  • q2-deblur
    • @Oddant1 Made it a new warning to prevent allowing underscores in sample ids (which deblur can't handle. :hammer:
    • @EmFord (GH) got rid of a leftover license file lurking in the source code for this plugin! :spider_web:
  • q2-emperor
    • This release is using the latest release version of emperor (1.0.0-beta.20).
    • Miscellaneous bug fixes: :bug: and performance improvements :speedboat::
      • Improve error messages for missing or mismatching feature metatdata.
      • For large plots fix a problem where hidden objects would still be clickable.
      • Fixes a bug where axes orientation would be incorrectly loaded from a settings file.
      • Daniel Hakim: Improved performance for animations with a large number of timepoints.
    • New features :sparkles::
      • Add a search bar to every tab to narrow down the values you are interested in for any metadata column.
      • Xuhan Yang: Adds a list of colors to the color selector based on the currently selected color palette.
      • Daniel Hakim: Add the ability to visualize multiple dimensions using parallel plots. Go to the Axes tab and click on Parallel or Scatter to change the plot type.
      • The following animated GIF showcases all three new features:
        release
        release1904×1042 590 KB
  • q2-diversity
    • @tomasz implemented a new intersect_ids feature in the alpha-correlation visualizer. This parameter allows for filtering out IDs in sample metadata or the SampleData[AlphaDiversity] that aren't found in both inputs (which would previously terminate as an error). :red_gift_envelope:
    • @yoshiki got the core-metrics/core-metrics-phylogenetics pipelines outfitted with the ability to subsample-with-replacement while rarefying! :billed_cap:
    • @ebolyen fixed a bug in the beta-group-significance visualization that caused some of the boxplots and PDFs to not show up. This was primarily caused by having any kind of non-alphanumeric characters in the metadata used to compute the visualization: :pleading_face:
  • q2-vsearch
    • @colinbrislawn hooked us up with a sweet :candy: new feature in join-pairs --- the ability to specify how many execution threads :thread: with which to get the job done! Neat!
  • q2-taxa
    • @David-Rod took a magnifying glass :mag_right: to the barplot visualizer in this plugin - lo and behold, a simple, yet important typo was revealed - an error used the word "feature" when it should've used the word "sample". Funny how such a small change in language can yield :stop_sign: such different results!
  • q2-fragment-insertion
    • @thermokarst & @Stefan brought SILVA 128 to the sepp method, and there was much rejoicing! :game_die: This method no longer contains a "default" reference (it used to be GreenGenes) - now users can grab a handy SEPP database reference from the Docs (pick your favorite - SILVA or GG!). In the future we hope to expose new methods to support importing and prepping any database, stay tuned :tv: ! For a glimpse of this new action, well, uh, in action, check out the Parkinson's Mouse Tutorial!
  • q2-dada2
    • @Oddant1 updated the DADA2StatsFormat to include a few new computed columns, that way you don't have to do the mental math :brain: to figure out the percentage of reads merged. Phew!

Happy QIIME-ing! :sun_with_face:

4 Likes
10

We are going to push the release back by a few days, to November 1, 2019, that way we have enough time to get through our backlog. Thanks for your patience!

7 Likes