[Preview] QIIME 2 2019.4 development changelog

Developer Discussion
,
1

QIIME 2 2019.4 has been released!
Please see the official changelog for more details.

Original content of post

:exclamation: Important :exclamation:

The following is an early developer preview of the changes expected in 2019.4 This post is a live-document which will be updated throughout our development cycle. Any links will in this topic will be broken until the release is officially published. When we are ready for release, we’ll copy this changelog and create a new post in the Announcements category.

Important Developer Dates (please keep an :eye: on this post, these :calendar: might change):

  • PRs must be submitted by: April 24, 2019
  • PRs must be merged by: April 26, 2019
  • Repo Freeze and Package Building: April 29, 2019
  • Release Day: April 30, 2019

Here’s the highlights of the release:

  • QIIME 2 Framework
    • @ChrisKeefe fixed a bug caused by invalid primitive arguments :three::no_good_man:. Expanded type mismatch error messages to better describe various invalid-input situations. :speaking_head:
    • @ChrisKeefe fixed some pretty confusing language in the SemanticType docstring.
    • @Sleightly (GH) has added a new SDK utility named build_graph to qiime2.sdk.actiongraph. This will generate a directed NetworkX graph of QIIME 2 plugin actions and types. The graph is bipartite, with types on one conceptual side, and actions on the other. Edges between them indicate parameters.
    • The implementation of QIIME 2 types has been reworked by @ebolyen :martial_arts_uniform: . Some minor changes include:
      • Properties(['a', 'b']) => Properties('a', 'b')
      • Range(X, None) can be optionally written as Start(X) (it is converted to a Range)
      • Range(None, X) can be optionally written as End(X) (also converted to Range)
      • Choices({'a', 'b'}) => Choices('a', 'b') (this means order is now retained as well)
    • Unions of primitive types are now possible, thanks to @ebolyen! This means you can create a plugin with a parameter such as: Int % Start(0) | Str % Choices("ALL", "SOME", "NONE")
    • Intersection (&) of types is now possible :stop_sign: however most intersections will result in a "bottom" type - thanks for that, @ebolyen! (This operator is primarily for implementing internal type solving algorithms.)
      • real intesection: Start(0) & End(10) => Range(0, 10)
      • contradiction: FeatureTable[Frequency] & Phylogeny[Rooted] => UnionExp([]) (an empty union is the smallest, emptiest, type in QIIME 2)
    • The framework now supports different output types based on the received input type :grapes: . This feature is called TypeMap (and it has a sibling TypeMatch, and was created by @ebolyen):
      • This allows expressions such as:
        T, P, R = TypeMap({
    (TypeA, Bool % Choices(True)): OutputA,
    (TypeB, Bool % Chocies(False)): OutputB
})
      • For a direct example of TypeMatch, see implementation for feature-table filter-samples.
    • A utility to parse a type from the result of a to_ast() call now exists: qiime2.sdk.util.type_from_ast. This and parse_type are now significantly faster as loading plugins are no longer required to reconstruct the type. type_from_ast is capable of completely reconstructing a TypeMap.
    • @thermokarst added a new utility to qiime2.sdk.util named parse_primitive which takes on the role of <sometype>.decode(val), however it can handle collection types of varying style.
    • Other type utilities exposed in qiime2.sdk.util include: is_semantic_type, is_primitive_type, is_metadata_type, is_visualization_type, and interrogate_collection_type.
  • Library
    • @thermokarst fixed a bug that prevented admins from editing plugins (due to an over-specified database rule).
  • docs
    • @Nicholas_Bokulich fixed a typo in one of the overview tutorial images.
    • @Nicholas_Bokulich added an example using q2-sample-classifier's new heatmap pipeline to the q2-sample-classifier tutorial. :bookmark_tabs:
  • dev-docs
    • @ChrisKeefe busted out his editor's red pencil on the dev docs.
    • @cduvallet updated the plugin development tutorial to reflect the latest versions of the packages used in the document! :card_file_box:
  • q2cli
    • @epruesse cleaned up the --version output when looking at a plugin's details! :cherry_blossom:
    • @ebolyen put many hours of blood (:syringe:), sweat (:sweat_drops:), and tears (:cry:) into cleaning up the q2cli experience, changes include:
      • --cmd-config has been removed (it was underdocumented and added a lot of complexity). We encourage users of QIIME 2 who need programmatic control to use the Python API instead which is significantly more flexible.
      • --py-packages has been removed from qiime info (it was already broken). Use conda list instead. :dragon:
      • --output-dir and --o-<name> options now ensure the path is writable before executing the command. :writing_hand:
      • --output-dir no longer creates the directory before running the command. This means no more rm -rf'ing because there was a typo in a different parameter. :open_file_folder:
      • The underlying parser has been significantly improved. The following is now possible:
        • Missing arguments will now be recognized instead of swallowing the next option (very confusing for beginners). :open_mouth:
        • --help is now "hyper-greedy", if it appears anywhere in the command, the help text will be shown (no matter how many errors exist otherwise).
        • Options that are repeatable may also now receive an arbitrary number of arguments:
          --p-something 1 --p-something 2 --p-something 3
          can now also be written as:
          --p-something 1 2 3   # the original form is still supported
          This means that shell-expansion (like *.qza) will become more useful:
          qiime feature-table merge --i-tables dada2-outputs/*.qza ...
      • The CLI cache is now optimistically :butterfly: stored in your conda environment. If q2cli is not running in a conda environment, then it will fall back to XDG_CONFIG_HOME as before. This means that switching between multiple QIIME 2 installations will not constantly refresh the cache.
      • difflib is now used to recommend options when a typo is made in a command. :woman_shrugging:
      • Error handling has been made significantly more consistent between plugin commands and builtins (such as tools, dev, and info):
      • Errors are now enumerated. Parsing errors such as malformed commands will be re-parsed multiple times to collect as many errors as q2cli can (these are an incomplete list, and will be printed as (1/5?) instead of (1/5)) :100:
      • In order to support TypeMap, the --help text has a new layout system which includes the QIIME 2 type whenever relevant (this is especially helpful for ranged primitives). TypeMaps are indicated with a superscript number which will match between inputs and outputs. (In the future, a table may be a better way to represent this).
      • Help text is now colorized to improve readability :rainbow: :

https://forum-qiime2-org.s3.dualstack.us-west-2.amazonaws.com/original/2X/c/c83b5c662898d9addfee797017d1e087171042b8.webm

  • q2-diversity
    • @thermokarst fixed a bug :bug: in the adonis visualizer that prevented some metadata headers from working as expected.
    • @Nicholas_Bokulich fixed a bug in the adonis visualizer that caused an error if metadata values contained single quotes (e.g., apostrophes). :butterfly:
    • @Nicholas_Bokulich fixed a typo in the adonis visualizer help text. :alembic:
  • q2-longitudinal
    • @Nicholas_Bokulich added a new anova visualizer to the plugin! :tada:
    • The heatmap generated by maturity-index can now be controlled to limit the number of important features displayed. (Thanks @Nicholas_Bokulich!) :fire_engine:
  • q2-deblur
    • @thermokarst added a new left_trim_len parameter to the denoising methods of this plugin! :mountain_biking_man:
  • q2-demux
    • Fixed the per-sample sequence count table layout (that thing sure was bothering @ChrisKeefe!) :fish_cake:
    • @wasade brought Golay barcode correction to emp-single and emp-paired - the future is now! :baseball:
  • q2-cutadapt
    • @thermokarst added a new minimum_length parameter to trim_single and trim_paired! :scissors:
    • The ability to discard_untrimmed reads has been added to trim_single and trim_paired by @thermokarst. :grapes:
    • @thermokarst fixed a bug related to too many files being opened when demuxing datasets with many samples in it. Check out the new batch_size parameter on demux-paired and demux-single to control how many samples are demultiplexed at a time! :broccoli:
    • @ChrisKeefe & @thermokarst added support for dual-index demultiplexing! This functionality is enabled by preparing a second column in your Metadata file that contains the barcodes for the reverse reads. Check out the docs for more info!
  • q2-feature-classifier
  • q2-sample-classifier
    • @Nicholas_Bokulich added a heatmap pipeline to display the (normalized) abundances of predictive features per sample or per group, using FeatureData[Importance] scores output by supervised classifiers and regressors. :fire_extinguisher:
    • @Nicholas_Bokulich updated the citation information to include the report describing this plugin in the Journal of Open Source Software.
  • q2-feature-table
    • @gwarmstrong (GH) added a feature to allow disabling of clustering on both the sample and feature axes when creating a heatmap! :fire:
    • merge-taxa now supports multi-column FeatureData[Taxonomy] artifacts, e.g., to preserve classification confidence scores in the merged output. (thanks @Nicholas_Bokulich) :japanese_ogre:
    • @turanoo overhauled the interactive even-sampling depth visualization --- it is now built with vega! There were several improvements/bugfixes related:
      • Sample count summaries for each metadata group are now displayed in the plot
      • SVG and PNG exports are enabled for the plot
      • The visualization can be customized in the interactive vega editor
      • A bug related to scrolling in the visualization's tabs was fixed
    • As a result of TypeMap, it is now possible to filter out samples (feature-table filter-samples) of feature tables with the variant types of: RelativeFrequency, PresenceAbsence, and Composition (Frequency is of course still supported). The output type will match the input type provided.
  • q2-quality-control
    • @Nicholas_Bokulich fixed a bug in evaluate-composition that caused linear regression R values of zero to be reported when observed and expected feature tables contain only a single observation at any taxonomic level. (the correct answer is no R value should be reported). :bug:
  • q2-dada2
    • @benjjneb updated this plugin to use DADA2 1.10 - users can expect to see speed improvements from 2x to 20x! :racing_car: :fire: :sweet_potato:! Check out the DADA2 docs for more details on this version of DADA2.
  • q2-emperor
    • This plugin now uses the latest version of emperor (1.0.0-beta.19)
    • Miscellaneous bug fixes: :bug:
      • Fixes a bug where re-centering the camera wouldn't re-center the view.
      • Fixes a bug where two dimensional plots would fail to load.
      • Fixes a bug where sample metadata would fail to load if only sample identifiers were included.
    • Experimental support for exporting 2D vega plots has been added! This is currently only available for a subset of the available plot types, and can be accessed through the menu:
      image
      Any customization to the 3D plot should be inherited in the vega plot. For example, here is a customized 3D plot:
      image
      image1076×800 67.4 KB

      And here is the 2D vega version:
      visualization
      This functionality is still experimental --- please try it out and let us know how it goes for you! :balloon:

Happy QIIME-ing! :sun_with_face:

3 Likes
2

Still time to throw in a --discard-untrimmed parameter with this update @thermokarst? One of the things I realized was invaluable in my workflow was using the linked adapters feature in Cutadapt; however, I only wanted to keep the reads that were framed by those 5' and 3' primers, and that required passing that flag (I think/hope!?).

Perhaps a future request for 2019.8?

Thanks for the improvements everyone!

3 Likes
3

:bomb:

Ask, and you shall receive.

2 Likes
4

Just a friendly reminder, PRs must be submitted by April 19, 2019 in order to be considered for the 2019.4 release. Keep us posted if you have incoming work that might not be on our radar - thanks! :qiime2:

5

Scratch that, we are going to postpone the release by a few days so that we have time here to wrap up a few bigger projects:

  • PRs must be submitted by: April 24, 2019
  • PRs must be merged by: April 26, 2019
  • Repo Freeze and Package Building: April 29, 2019
  • Release Day: April 30, 2019
6

Hey all - we are still wrapping up a few exciting new features, so we are going to postpone the release by a few days. We hope to get it out as soon as possible --- please stay tuned! :t_rex:

1 Like
9

Thanks for being patient — we think the few extra days will be worth the wait! Head on over to the 2019.4 Announcement to learn more about some of the exciting new features in this release!

3 Likes