[Preview] QIIME 2 2018.4 development changelog

[q2d2]

QIIME 2 2018.4 has been release!
Please see the official changelog for more details.

Original content of post

Important

The following is an early developer preview of the changes expected in 2018.4 This post is a live-document which will be updated throughout our development cycle. Any links will in this topic will be broken until the release is officially published. When we are ready for release, we'll copy this changelog and create a new post in the Announcements category.

---

Important Developer Dates:

• PRs must be submitted by: 04/20
• PRs must be merged by: 04/24
• Release Day: 04/26

---

Here's the highlights of the release (so far):

• QIIME 2 Framework
  • QIIME 2 now supports citations for anything a plugin can define (except semantic type)! This is a large feature with several associated changes:
    • Citations are structured data, instead of plain-text. They are cannonically represented as BibTeX entries, as these are compatible with a wide variety of software and are human-readable (for the most part). These entries may be UTF8 or BibTeX-escaped, but will be normalized to unicode (UTF8) for easier consumption.
    • Provenance now stores an additional citations.bib file (normalized as above) which contains all of the associated citations which were recoreded in the process of completing that action. This file will be present in ancestral nodes as well, allowing interfaces to collate all citations used from the start to end of an analysis (see qiime tools citations). Additionally, /provenance/action/action.yaml files now include a !cite custom type which references the citations.bib file, allowing interfaces to show exactly where a citation was referenced.
    • There is now a qiime2.plugin.Citations object available. It provides a classmethod Citations.load() which will parse a BibTeX file (optionally from package data). It also has a __getitem__ (obj['key']) and __iter__ (iter(obj)) API for use during registration.
    • There is a new .citations property available on Artifact and Visualization objects. It will contain a Citations object containing a collated list of entries which can be saved to a file via .citations.save('file.bib').
    • A minor note: transformers are now recorded in provenance as a result of these changes.
  • QIIME 2 can now render visualizations inside a Jupyter notebook (or lab).
    This requires installing a Jupyter server extension which is distributed alongside QIIME 2. Use this command to enable: jupyter serverextension enable --py qiime2 --sys-prefix (requires Jupyter Notebook or JupyterLab to be installed in your environment).
  • The qiime2.metadata subpackage now has numpydoc docstrings on all public-facing APIs. The subpackage also has extensive unit tests and 100% code coverage. Metadata-related error messages have been improved for clarity and contain more details.
  • qiime2.Metadata.filter_columns() now ignores missing data in its uniqueness/zero-variance filters.
  • qiime2.MetadataColumn now includes equality operators (__eq__ and __ne__) for object comparisons.
  • QIIME 2 environments will now set $R_LIBS_USER and $PYTHONNOUSERSITE environment variables avoiding library/import conflicts with locally installed packages. In practice, this means you shouldn't see:

    Loading required package: Rcpp
    
     *** caught segfault ***
    address 0x18, cause 'memory not mapped'
    

    ever again!
• docs
  • The QIIME 2 Metadata file format spec includes details about number formatting in metadata files, as well as a new "advanced details" section about the format.
  • Updated some outdated links - thanks for the contribution, @colinbrislawn!
  • Ported the OTU Picking Community Tutorial to the official docs.
  • Added 'copy-to-clipboard' buttons to all code/command blocks! These buttons make copy-and-pasting commands from the docs a one-click operation! Thanks @patthehat033!!
  • Updated the feature classifier tutorial to include details on working with paired-end reads, clarifications about read extraction, and some notes on ITS handling.
  • Updated the q2-longitudinal tutorial to describe the new interactive volatility plots incorporated in the volatility visualizer.
  • Updated the sample classifier tutorial to describe rule-of-thumb sample size requirements for using supervised learning actions.
  • Citations are now listed in a normalized format and are included on both the plugin and action pages.
• q2view
  • Fixed a bug that caused certain plots to not render, specifically plots that utilize user-supplied metadata, when that metadata had spaces or slashes in the MetadataColumn labels.
• q2cli
  • A new tool has been added: qiime tools citations which will print all citations associated with a given QIIME 2 Artifact or Visualization, including those citations associated with ancestors in provenance. This will only work on Artifacts/Visualizations and ancestors which were generated by this release or later. Older results will not have citation information and so qiime tools citations will have nothing to report.
  • A new --citations flag has been added to plugin commands and plugin actions (e.g. qiime <plugin> --citations and qiime <plugin> <action> --citations). This will print a BibTeX file to standard out and will list what citations have been associated with the command in question.
  • qiime info --citations has been removed in favor of the above mechanisms.
  • Fixed a bug with --verbose not always working as expected.
• q2templates
  • Updated the base template for QIIME 2 visualizations to now include a footer section - this feature is targeting QIIME 2 visualization developers.
• q2-dada2
  • This plugin now supports denoising pyrosequencing reads! The new denoise_pyro method supports both 454 and single-end IonTorrent sequencing products.
  • A new output is available named denoising-stats on denoise-single/paired/pyro which can be visualized with qiime metadata tabulate. It contains information about how many reads are retained through the denoising process for every sample (this was formerly available via --verbose, but only for a few samples).
• q2-diversity
  • Added a new method - procrustes-analysis, for fitting two ordination matrices with a Procrustes analysis.
  • Fixed a bug that caused alpha-group-significance visualizations to break if Metadata Column labels contained a forward-slash character (/).
• q2-longitudinal
  • Made group-column optional when running pairwise-differences, allowing paired tests to be performed in a single group (e.g., does metric X change between states 1 and 2 in ALL samples).
  • volatility plots are now interactive and more fun than ever. Check out the q2-longitudinal tutorial for more details!
• q2-emperor
  This release includes a number of changes to the user interface, bug fixes and new features.
  • Bug Fixes
    • Fix a bug where the panel on the right would accidentally hide when it was double-clicked. This is now only triggered by double-clicking the separator between the plot and the tabs.
    • Fix a bug where the axes labels would sometimes be unreadable or too small.
    • Fix a bug where the media buttons in the Animations tab would not always update according to the current state of the animation.
    • Fix a bug where samples without metadata were included in the ordination. This release adds additional checks to validate that all samples included in an ordination are described as part of the sample metadata. Previously Emperor would silently pad the data with a nan placeholder.
  • New Features
    • Add a button on the Axes tab to download the scree plot into an SVG file.
    • Add the ability to download a colorbar when saving the plot as an SVG. Previously Emperor would download a discrete legend of a continuous coloring scheme.
    • Add a button on the top right of the plot to show the context menu options (save settings, load setting, save to SVG, etc).
    • Add a Star geometry to the options in the Shapes tab.
    • Add a warning to the plot when users hide samples. This serves as a reminder that ordination plots heavily depend on all the samples used during the dimensionality reduction (i.e. PCoA).
    • Metadata categories are now alphabetically sorted, this helps to make plots consistent across runs of multiple filterings.
    • Added a new visualization, procrustes-plot, for plotting two procrustes-fitted matrices. See also q2-diversity's procrustes-analysis method to generate the fitted matrices.
  • The animated GIF below showcases some of these features:
    
    
• q2-deblur
  • Upgraded to deblur 1.0.4, which includes bugfixes for sequences with lower-case nucleotide characters.

Happy QIIME-ing!