[Preview] QIIME 2 2017.10 development changelog

Developer Discussion
,
1

QIIME 2 2017.10 has been released!
Please see the official changelog for more.

Original content of post

:exclamation: Important :exclamation:

The following is an early developer preview of the changes expected in 2017.10 This post is a live-document which will be updated throughout our development cycle. Any links will in this topic will be broken until the release is officially published. When we are ready for release, we’ll copy this changelog and create a new post in the Announcements category.

Here’s the highlights of the release (so far):

  • QIIME 2 Framework
    • Added Artifact.validate method for validating an artifact’s data. There are currently two levels of validation available: min and max.
    • Added a new plugin developer API for defining minimal and maximal validation for file and directory formats. This validation API also gives the plugin developer an opportunity to describe why data is invalid, which was previously not possible with the sniff API (binary yes/no answer only). sniff has been deprecated in favor of this new validation API. See q2_types.per_sample_sequences.FastqGzFormat for an example using the new API (i.e. define a _validate_ method instead of sniff). Existing formats with sniff will have the sniffer used for all levels of validation, and will continue to provide binary yes/no answers for backwards-compatibility.
    • Fixed a bug with the repr of Range which made the start and end parameters indistinguishable. Thanks @hunter-cameron!
    • Added a new action type called Pipeline. This action type supports calling other actions and returning both Artifacts and Visualizations. Check out q2-diversity core-metrics for a complete example, but minimally the API is as follows:
      • Plugin objects can now register a pipeline with plugin.pipelines.register_function, it works the same as the other action registrations.
      • Pipelines do not use function annotations and instead receive Artifact objects as input and return Artifact and Visualization objects as output.
      • Pipelines must have ctx as their first parameter, it provides the following API:
        • ctx.get_action(plugin: str, action: str) this returns a “sub-action” that can be called like a normal Artifact API call.
        • ctx.make_artifact(type, view, view_type=None) this has the same behavior as Artifact.import_data. It is wrapped by ctx for pipeline book-keeping.
    • Pipelines are available from the Artifact API via qiime2.plugins.PLUGIN_NAME.pipelines or from qiime2.plugins.PLUGIN_NAME.actions.
    • Pipelines provide nested provenance. This is recorded as a new pipeline action type and an alias-of property. The alias-of has a UUID which will point to a nested DAG which represents the actions that occurred within a pipeline.
    • A long-standing resource leak of file descriptors has been fixed. Previously, if many transformers or format instances were created, a process would eventually run out of file descriptors.
  • q2-quality-control
    • A new plugin containing methods for quality control of sequencing data. In the initial release, this plugin contains one method, exclude_seqs, which aligns query sequences in a FeatureData[Sequences] against a user-defined reference sequence set, and outputs separate files of sequences that hit or do not align to the reference sequences. This method is useful for excluding specific known contaminants, or removing host DNA from sequences.
    • Check out the accompanying q2-quality-control tutorial!
  • docs
    • Added a tutorial describing how to use the new q2-quality-control plugin.
    • Added a section to the q2-longitudinal tutorial showing how to use the new first-differences and first-distances methods.
    • Added exercises to the FMT tutorial focused on longitudinal analyses.
    • Added commands to the Moving Pictures tutorial to generate stats for quality-filter q-score and deblur denoise-16S outputs.
    • Added a section to the Moving Pictures tutorial describing how to generate and interpret alpha rarefaction plots using diversity alpha-rarefaction.
    • Added a section to the Filtering tutorial explaining how to filter feature-tables and representative sequences via taxonomic assignments.
  • q2-taxa
    • Added two new methods, filter-table and filter-seqs, for filtering FeatureTable[Frequency] and FeatureData[Sequence] (respectively) based on taxonomic annotations in a provided FeatureData[Taxonomy] artifact. Applications of these new methods include: removing features with specific annotations (e.g., chloroplast or mitochondrial 16S rRNA from a 16S dataset); retaining only features annotated as being part of a specific taxonomic group (e.g., the bacteroidetes); or removing features that were not able to be assigned taxonomy at sufficient depth for a downstream application (e.g., filtering all features that do not contain at least a phylum-level assignment).
  • q2-diversity
    • core-metrics and core-metrics-phylogenetic are now QIIME 2 pipelines and additionally return Emperor plots for their beta-diversity metrics.
    • alpha-rarefaction now takes a set of metrics. (metric has been renamed to metrics)
    • Added new mantel visualizer, which performs a Mantel test on two distance matrices. This new visualizer replaces beta-correlation and provides a more general interface to the Mantel test. To reproduce the behavior of beta-correlation, use the new metadata distance-matrix command to create a distance matrix from a numeric metadata category, and then compare that distance matrix to another with diversity mantel. Note: a beta-correlation pipeline may be included in the future to essentially chain these two commands together.
  • q2-metadata
    • Added a new method, distance-matrix, for computing a Euclidean distance matrix from a numeric metadata category.
  • q2-longitudinal
    • Added two new methods, first-differences and first-distances, for computing the magnitude of change in a user-defined metric between successive time points. There is also an accompanying section of the q2-longitudinal tutorial showing how to use these methods.
    • Fixed a bug that caused volatility charts to have inconsistently ordered labels.
    • Added an optional parameter to volatility that allows for a user to control the step-size/interval of the plot x-axis labels.
  • q2cli
    • Added a qiime tools validate command for validating an artifact’s data. There is an option to control the level of validation (e.g. minimal vs. maximal validation).
    • Set and List are now supported for input parameters.
  • q2-feature-table
    • rarefy performs better validation of sampling depths (depths must be greater than or equal to 1).
    • A new method subsample has been added which can randomly subsample (without replacement) features or samples from a table.
  • q2-gneiss
    • assign-ids now takes a FeatureTable[Frequency] aligns it to the Phylogeny[Rooted], and returns the newly aligned table.
    • balance-taxonomy now takes a FeatureTable[Composition] instead of FeatureTable[Balance]. The parameter was also renamed to table instead of balance.
    • balance-taxonomy now displays a proportion plot which displays relative abundance of individual taxa across two groups for either side of a balance.
  • q2-sample-classifier
    • Regression outputs now report r-squared (coefficient of determination) in addition to r (correlation coefficient).
    • The color palettes used by classify-samples have been expanded (and renamed to be more intuitive).
    • The “Model Accuracy” section of visualizations now include a link to download the table as a TSV file.
  • q2-types
    • Added more robust validation for many of the demultiplexed sequence formats, which will help users identify issues with their data (e.g. when importing) instead of receiving a binary yes/no answer about the data’s validity.
  • q2-alignment
    • alignment mafft now raises a more meaningful error message when a process runs out of memory. This was fixed by updating the minimum required version of mafft to 7.310.

Happy QIIME-ing! :sun_with_face:

3 Likes