[Preview] QIIME 2 2018.8 development changelog

changelog
preview

(Q2-D2) #1

QIIME 2 2018.8 has been release!
Please see the official changelog for more details.

Original content of post

:exclamation: Important :exclamation:

The following is an early developer preview of the changes expected in 2018.8 This post is a live-document which will be updated throughout our development cycle. Any links will in this topic will be broken until the release is officially published. When we are ready for release, we’ll copy this changelog and create a new post in the Announcements category.


Important Developer Dates (please keep an :eye: on this post, these :calendar: might change):

  • PRs must be submitted by: 08/22
  • PRs must be merged by: 08/27
  • Release Day: 08/29

GitHub Project Board: https://github.com/orgs/qiime2/projects/11


Here’s the highlights of the release:

  • QIIME 2 Framework
    • @ChrisKeefe - When calling docstrings through the Artifact API, arguments now appear in the same order used in the function signature itself, making documentation more intuitive and creating parity with the CLI.
  • docs
    • @turanoo - Updated the tutorial docs to reflect the new flags added to qiime tools (import|export|extract).
    • @John_Chase - Added metadata example to the Artifact API interface page. :balloon:
    • @turanoo - Fixed a typo in the OTU Clustering tutorial. :keyboard:
    • @ChrisKeefe - Plugin and Action pages in the docs now include import statements to make access more transparent for users of the Artifact API, and copy-buttons for convenience. :eyeglasses:
    • @turanoo - Modified the Moving Pictures tutorial to reflect the addition of a new pipeline in q2-phylogeny.
    • @Nicholas_Bokulich - Updated the q2-sample-classifier tutorial to describe various new methods, changes in outputs, and add FLOWCHARTS! :twisted_rightwards_arrows:
    • @Nicholas_Bokulich - Updated the q2-feature-classifier tutorial to describe requirements that reference sequences and taxonomy should be paired.
    • @Nicholas_Bokulich - Updated the diversity analysis flowchart in the overview tutorial to show where feature table subsampling occurs in the core-metrics* pipelines.
    • @Nicholas_Bokulich - Updated the q2-longitudinal tutorial to describe the new actions added to that plugin in this release.
    • @lisa55asil - Updated the gneiss tutorial
    • @colinbrislawn - Added “updating by installing again” instructions to the install docs. :arrow_up:
  • q2view
    • @turanoo - Added more examples to the gallery :part_alternation_mark: :straight_ruler:
  • q2-diversity
    • @ChrisKeefe - In 2017, the beta-correlation pipeline was split into two separate functions distance-matrix and mantel. A new beta-correlation pipeline recreates the original functionality.
    • @HannesHolste - Added an approximation method to PCoA to dramatically improve the computational performance for large distance matrices (tens of thousands of samples). A new parameter in qiime diversity pcoa lets users specify the number of dimensions you want to compute, by default all dimensions are computed (as in previous versions of QIIME).
    • @Yoshiki - Added qiime diversity pcoa-biplot to compute PCoA biplots that project a feature table into an ordination space.
    • @thermokarst - Fixed a problem in the alpha-rarefaction visualization that prevented scrolling on the legend.
    • @mortonjt - Added aitchison distance as a new metric to beta. This change adds an optional pseudocount parameter to the method.
    • @yoshiki - Added permdisp as a method for beta-group-significance.
    • @jakereps - Improved error message when metadata is empty as a result of filtering.
  • q2cli
    • Tab completion is now automatically available after activating your conda environment in bash and ZSH.
    • :exclamation: Breaking changes in q2cli :exclamation:
      As part of our efforts to keep the builtin commands of q2cli consistent we have updated the subcommands available under the tools category. The commands impacted are:

      qiime tools import

      old:

      qiime tools import \
       --type 'SampleData[PairedEndSequencesWithQuality]' \
       --input-path pe-64-manifest \
       --output-path paired-end-demux.qza \
       --source-format PairedEndFastqManifestPhred64
      

      new:

      Please note that --source-format has been renamed to --input-format.
      qiime tools import \
        --type 'SampleData[PairedEndSequencesWithQuality]' \
        --input-path pe-64-manifest \
        --output-path paired-end-demux.qza \
        --input-format PairedEndFastqManifestPhred6
      

      qiime tools export

      old:

         qiime tools export \
           unrooted-tree.qza \
           --output-dir exported-tree
      

      new:

      Please note the positional INPUT argument has been replaced with an
      --input-path option, and the --output-dir option has been renamed to
      --output-path.
      qiime tools export \
        --input-path unrooted-tree.qza \
        --output-path exported-tree
      

      qiime tools extract

      old:

      qiime tools extract \
        feature-table.qza \
        --output-dir extracted-feature-table
      

      new:

      Please note the positional INPUT argument has been replaced with an
      --input-path option, and the --output-dir option has been renamed to
      --output-path.
      qiime tools extract \
        --input-path feature-table.qza
        --output-path extracted-feature-table
      
      Thanks @turanoo!
    • @turanoo - qiime tools export now supports exporting a file or a directory into another format. :hammer_and_wrench:
    • @turanoo - Built-ins now support printing a success message after completion of a command. :pretzel: :green_apple:
  • q2-quality-control
    • @Nicholas_Bokulich - fixed a bug where the TAR and TDR values were transposed in evaluate-composition. See this post for more details.
  • q2-sample-classifier
    • @Nicholas_Bokulich - Added two new visualizers — scatterplot and confusion-matrix — to plot accuracy results from regressors and classifiers, respectively.
    • @Nicholas_Bokulich - Random splitting into training/test data is now performed on the metadata, not on the feature table. Effect on users: faster, lower memory, more efficient splitting!
    • @Nicholas_Bokulich - classify-samples and regress-samples are now :qiime2: Pipelines! Effect on users: multiple different outputs are now produced, including trained classifiers for re-use!
    • @Nicholas_Bokulich - :exclamation: API Breaking Changes :exclamation: predict has been split into two methods: predict-classification and predict-regression; and the SampleEstimator semantic type now must be either SampleEstimator[Classifier] or SampleEstimator[Regressor] formats. This makes up/downstream handling more transparent for users.
    • @Nicholas_Bokulich - New visualizers Summarize for extracting parameter settings and RFE results from trained SampleEstimators. :chart_with_upwards_trend:
    • :exclamation: API Breaking Changes :exclamation: @Nicholas_Bokulich - maturity-index is no longer an action in q2-sample-classifier, it is now a pipeline in the q2-longitudinal plugin! (see below for more details).
    • A new pipeline, classify-samples-from-dist, has been added. This uses a kNN classifier to classify samples on a precomputed distance matrix, rather than from a feature table. Thanks Justin Kuczynski and Alena Kuczynski for creating this action!
  • q2-types
    • @HannesHolste and @Yoshiki - bumped scikit-bio to the latest version, 0.5.4. :accept:
    • @Pauline_Trinh - Added the ability for AlphaDiversityFormat to contain additional columns as metadata.
    • @gregcaporaso - Updated the alpha diversity format transformer to include metadata.
    • @cduvallet - Added PercentileNormalized as a varient for FeatureTable. :diamond_shape_with_a_dot_inside:
  • q2-phylogeny
    • @SoilRotifer - Added new tree-building methods based on IQ-TREE! :tada:
    • @SoilRotifer - Added access to SSE3 / AVX2 versions of RAxML via --p-raxml-version. These CPU-optimized versions of RAxML will reduce run time. :fast_forward:
    • @turanoo - Added a new pipeline to build phylogenetic trees using mafft and fasttree. :evergreen_tree: :t_rex:
  • q2-alignment
    • @turanoo - Updated q2-alignment mafft unit tests to remove issue caused by a newer MAFFT version. :bug:
    • @turanoo - Added the parttree parameter to mafft allowing alignments of greater than 1 million sequences in the QIIME 2 plugin.
    • @turanoo - Fixed a regression in the n-threads parameter that prevented MAFFT from using all cores.
  • q2-gneiss
    • :exclamation: API Breaking Changes :exclamation:
      • ilr-transform has been replaced by ilr-hierarchical and ilr-phylogenetic. Thanks @qiyunzhu!
      • This plugin no longer makes use of the FeatureTable[Composition] type, instead pseudocounts have been added as parameters to the following methods:
        • ilr-hierarchical
        • ilr-phylogenetic
        • balance-taxonomy
        • dendrogram-heatmap
        • correlation-clusterin
      • Additionally the pseudocount behavior now defaults to replacement of 0 values with 0.5. This can be adjusted with the pseudocount parameter. This value is not added to non-zero observations (like in composition add-pseudocount).
  • q2-longitudinal
    • Added a new pipeline, feature-volatility for identifying features that are predictive of sampling time (or other continuous metadata) — see the tutorial for details! Thank you @thermokarst for building the plot-feature-volatility visualizer, the crowning gem of this pipeline. :crown:
    • @Nicholas_Bokulich - individual_id_column is now an optional parameter for the visualizer volatility! This means that volatility can be used for visualizing changes in metrics across any type of continuous data, not only longitudinal experiments where the same subjects were sampled repeatedly! :spaghetti:
    • @Nicholas_Bokulich - updated state_column parameter descriptions for several methods to specify how that metadata column is handled by paired-sample vs. longitudinal methods.
    • :exclamation: API Breaking Changes :exclamation: @Nicholas_Bokulich - maturity-index, formerly in q2-sample-classifier, is now a pipeline in the q2-longitudinal plugin! This is because it is almost exclusively a longitudinal method (and so it was designed to be). Additionally, it is now a pipeline and the outputs and tests performed internally have been modified (and improved) extensively.
  • Virtual Machines
    • @johnbradley - More versatile configuration of Docker image! :ping_pong:
  • q2-vsearch
  • q2-composition
  • q2-emperor - by @Yoshiki and @Antgonza
    • This plugin is now using the latest version of emperor (1.0.0beta17).
    • For very large datasets responsive plots can now be rendered live in the browser. Checkout this example with ~114K samples (courtesy of @wasade).
    • Fixes a glitch that would happen after re-centering the camera and panning around the scene.
    • Adds a new visualizer (qiime emperor biplot) to plot the results from qiime diversity pcoa-biplot. This method optionally expects feature metadata, this will allow you to modify visual aspects of the Biplot arrows. For an example see this animated GIF:

Happy QIIME-ing! :sun_with_face:


Emperor Plot Animated Download
Choosing a seed number when using RAxML for tree building
(Matthew Ryan Dillon) #2

Just a quick reminder, if you are interested in potentially getting changes in for the upcoming 2018.8 release, today is the last day to submit a PR. Thanks!


(Matthew Ryan Dillon) #3

Hey just a head’s up - it looks like we might postpone the release by a day or two so that we can work through some backlog here. Some exciting changes are afoot… :foot: Stay tuned!

:qiime2: :t_rex:


(Matthew Ryan Dillon) #8

(Matthew Ryan Dillon) #9