QIIME 2 2017.12 release is now live!

[q2d2]

The QIIME 2 2017.12 release is now live! There are lots of exciting new changes packed into this release that are described below.

QIIME 2 2017.12 is the last QIIME 2 release before we end our support of QIIME 1. As a reminder, QIIME 2 will succeed QIIME 1 on January 1, 2018. QIIME 1 will no longer be officially supported at that time, as development and support effort for QIIME will be focused entirely on QIIME 2. To prepare for this transition, we have been working toward "feature parity" with QIIME 1 over the last year. We have made a huge amount of progress toward this goal, but there may still be some things that you want to do with QIIME 1 that are not possible with QIIME 2. If you identify things that you did with QIIME 1 but can't do with QIIME 2, please get in touch about them on the QIIME 2 forum. We'll be happy to help with work-arounds as we wrap up loose ends.

Because we still anticipate a lot of new features going into QIIME 2 over the next few months, we will continue to have monthly releases for at least the first quarter of 2018. We highly recommend staying up-to-date with the latest releases to get access to new features, bug fixes, documentation, etc. We will only provide user support for the latest monthly release of QIIME 2 as these monthly releases continue. Later in 2018, we expect to develop and implement a strategy for "long-term support" releases that we will support for longer periods of time.

Check out the QIIME 2 2017.12 docs for details on installing the latest QIIME 2 release, as well as tutorials and other resources. Get in touch on the QIIME 2 forum if you run into any issues!

We will provide official conda builds on release day for native QIIME 2 installations. Virtual machine builds such as AWS and Docker will be released within the following week. We will post a follow-up announcement here when those builds become available.

---

Here's the highlights of the release:

• q2-cutadapt
  • This new plugin wraps cutadapt!
  • demux-single supports demultiplexing single-end reads, where the barcodes are still inline in the sequence.
  • demux-paired supports demultiplexing paired-end reads, where the barcodes are still inline and occur on the 5' end of the forward read.
  • trim-single & trim-paired support finding and removing adapters in demultiplexed reads.
• q2-dada2
  • Upgraded dada2 to the latest, version 1.6! This version brings many performance enhancements, so if you previously experienced slow q2-dada2 performance, check out this new version and see for yourself!
  • Added addition statistics to stdout log (--verbose for users of q2cli) that indicates how many reads were filtered, denoised, merged, and non-chimeric.
• q2-feature-classifier
  • In classify-sklearn, the reads-per-batch parameter is now automatically scaled based on the number of query sequences and n_jobs, making parallel jobs with this action much more efficient (without requiring user input). Autoscaling is set by default (when reads-per-batch == 0), but this parameter can still be defined by the user (e.g., to reduce batch size to address memory constraints).
• q2-feature-table
  • Fixed a bug that accidentally allowed users to specify a negative rarefaction depth in the summarize visualization. Thanks @patthehat033!
  • Fixed a performance issue in the summarize visualization - tables with lots of features should now render much more quickly (orders of magnitude faster in our tests).
  • merge now accepts arbitrarily many tables provided via the new tables input (table1 and table2 have been removed as input).
  • merge-seq-data has been renamed to merge-seqs and now accepts arbitrarily many FeatureData[Sequence] artifacts via the new data input (data1 and data2 have been removed as input).
  • merge-taxa-data has been renamed to merge-taxa and now accepts arbitrarily many FeatureData[Taxonomy] artifacts via the new data input (data1 and data2 have been removed as input).
  • filter-seqs can now take a FeatureTable[Frequency] artifact as an optional input, in which case it filters features that are not present in the FeatureTable[Frequency] artifact (or, if exclude-ids is passed as well, it filters features that are present in the FeatureTable[Frequency] artifact).
• QIIME 2 Framework
  • Variadic input is now supported in the framework for Artifact inputs. Plugin developers can use List[...] and Set[...] with a semantic type to indicate that they would like to receive 1 or more artifacts with that type.
    • Set is unordered but unique
    • List is ordered but not necessarily unique
    • View annotations (function annotations) for a method are unchanged. So for a registration of List[FeatureTable[Frequency]] the view annotation would still be biom.Table (or whatever you prefer). In the future we would like this to be typing.List[biom.Table] but there are some technical challenges preventing that at this time.
• docs
  • The minimum disk space requirements recommendation for the VirtualBox images have been revised --- recent versions of QIIME 2 are larger than the versions that existed at the time.
  • Added a step to the VirtualBox install guide to demonstrate how to set up shared copy-and-paste between the host and guest machines.
  • Updated the VirtualBox install guide to make users aware of VirtualBox version requirements.
• q2cli
  • Help text has been rearranged, this should make the types and descriptions more readable.
  • Variadic input is now supported for input artifacts. The help text will show MULTIPLE ARTIFACT PATH or MULTIPLE <TYPE> to indicate that the parameter can be provided multiple times.
• q2-quality-filter
  • Removed the visualize-stats visualizer in favor of viewing QualityFilterStats as Metadata, which means users can now visualize these stats with the generic metadata tabulate visualizer!
  • Added support for SampleData[PairedEndSequencesWithQuality] as an additional input type to q-score - reverse reads are ignored and not included in the output (so the output is always SampleData[SequencesWithQuality]). This is similar to how dada2 denoise-single behaves - if you provide paired-end reads it only operates on the forward reads.
• q2-emperor
  • Bug Fixes
    • Fix a problem where cubes would be rendered in a slightly different color from all the other shapes.
    • Fix a bug where plots would rotate slightly when switching tabs, or changing a visible attribute.
  • New Features
    • Add the ability to copy sample names to the clipboard. This can be done in one of two ways: by clicking on a sample and then clicking on the bottom left sample name legend, or by double-clicking a sample.
    • Add the ability to resize the plot. This can be done by clicking and dragging the line between the plot and the tabs space. Double clicking will toggle the visibility of the menu. See this animated gif:
    • 
    • New shapes are now available in the Shapes tab.
    • Add the ability to create animated PCoA plots via the Animations tab. This features is useful to visualize how multiple samples change over a gradient. This tutorial explains how to use this functionality, and this this paper explains the motivation for this feature.
• q2-demux
  • A bug in summarize has been fixed in the interactive tab where the shortest global read would be reported in the legend instead of the shortest sampled read. The plot was not impacted by this bug.
• q2-longitudinal
  • Updated plugin to cite the preprint for the upcoming q2-longitudinal paper.
• q2-diversity
  • Fixed a bug which would crash core-metrics-phylogenetic when n-jobs was set to any value other than 1.
  • beta-rarefaction now supports jackknifed PCoA plots with Emperor, and sample clustering (with support calculation) via neighbor joining and UPGMA. Two new required parameters were added to that end: metadata (same as always) and clustering-method (accepts "nj" or "upgma").
• q2-vsearch
  • Added two new methods uchime-ref and uchime-denovo, which support the Uchime chimera checking algorithm as implemented in vsearch. A community tutorial has also been added which illustrates how to use this new functionality.

Happy QIIME-ing!

[ebolyen (Evan Bolyen)]

Hi all!

We discovered that conda has just updated to 4.4.0 within the last hour or so of building this release.
This latest conda release has a lot of really exciting features, unfortunately it also looks like there’s a couple of issues that need to be fixed before it will work with our typical installation instructions.

With that in mind, instead of running the typical conda update conda that we always recommend before installing a new version of QIIME 2, we’re recommending you run the following:

conda install conda=4.3

We expect this will be resolved in the near future, but since the holiday season is nearly upon us, we won’t be able to explore this too deeply until January.

Sorry for the inconvenience and Happy Holidays !

[q2d2]

The QIIME 2 Virtual Machines are now available! We have QIIME 2 2017.12 available on AWS, Docker, and VirtualBox, please see the docs for more information. Thanks!

[Micro_Biologist (Jono Warren)]

Erm, I think the 2017.11 version was included in the zip not the 2017.12 version (or its just labelled as so)

EDIT: yes I checked and it is the 2017.11 version

[thermokarst]

Thanks for the notice @Micro_Biologist! I accidentally zipped the wrong file, sorry! I just rezipped and reuploaded - hopefully I got it right this time!

[thermokarst]

A quick note regarding q2-cutadapt — we plan on writing a Community Tutorial to illustrate how to use this plugin (there are two new import formats, plus methods for demultiplexing multiplexed reads and filtering demultiplexed reads). There is a skeleton of that tutorial available now, but stay tuned for something more detailed in the next few weeks!

[thermokarst]

Release Update

If you have encountered an error message when running dada2 denoise-single or dada2 denoise-paired in QIIME 2 2017.12 that contains the following:

 *** caught segfault ***
address 0x8, cause 'memory not mapped'

then this update is for you! We have just released a patched version of QIIME 2 2017.12 that attempts to fix this problem. If you would like to install this update, first remove your existing 2017.12 environment:

$ conda env remove -n qiime2-2017.12

Then, follow the native installation guide to reinstall - this should fetch the patched release automatically.

We will try and get rebuilt virtual machines out some time next week.

As always, feel free to post any questions! Thanks for your support and happy QIIMEing!

[thermokarst]

Release Update

If you have encountered an error message when attempting to install this release that contains the following:

“CondaHTTPError: HTTP 404 NOT FOUND for url: https://conda.anaconda.org/conda-forge/noarch/wheel-0.30.0-py_1.tar.bz2”

then this update is for you! This error is because the folks at conda-forge replaced the wheel package with a newer build, and took down the version that the QIIME 2 2017.12 environment file was referencing. We have updated this file to point to the latest build of wheel, so if you were experiencing this error earlier, you should be able to re-attempt to install with better success . Good luck and let us know if you run into any issues!

[uria (Uria Mor)]

https://anaconda.org/conda-forge/pysocks/1.6.8/download/noarch/pysocks-1.6.8-py_0.tar.bz2 is broken (404 not found).
I suggest changing it to https://anaconda.org/conda-forge/pysocks/1.6.7/download/linux-64/pysocks-1.6.7-py36_0.tar.bz2

[Miguelcs]

I just installed Qiime2 natively on my mac. I found several troublesome links in file with the address to the packages (https://data.qiime2.org/distro/core/qiime2-2017.12-conda-osx-64.txt), which caused the installation to abort. I was able to solve this issue by creating a new environment and installing all packages by batches (pasting the addresses in the terminal conda install --name qiime2017_12 package_address). I was able in this way to determine which links were causing problems and address specific issues manually.

Several of the fixed links:
https://conda.anaconda.org/conda-forge/osx-64/jupyter_core-4.3.0-py36_0.tar.bz2
https://conda.anaconda.org/conda-forge/osx-64/pysocks-1.6.7-py36_0.tar.bz2
https://conda.anaconda.org/conda-forge/osx-64/jsonschema-2.6.0-py36_0.tar.bz2
https://conda.anaconda.org/conda-forge/osx-64/r-rcppparallel-4.3.20-r3.4.1_0.tar.bz2

[thermokarst]

Thanks @uria & @Miguelcs - conda-forge removed a handful of packages, leaving the environment files in a bad place. We just released updated versions of the environment files, so you should be able to install with more success . In the meantime, we are working hard on building up a new install solution that will circumvent these problems when conda-forge removes packages from their conda channel. Stay tuned, and thanks for reporting in!

[thermokarst]

Okay, as I was writing that message, conda-forge removed even more packages, breaking the environment file, again. We are working hard to replace these files with a more robust solution, please stay tuned.

[thermokarst]

This is the QIIME 2 dev team right now:

[ebolyen (Evan Bolyen)]

I think we have fixed this Once and For All ™. Thanks everyone!

[q2d2]

The QIIME 2 Virtual Machines are now available! We have QIIME 2 2017.12 available on AWS, Docker, and VirtualBox, please see the docs for more information. Thanks!