QIIME 2 2023.5 is now available!

[q2d2]

The QIIME 2 2023.5 release is now available! Thanks to everyone involved for their hard work!

As a reminder, our next planned QIIME 2 release is scheduled for August 2023 (QIIME 2 2023.8), but please stay tuned for updates.

Check out the QIIME 2 2023.5 docs for details on installing the latest QIIME 2 release, as well as tutorials and other resources. Get in touch on the QIIME 2 Forum if you run into any issues!

We will follow up here once Docker images have been built and are ready for use

---

Exciting Announcements!

Parsl

QIIME 2 pipelines can now be parallelized via parsl Huge thanks to @Oddant1 for implementing this!

On the CLI, pass the --parallel flag to parallelize a given pipeline using a basic parsl configuration that should work on most non HPC systems.

In the Python API, call .parallel on the pipeline to get the same result (ex. diversity.pipelines.core_metrics.parallel(*args, **kwargs)).

Parsl allows for more detailed configuration for HPC systems. More documentation on how to do this in QIIME 2 may be found in the dev docs linked above.

Pipeline Resumption

QIIME 2 pipelines that fail part way through can now be resumed from their point of failure instead of needing to restart from the beginning. Another huge thanks to @Oddant1 for implementing this!

This behaviour is enabled by default on the CLI. QIIME 2 will create a pool in your default cache (or the cache indicated by the new --use-cache flag on pipelines) that will store all intermediate results from a pipeline that is running and will attempt to reuse the results in this pool should you rerun the pipeline after a failure. The pool will be deleted on pipeline success.

If you want to specify a pool to use (that will not be deleted automatically on pipeline success) provide the --recycle-pool flag followed by a key to be used for the pool in the cache. If you want to opt out of this behavior, pass the --no-recycle flag.

In the Python API, you must with in a pool to be used for pipeline resumption using the usual syntax for withing in a pool.

from qiime2 import Cache

cache = Cache('cache_path')
pool = Cache.create_pool('pool', reuse=True)

with pool:
    diversity.pipelines.core_metrics(*args, **kwargs)

This will do the exact same thing as

qiime diversity core-metrics <inputs and params> --use-cache 'cache_path' --recycle-pool 'pool' <outputs>

This will hopefully make it so that if you hit your wall time or some other transient error while executing a pipeline you do not lose all the progress the pipeline made before it failed.

NOTE: If you change any of your inputs or parameters to a pipeline it may not be possible to reuse all of the intermediate results created by the previous run; however, QIIME 2 will still reuse any results not implicated by the changed arguments.

Output Collections

It is now possible (also huge thanks to @Oddant1 ) to return collections of artifacts as a single output.

On the CLI, output collections will need to be given a directory that does not exist yet (the same as --output-dir). They will create this directory then write all artifacts to it along with a .order file that simply contains the names of all of the artifacts in the collection in order.

In the Python API, a ResultCollection object will be returned that can be accessed in much the same way as a dictionary with the addition of a validate method that will run validate on all artifacts in the collection. .save can be called on ResultCollections to save them to disk using the exact same rules as the CLI. ResultCollection.load may be called to load a directory into a ResultCollection object in the same way as a single artifact may be loaded.

q2-quality-control

@jordenrabasco added three new commands for using decontam in QIIME 2:

• decontam-identify - Supports identifying contaminants based on negative controls using either frequency information (quantitative measures) or feature prevalence in controls.
• decontam-score-viz - Histrogram summary of contaminants with optional normalization of feature-counts.
• decontam-remove - (experimental) Filter feature table by the scores. This may be replaced by feature-table's filter-features in the future.

Provenance replay

@gregcaporaso, @ebolyen, and @colinvwood added provenance-lib to the core distribution! provenance-lib provides access to provenance replay, which enables users to generate new executable scripts from the data provenance stored in all QIIME 2 Results. Want to learn more? Check out @ChrisKeefe's tutorial on the QIIME 2 Forum, and read our new pre-print, Facilitating Bioinformatics Reproducibility. Get started using this today to document your bioinformatics, or to learn from what others did by applying provenance replay to their QIIME 2 results. After activating your QIIME 2 2023.5 conda environment, access this functionality by running replay --help.

---

BREAKING CHANGES

• qiime2

  • There has been a change to artifact provenance to support output collections that makes artifacts created with QIIME 2 2023.5 and onwards not backwards compatible with versions of QIIME 2 older than 2023.5.

• q2cli

  • @colinvwood added the commands qiime tools list-types and qiime tools list-formats that replace the --show-importable-types and --show-importable-formats flags to the qiime tools import command. The new commands list descriptions for each available semantic type or format where available and allow only queries of interest to be listed.

• q2-composition

  • @gregcaporaso addressed an issue in da-barplot where the visualization made
    assumptions about the feature id schema. da-barplot previously split feature ids on semicolons for readability in figures, assuming that the different semicolon-delimited fields were different taxonomic levels. However, there is no guarantee that semicolons in feature ids are always intended to be level delimiters, or that if there are intended to be level delimiters that they would always be semicolons (for example, | is a commonly used delimiter as well). Users must now provide the --p-level-delimiter ';' parameter to achieve the previous behavior.

---

Here are the highlights of the release:

• QIIME 2 Framework

  • @Oddant1 fixed a race condition that could occur when processes were cleaning up on exit

• q2-composition

  • @cherman2 fixed a bug in da-barplot where links to subplots with metadata values that included spaces were broken.
  • @lizgehret fixed a in ancombc that caused undesirable string splitting in the tabulate visualizer when a single reference_level column::value pair was provided. Thanks to @arwqiime for bringing this to our attention!
  • @lizgehret added metadata column type enforcement in ancombc, allowing for CategoricalMetadata columns containing integer values to be treated as discrete groups when included in the formula
  • @lizgehret added a unit test suite to the tabulate visualizer

• q2-feature-table

  • @cherman2 added support for all FeatureTable types to transpose. Now any feature table can be transposed This will address issues like the one @emmlemore detailed in their post on the forum! Thanks @emmlemore

• q2-fmt

  • @cherman2 added a method called feature-peds. This calculates what proportion of subjects engrafted each donor feature.
  • @cherman2 refactored sample-peds to match the implementation of feature-peds.
  • @cherman2 fixed a bug that allowed FeatureTable[Composition] in as an input for sample-peds.
  • @cherman2 added a drop_incomplete_timepoint parameter to sample-peds. This will enable dropping any time points with large numbers of samples missing!
  • @cherman2 added a level delimiter parameter in plot-heatmap that allows users to split taxonomic strings

• q2-longitudinal

  • @lizgehret and @colinvwood fixed a in the feature-volatility visualizer caused by blank values in NumericMetadata columns

• q2-sample-classifier

  • @crusher083 added support for additional base estimators in Adaboost estimators

• q2-taxa

  • @gregcaporaso added support for FeatureTable[PresenceAbsence] as input to barplot. This is useful to support some of our QIIME 2 end-to-end shotgun metagenomics workflows (which are coming soon!).
  • @nicholas_bokulich updated barplot by making FeatureData[Taxonomy] an optional input. For this use-case, feature labels are parsed from the Feature s.

• q2-types

  • @gregcaporaso added the ImmutableMetadata type, which is intended to house QIIME 2 metadata in an artifact. This enables actions to output metadata, which previous wasn't possible since QIIME 2 actions can only output artifacts and visualizations. If an ImmutableMetadata artifact is exported, it will be a plain-old (mutable) metadata file.
  • @Oddant1 added support for numeric sample-ids

---

Documentation Updates

• Cancer Microbiome Intervention Tutorial

  • Fixed typos in the Cancer Microbiome Intervention Tutorial pointed out by Amanda Birmingham. Thanks Amanda!

• User docs

  • @gregcaporaso added a note about the Silva taxonomy classifiers to the Data Resources page of the docs. Specifically, the Silva classifiers and reference files provided on the QIIME 2 webiste include species-level taxonomy. While Silva annotations do include species, Silva does not curate the species-level taxonomy so this information may be unreliable. In a future version of QIIME 2 we will no longer include species-level information in our Silva taxonomy classifiers. This is discussed on the QIIME 2 Forum here (see Species-labels: caveat emptor!) and on GitHub. Thanks to @wasade for bringing this to our attention!

• Developer docs

  • @crusher083 added some new section headers and examples to the Python 3 API for improved readability. Thanks @crusher083!
  • @oddant1 added documentation for Parsl, Pipeline Resumption, and Collections

---

Patch Updates (2023.5.1)

• q2cli
  • Fixed a bug which crashed qiime tools view with any input.
  • Fixed a bug which caused recycle pools to use the cache associated with a output destination, rather than the default/--use-cache provided value.
• qiime2
  • Fix deprecation warning when setting a value in the parallel/parsl config to None.
• q2-cutadapt
  • @vaamb added support for mixed-orientation reads to demux-paired

[jack2017 (Jack)]

Dear all,

Thank you for your hard work and for the updates!!
For me as infrequent user the update frequency is dazzling. Would it be possible to reduce that, or make less frequent "long term support" versions like Ubuntu, or a rolling release version like Debian, and/or indicate what we can use from earlier Qiime2 runs (e.g. databases and other one/few-time time consuming analyses that do not need to be repeated often)? Maybe I missed such options, then my apologies.
Thanks.

[lizgehret]

Hi @jack2017!

Thanks for your interest and questions!

We are planning to start providing long-term support releases either at the end of this year or beginning of next year, but we will continue to do these smaller feature releases ~4x a year.

All of our previous versions are still available for use, and you can check our release changelogs to see what is available and/or any deprecations or breaking changes between releases.

Cheers

[Sam_Degregori]

@jack2017 I used 2021.11 the entire year last year. Before that I used 2019.7 for three years straight. No issues. And the version specific documentation is easily accessible on the qiime2 website

[lizgehret]

NOTE: For anyone who has already installed QIIME 2 2023.5, you may have noticed these warnings pop up in your terminal when running any command:

/home/bt140047/miniconda3/envs/qiime2-2023.5/lib/python3.8/site-packages/umap/distances.py:1086: NumbaDeprecationWarning:
The 'nopython' keyword argument was not supplied to the 'numba.jit' decorator. The implicit default value for this argument is currently False, but it will be changed to True in Numba 0.59.0. See Deprecation Notices — Numba 0.57.0+0.g4fd4e39c6.dirty documentation 1 for details.
@numba.jit()

We have rolled back a couple of our dependencies that caused these (harmless but annoying) warning messages, so please re-install QIIME 2 2023.5 and these warnings will no longer show up.

Cheers

[lizgehret]

NOTE: We have found a bug in q2cli (the command line interface) that produces an error when running qiime tools view. We are currently working on a fix for this and will get a patch release out after the holiday weekend. In the meantime, please use q2view via view.qiime2.org for viewing all visualizations. We apologize for the inconvenience!

[Omar_tak (Omar Gregory Melo Calero)]

I was having that problem with qiime tools view, thanks for the advice. I'll be waiting for the patch.

[colinbrislawn (Colin J Brislawn)]

I have a request along these lines. (Feel free to move this to a new topic!)

Microsoft Azure publishes images with two labels:

• Ubuntu 22.04 ubuntu-latest OR ubuntu-22.04
• macOS 12 Monterey macOS-latest OR macOS-12

It can be nice to target a specific version
It can be nice to target -latest as a CI/CD moving target

I'm requesting qiime2-latest
(Looks like we already have a conda env for qiime2-latest?)

[propolis (John Hasty)]

Check out this Docker image

propolis/qiime2-2023.5-greengenes2-rescript-jupyter

With suggestion for mounting an external drive for data storage.

[ebolyen (Evan Bolyen)]

Hey all,

We just released a patch which fixes the issues with qiime tools view and adds a few other things.

See the addendum to the release notes above for details!

[fenny]

Is there a way to apply the patches to an existing virtual environment or should we build a complete new environment?

[colinvwood (Colin Wood)]

Hello @fenny,

You can apply a patch by updating a package in your environment. For example, to fix the q2cli bug:

conda update q2cli -c https://packages.qiime2.org/qiime2/2023.5/passed/core/

Or to apply all the patches (at the time of writing!):

conda update q2cli qiime2 q2-cutadapt -c https://packages.qiime2.org/qiime2/2023.5/passed/core/

Remember to do this inside of the environment you want to update. Otherwise add the --name <your-env> parameter.

There is always a small chance that this breaks your environment, so don't do this if the risk isn't worth it.