QIIME 2 2022.8 is now available!

[q2d2]

The QIIME 2 2022.8 release is now available! Thanks to everyone involved for their hard work!

As a reminder, our next planned QIIME 2 release is scheduled for November 2022 (QIIME 2 2022.11), but please stay tuned for updates.

Check out the QIIME 2 2022.8 docs for details on installing the latest QIIME 2 release, as well as tutorials and other resources. Get in touch on the QIIME 2 Forum if you run into any issues!

Virtual machine builds will be available sometime next week - watch this topic thread for an update!

---

Exciting Announcements!

• Community Distribution!

  • NEW! In our continued efforts to support plugin developers and the QIIMEmunity (pun intended), we have added a new distribution to our infrastructure that includes all active, currently maintained community plugins This allows for all of these plugins to undergo regular integration tests so that we can quickly triage and debug any issues that may arise due to dependency conflicts or other unforeseen !

  • Please note that we are still beta testing this distribution, so there is no formal release but we wanted to share the exciting news with all of you To install, simply run the following command:

  conda create -n qiime2-community-2022.8-BETA \
    -c conda-forge -c bioconda \
    -c https://packages.qiime2.org/qiime2/2022.8/passed/community/ \
    qiime2-community
  

  • If you are a developer and you would like to see your plugin added to the community distribution, please get in touch with us! This is our goal for this project - to have a distribution with all of our community plugins available all in one place

• q2galaxy

  • NEW! We're very excited to announce that QIIME 2 is now officially supported in Galaxy release 22.05, which is now available! What this means is that you can upload QIIME 2 artifacts/visualizations and Galaxy will recognize our file format This does not include any QIIME 2 actions, which is the next exciting announcement.

  • We now have a Galaxy toolshed repository for QIIME 2 actions which can be used by any Galaxy instance running release 22.05 with Docker support. This makes it easy to install the latest QIIME 2 release directly into your Galaxy server straight from the official Galaxy Toolshed

  • Please note that the toolshed will be made available along with the virtual machines on our usual post-release timeline of 1-2 weeks after release day

BREAKING CHANGES

• q2-feature-classifier

  @Nicholas_Bokulich completed a large refactor for improved readability and efficiency that includes the following new features:

  1. Separated blast + vsearch aligners from the consensus classification step. Now we have thin wrappers for database searching, as well as a separate action for generating consensus annotations from BLAST6Format search results
  2. classify-consensus-* methods are now pipelines, linking together the separate actions mentioned above
  3. The Search results are output as part of the classify-consensus-* pipelines for convenient post-mortem
  4. New + improved unit tests

• q2studio

  • DEPRECIATION NOTICE: q2studio will be removed from our user documentation and will no longer be maintained after the 2022.11 release. It is being phased out and will be replaced with a fully supported release of q2galaxy!

---

Here are the highlights of the release:

• QIIME 2 Framework

  • @ebolyen added support for all INSDC missing values to be processed as NULL within QIIME 2 Metadata
  • @misialq added support for typing.Union in view types by selecting the first available transformer. This adds the ability for transformers to solve to a typing type like Union which will make paired-end visualizations simpler
  • @ebolyen fixed a long-standing annoyance for plugin developers by allowing for dictionaries to be registered outputs! Additionally, Evan added sorting for sets provided to Choices so that interfaces which cache the types will maintain a consistent ordering.

• q2-cutadapt

  • @hyphaltip added multi-core support to demux-paired with the --p-cores parameter This speeds up demux dramatically based on the number of cores selected!

• q2-dada2

  • @jordanrabasco submitted a huge refactor in q2-dada2 that combines all of the R scripts into one file for single, paired end, and ccs reads - for easier readability and reduced code duplication!
  • @leasi added the allowOneOff argument in the runBimeraDenovo() method, which enables users to allow for one mismatch or indel in bimera detection when set to True

• q2-demux

  • @lizgehret and @Keegan-Evans added more concise error handling for metadata file handling when running qiime demux emp-single to include the specific column where any metadata.tsv (or sequences.qza) errors occur

• q2-diversity

  • @yoshiki added support for sub-sampling with replacement in phylogenetic-core-metrics, which is useful for metagenomic studies where there's frequently over 1M counts per sample
  • @ChrisKeefe updated the Halko citation and added a DOI field for a number of other citations

• q2-diversity-lib

  • @ChrisKeefe added some Usage API examples on Faith's Phylogenetic Diversity, Observed Features, Pielou's Evenness, and Shannon's Entropy
  • @wasade added new citations for in-press UniFrac and optimized Faith's PD papers
  • @wasade made an improvement to how alpha diversity metrics are calculated. These calculations are now made either directly from sparse data, or operate on partitions to minimize memory use on dense conversions.
  • @thermokarst added new unifrac OMP support, this resolves the slowness/crashing that was experienced during the FAES workshop.

• q2-FMT (Available in the community distribution as a beta release)

  • NEW! Developed by @ebolyen and @lizgehret and currently maintained by @cherman2, this plugin provides tools for analyzing the efficacy of fecal microbiota transplants The current methods that are supported are as follows:

    1. group_timepoints: Produce timepoint and reference distributions for use in statistical calculations, based on specified group data over discrete timepoints. Both alpha and beta diversity data are supported for this method.
    2. engraftment pipeline: Applies group timepoints to Fecal Microbiota Transplant data and then applies statistical tests (Wilcoxon signed rank test or Mann–Whitney U test) on alpha or beta diversity metrics to assess engraftment success.

• q2-longitudinal

  • @lizgehret fixed a in pairwise-distances and pairwise-distances that raised an error if samples in the metadata but not the data were missing group data. Samples not present in the data (e.g., an alpha diversity vector or distance matrix), they wouldn't be included in the analysis, so missing group data is irrelevant for those samples.

• q2-metadata

  • @gregcaporaso added an action that supports the generation of shuffled metadata groups. This allows users to create one or more sample metadata columns by shuffling the values in an actual metadata column. The shuffled metadata columns can be useful for assessing whether an analysis of the corresponding actual metadata column is producing an interesting signal or should be considered noise. For example, if a differential abundance test identifies 10 features of interest with an actual metadata column, and 5-15 features of interest with a shuffled variant of that column, the actual test results should likely be considered unreliable.

• q2-phylogeny

  • @SoilRotifer did some maintenance that restored some previously deprecated methods.

• q2-stats (Available in the community distribution as a beta release)

  • NEW! Currently in development by @ebolyen and @lizgehret this plugin provides statistical tools, methods, and visualizers that can be used on a variety of different data.

• q2-types

  • @Nicholas_Bokulich added a transformer that allows for BLAST6Format data to be transformed into QIIME 2 Metadata

---

Documentation:

• @thermokarst published new developer tutorials on the following:

  • How to publish a plugin using Library
  • An overview of our packaging infrastructure

• @ChrisKeefe published a new developer tutorial on the usage API and how to write basic usage examples

• @ChrisKeefe published a new user tutorial on the Artifact API

[Teonexone]

Thank you!!
Did you take in consideration the option for systems having conda versions with native ARM64 support (M1 Macs)?

[ebolyen (Evan Bolyen)]

Hey @Teonexone,

Unfortunately M1 remains a huge hardware problem for everyone (ourselves included). OS X licensing is also quite restrictive, which additionally limits how much you could use any given piece of hardware for these purposes (I believe the best you can secure is 3 VMs on a single host):

Bioconda (who is upstream of us) has some discussion about this here and here.

I think @colinbrislawn may have a more up-to-date view of the landscape than I however.

Until then, I'm afraid emulation through Rosetta remains the best option. I know that's something a lot of users struggle with, so if anyone has a nice writeup, I think a lot of people would find it valuable.

[rrichter (Alex Richter)]

With regards to the Rosetta emulation, it's reasonably straightforward in anaconda to create an x86 environment. As adapted from a Towards Data Science post

CONDA_SUBDIR=osx-64 conda env create -n qiime2-2022.8 --file qiime2-2022.8-py38-osx-conda.yml
conda activate qiime2-2022.8
conda config --env --set subdir osx-64

has worked for me (for 2022.2, but the principle holds).

[colinbrislawn (Colin J Brislawn)]

@ebolyen

I think @colinbrislawn may have a more up-to-date view of the landscape than I however.

Nope, lol. I'm still running Qiime2 on x86 and using my M1 mac for downstream stuff. The only team successfully shipping osx-arm64 is conda-forge and they are commercially supported.

osx-arm64 support is on the roadmap for GitHub actions! More discussion here.|

[q2d2]

We have finished uploading our tools to the Galaxy Toolshed!

You can install the core distribution by searching for suite_qiime2_core, or you can install each plugin suite individually. It is also possible to install each action as it's own tool. We will be uploading these tools using our friendly helper @q2d2, so if you see them on the toolshed, that's us!

Note that installing the full core distribution will take some time, so please be patient. Additionally, these tools require your Galaxy deployment to support docker containers. This is a first step for getting QIIME 2 available. In the future, we may support conda as well (through the bioconda channel).

[q2d2]

Hello everyone!

We have just finished up our patch release for 2022.8. This includes a couple of important updates and fixes to the QIIME 2 Framework, and a bug that we found within q2-feature-table. Resolved issues are linked below for anyone interested:

QIIME 2 Framework

qiime2 import error due to multiple users needing to access /tmp/qiime2 on Ubuntu server

[BACKPORT] BUG: fix Jupyter reprs not loading because of Cache (#637)

q2-feature-table

Empty cells in numeric metadata column causing error with viewing feature-table.qzv

An updated Docker VM is also currently available for this patch release.

Happy ing!

[q2d2]

Hey everyone!

We've just updated our release with a fix for OS X which would produce errors such as:

ValueError: Path: 'None' already exists and is not a cache

This release has been a trial-by-fire for a new feature that will be officially announced in November, called "Artifact Cache". We're sorry for the bumpy ride, but appreciate everyone's patience as we iron out these issues.

Briefly, this occured because we now create a single directory structure in your $TMPDIR instead of littering it with lots of small directories. However, to make sure things worked right, we would include a VERSION file at the root to double check that the directory is what we expect.

Unfortunately, OS X deletes temporary files that have not been accessed in at least 3 days, which meant if you closed your laptop for a weekend, there would be a good chance OS X had deleted this file and QIIME 2 would choose not to proceed with loading anything out of an abundance of concern for things no longer matching. (We now allow this mismatch for the temporary cache and just recreate it, you should see a user-warning whenever we do this.)

Some of you may recognize this unfortunate behavior if you've ever run a 3+ day long job on OS X . As QIIME 2 generally could not, depending on the specific action, rely on any of it's data to still exist, you would see some really disappointing error about a file not existing.

The really great news, is we are pretty sure we've fixed this long-standing bug as well. We now spawn a very light-weight thread which accesses each file in the new cache every 6 hours to defeat this behavior of OS X (or potentially a particularly aggressive sysadmin's scripts on linux). Once your action is done, the thread ends and OS X is free to destroy anything (although we've already cleaned up at that point).

[Brett_Loman]

@rrichter

CONDA_SUBDIR=osx-64 conda env create -n qiime2-2022.8 --file qiime2-2022.8-py38-osx-conda.yml
conda activate qiime2-2022.8
conda config --env --set subdir osx-64

Responding to this simply to state that this is a "hidden" problem, and to say this worked for me with my M1 chip laptop after trying the standard installation instructions and thinking I was missing something for about an hour. Would HIGHLY recommend putting this as an option (among the Mac, linux, and windows options) on the native install page!!! Thanks for the continued support.

[lizgehret]

Thanks so much for this information @Brett_Loman, we'll get this added to our documentation!