[Preview] QIIME 2 2024.10 development changelog

Developer Discussion
,
1

QIIME 2 2024.10 has been released!
Please see the official changelog for more details.

Original content of post

exclamation: Important :exclamation:

The following is an early developer preview of the changes expected in 2024.10

This post is a live-document which will be updated throughout our development cycle. Any links will in this topic will be broken until the release is officially published. When we are ready for release, we’ll copy this changelog and create a new post in the Announcements category.

Developer Project Board:

QIIME 2 2024.10 Project on Github

:rotating_light: Important Announcements :rotating_light:

Interface Changes coming in QIIME 2 2025.4 :boom:

  • @jordenrabasco developed some big changes to the q2-dada2 error model output which will be coming in the 2025.4 release. These include:
    • q2-dada2 denoise- commands output a collection[DADA2stats] rather than a DADA2STATS object Note: this will be a breaking change
    • New q2-dada2 action stats_viz which will visualize all DADA2stats in the collection[DADA2stats] (denoised stats, and error model stats) as a singular visualization with different tabs for each DADA2STATs object. :sparkler:
  • Expect changes to q2-diversity's alpha-group-significance and beta-group-significance as they will be updated to pipelines using q2-stats. Specifically, these visualizers will soon return multiple results which will include statistical outputs and visualizations. Additionally these will require selecting the desired columns to compare, instead of using all available.

Python Versioning :snake:

  • Moving forward, with each release we will announce what the subsequent Python version will be for the upcoming (n+1) release. For 2025.4, we will stay with python 3.10 (as 3.11 builds are not yet available on Bioconda for certain dependencies) - but we are hoping to bump to >=3.11 for 2025.10 (depending on build availability at the start of the release cycle).
    • Target for 2025.4: Python 3.10
    • Target for 2025.10: Python >=3.11 (TBD)

:bangbang: Breaking Changes :bangbang:

QIIME 2 Framework

  • @Oddant1 disallowed registering List[<Type>] as an output of a plugin action, as this was never intended; use Collection[<Type>] instead.
  • @Oddant1 corrected action-executor mappings to use plugin-name + action-name in the configuration. See below:
    # previously the toml group was only [parsl.executor_mapping]
[parsl.executor_mapping.plugin_name]
action_name = "executor_name"

Here are the highlights of the release! :sparkler:

qiime2.org updates :house:

  • The QIIME 2 homepage was updated - thanks to the Curvenote team for helping with the refactor! This page is now easier for us to keep up-to-date and to integrate dynamic content, like the Forum Announcements feed.

QIIME 2 View Updates :mag:

We are now fully transitioning to the new version of q2view. For the time being, the old version is still being hosted at old-view.qiime2.org, but we are no longer supporting it or linking to it from view.qiime2.org :chains:.

  • If you have created cool new visualizations in QIIME 2 it is now easy to add them to the q2view gallery and have them show up on q2view in real time! Follow the instructions in the README of the q2view-gallery repo; if we merge your PR, your visualization will be added to the gallery :framed_picture:.
  • It is now possible to view links to QIIME 2 results hosted on Zenodo. Simply choose to view a "file on Zenodo" and follow the instructions :bar_chart:.

QIIME 2 Library Updates :books:

@Oddant1 completely rewrote QIIME 2 Library to make it easier to share your plugins and keep them up-to-date and installable :boom:. The new version of library is currently in an alpha state, and the old version is in the process of being deprecated and removed.

  • If you have a plugin you would like to add to the QIIME 2 Library follow the instructions in the README of the library-plugins repo; if we merge your PR, your plugin will be added to the library :ladder:.

QIIME 2 Forum Updates :page_facing_up:

Distribution Updates :package:

Framework Updates :deciduous_tree:

  • @Oddant1 made it so cache validation will now return false on things that aren't directories instead of erroring :file_cabinet:
  • @Oddant1 made parallel pipelines tolerant of mixed artifacts and proxies in a collection :abcd: :1234:
  • @Oddant1 made it so actions will now show up with their real names when run in parallel :railway_track:
  • @Oddant1 made the highthroughput executor the default instead of threadpool :hotsprings:
  • @Oddant1 made the QIIME 2 duplicate util detect when os.link is unsupported and copy instead :scroll:
  • @Oddant1 made it so Collection[Visualizaion] usage variables are properly typed.
  • @lizgehret fixed a :lady_beetle: in Artifact.import_data() where checksums weren't being stored when the view_type was a sub-class of FormatBase :abacus:
  • @yoshiki updated the jupyter server extension command (formerly jupyter serverextension) :ringer_planet:
  • @Oddant1 improved support for collections of visualizations and collections in general in the usage API :closed_book:
  • @Oddant1 allowed nested pipelines to parallelize properly :plunger:
  • @Oddant1 fixed a :bug: that was causing parallelized pipelines to fail trying to shutil.copytree a directory that did not exist :evergreen_tree:
  • @Oddant1 fixed a :beetle: that caused multiple of the same pipeline run simultaneously to fail when the first pipeline to finish succeeded and removed the recycle pool. A warning will now be raised, but the actions will finish executing as expected :recycle: :ocean:
  • @gregcaporaso added qiime.util.get_filepath_from_package, to standardize accessing files in QIIME 2 packages. This can facilitate, for example, re-using test data in usage examples.

Interface Updates :tv:

  • q2cli

    • @Oddant1 added the --use-cache flag to all actions not just pipelines :money_with_wings:
    • @Oddant1 added plugin level citations to the --citation flag on actions :clapper:
    • @Oddant1 added information about the current default parallel config to the output of the qiime info command :gear:
    • @Oddant1 improved support for collections of visualizations and collections in general in the cli usage API :books:

  • q2galaxy

    • @bebatut added a bio.tools xref to rendered tools for QIIME 2, which will add support some new ontology operations (grouping and filtering tools in Galaxy).

Plugin Updates :electric_plug:

  • q2-alignment

    • @SoilRotifer added the --keeplength option to force the original alignment length to remain unchanged while adding new sequences. Note, this means that in order to keep the original alignment length unchanged, the newly added sequences will have any insertions deleted.

  • q2-dada2

    • @colinvwood changed the denoise-ccs --p-adapter parameter to optional to support a wider set of input data.

  • q2-demux

    • @cherman2 updated partition_samples_single to also accept SampleData[JoinedSequenceswithQuality] :safety_pin:

  • q2-diversity

    • @lizgehret fixed a :beetle: in the alpha-rarefaction visualization (due to an updated version of pandas) that removed all of the sequencing depth data :exploding_head:
    • @lizgehret fixed a :bug: in the bioenv visualizer that was dropping samples with rows containing any empty values. It now drops sparse columns instead :bar_chart: Thanks to @jacobs for reporting this! :raised_hands:t3:

  • q2-diversity-lib

    • @lizgehret fixed a :bug: in the faith-pd output that left #SampleID as the index name in the resultant alpha diversity vector. Thanks to @Mehrbod_Estaki for reporting this! :pray:

  • q2-feature-table

    • @gregcaporaso updated split to ignore samples that are in the metadata but not the feature table. :axe:

  • q2-longitudinal

    • @Oddant1 added tests to volatility as part of an ongoing effort to ensure that visualizations are rendering correctly :chart_with_upwards_trend:

  • q2-types

    • Many split and combine actions were transferred into q2-types, facilitating the development of parallel Pipelines for plugin developers. (Learn more about using QIIME 2 in parallel here, and developing parallel Pipelines here.)
    • @Sann5 added the ReferenceDB[HMMER] semantic type to allow for orthologue prediction with a HMMER database through eggNOG in q2-moshpit :metal:
    • @misialq refactored some of the existing GenomeData types and added some new types that will be utilized in q2-moshpit as follows:
      • Removed SampleData[BLAST6] in favor of a new type: SampleData[Orthologs]
      • Removed OG and KEGG types as they are unused in the current methods.
      • Added SampleData[Orthologs] and GenomeData[Orthologs] to store orthologs produced by EggNOG+Diamond.
      • Added GenomeData[DNASequence] to represent collections of genome sequences fetched from various sources. :dna:
    • @Vinzent_Risch added FeatureTable[Normalized], the output of a new action that can normalize FeatureTable[Frequency] artifacts. This also includes properties that indicate the specific normalization method that was used. :dizzy:
    • @Vinzent_Risch added a new directory format called GenomeDataDirectoryFormat that GenesDirectoryFormat, ProteinsDirectoryFormat, LociDirectoryFormat and DNASequencesDirectoryFormat all inherit from :bowing_man:

  • q2-vizard

    • @lizgehret developed a new suite of visualization tools (included in the amplicon distribution) that can be used to visualize your metadata! :framed_picture: Each of these tools include a full test suite that utilizes Selenium to ensure that the visual representation of the data is accurate.
    • The following new visualizations are available (with interactive demos of each method available here):
      • 2D scatterplot (qiime vizard scatterplot-2d) :dotted_line_face:
        • This visualizer provides an exploratory view of your Metadata - allowing for any two numeric measures to be plotted against each other, with an optional third categorical measure used for color-coding. You can easily toggle between different measures using the drop downs for X, Y, and colorBy.
      • heatmap (qiime vizard heatmap) :hot_face:
        • This visualizer generates a heatmap displaying relationships between three Metadata measures. Two of the measures (which can be either categorical or numeric) are mapped to the x and y axes. The third measure (which must be numeric) defines the color gradient of the heatmap, illustrating the intensity or distribution of values across the grid.
      • lineplot (qiime vizard lineplot) :chart_with_upwards_trend:
        • This visualizer generates a lineplot displaying relationships between two numeric Metadata measures, with an optional third categorical measure for grouping your data into separate lines. If replicates are present within your first numeric measure (plotted on the X-axis), you can select either 'median' or 'mean' for replicate handling, which will create line(s) with the average at each point where replicates are present. All numeric columns present within your Metadata will be available as drop-down options on the Y-axis, but the chosen measure for the X-axis will remain fixed.
      • boxplot (qiime vizard boxplot) :bar_chart:
        • This visualizer generates boxplot(s) displaying relationships between a numerical Metadata measure and a categorical Metadata measure. Users can choose from three whisker calculation methods: percentile-based (9th/91st), min-max, and Tukey's Interquartile Range (IQR).

  • q2-stats

    • @ebolyen has finalized a set of methods to manipulate distributions and run permutational pairwise tests (Wilcoxon's Signed Rank Test and the Mann-Whitney U Test). The general purpose of q2-stats is to provide a foundation for manipulating distributions of data with appropriate statistical tests and more opinionated visualizations. :nerd_face: These methods are modular to support other plugins utilizing these methods. All that is needed is an appropriate "prep" method to set up the distribution with relevant categories and groups. (see q2-fmt for a great example of this.) :paperclip:
    • The following methods are available:
      • facet-across - Facet across outer group to create a collection of smaller distributions
      • facet-within - Facet within outer group to create a collection of smaller distributions
      • mann-whitney-u - Mann-Whitney U Test
      • mann-whitney-u-facet - Per-facet Mann-Whitney U Test
      • wilcoxon-srt - Wilcoxon Signed Rank Test
      • wilcoxon-srt-facet - Per-facet Wilcoxon Signed Rank Test
      • collate-stats - Combine and FDR correct multiple stats result tables
      • plot-rainclouds - Creates raincloud plots from any distribution supported by q2-stats.
    • Additionally two experimental (and deprecated actions) are available as a preview :t_rex:
      • alpha-group-significance - Will eventually be replaced with the regular alpha-group-significance method in q2-diversity.
      • prep-alpha-distribution - A "prep" method which converts alpha diversity into a distribution which q2-stats can work with.

  • q2-vsearch

    • @jphagen added a usage example for cluster-features-de-novo.

  • RESCRIPt

    • @lizgehret updated the evaluate-classifications and evaluate-taxonomy visualizations - they now both use the newly available lineplot from q2-vizard instead of q2-longitudinal's volatility plot :chart_with_upwards_trend:

Community Plugin Updates :electric_plug:

  • q2-fmt :poop:

    • @cherman2 added a to-baseline parameter in fmt group-timepoints and fmt cc. :poop: This new parameter will allow users to investigate engraftment by comparing diversity metrics to a baseline microbiome to illustrate microbial shifts away from baseline after FMT. :chart_with_downwards_trend:
    • @cherman2 refactored fmt peds-heatmap to encode sample size! This will help users determine which individuals/features have the best engraftment while considering population size of the engraftment! :bar_chart:
    • @cherman2 refactored group-timepoints so that it now accepts an additional group parameter (like a treatment group :dark_sunglasses: ). This means you can investigate the engraftment extent of any groups of interest and compare engraftment extent between your groups!
    • @cherman2 add an action called peds-simulation, this action randomizes the relationships between donors and recipients, to test whether the PEDS score between a recipient and their actual donor issignificantly higher than PEDS scores between other recipients paired with random donors. :star_struck: This method attempts to quantify the extent to which indicator features that are unique to a given donor are transferred to their recipients, as opposed to features that are not indicative of any specific donor.
      • Note: PEDS Monte Carlo simulation may have dependency issues between samples and the simulated background distribution that can make the test overly conservative. This can be fixed by filtering down to a single timepoint before running this method. Additionally if there are many baseline timepoints, the global test may be too conservative and this can be addressed by filtering out baseline samples prior to running this method. :+1:
    • @cherman2 added an action called sample-pprs (Proportional Persistence of Recipient Strains). This action will help identify which features are persistant in the recipient after FMT intervention. :mag_right:

  • q2-boots (docs, pre-print) :hiking_boot:

    • @isaiah-ghost and @gregcaporaso released a new plugin, q2-boots, that provides bootstrapped and rarefaction-based (collectively, resampled) alpha and beta diversity analyses, designed to mirror the interface of q2-diversity. This was developed for a Master's project by @isaiah-ghost.
    • Some aspects of the q2-boots interface are still subject to change, but you can expect this to eventually migrate into some of the distributions.
    • We strongly recommend starting to use the actions in this plugin where you previously would have used the qiime diversity core-metrics-* actions. While they are computationally more expensive, the results are more robust.

Documentation Updates :open_book:

Docker Container Updates :jar:

  • Recently, we have heard from multiple teams who are interested in teaching QIIME 2 workshops, either to promote their own plugins or to educate on microbiome data analysis generally. Henceforth, with each release we will provide Docker containers designed specifically to aid in this process. The key distinctions of these new workshop containers include default hosting of Jupyter Lab and the installation of several useful command line tools. These containers are separate from the previously provided Docker containers, which will continue to be updated with each release. The workshop containers are available from the following quay URLs: quay.io/qiime2/<distribution>-workshop:<epoch>, for example: quay.io/qiime2/amplicon-workshop:2024.10. To learn more see here.
  • The following distributions are now available as Docker images (in addition to amplicon and metagenome):
    • tiny
    • pathogenome
3 Likes
2

Our current development process for tracking and assigning incoming issues and pull requests can be found here :cowboy_hat_face:

1 Like