QIIME 2 2019.4 is now available!

(Q2-D2) #1

The QIIME 2 2019.4 release is now available! Thanks to everyone involved for their hard work!

As a reminder, our next planned QIIME 2 release is scheduled for late July 2019 (QIIME 2 2019.7), but please stay tuned. Other proposed releases this year include:

  • QIIME 2 2019.7
  • QIIME 2 2019.10

Check out the QIIME 2 2019.4 docs for details on installing the latest QIIME 2 release, as well as tutorials and other resources. Get in touch on the QIIME 2 forum 2 if you run into any issues!

Virtual machine builds will be available sometime next week - watch this topic thread for an update!

Here’s the highlights of the release:

  • QIIME 2 Framework
    • @ChrisKeefe fixed a bug caused by invalid primitive arguments :three::no_good_man:. Expanded type mismatch error messages to better describe various invalid-input situations. :speaking_head:
    • @ChrisKeefe fixed some pretty confusing language in the SemanticType docstring.
    • @Sleightly (GH) has added a new SDK utility named build_graph to qiime2.sdk.actiongraph. This will generate a directed NetworkX graph of QIIME 2 plugin actions and types. The graph is bipartite, with types on one conceptual side, and actions on the other. Edges between them indicate parameters.
    • The implementation of QIIME 2 types has been reworked by @ebolyen :martial_arts_uniform: . Some minor changes include:
      • Properties(['a', 'b']) => Properties('a', 'b')
      • Range(X, None) can be optionally written as Start(X) (it is converted to a Range)
      • Range(None, X) can be optionally written as End(X) (also converted to Range)
      • Choices({'a', 'b'}) => Choices('a', 'b') (this means order is now retained as well)
    • Unions of primitive types are now possible, thanks to @ebolyen! This means you can create a plugin with a parameter such as: Int % Start(0) | Str % Choices("ALL", "SOME", "NONE")
    • Intersection (&) of types is now possible :stop_sign: however most intersections will result in a “bottom” type - thanks for that, @ebolyen! (This operator is primarily for implementing internal type solving algorithms.)
      • real intesection: Start(0) & End(10) => Range(0, 10)
      • contradiction: FeatureTable[Frequency] & Phylogeny[Rooted] => UnionExp([]) (an empty union is the smallest, emptiest, type in QIIME 2)
    • The framework now supports different output types based on the received input type :grapes: . This feature is called TypeMap (and it has a sibling TypeMatch, and was created by @ebolyen):
      • This allows expressions such as:
        T, P, R = TypeMap({
            (TypeA, Bool % Choices(True)): OutputA,
            (TypeB, Bool % Choices(False)): OutputB
        })
        
      • For a direct example of TypeMatch, see implementation for feature-table filter-samples.
    • A utility to parse a type from the result of a to_ast() call now exists: qiime2.sdk.util.type_from_ast. This and parse_type are now significantly faster as loading plugins are no longer required to reconstruct the type. type_from_ast is capable of completely reconstructing a TypeMap.
    • @thermokarst added a new utility to qiime2.sdk.util named parse_primitive which takes on the role of <sometype>.decode(val), however it can handle collection types of varying style.
    • Other type utilities exposed in qiime2.sdk.util include: is_semantic_type, is_primitive_type, is_metadata_type, is_visualization_type, and interrogate_collection_type.
  • Library
    • @thermokarst fixed a bug that prevented admins from editing plugins (due to an over-specified database rule).
  • docs
    • @Nicholas_Bokulich fixed a typo in one of the overview tutorial images.
    • @Nicholas_Bokulich added an example using q2-sample-classifier's new heatmap pipeline to the q2-sample-classifier tutorial. :bookmark_tabs:
  • dev-docs
    • @ChrisKeefe busted out his editor’s red pencil on the dev docs.
    • @cduvallet updated the plugin development tutorial to reflect the latest versions of the packages used in the document! :card_file_box:
  • q2cli
    • @epruesse cleaned up the --version output when looking at a plugin’s details! :cherry_blossom:
    • @ebolyen put many hours of blood (:syringe:), sweat (:sweat_drops:), and tears (:cry:) into cleaning up the q2cli experience, changes include:
      • --cmd-config has been removed (it was underdocumented and added a lot of complexity). We encourage users of QIIME 2 who need programmatic control to use the Python API instead which is significantly more flexible.
      • --py-packages has been removed from qiime info (it was already broken). Use conda list instead. :dragon:
      • --output-dir and --o-<name> options now ensure the path is writable before executing the command. :writing_hand:
      • --output-dir no longer creates the directory before running the command. This means no more rm -rf'ing because there was a typo in a different parameter. :open_file_folder:
      • The underlying parser has been significantly improved. The following is now possible:
        • Missing arguments will now be recognized instead of swallowing the next option (very confusing for beginners). :open_mouth:
        • --help is now “hyper-greedy”, if it appears anywhere in the command, the help text will be shown (no matter how many errors exist otherwise).
        • Options that are repeatable may also now receive an arbitrary number of arguments:
          --p-something 1 --p-something 2 --p-something 3
          
          can now also be written as:
          --p-something 1 2 3   # the original form is still supported
          
          This means that shell-expansion (like *.qza) will become more useful:
          qiime feature-table merge --i-tables dada2-outputs/*.qza ...
          
      • The CLI cache is now optimistically :butterfly: stored in your conda environment. If q2cli is not running in a conda environment, then it will fall back to XDG_CONFIG_HOME as before. This means that switching between multiple QIIME 2 installations will not constantly refresh the cache.
      • difflib is now used to recommend options when a typo is made in a command. :woman_shrugging:
      • Error handling has been made significantly more consistent between plugin commands and builtins (such as tools, dev, and info):
      • Errors are now enumerated. Parsing errors such as malformed commands will be re-parsed multiple times to collect as many errors as q2cli can (these are an incomplete list, and will be printed as (1/5?) instead of (1/5)) :100:
      • In order to support TypeMap, the --help text has a new layout system which includes the QIIME 2 type whenever relevant (this is especially helpful for ranged primitives). TypeMaps are indicated with a superscript number which will match between inputs and outputs. (In the future, a table may be a better way to represent this).
      • Help text is now colorized to improve readability :rainbow: :

  • q2-diversity
    • @thermokarst fixed a bug :bug: in the adonis visualizer that prevented some metadata headers from working as expected.
    • @Nicholas_Bokulich fixed a bug in the adonis visualizer that caused an error if metadata values contained single quotes (e.g., apostrophes). :butterfly:
    • @Nicholas_Bokulich fixed a typo in the adonis visualizer help text. :alembic:
  • q2-longitudinal
    • @Nicholas_Bokulich added a new anova visualizer to the plugin! :tada:
    • The heatmap generated by maturity-index can now be controlled to limit the number of important features displayed. (Thanks @Nicholas_Bokulich!) :fire_engine:
  • q2-deblur
    • @thermokarst added a new left_trim_len parameter to the denoising methods of this plugin! :mountain_biking_man:
  • q2-demux
    • Fixed the per-sample sequence count table layout (that thing sure was bothering @ChrisKeefe!) :fish_cake:
    • @wasade brought Golay barcode correction to emp-single and emp-paired - the future is now! :baseball:
  • q2-cutadapt
    • @thermokarst added a new minimum_length parameter to trim_single and trim_paired! :scissors:
    • The ability to discard_untrimmed reads has been added to trim_single and trim_paired by @thermokarst. :grapes:
    • @thermokarst fixed a bug related to too many files being opened when demuxing datasets with many samples in it. Check out the new batch_size parameter on demux-paired and demux-single to control how many samples are demultiplexed at a time! :broccoli:
    • @ChrisKeefe & @thermokarst added support for dual-index demultiplexing! This functionality is enabled by preparing a second column in your Metadata file that contains the barcodes for the reverse reads. Check out the docs for more info!
  • q2-feature-classifier
  • q2-sample-classifier
    • @Nicholas_Bokulich added a heatmap pipeline to display the (normalized) abundances of predictive features per sample or per group, using FeatureData[Importance] scores output by supervised classifiers and regressors. :fire_extinguisher:
    • @Nicholas_Bokulich updated the citation information to include the report describing this plugin in the Journal of Open Source Software.
  • q2-feature-table
    • @gwarmstrong (GH) added a feature to allow disabling of clustering on both the sample and feature axes when creating a heatmap! :fire:
    • merge-taxa now supports multi-column FeatureData[Taxonomy] artifacts, e.g., to preserve classification confidence scores in the merged output. (thanks @Nicholas_Bokulich) :japanese_ogre:
    • @turanoo overhauled the interactive even-sampling depth visualization — it is now built with vega! There were several improvements/bugfixes related:
      • Sample count summaries for each metadata group are now displayed in the plot
      • SVG and PNG exports are enabled for the plot
      • The visualization can be customized in the interactive vega editor
      • A bug related to scrolling in the visualization’s tabs was fixed
    • As a result of TypeMap, it is now possible to filter out samples (feature-table filter-samples) of feature tables with the variant types of: RelativeFrequency, PresenceAbsence, and Composition (Frequency is of course still supported). The output type will match the input type provided.
  • q2-quality-control
    • @Nicholas_Bokulich fixed a bug in evaluate-composition that caused linear regression R values of zero to be reported when observed and expected feature tables contain only a single observation at any taxonomic level. (the correct answer is no R value should be reported). :bug:
  • q2-dada2
    • @benjjneb updated this plugin to use DADA2 1.10 - users can expect to see speed improvements from 2x to 20x! :racing_car: :fire: :sweet_potato:! Check out the DADA2 docs for more details on this version of DADA2.
  • q2-emperor
    • This plugin now uses the latest version of emperor (1.0.0-beta.19)
    • Miscellaneous bug fixes: :bug:
      • Fixes a bug where re-centering the camera wouldn’t re-center the view.
      • Fixes a bug where two dimensional plots would fail to load.
      • Fixes a bug where sample metadata would fail to load if only sample identifiers were included.
    • Experimental support for exporting 2D vega plots has been added by @thermokarst ! This is currently only available for a subset of the available plot types, and can be accessed through the menu:
      image
      Any customization to the 3D plot should be inherited in the vega plot. For example, here is a customized 3D plot:

      And here is the 2D vega version:
      visualization
      This functionality is still experimental — please try it out and let us know how it goes for you! :balloon:
  • q2-types

Keep on QIIME-ing! :sun_with_face: :qiime2: :t_rex:

17 Likes
[Preview] QIIME 2 2019.4 development changelog
[Preview] QIIME 2 2019.4 development changelog
(benjamin w.) #5

Sweet 2-D plot, it looks like those in Phyloseq which we really like. Great job.

3 Likes
(Matthew Ryan Dillon) #6

Thanks! You can check out the interactive vega spec for that example viz here:

Emperor/Vega demo

Enjoy!

2 Likes
(Justine) #7

Thank you! :smile::tada:

3 Likes
#8

Thank you for a new release! Great work

2 Likes
#9

Just wanted to thank the team for their hard work. Qiime2-2019.4 and the tool as a whole is a huge deal and I am privileged to use it. Definitely looking forward to the cutadapt bug fixes, new filter parameters, trim left on Deblur.

Really exciting stuff! Thank you!

4 Likes
(Liyah Smith) #10

Hello, I am excited for the new release! I am only able to use QIIME2 through the Virtual Box, so I was wondering when the disk image file will be available so that I am able to upgrade?

(Nicholas Bokulich) #11

The disk images for virtual box are usually released a few days after the main release, so that should be available some time this week. Thanks for your patience @Liyah_Smith!

Deicode RPCA biplot - Are the vectors distorting the plot?
(Matthew Ryan Dillon) #12

We just deployed new slightly tweaked QIIME 2 2019.4 environment files — these pin the version of numpy to 1.15.4. Maybe @mortonjt can say a few words here about why we made this change, and why it might matter to you. Thanks!

(Jamie Morton) #13

Thanks @thermokarst for deploying an older version.

We’ve seen that the SVD operation in the newest release of numpy and scipy are broken: https://github.com/numpy/numpy/issues/13401

What this means is that all beta diversity calculations involving svd (i.e. pcoa), in addition to the balances will output wrong results using qiime2-2019.4

The newest qiime2-2019.4 environment files should avoid this issue by pinning numpy to an older version. So make sure your qiime2 environments are up-to-date if you are using qiime2-2019.4

QIIME 2 2019.4.0 has faulty OpenBLAS results on certain CPUs
(Eman Khalaf) #15

Hello,

How could I know if the results (diversity calculations, gneiss) are correct or not? I already installed the new release 2-2019.4 successfully and am using it since yesterday.

Thank you

(Matthew Ryan Dillon) #16

As @mortonjt mentioned, the issue to do with a bug in a specific version of numpy — the best way to ensure that everything is okay is to re-install 2019.4, being sure to download the newest environment file.

(Eman Khalaf) #17

Okay, so after installing the newest environment file, all what I need is to re-do diversity analysis and gneiss. I mean other plugins are still good and no need to re-run them.

Thank you

#18

Great timing! I installed 2019.4 only yesterday. I checked the yml file and the numpy version listed was 1.16.3, so I removed the environment and reinstalled. FYI, I got a conda error when trying to remove the environment:

pre-unlink script failed for package conda-forge::widgetsnbextension-3.4.2-py36_1000

I solved this based on suggestions here and here

Sorry to post conda issues here, but in case anyone else gets this error.

Thanks for continued development!

4 Likes
(Didit) #19

@mortonjt: can we just install the older version of numpy instead reinstall the new updated QIIME2-2019.4 environment? And how?

Thanks.

(Evan Bolyen) #20

Correct! Only analysis downstream of your feature table could have been impacted by this. Since those steps are (usually) very fast, its best to just re-run them so you don’t have to worry about it.

Yes, the only difference between the two environment files is the numpy version.

You can correct that by running:

conda install -c conda-forge numpy=1.15.4

in your QIIME 2 environment. If you are at all nervous about that or conda, the safest thing to do is still to reinstall.


As a general note to anyone: if you or someone you know may have been impacted by this bug, you can always check provenance at the bottom. Just click on python-packages and expand the list until you see numpy. You want to see numpy 1.15.4 with QIIME 2 2019.4. If you see 1.16.3, that could be an issue, so you should verify that the steps recorded do not use some of the problematic OpenBLAS bindings (this is hard to do, so best just re-run).

1 Like
(Didit) #21

After searching numpy files on my desktop pc (installed 2019.4 envs on 6 May '19), i got this…


There is both of the versions (1.16.3 and 1.15.4).
Is it ok with this condition on my desktop pc? or should I uninstall the 1.16.3 version?

And on my laptop (installed 2019.4 envs on 5 May '19), there is only numpy-1.16.3 version. I will install numpy-1.15.4 on my laptop. thanks @ebolyen.

Thanks all.

(Evan Bolyen) #22

Your search is showing your miniconda3/pkgs which is an internal cache for conda, it keeps all versions, but doesn’t reflect which version is actually used. Use conda list to identify that. Otherwise numpy 1.16.3 isn’t itself causing any harm by existing on the hard drive so just verify the environment is correct and you will be fine.

(Didit) #23

@ebolyen: OK, I got it. It only shows 1.15.4 version in conda list. thanks alot.

2 Likes
(Evan Bolyen) #24

Update: the situation appears to be a bit worse than originally thought. See this announcement for details. We will be releasing a new environment shortly.

1 Like