QIIME 2 2019.4 has been released!
Please see the official changelog for more details.
Original content of post
Important
The following is an early developer preview of the changes expected in 2019.4 This post is a live-document which will be updated throughout our development cycle. Any links will in this topic will be broken until the release is officially published. When we are ready for release, we’ll copy this changelog and create a new post in the Announcements category.
Important Developer Dates (please keep an on this post, these might change):
- PRs must be submitted by: April 24, 2019
- PRs must be merged by: April 26, 2019
- Repo Freeze and Package Building: April 29, 2019
- Release Day: April 30, 2019
Here’s the highlights of the release:
- QIIME 2 Framework
- @ChrisKeefe fixed a bug caused by invalid primitive arguments . Expanded type mismatch error messages to better describe various invalid-input situations.
- @ChrisKeefe fixed some pretty confusing language in the
SemanticType
docstring. @Sleightly (GH)
has added a new SDK utility namedbuild_graph
toqiime2.sdk.actiongraph
. This will generate a directed NetworkX graph of QIIME 2 plugin actions and types. The graph is bipartite, with types on one conceptual side, and actions on the other. Edges between them indicate parameters.- The implementation of QIIME 2 types has been reworked by @ebolyen . Some minor changes include:
Properties(['a', 'b'])
=>Properties('a', 'b')
Range(X, None)
can be optionally written asStart(X)
(it is converted to aRange
)Range(None, X)
can be optionally written asEnd(X)
(also converted toRange
)Choices({'a', 'b'})
=>Choices('a', 'b')
(this means order is now retained as well)
- Unions of primitive types are now possible, thanks to @ebolyen! This means you can create a plugin with a parameter such as:
Int % Start(0) | Str % Choices("ALL", "SOME", "NONE")
- Intersection (
&
) of types is now possible however most intersections will result in a "bottom" type - thanks for that, @ebolyen! (This operator is primarily for implementing internal type solving algorithms.)- real intesection:
Start(0) & End(10)
=>Range(0, 10)
- contradiction:
FeatureTable[Frequency] & Phylogeny[Rooted]
=>UnionExp([])
(an empty union is the smallest, emptiest, type in QIIME 2)
- real intesection:
- The framework now supports different output types based on the received input type . This feature is called
TypeMap
(and it has a siblingTypeMatch
, and was created by @ebolyen):- This allows expressions such as:
T, P, R = TypeMap({ (TypeA, Bool % Choices(True)): OutputA, (TypeB, Bool % Chocies(False)): OutputB })
- For a direct example of TypeMatch, see implementation for feature-table filter-samples.
- This allows expressions such as:
- A utility to parse a type from the result of a
to_ast()
call now exists:qiime2.sdk.util.type_from_ast
. This andparse_type
are now significantly faster as loading plugins are no longer required to reconstruct the type.type_from_ast
is capable of completely reconstructing aTypeMap
. - @thermokarst added a new utility to
qiime2.sdk.util
namedparse_primitive
which takes on the role of<sometype>.decode(val)
, however it can handle collection types of varying style. - Other type utilities exposed in
qiime2.sdk.util
include:is_semantic_type
,is_primitive_type
,is_metadata_type
,is_visualization_type
, andinterrogate_collection_type
.
- Library
- @thermokarst fixed a bug that prevented admins from editing plugins (due to an over-specified database rule).
- docs
- @Nicholas_Bokulich fixed a typo in one of the overview tutorial images.
- @Nicholas_Bokulich added an example using
q2-sample-classifier
's newheatmap
pipeline to theq2-sample-classifier
tutorial.
- dev-docs
- @ChrisKeefe busted out his editor's red pencil on the dev docs.
- @cduvallet updated the plugin development tutorial to reflect the latest versions of the packages used in the document!
- q2cli
- @epruesse cleaned up the
--version
output when looking at a plugin's details! - @ebolyen put many hours of blood (), sweat (), and tears () into cleaning up the q2cli experience, changes include:
--cmd-config
has been removed (it was underdocumented and added a lot of complexity). We encourage users of QIIME 2 who need programmatic control to use the Python API instead which is significantly more flexible.--py-packages
has been removed fromqiime info
(it was already broken). Useconda list
instead.--output-dir
and--o-<name>
options now ensure the path is writable before executing the command.--output-dir
no longer creates the directory before running the command. This means no morerm -rf
'ing because there was a typo in a different parameter.- The underlying parser has been significantly improved. The following is now possible:
- Missing arguments will now be recognized instead of swallowing the next option (very confusing for beginners).
--help
is now "hyper-greedy", if it appears anywhere in the command, the help text will be shown (no matter how many errors exist otherwise).- Options that are repeatable may also now receive an arbitrary number of arguments:
can now also be written as:--p-something 1 --p-something 2 --p-something 3
This means that shell-expansion (like--p-something 1 2 3 # the original form is still supported
*.qza
) will become more useful:qiime feature-table merge --i-tables dada2-outputs/*.qza ...
- The CLI cache is now optimistically stored in your conda environment. If q2cli is not running in a conda environment, then it will fall back to
XDG_CONFIG_HOME
as before. This means that switching between multiple QIIME 2 installations will not constantly refresh the cache. difflib
is now used to recommend options when a typo is made in a command.- Error handling has been made significantly more consistent between plugin commands and builtins (such as
tools
,dev
, andinfo
): - Errors are now enumerated. Parsing errors such as malformed commands will be re-parsed multiple times to collect as many errors as q2cli can (these are an incomplete list, and will be printed as
(1/5?)
instead of(1/5)
) - In order to support TypeMap, the
--help
text has a new layout system which includes the QIIME 2 type whenever relevant (this is especially helpful for ranged primitives). TypeMaps are indicated with a superscript number which will match between inputs and outputs. (In the future, a table may be a better way to represent this). - Help text is now colorized to improve readability :
- @epruesse cleaned up the
https:/uploads/qiime21/original/2X/c/c83b5c662898d9addfee797017d1e087171042b8.webm
- q2-diversity
- @thermokarst fixed a bug in the
adonis
visualizer that prevented some metadata headers from working as expected. - @Nicholas_Bokulich fixed a bug in the
adonis
visualizer that caused an error if metadata values contained single quotes (e.g., apostrophes). - @Nicholas_Bokulich fixed a typo in the
adonis
visualizer help text.
- @thermokarst fixed a bug in the
- q2-longitudinal
- @Nicholas_Bokulich added a new
anova
visualizer to the plugin! - The heatmap generated by
maturity-index
can now be controlled to limit the number of important features displayed. (Thanks @Nicholas_Bokulich!)
- @Nicholas_Bokulich added a new
- q2-deblur
- @thermokarst added a new
left_trim_len
parameter to the denoising methods of this plugin!
- @thermokarst added a new
- q2-demux
- Fixed the per-sample sequence count table layout (that thing sure was bothering @ChrisKeefe!)
- @wasade brought Golay barcode correction to
emp-single
andemp-paired
- the future is now!
- q2-cutadapt
- @thermokarst added a new
minimum_length
parameter totrim_single
andtrim_paired
! - The ability to
discard_untrimmed
reads has been added totrim_single
andtrim_paired
by @thermokarst. - @thermokarst fixed a bug related to too many files being opened when demuxing datasets with many samples in it. Check out the new
batch_size
parameter ondemux-paired
anddemux-single
to control how many samples are demultiplexed at a time! - @ChrisKeefe & @thermokarst added support for dual-index demultiplexing! This functionality is enabled by preparing a second column in your Metadata file that contains the barcodes for the reverse reads. Check out the docs for more info!
- @thermokarst added a new
- q2-feature-classifier
- @BenKaehler & @Nicholas_Bokulich updated the default value for the
reads_per_batch
parameter, in an effort to reduce memory usage!
- @BenKaehler & @Nicholas_Bokulich updated the default value for the
- q2-sample-classifier
- @Nicholas_Bokulich added a
heatmap
pipeline to display the (normalized) abundances of predictive features per sample or per group, usingFeatureData[Importance]
scores output by supervised classifiers and regressors. - @Nicholas_Bokulich updated the citation information to include the report describing this plugin in the Journal of Open Source Software.
- @Nicholas_Bokulich added a
- q2-feature-table
@gwarmstrong (GH)
added a feature to allow disabling of clustering on both the sample and feature axes when creating aheatmap
!merge-taxa
now supports multi-columnFeatureData[Taxonomy]
artifacts, e.g., to preserve classification confidence scores in the merged output. (thanks @Nicholas_Bokulich)- @turanoo overhauled the interactive even-sampling depth visualization --- it is now built with vega! There were several improvements/bugfixes related:
- Sample count summaries for each metadata group are now displayed in the plot
- SVG and PNG exports are enabled for the plot
- The visualization can be customized in the interactive vega editor
- A bug related to scrolling in the visualization's tabs was fixed
- As a result of TypeMap, it is now possible to filter out samples (
feature-table filter-samples
) of feature tables with the variant types of:RelativeFrequency
,PresenceAbsence
, andComposition
(Frequency
is of course still supported). The output type will match the input type provided.
- q2-quality-control
- @Nicholas_Bokulich fixed a bug in
evaluate-composition
that caused linear regression R values of zero to be reported when observed and expected feature tables contain only a single observation at any taxonomic level. (the correct answer is no R value should be reported).
- @Nicholas_Bokulich fixed a bug in
- q2-dada2
- @benjjneb updated this plugin to use DADA2 1.10 - users can expect to see speed improvements from 2x to 20x! ! Check out the DADA2 docs for more details on this version of DADA2.
- q2-emperor
- This plugin now uses the latest version of emperor (
1.0.0-beta.19
) - Miscellaneous bug fixes:
- Fixes a bug where re-centering the camera wouldn't re-center the view.
- Fixes a bug where two dimensional plots would fail to load.
- Fixes a bug where sample metadata would fail to load if only sample identifiers were included.
- Experimental support for exporting 2D vega plots has been added! This is currently only available for a subset of the available plot types, and can be accessed through the menu:
Any customization to the 3D plot should be inherited in the vega plot. For example, here is a customized 3D plot:
And here is the 2D vega version:
This functionality is still experimental --- please try it out and let us know how it goes for you!
- This plugin now uses the latest version of emperor (
Happy QIIME-ing!