All the qiime 2 plugins are conveniently listed on the available plugins page. This is so perfect!
In contrast, I can't find a page listing these 'built-ins'. The command qiime info is only mentioned twice, and I can't find a single mention of qiime dev in the official docs.
The crucially important qiime tools import has a great tutorial, but no comprehensive docs. This has confused some users:
Yes, +1 very much agree that this is quite a confusing aspect of qiime 2.
We'd actually discussed something similar to this with @ebolyen and @Nicholas_Bokulich during this pull request discussion - not sure if this was followed up on or an issue was ever created as a result though.
Thanks for the update, Claire! I'm glad this issue is still on their radar, and until then, the importing tutorial is a great place for most users.
As it sounds like you are a bit more plugged into development than I am, can you help me find a discussion on what should be a built-in vs a plugin? Essential steps like denoising / clustering are all presented to users as plugins, so I guess I'm struggling to understand the role of 'built-ins' outside of the API.
I should probably wait for other folks to :qiime2:in about this. Until then, see you all in 2019!
Remember, these questions are specific to q2cli. QIIME 2 is "interface agnostic," which means that multiple interfaces expose the same plugin functionality. In the case of q2cli, we need to expose some additional tools in order to allow user to import and export data, as well as a few other "odds and ends" things. These same bits of functionality are also available in other interfaces, but via mechanisms that are most appropriate for those interfaces. Take a look at the import tutorial for q2cli, then look at the Artifact API example for comparison!
The other thing to keep in mind @colinbrislawn is that QIIME 2 itself knows nothing about microbiome bioinformatics. All of that knowledge comes from plugins. So, with that in mind, the the commands available in q2cli's tools command should be considered "lower-level" than microbiome science-ey things --- instead, consider these pieces are tools relevant for getting data in and out of the system, for example. Those ideas transcend microbiome bioinformatics. You could imagine a q2-astronomy plugin that exposes new formats and actions relevant for analyzing the sky --- the commands in that plugin are specific to that domain, but, getting data in and out of QIIME 2 looks the exact same as with the microbiome formats you know and love, the only thing different is that the types and formats will be different.
Got it! This answer my question about plugins vs 'built-ins':
"Built-ins" provide low level access to Qiime 2 data structures, while plugins provide additional functionality, often by making use of programs that are not part of Qiime 2.
Got it.
Are you willing to humor me and let me play devil's advocate?
You already answered my question really well, so you totally don't have to.
I'm all excited about q2-astronomy, but I'm still worried about plugins.
I agree! And yet...
...right now, the first four items on that page are specific to genomics:
Sequence data with sequence quality information (i.e. FASTQ)
Sequences without quality information (i.e. FASTA)
Per-feature unaligned sequence data (i.e., representative FASTA sequences)
Per-feature aligned sequence data (i.e., aligned representative FASTA sequences)
A q2-astronomy plugin would never use any of these...
Arn't these the "microbiome science-ey" data types you are talking about?
Yep - those are all examples of how to import common formats in the field of microbiome bioinformatics. The point I was making above though is that the QIIME 2 Framework (the heart and soul of the QIIME 2 ecosystem) knows nothing about those types and formats --- those are all defined by a plugin (in this case, q2-types). Plugins teach QIIME 2 how to do things.
I have discovered the source of my own confusion. This statement helped clear it up:
I just realized that while qiime tools import is a "built-in," all of the semantic types being imported by this tool are defined in plugins.
(Am I getting this right?)
Here is what caused my confusion: the documentation for these plugins is on the page of the builtin.
Refactoring needed indeed!
Based on my new understanding, I guess I would like to see the tutorial on importing q2-types under the q2-types plugin: types — QIIME 2 2018.11.0 documentation
Maybe more broadly, it would be cool if each plugin not only got a page for documentation, but also a dedicated page for a tutorial. Or maybe add a new link (along with version, website, support) to a tutorial. This would allow us to ship technical and tutorial documentation for each plug-in.
