Using Qiime 2 as an API / documentation on q2-types

Hi,

first of all thanks a lot for your work on Qiime2.

I Was trying to use Qiime2 as an API in order to have a single Python file running a set of Qiime2 steps rather than calling a series of cli commands. This seems to work in general since you can import artifacts into Python and then call the respective functions from the plugins directly. That should also be a little more efficient since you can pass the artifacts by reference and don’t need to serialize to disk.

However, I struggle a bit with creating the input and extracting information from the returned artifacts/types and converting them to other Python objects. For instance the demux function has an EMPPairedEndSequences object as parameter and returns SampleData[PairedEndSequencesWithQuality], but I could not find information how those objects can be created, modified or converted. I looked through the q2-types source but there are no inline docs and the large inheritance patterns makes it a bit difficult to understand which attributes and methods each type has. Also I could not find a straight-forward way to get a list of permissible classes to pass to the .view() method (I think I found one dict that is generated when registering plugins but can’t remember).

So my actual questions would be if there is some additional documentation on q2-types and how to use them together with Qiime2 to run plugins with out the CLI mode.

Hope that was more or less clear. Thanks a lot for your help!

Christian

4 Likes

Hey there @cdiener!

Just to clarify, you are using the Artifact API, right? If not, I would recommend taking a peek! There are also a few posts floating around this forum with examples.

This is a huge documentation hole on our part --- this is something that has been on the TODO list for far too long. We have plans for integrated docs (or docstrings), but that won't land until at least some time in 2019. I am thinking in the meantime to start up a wiki post here on this forum to start documenting the types, formats, and transformers between formats that are known in the QIIME 2 ecosystem.

In the meantime, if you have specific questions, I would recommend asking here, that is probably the quickest way you will get info.

So sorry for the bottleneck here!

:t_rex: :qiime2:

1 Like

Thanks for the quick reply @thermokarst.

Yes I am using the Artifact API, and actually only want to use that exclusively. There were some other hickups like the plugin functions not being thread safe (overwrite the same tempdir) so I put that on hold a little bit.

Any docs whether they are inline or wiki posts on that would be great. I will ask in the forum if I have more specific questions.

Thanks a lot
Christian

1 Like

Wait, that doesn't sound possible, unless you are somehow not actually using the Artifact API, but rather just importing the commands directly from the plugins. Would you be able to share a snippet of what you are doing? We haven't ever run into a situation where the same tempdir is used (neither in our own work, or reported here on the forum). Curious to see the conditions that would lead to that. Thanks!

1 Like

Okay so maybe just don’t know how to use the Artifact API. So basically what I tried to do is to import the input artifacts with Artifact.load and then pass them to qiime2.plugins.dada2.denoise_paired within a multiprocessing.Pool which led to some of the processes using the same files or something like that (sorry already a little while ago).

But I should create a reproducible example and see if that persists.

Calling qiime2 CLI commands directly with subprocess.run did work though.

1 Like

This topic was automatically closed 31 days after the last reply. New replies are no longer allowed.