Discovering plugins via importlib instead of pkg_resources

Developer Discussion
1

Hi,

This is kind of an exotic use case but it might be nice for teaching and making Qiime tutorials more accessible.

Motivation

We recently held a course teaching Qiime2 and using Google Colab. Like other (free) managed Jupyter environments you need to install packages into the running Python process and can use them on the fly.

See GitHub - Gibbons-Lab/isb_course_2020: Materials and tutorials for the 2020 ISB Microbiome Course for a link to the notebook and the setup_qiime2.py script we use to install Qiime2 into the running environment. Google Colab will delete all data when you close the Jupyter instance and there is no way to restart the underlying jupyter/Python process without losing the installed packages. Everything works fine except the Qiime2 python API. Basically, no plugins are registered, which means that none of the default types is available and you can't really use the Artifact API.

In this environment, the following will give an empty list:

import pkg_resources
list(pkg_resources.iter_entry_points(group="qiime2.plugins"))

Qiime2 is installed however and can be imported with import qiime2 etc.

Proposed solution

I am not super familiar with Python entry points. However, it looks like setuptools now recommends to use importlib metadata to get the entry points and that seems to work in those environments:

from importlib_metadata import entry_points
entry_points()["qiime2.plugins"]

gives

(EntryPoint(name='q2-quality-filter', value='q2_quality_filter.plugin_setup:plugin', group='qiime2.plugins'),
 EntryPoint(name='q2-diversity-lib', value='q2_diversity_lib.plugin_setup:plugin', group='qiime2.plugins'),
 EntryPoint(name='q2-demux', value='q2_demux.plugin_setup:plugin', group='qiime2.plugins'),
 EntryPoint(name='q2-composition', value='q2_composition.plugin_setup:plugin', group='qiime2.plugins'),
 EntryPoint(name='q2-deblur', value='q2_deblur.plugin_setup:plugin', group='qiime2.plugins'),
 EntryPoint(name='q2-cutadapt', value='q2_cutadapt.plugin_setup:plugin', group='qiime2.plugins'),
 EntryPoint(name='dummy-plugin', value='qiime2.core.testing.plugin:dummy_plugin', group='qiime2.plugins'),
 EntryPoint(name='q2-gneiss', value='q2_gneiss.plugin_setup:plugin', group='qiime2.plugins'),
 EntryPoint(name='q2-longitudinal', value='q2_longitudinal.plugin_setup:plugin', group='qiime2.plugins'),
 EntryPoint(name='q2-feature-classifier', value='q2_feature_classifier.plugin_setup:plugin', group='qiime2.plugins'),
 EntryPoint(name='q2-vsearch', value='q2_vsearch.plugin_setup:plugin', group='qiime2.plugins'),
 EntryPoint(name='q2-feature-table', value='q2_feature_table.plugin_setup:plugin', group='qiime2.plugins'),
 EntryPoint(name='q2-fragment-insertion', value='q2_fragment_insertion.plugin_setup:plugin', group='qiime2.plugins'),
 EntryPoint(name='q2-types', value='q2_types.plugin_setup:plugin', group='qiime2.plugins'),
 EntryPoint(name='q2-taxa', value='q2_taxa.plugin_setup:plugin', group='qiime2.plugins'),
 EntryPoint(name='q2-emperor', value='q2_emperor.plugin_setup:plugin', group='qiime2.plugins'),
 EntryPoint(name='q2-sample-classifier', value='q2_sample_classifier.plugin_setup:plugin', group='qiime2.plugins'),
 EntryPoint(name='q2-alignment', value='q2_alignment.plugin_setup:plugin', group='qiime2.plugins'),
 EntryPoint(name='q2-diversity', value='q2_diversity.plugin_setup:plugin', group='qiime2.plugins'),
 EntryPoint(name='q2-metadata', value='q2_metadata.plugin_setup:plugin', group='qiime2.plugins'),
 EntryPoint(name='q2-dada2', value='q2_dada2.plugin_setup:plugin', group='qiime2.plugins'),
 EntryPoint(name='q2-phylogeny', value='q2_phylogeny.plugin_setup:plugin', group='qiime2.plugins'),
 EntryPoint(name='q2-quality-control', value='q2_quality_control.plugin_setup:plugin', group='qiime2.plugins'))

which seems to be correct.

Discussion

Since I am sure there is more that is connected to the PluginManager, do you think that would be reasonable? It seems like it can be used as a drop-in replacement but I might be wrong.

Thanks and keep up the great work!

3 Likes
3

Hey @cdiener!

Thanks for the heads up, this actually looks like a pretty trivial change to make, we aren't doing that much:

It looks like the new EntryPoint object has basically the same API:

the only real change is entry_point.dist appears to now be metadata(name) but we were only using that to get a name, so we can probably just use the name attribute now.

As an immediate workaround, there is now an add_plugin() method on the plugin manager which you could use to set one up manually, it would look a bit like this:

def _hack_in_the_plugins():
    import qiime2.sdk as sdk
    from importlib_metadata import entry_points
    
    pm = sdk.PluginManager(add_plugins=False)
    for entry in entry_points()["qiime2.plugins"]:
        plugin = entry.load()
        package = entry.value.split(':')[0].split('.')[0]
        pm.add_plugin(plugin, package, entry.name)

_hack_in_the_plugins()

from qiime2.plugins import feature_table  # or however
3 Likes
4

Oh, importantly, the PluginManager is a singleton, so you can set it up basically however you need to at the start, and then anything else that needs that will end up with the same instance later on (which prevents passing this object around to everything).

3 Likes
6

Awesome I'll add the workaround to our setup script, thanks!

2 Likes