# Discovering plugins via importlib instead of pkg\_resources

**URL:** <https://forum.qiime2.org/t/discovering-plugins-via-importlib-instead-of-pkg-resources/18358>\
**Category:** Developer Discussion\
**Tags:** plugin-development\
**Created:** [February 3, 2021, 1:01am UTC](https://forum.qiime2.org/t/discovering-plugins-via-importlib-instead-of-pkg-resources/18358 "2021-02-03T01:01:10Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![cdiener](https://forum.qiime2.org/user_avatar/forum.qiime2.org/cdiener/32/7026_2.png) [@cdiener](https://forum.qiime2.org/u/cdiener)\
**Post date:** [February 3, 2021, 1:01am UTC](https://forum.qiime2.org/t/discovering-plugins-via-importlib-instead-of-pkg-resources/18358/1 "2021-02-03T01:01:10Z")

</div>

Hi,

This is kind of an exotic use case but it might be nice for teaching and making Qiime tutorials more accessible.

## Motivation

We recently held a course teaching Qiime2 and using Google Colab. Like other (free) managed Jupyter environments you need to install packages into the running Python process and can use them on the fly.

See [GitHub - Gibbons-Lab/isb\_course\_2020: Materials and tutorials for the 2020 ISB Microbiome Course](https://github.com/Gibbons-Lab/isb_course_2020) for a link to the notebook and the `setup_qiime2.py` script we use to install Qiime2 into the running environment. Google Colab will delete all data when you close the Jupyter instance and there is no way to restart the underlying jupyter/Python process without losing the installed packages. Everything works fine except the Qiime2 python API. Basically, no plugins are registered, which means that none of the default types is available and you can't really use the Artifact API.

In this environment, the following will give an empty list:

```Python
import pkg_resources
list(pkg_resources.iter_entry_points(group="qiime2.plugins"))

```

Qiime2 is installed however and can be imported with `import qiime2` etc.

## Proposed solution

I am not super familiar with Python entry points. However, it looks like setuptools [now recommends to use importlib metadata](https://setuptools.readthedocs.io/en/latest/userguide/entry_point.html#advertising-behavior) to get the entry points and that seems to work in those environments:

```Python
from importlib_metadata import entry_points
entry_points()["qiime2.plugins"]

```

gives

```auto
(EntryPoint(name='q2-quality-filter', value='q2_quality_filter.plugin_setup:plugin', group='qiime2.plugins'),
 EntryPoint(name='q2-diversity-lib', value='q2_diversity_lib.plugin_setup:plugin', group='qiime2.plugins'),
 EntryPoint(name='q2-demux', value='q2_demux.plugin_setup:plugin', group='qiime2.plugins'),
 EntryPoint(name='q2-composition', value='q2_composition.plugin_setup:plugin', group='qiime2.plugins'),
 EntryPoint(name='q2-deblur', value='q2_deblur.plugin_setup:plugin', group='qiime2.plugins'),
 EntryPoint(name='q2-cutadapt', value='q2_cutadapt.plugin_setup:plugin', group='qiime2.plugins'),
 EntryPoint(name='dummy-plugin', value='qiime2.core.testing.plugin:dummy_plugin', group='qiime2.plugins'),
 EntryPoint(name='q2-gneiss', value='q2_gneiss.plugin_setup:plugin', group='qiime2.plugins'),
 EntryPoint(name='q2-longitudinal', value='q2_longitudinal.plugin_setup:plugin', group='qiime2.plugins'),
 EntryPoint(name='q2-feature-classifier', value='q2_feature_classifier.plugin_setup:plugin', group='qiime2.plugins'),
 EntryPoint(name='q2-vsearch', value='q2_vsearch.plugin_setup:plugin', group='qiime2.plugins'),
 EntryPoint(name='q2-feature-table', value='q2_feature_table.plugin_setup:plugin', group='qiime2.plugins'),
 EntryPoint(name='q2-fragment-insertion', value='q2_fragment_insertion.plugin_setup:plugin', group='qiime2.plugins'),
 EntryPoint(name='q2-types', value='q2_types.plugin_setup:plugin', group='qiime2.plugins'),
 EntryPoint(name='q2-taxa', value='q2_taxa.plugin_setup:plugin', group='qiime2.plugins'),
 EntryPoint(name='q2-emperor', value='q2_emperor.plugin_setup:plugin', group='qiime2.plugins'),
 EntryPoint(name='q2-sample-classifier', value='q2_sample_classifier.plugin_setup:plugin', group='qiime2.plugins'),
 EntryPoint(name='q2-alignment', value='q2_alignment.plugin_setup:plugin', group='qiime2.plugins'),
 EntryPoint(name='q2-diversity', value='q2_diversity.plugin_setup:plugin', group='qiime2.plugins'),
 EntryPoint(name='q2-metadata', value='q2_metadata.plugin_setup:plugin', group='qiime2.plugins'),
 EntryPoint(name='q2-dada2', value='q2_dada2.plugin_setup:plugin', group='qiime2.plugins'),
 EntryPoint(name='q2-phylogeny', value='q2_phylogeny.plugin_setup:plugin', group='qiime2.plugins'),
 EntryPoint(name='q2-quality-control', value='q2_quality_control.plugin_setup:plugin', group='qiime2.plugins'))

```

which seems to be correct.

## Discussion

Since I am sure there is more that is connected to the PluginManager, do you think that would be reasonable? It seems like it can be used as a drop-in replacement but I might be wrong.

Thanks and keep up the great work!

---

<div class="post-metadata">

**Author:** ![thermokarst](https://forum.qiime2.org/letter_avatar_proxy/v4/letter/t/8e7dd6/32.png) [@thermokarst](https://forum.qiime2.org/u/thermokarst)\
**Post date:** [February 3, 2021, 2:16am UTC](https://forum.qiime2.org/t/discovering-plugins-via-importlib-instead-of-pkg-resources/18358/2 "2021-02-03T02:16:11Z")

</div>



---

<div class="post-metadata">

**Author:** ![ebolyen](https://forum.qiime2.org/user_avatar/forum.qiime2.org/ebolyen/32/11_2.png) [@ebolyen](https://forum.qiime2.org/u/ebolyen)\
**Post date:** [February 5, 2021, 5:46pm UTC](https://forum.qiime2.org/t/discovering-plugins-via-importlib-instead-of-pkg-resources/18358/3 "2021-02-05T17:46:26Z")

</div>

Hey @cdiener!

Thanks for the heads up, this actually looks like a pretty trivial change to make, we aren't doing that much:

> <https://github.com/qiime2/qiime2/blob/66ea209e136fc2e1a69268cdba81d02e1130d137/qiime2/sdk/plugin_manager.py#L78-L81>

It looks like the new EntryPoint object has basically the same API:

the only real change is `entry_point.dist` appears to now be `metadata(name)` but we were only using that to get a name, so we can probably just use the `name` attribute now.

* * *

As an immediate workaround, there is now an `add_plugin()` method on the plugin manager which you could use to set one up manually, it would look a bit like this:

```python
def _hack_in_the_plugins():
    import qiime2.sdk as sdk
    from importlib_metadata import entry_points
    
    pm = sdk.PluginManager(add_plugins=False)
    for entry in entry_points()["qiime2.plugins"]:
        plugin = entry.load()
        package = entry.value.split(':')[0].split('.')[0]
        pm.add_plugin(plugin, package, entry.name)

_hack_in_the_plugins()

from qiime2.plugins import feature_table # or however

```

---

<div class="post-metadata">

**Author:** ![ebolyen](https://forum.qiime2.org/user_avatar/forum.qiime2.org/ebolyen/32/11_2.png) [@ebolyen](https://forum.qiime2.org/u/ebolyen)\
**Post date:** [February 5, 2021, 5:54pm UTC](https://forum.qiime2.org/t/discovering-plugins-via-importlib-instead-of-pkg-resources/18358/4 "2021-02-05T17:54:41Z")

</div>

Oh, importantly, the PluginManager is a singleton, so you can set it up basically however you need to at the start, and then anything else that needs that will end up with the same instance later on (which prevents passing this object around to everything).

---

<div class="post-metadata">

**Author:** ![ebolyen](https://forum.qiime2.org/user_avatar/forum.qiime2.org/ebolyen/32/11_2.png) [@ebolyen](https://forum.qiime2.org/u/ebolyen)\
**Post date:** [February 5, 2021, 5:54pm UTC](https://forum.qiime2.org/t/discovering-plugins-via-importlib-instead-of-pkg-resources/18358/5 "2021-02-05T17:54:51Z")

</div>



---

<div class="post-metadata">

**Author:** ![cdiener](https://forum.qiime2.org/user_avatar/forum.qiime2.org/cdiener/32/7026_2.png) [@cdiener](https://forum.qiime2.org/u/cdiener)\
**Post date:** [February 5, 2021, 6:23pm UTC](https://forum.qiime2.org/t/discovering-plugins-via-importlib-instead-of-pkg-resources/18358/6 "2021-02-05T18:23:44Z")

</div>

Awesome I'll add the workaround to our setup script, thanks!