Error during MAG dereplication in QIIME2-shotgun-2024.2 (maybe due to remove types_genomics plugin)

Technical Support
1

Hello, I have tried to MAG-based analysis for shotgun metagenomics data.
And I tried to import 'sourmash' for MAGs dereplication following the instruction in Metagenomics with QIIME2 - Spring 2024 Leiden & Zurich Tutorial.

First I got an error message that there is no plugin called 'sourmash'.
So, I installed manually it in my QIIME2-shotgun-2024.2 environment. And then, I checked that 'sourmash' is installed but I failed to import it by qiime2 command. I guess this is due to lack of 'types_genomics' plugin in my environment, so I manually installed it as following command (I didn't recognize that 'types_genomics' has been removed/merged to 'types' plugin for the 2024.2 version release.)

  1. try to install q2-types-genomics by mamba
mamba install -c conda-forge -c bioconda -c https://packages.qiime2.org/qiime2/2023.5/tested -c defaults q2-types-genomics

-> but if failed to solve environment with error message

Could not solve for environment specs
The following packages are incompatible
├─ q2-sapienns is installable and it requires
│  └─ q2-types 2024.2.* , which can be installed;
└─ q2-types-genomics is not installable because it requires
   └─ q2-types 2023.5.* , which conflicts with any installable versions previously reported.
  1. try to install q2-sourmash by pip
pip install https://github.com/dib-lab/q2-sourmash/archive/master.zip
  Preparing metadata (setup.py) ... done
Building wheels for collected packages: q2-sourmash
  Building wheel for q2-sourmash (setup.py) ... done
  Created wheel for q2-sourmash: filename=q2_sourmash-0+unknown-py3-none-any.whl size=8262 sha256=0bb195cf5e6877a153a4e94760b2a7da777131aa26cd0208ab3fc5c309ef5ac2
  Stored in directory: /tmp/pip-ephem-wheel-cache-tzxti50p/wheels/06/ef/94/aefc13235aa86475a6c51af73929775fd0148e9fe208bd5f1e
Successfully built q2-sourmash
Installing collected packages: q2-sourmash
Successfully installed q2-sourmash-0+unknown

-> after this step, qiime2 itself are not executable

QIIME is caching your current deployment for improved performance. This may take a few moments and should only happen once per deployment.
Traceback (most recent call last):
  File "/home/sbml/mambaforge/envs/qiime2-shotgun-2024.2/lib/python3.8/site-packages/q2cli/util.py", line 279, in get_plugin_manager
    return qiime2.sdk.PluginManager.reuse_existing()
  File "/home/sbml/mambaforge/envs/qiime2-shotgun-2024.2/lib/python3.8/site-packages/qiime2/sdk/plugin_manager.py", line 58, in reuse_existing
    raise UninitializedPluginManagerError
qiime2.sdk.plugin_manager.UninitializedPluginManagerError

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/sbml/mambaforge/envs/qiime2-shotgun-2024.2/bin/qiime", line 11, in <module>
    sys.exit(qiime())
  File "/home/sbml/mambaforge/envs/qiime2-shotgun-2024.2/lib/python3.8/site-packages/click/core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
  File "/home/sbml/mambaforge/envs/qiime2-shotgun-2024.2/lib/python3.8/site-packages/click/core.py", line 1077, in main
    with self.make_context(prog_name, args, **extra) as ctx:
  File "/home/sbml/mambaforge/envs/qiime2-shotgun-2024.2/lib/python3.8/site-packages/click/core.py", line 943, in make_context
    self.parse_args(ctx, args)
  File "/home/sbml/mambaforge/envs/qiime2-shotgun-2024.2/lib/python3.8/site-packages/q2cli/click/command.py", line 43, in parse_args
    return super().parse_args(ctx, args)
  File "/home/sbml/mambaforge/envs/qiime2-shotgun-2024.2/lib/python3.8/site-packages/click/core.py", line 1641, in parse_args
    echo(ctx.get_help(), color=ctx.color)
  File "/home/sbml/mambaforge/envs/qiime2-shotgun-2024.2/lib/python3.8/site-packages/click/core.py", line 704, in get_help
    return self.command.get_help(self)
  File "/home/sbml/mambaforge/envs/qiime2-shotgun-2024.2/lib/python3.8/site-packages/click/core.py", line 1325, in get_help
    self.format_help(ctx, formatter)
  File "/home/sbml/mambaforge/envs/qiime2-shotgun-2024.2/lib/python3.8/site-packages/click/core.py", line 1358, in format_help
    self.format_options(ctx, formatter)
  File "/home/sbml/mambaforge/envs/qiime2-shotgun-2024.2/lib/python3.8/site-packages/q2cli/click/command.py", line 157, in format_options
    for subcommand in self.list_commands(ctx):
  File "/home/sbml/mambaforge/envs/qiime2-shotgun-2024.2/lib/python3.8/site-packages/q2cli/commands.py", line 92, in list_commands
    plugins = sorted(self._plugin_lookup)
  File "/home/sbml/mambaforge/envs/qiime2-shotgun-2024.2/lib/python3.8/site-packages/q2cli/commands.py", line 76, in _plugin_lookup
    import q2cli.core.cache
  File "/home/sbml/mambaforge/envs/qiime2-shotgun-2024.2/lib/python3.8/site-packages/q2cli/core/cache.py", line 285, in <module>
    CACHE = DeploymentCache()
  File "/home/sbml/mambaforge/envs/qiime2-shotgun-2024.2/lib/python3.8/site-packages/q2cli/core/cache.py", line 61, in __init__
    self._state = self._get_cached_state(refresh=refresh)
  File "/home/sbml/mambaforge/envs/qiime2-shotgun-2024.2/lib/python3.8/site-packages/q2cli/core/cache.py", line 107, in _get_cached_state
    self._cache_current_state(current_requirements)
  File "/home/sbml/mambaforge/envs/qiime2-shotgun-2024.2/lib/python3.8/site-packages/q2cli/core/cache.py", line 205, in _cache_current_state
    state = self._get_current_state()
  File "/home/sbml/mambaforge/envs/qiime2-shotgun-2024.2/lib/python3.8/site-packages/q2cli/core/cache.py", line 253, in _get_current_state
    plugin_manager = q2cli.util.get_plugin_manager()
  File "/home/sbml/mambaforge/envs/qiime2-shotgun-2024.2/lib/python3.8/site-packages/q2cli/util.py", line 291, in get_plugin_manager
    return qiime2.sdk.PluginManager()
  File "/home/sbml/mambaforge/envs/qiime2-shotgun-2024.2/lib/python3.8/site-packages/qiime2/sdk/plugin_manager.py", line 67, in __new__
    self._init(add_plugins=add_plugins)
  File "/home/sbml/mambaforge/envs/qiime2-shotgun-2024.2/lib/python3.8/site-packages/qiime2/sdk/plugin_manager.py", line 105, in _init
    plugin = entry_point.load()
  File "/home/sbml/mambaforge/envs/qiime2-shotgun-2024.2/lib/python3.8/site-packages/pkg_resources/__init__.py", line 2496, in load
    return self.resolve()
  File "/home/sbml/mambaforge/envs/qiime2-shotgun-2024.2/lib/python3.8/site-packages/pkg_resources/__init__.py", line 2502, in resolve
    module = __import__(self.module_name, fromlist=['__name__'], level=0)
  File "/home/sbml/mambaforge/envs/qiime2-shotgun-2024.2/lib/python3.8/site-packages/q2_sourmash/plugin_setup.py", line 12, in <module>
    from q2_types_genomics.per_sample_data import MAGs
ModuleNotFoundError: No module named 'q2_types_genomics'
  1. So I tried to download&decompress q2-types-genomics pacakge and move to my qiime env
wget https://packages.qiime2.org/qiime2/2024.2/shotgun/passed/linux-64/q2-types-genomics-2023.11.0.dev0+8.g3b993ed-py38hd426f94_0.tar.bz2
--2024-06-02 22:45:39--  https://packages.qiime2.org/qiime2/2024.2/shotgun/passed/linux-64/q2-types-genomics-2023.11.0.dev0+8.g3b993ed-py38hd426f94_0.tar.bz2
Resolving packages.qiime2.org (packages.qiime2.org)... 100.20.96.200
접속 packages.qiime2.org (packages.qiime2.org)|100.20.96.200|:443... 접속됨.
HTTP request sent, awaiting response... 200 OK
Length: 12243803 (12M) [application/octet-stream]
Saving to: ‘q2-types-genomics-2023.11.0.dev0+8.g3b993ed-py38hd426f94_0.tar.bz2’

q2-types-genomics-2023.11.0.dev0+8.g3b993ed-py38hd426f94_0.tar.bz2            100%[=================================================================================================================================================================================================>]  11.68M  2.57MB/s    in 10s     

2024-06-02 22:45:50 (1.17 MB/s) - ‘q2-types-genomics-2023.11.0.dev0+8.g3b993ed-py38hd426f94_0.tar.bz2’ saved [12243803/12243803]

tar -jxvf q2-types-genomics-2023.11.0.dev0+8.g3b993ed-py38hd426f94_0.tar.bz2
*[Messages for installation...]*
mv lib/python3.8/site-packages/q2_types_genomics /home/sbml/mambaforge/envs/qiime2-shotgun-2024.2/lib/python3.8/site-packages/
mv lib/python3.8/site-packages/q2_types_genomics-2023.11.0.dev0+8.g3b993ed-py3.8.egg-info /home/sbml/mambaforge/envs/qiime2-shotgun-2024.2/lib/python3.8/site-packages/

And then now, I got an error message guessing 'types' and 'types-genomics' are conflicting..

qiime

QIIME is caching your current deployment for improved performance. This may take a few moments and should only happen once per deployment.

Traceback (most recent call last):
  File "/home/sbml/mambaforge/envs/qiime2-shotgun-2024.2/lib/python3.8/site-packages/q2cli/util.py", line 279, in get_plugin_manager
    return qiime2.sdk.PluginManager.reuse_existing()
  File "/home/sbml/mambaforge/envs/qiime2-shotgun-2024.2/lib/python3.8/site-packages/qiime2/sdk/plugin_manager.py", line 58, in reuse_existing
    raise UninitializedPluginManagerError
qiime2.sdk.plugin_manager.UninitializedPluginManagerError

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/sbml/mambaforge/envs/qiime2-shotgun-2024.2/bin/qiime", line 11, in <module>
    sys.exit(qiime())
  File "/home/sbml/mambaforge/envs/qiime2-shotgun-2024.2/lib/python3.8/site-packages/click/core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
  File "/home/sbml/mambaforge/envs/qiime2-shotgun-2024.2/lib/python3.8/site-packages/click/core.py", line 1077, in main
    with self.make_context(prog_name, args, **extra) as ctx:
  File "/home/sbml/mambaforge/envs/qiime2-shotgun-2024.2/lib/python3.8/site-packages/click/core.py", line 943, in make_context
    self.parse_args(ctx, args)
  File "/home/sbml/mambaforge/envs/qiime2-shotgun-2024.2/lib/python3.8/site-packages/q2cli/click/command.py", line 43, in parse_args
    return super().parse_args(ctx, args)
  File "/home/sbml/mambaforge/envs/qiime2-shotgun-2024.2/lib/python3.8/site-packages/click/core.py", line 1641, in parse_args
    echo(ctx.get_help(), color=ctx.color)
  File "/home/sbml/mambaforge/envs/qiime2-shotgun-2024.2/lib/python3.8/site-packages/click/core.py", line 704, in get_help
    return self.command.get_help(self)
  File "/home/sbml/mambaforge/envs/qiime2-shotgun-2024.2/lib/python3.8/site-packages/click/core.py", line 1325, in get_help
    self.format_help(ctx, formatter)
  File "/home/sbml/mambaforge/envs/qiime2-shotgun-2024.2/lib/python3.8/site-packages/click/core.py", line 1358, in format_help
    self.format_options(ctx, formatter)
  File "/home/sbml/mambaforge/envs/qiime2-shotgun-2024.2/lib/python3.8/site-packages/q2cli/click/command.py", line 157, in format_options
    for subcommand in self.list_commands(ctx):
  File "/home/sbml/mambaforge/envs/qiime2-shotgun-2024.2/lib/python3.8/site-packages/q2cli/commands.py", line 92, in list_commands
    plugins = sorted(self._plugin_lookup)
  File "/home/sbml/mambaforge/envs/qiime2-shotgun-2024.2/lib/python3.8/site-packages/q2cli/commands.py", line 76, in _plugin_lookup
    import q2cli.core.cache
  File "/home/sbml/mambaforge/envs/qiime2-shotgun-2024.2/lib/python3.8/site-packages/q2cli/core/cache.py", line 285, in <module>
    CACHE = DeploymentCache()
  File "/home/sbml/mambaforge/envs/qiime2-shotgun-2024.2/lib/python3.8/site-packages/q2cli/core/cache.py", line 61, in __init__
    self._state = self._get_cached_state(refresh=refresh)
  File "/home/sbml/mambaforge/envs/qiime2-shotgun-2024.2/lib/python3.8/site-packages/q2cli/core/cache.py", line 107, in _get_cached_state
    self._cache_current_state(current_requirements)
  File "/home/sbml/mambaforge/envs/qiime2-shotgun-2024.2/lib/python3.8/site-packages/q2cli/core/cache.py", line 205, in _cache_current_state
    state = self._get_current_state()
  File "/home/sbml/mambaforge/envs/qiime2-shotgun-2024.2/lib/python3.8/site-packages/q2cli/core/cache.py", line 253, in _get_current_state
    plugin_manager = q2cli.util.get_plugin_manager()
  File "/home/sbml/mambaforge/envs/qiime2-shotgun-2024.2/lib/python3.8/site-packages/q2cli/util.py", line 291, in get_plugin_manager
    return qiime2.sdk.PluginManager()
  File "/home/sbml/mambaforge/envs/qiime2-shotgun-2024.2/lib/python3.8/site-packages/qiime2/sdk/plugin_manager.py", line 67, in __new__
    self._init(add_plugins=add_plugins)
  File "/home/sbml/mambaforge/envs/qiime2-shotgun-2024.2/lib/python3.8/site-packages/qiime2/sdk/plugin_manager.py", line 107, in _init
    self.add_plugin(plugin, package, project_name,
  File "/home/sbml/mambaforge/envs/qiime2-shotgun-2024.2/lib/python3.8/site-packages/qiime2/sdk/plugin_manager.py", line 136, in add_plugin
    self._integrate_plugin(plugin)
  File "/home/sbml/mambaforge/envs/qiime2-shotgun-2024.2/lib/python3.8/site-packages/qiime2/sdk/plugin_manager.py", line 162, in _integrate_plugin
    raise ValueError("Duplicate semantic type (%r) defined in"
ValueError: Duplicate semantic type ('Kraken2Report') defined in plugins: 'types-genomics' and 'types'

So, could anyone give me some advice to use 'sourmash', 'types'(or 'types-genomics') for MAG dereplication?

2

(May not be best option but) I could use qiime2 sourmash plugin by installing another conda/mamba environment for qiime2-shotgun-2023.9 which containing types_genomics plugin.

5

Hi @Sang-Mok_Lee,

The q2-sourmash plugin is not currently compatible with the shotgun (now metagenome) distribution. This will be added to the metagenome distribution for 2024.10, but in the meantime, here's how I'd recommend installing:

Deactivate your current conda environment and then run the following command:

conda create -n q2community-2023.5 -c https://packages.qiime2.org/qiime2/2023.5/passed/community/ -c conda-forge -c bioconda qiime2-community

This will create an environment that contains the 2023.5 community distribution (which is no longer supported for current versions of QIIME 2, but which does contain all of the requisite packages/versions you'll need for sourmash).

You'll then activate this environment (conda activate q2community-2023.5) and run the following commands:

conda install -c bioconda sourmash

pip install https://github.com/dib-lab/q2-sourmash/archive/master.zip

You can confirm that sourmash was installed successfully by running qiime sourmash --help. As a sanity check, I also tested this all locally on my Mac OS (Intel) and was successful.

A friendly reminder that this is your interim solution for utilizing q2-sourmash, but that an updated version will be available in our 2024.10 release within the metagenome distribution. :slightly_smiling_face:

Hope this helps! Cheers :lizard:

3 Likes
7

Thanks for your quick reply @lizgehret !

I guess 2023.9 also maintains the q2-sourmash plugin.
(Please tell me if the version should not be used for analysis)
Anyways, I think the 2024.10 ver is suitable for not only 16S but also shotgun metagenomics data.
It will be really good news for every scientist struggling to process their metagenomics data.
Thank you for your teams' dedication to the field of metagenome research!

2 Likes
8

Oh, I mean that q2-sourmash plugin can be installed in the 2023.9 environment.

9

This topic was automatically closed 31 days after the last reply. New replies are no longer allowed.