TypeError with qiime info (2023.5 conda installation)

nitishnih · June 16, 2023, 5:30pm

I installed QIIME 2 2023.5 in a container (using singularity) as conda environment using the instructions posted for Linux:

wget https://data.qiime2.org/distro/core/qiime2-2023.5-py38-linux-conda.yml
conda env create -n qiime2-2023.5 --file qiime2-2023.5-py38-linux-conda.yml

There were no errors with this installation. Previous container installations have worked seamlessly, last installation on cluster being 2-2022.11. However, running qiime info or qiime --help with 2023.5 gives this error:

Error Message

QIIME is caching your current deployment for improved performance. This may take a few moments and should only happen once per deployment.
Traceback (most recent call last):
File "/opt/conda/envs/qiime2-2023.5/bin/qiime", line 11, in
sys.exit(qiime())
File "/opt/conda/envs/qiime2-2023.5/lib/python3.8/site-packages/click/core.py", line 1130, in call
return self.main(*args, **kwargs)
File "/opt/conda/envs/qiime2-2023.5/lib/python3.8/site-packages/click/core.py", line 1054, in main
with self.make_context(prog_name, args, **extra) as ctx:
File "/opt/conda/envs/qiime2-2023.5/lib/python3.8/site-packages/click/core.py", line 920, in make_context
self.parse_args(ctx, args)
File "/opt/conda/envs/qiime2-2023.5/lib/python3.8/site-packages/q2cli/click/command.py", line 43, in parse_args
return super().parse_args(ctx, args)
File "/opt/conda/envs/qiime2-2023.5/lib/python3.8/site-packages/click/core.py", line 1613, in parse_args
rest = super().parse_args(ctx, args)
File "/opt/conda/envs/qiime2-2023.5/lib/python3.8/site-packages/click/core.py", line 1378, in parse_args
value, args = param.handle_parse_result(ctx, opts, args)
File "/opt/conda/envs/qiime2-2023.5/lib/python3.8/site-packages/click/core.py", line 2360, in handle_parse_result
value = self.process_value(ctx, value)
File "/opt/conda/envs/qiime2-2023.5/lib/python3.8/site-packages/click/core.py", line 2322, in process_value
value = self.callback(ctx, self, value)
File "/opt/conda/envs/qiime2-2023.5/lib/python3.8/site-packages/click/core.py", line 1273, in show_help
echo(ctx.get_help(), color=ctx.color)
File "/opt/conda/envs/qiime2-2023.5/lib/python3.8/site-packages/click/core.py", line 699, in get_help
return self.command.get_help(self)
File "/opt/conda/envs/qiime2-2023.5/lib/python3.8/site-packages/click/core.py", line 1298, in get_help
self.format_help(ctx, formatter)
File "/opt/conda/envs/qiime2-2023.5/lib/python3.8/site-packages/click/core.py", line 1331, in format_help
self.format_options(ctx, formatter)
File "/opt/conda/envs/qiime2-2023.5/lib/python3.8/site-packages/q2cli/click/command.py", line 157, in format_options
for subcommand in self.list_commands(ctx):
File "/opt/conda/envs/qiime2-2023.5/lib/python3.8/site-packages/q2cli/commands.py", line 92, in list_commands
plugins = sorted(self._plugin_lookup)
File "/opt/conda/envs/qiime2-2023.5/lib/python3.8/site-packages/q2cli/commands.py", line 76, in _plugin_lookup
import q2cli.core.cache
File "/opt/conda/envs/qiime2-2023.5/lib/python3.8/site-packages/q2cli/core/cache.py", line 285, in
CACHE = DeploymentCache()
File "/opt/conda/envs/qiime2-2023.5/lib/python3.8/site-packages/q2cli/core/cache.py", line 61, in init
self._state = self._get_cached_state(refresh=refresh)
File "/opt/conda/envs/qiime2-2023.5/lib/python3.8/site-packages/q2cli/core/cache.py", line 107, in _get_cached_state
self._cache_current_state(current_requirements)
File "/opt/conda/envs/qiime2-2023.5/lib/python3.8/site-packages/q2cli/core/cache.py", line 205, in _cache_current_state
state = self._get_current_state()
File "/opt/conda/envs/qiime2-2023.5/lib/python3.8/site-packages/q2cli/core/cache.py", line 253, in _get_current_state
plugin_manager = q2cli.util.get_plugin_manager()
File "/opt/conda/envs/qiime2-2023.5/lib/python3.8/site-packages/q2cli/util.py", line 263, in get_plugin_manager
import qiime2.sdk
File "/opt/conda/envs/qiime2-2023.5/lib/python3.8/site-packages/qiime2/init.py", line 9, in
from qiime2.sdk import Artifact, Visualization, ResultCollection
File "/opt/conda/envs/qiime2-2023.5/lib/python3.8/site-packages/qiime2/sdk/init.py", line 9, in
from .context import Context
File "/opt/conda/envs/qiime2-2023.5/lib/python3.8/site-packages/qiime2/sdk/context.py", line 13, in
from qiime2.sdk.parallel_config import PARALLEL_CONFIG
File "/opt/conda/envs/qiime2-2023.5/lib/python3.8/site-packages/qiime2/sdk/parallel_config.py", line 40, in
VENDORED_FP = os.path.join(
File "/opt/conda/envs/qiime2-2023.5/lib/python3.8/posixpath.py", line 76, in join
a = os.fspath(a)
TypeError: expected str, bytes or os.PathLike object, not NoneType

Does this have something to do with caching? How should I troubleshoot this?

colinvwood · June 20, 2023, 3:25pm

Hello @nitishnih,

Basically what's happening is that the $CONDA_PREFIX environment variable is not set. This is a path to the root of the active conda environment. Let us know if this is enough information to troubleshoot your installation.

Thanks.

nitishnih · June 21, 2023, 4:01pm

Thanks @colinvwood

Here's the relevant section of the singularity recipe / definition file that install conda and then creates the environment

mkdir -p /opt
wget https://github.com/conda-forge/miniforge/releases/latest/download/Mambaforge-Linux-x86_64.sh
bash Mambaforge-Linux-x86_64.sh -p /opt/conda -b
rm Mambaforge-Linux-x86_64.sh
. /opt/conda/etc/profile.d/conda.sh
. /opt/conda/etc/profile.d/mamba.sh
mamba activate base
mamba update --yes --all

wget https://data.qiime2.org/distro/core/qiime2-2023.5-py38-linux-conda.yml
mamba env create -n qiime2-2023.5 --file qiime2-2023.5-py38-linux-conda.yml
mamba clean --all --yes

From my understanding, once I activate the base environment, $CONDA_PREFIX is set. I just tried this again, and added echo $CONDA_PREFIX before running the mamba env create statement and it gave the correct output: /opt/conda.

We have used this recipe for 2022.11 so I am not sure what has changed since. I'll run some more tests on my side. Let me know if you have more clues. Thank you.

colinvwood · June 21, 2023, 4:48pm

Hello @nitishnih,

From reading your traceback and the corresponding code, I can essentially guarantee you that CONDA_PREFIX was not set in whatever process environment existed. You say that echo $CONDA_PREFIX printed something before you create the qiime environment, but what about once it's activated, can you confirm this?

nitishnih · June 21, 2023, 5:33pm

Apologies, I misunderstood. I thought the installation was broken because of the CONDA_PREFIX issue.

With the container setup on our HPC cluster, we "hide" the execution of the QIIME container with a wrapper script. This doesn't activate the environment when running the process in the container environment, simply calls /opt/conda/envs/qiime2-2023.5/bin/qiime.

I guess with the new releases running qiime requires CONDA_PREFIX to be set appropriately. Is that right? I just updated the wrapper to set this and it works.

Thanks and sorry for the confusion.

colinvwood · June 21, 2023, 5:47pm

Yes, the way the code is written at the moment requires this environment variable to be set. Glad to hear that it's working now.

system · July 22, 2023, 11:48pm

This topic was automatically closed 31 days after the last reply. New replies are no longer allowed.