qiime info FileNotFoundError on 2024.10 amplicon Docker

User Support
1

Hello,
I just pull the latest 2024.10 amplicon docker image and I got an error when running the qiime info command, here's the command result

docker run -t -i -v $(pwd):/data Quay qiime info
System versions
Python version: 3.10.14
QIIME 2 release: 2024.10
QIIME 2 version: 2024.10.1
q2cli version: 2024.10.1

Installed plugins
alignment: 2024.10.0
composition: 2024.10.0
cutadapt: 2024.10.0
dada2: 2024.10.0
deblur: 2024.10.0
demux: 2024.10.0
diversity: 2024.10.0
diversity-lib: 2024.10.0
emperor: 2024.10.0
feature-classifier: 2024.10.0
feature-table: 2024.10.0
fragment-insertion: 2024.10.0
longitudinal: 2024.10.0
metadata: 2024.10.0
phylogeny: 2024.10.0
quality-control: 2024.10.0
quality-filter: 2024.10.0
rescript: 2024.10.0
sample-classifier: 2024.10.0
stats: 0+unknown
taxa: 2024.10.0
types: 2024.10.0
vizard: 0.0.1.dev0
vsearch: 2024.10.0

Application config directory
/home/qiime2/q2cli

Config
Traceback (most recent call last):
File "/opt/conda/envs/qiime2-amplicon-2024.10/bin/qiime", line 11, in
sys.exit(qiime())
File "/opt/conda/envs/qiime2-amplicon-2024.10/lib/python3.10/site-packages/click/core.py", line 1157, in call
return self.main(*args, **kwargs)
File "/opt/conda/envs/qiime2-amplicon-2024.10/lib/python3.10/site-packages/click/core.py", line 1078, in main
rv = self.invoke(ctx)
File "/opt/conda/envs/qiime2-amplicon-2024.10/lib/python3.10/site-packages/click/core.py", line 1688, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/opt/conda/envs/qiime2-amplicon-2024.10/lib/python3.10/site-packages/click/core.py", line 1434, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/opt/conda/envs/qiime2-amplicon-2024.10/lib/python3.10/site-packages/click/core.py", line 783, in invoke
return __callback(*args, **kwargs)
File "/opt/conda/envs/qiime2-amplicon-2024.10/lib/python3.10/site-packages/q2cli/builtin/info.py", line 68, in info
get_vendored_config()
File "/opt/conda/envs/qiime2-amplicon-2024.10/lib/python3.10/site-packages/qiime2/sdk/parallel_config.py", line 106, in get_vendored_config
config_fp = _get_vendored_config_path()
File "/opt/conda/envs/qiime2-amplicon-2024.10/lib/python3.10/site-packages/qiime2/sdk/parallel_config.py", line 168, in _get_vendored_config_path
with open(VENDORED_FP, 'w') as fh:
FileNotFoundError: [Errno 2] No such file or directory: '/home/qiime2/.conda/qiime2-amplicon-2024.10/etc/qiime2_config.toml'

This bug was not present in amplicon 2024.5 docker image.

Thank you !

3

Hey @Patg13,

Welcome to the :qiime2: forum!

Thanks for reaching out and bringing this to our attention! I was able to reproduce this on my end, and you're absolutely right - this is only an issue in 2024.10. I'm working on a fix for this as we speak, and will circle back as soon as it's been resolved. Thanks so much for your patience!

2 Likes
4

Hey @Patg13,

Thanks for your patience! This issue has been fixed - go ahead and pull the updated 2024.10 amplicon image, and you should be all set.

Cheers :lizard:

1 Like
6

Hi, I tested the new image, it's working...But something is different, normally, I bootstrap this docker image with Apptainer to generate a frozen SIF image, which I use on systems that does not support Docker, but now when I launch any qiime2 command, I hit this error
Traceback (most recent call last):
File "/opt/conda/envs/qiime2-amplicon-2024.10/bin/qiime", line 11, in
sys.exit(qiime())
File "/opt/conda/envs/qiime2-amplicon-2024.10/lib/python3.10/site-packages/click/core.py", line 1157, in call
return self.main(*args, **kwargs)
File "/opt/conda/envs/qiime2-amplicon-2024.10/lib/python3.10/site-packages/click/core.py", line 1077, in main
with self.make_context(prog_name, args, **extra) as ctx:
File "/opt/conda/envs/qiime2-amplicon-2024.10/lib/python3.10/site-packages/click/core.py", line 943, in make_context
self.parse_args(ctx, args)
File "/opt/conda/envs/qiime2-amplicon-2024.10/lib/python3.10/site-packages/q2cli/click/command.py", line 43, in parse_args
return super().parse_args(ctx, args)
File "/opt/conda/envs/qiime2-amplicon-2024.10/lib/python3.10/site-packages/click/core.py", line 1641, in parse_args
echo(ctx.get_help(), color=ctx.color)
File "/opt/conda/envs/qiime2-amplicon-2024.10/lib/python3.10/site-packages/click/core.py", line 704, in get_help
return self.command.get_help(self)
File "/opt/conda/envs/qiime2-amplicon-2024.10/lib/python3.10/site-packages/click/core.py", line 1325, in get_help
self.format_help(ctx, formatter)
File "/opt/conda/envs/qiime2-amplicon-2024.10/lib/python3.10/site-packages/click/core.py", line 1358, in format_help
self.format_options(ctx, formatter)
File "/opt/conda/envs/qiime2-amplicon-2024.10/lib/python3.10/site-packages/q2cli/click/command.py", line 157, in format_options
for subcommand in self.list_commands(ctx):
File "/opt/conda/envs/qiime2-amplicon-2024.10/lib/python3.10/site-packages/q2cli/commands.py", line 92, in list_commands
plugins = sorted(self._plugin_lookup)
File "/opt/conda/envs/qiime2-amplicon-2024.10/lib/python3.10/site-packages/q2cli/commands.py", line 76, in _plugin_lookup
import q2cli.core.cache
File "/opt/conda/envs/qiime2-amplicon-2024.10/lib/python3.10/site-packages/q2cli/core/cache.py", line 285, in
CACHE = DeploymentCache()
File "/opt/conda/envs/qiime2-amplicon-2024.10/lib/python3.10/site-packages/q2cli/core/cache.py", line 58, in init
self._cache_dir = self._get_cache_dir()
File "/opt/conda/envs/qiime2-amplicon-2024.10/lib/python3.10/site-packages/q2cli/core/cache.py", line 85, in _get_cache_dir
os.makedirs(cache_dir, exist_ok=True)
File "/opt/conda/envs/qiime2-amplicon-2024.10/lib/python3.10/os.py", line 215, in makedirs
makedirs(head, exist_ok=exist_ok)
File "/opt/conda/envs/qiime2-amplicon-2024.10/lib/python3.10/os.py", line 225, in makedirs
mkdir(name, mode)
OSError: [Errno 30] Read-only file system: '/home/qiime2/q2cli'

I'm not sure what changed, but since /home/qiime2 is inside the container, it can only be written during the building phase. I tried to force cache refresh during the build, but it didn't help. Do you have any ideas about why it's reacting differently than before ?

Thank you

9

Hey @Patg13,

Sorry for the delayed reply!

Would you mind sharing your Apptainer definition file? I'll see if I can reproduce this on my end. For full transparency I don't have much experience with Apptainer - but I'm curious if this is a general issue in Apptainer containers, or if there's any additional customization you have that might be causing this.

Cheers :lizard:

11

Apptainer_Qiime2_NRCan.recipe.txt (3.3 KB)
Here's my definition file, thank you very much for your help !
FYI, It is the same definition file I used for 2024.5 and it was working fine.

13

Hi,

I did some test on my Apptainer to try to find a reason for this bug and if I copy the folder of /opt/conda/envs/qiime2-amplicon-2024.10/var/q2cli to /home/qiime2/q2cli, Qiime2 work correctly, it look like something is overriding the Application config directory during the apptainer container building, but I can't seems to find what it is.

In the Amplicon Docker image v2024.5, the Application config directory was also /opt/conda/envs/qiime2-amplicon-2024.10/var/q2cli, but the /home/qiime2/q2cli was also present, it's not the case in 2024.10.

14

Hey @Patg13,

Sorry for the delay (x2)!

Thanks for sharing that, and thanks for doing some extra digging - we did in fact move the location for the config directory, which was actually the source of the original error you reported. Our conda prefix was previously set as follows:

ENV CONDA_PREFIX /home/qiime2/.conda/qiime2-${DISTRO}-${EPOCH}/

This was working up until this release (and for full disclosure, it's unclear exactly what changed with the Docker container creation process) but when I poked around in the 2024.10 environment, the location where the conda prefix needed to be set to had been changed to /opt/conda/envs/qiime2-${DISTRO}-${EPOCH}/.

So I'm guessing your configuration within Apptainer is expecting this previous configuration (based on what you shared above). I don't see anything in your definition file that is looking for that path specifically (but I'm also not very familiar with Apptainer, so I may be missing something there). I wasn't able to replicate this just using the Docker image directly - so I'm guessing there's something unique going on with your configuration that may need to be adjusted. I would try either including this copy command in your definition file (if that's possible) or modifying the expected config directory (again if that's something that's possible).

Hopefully one of these options will be a feasible solution for you - and any other @moderators more familiar with Apptainer than me are welcome to jump in with any suggestions!

Cheers :lizard:

2 Likes
16

I tried to force the CONDA_PREFIX in my Apptainer container, but it does not change the result (Application config dir is still /home/qiime/q2cli), but I think I found out why.

I checked the q2cli source code ( and found this: /opt/conda/envs/qiime2-amplicon-2024.10/lib/python3.10/site-packages/q2cli/utils.py)

def get_app_dir():
    import os
    conda_prefix = os.environ.get('CONDA_PREFIX')
    if conda_prefix is not None and os.access(conda_prefix, os.W_OK | os.X_OK):
        return os.path.join(conda_prefix, 'var', 'q2cli')
    else:
        import click
        return click.get_app_dir('q2cli', roaming=False)

If I understand correctly, the function is checking if $CONDA_PREFIX folder is writable with os.access(conda_prefix, os.W_OK | os.X_OK) and if it's not, it will fallback on /home/qiime/q2cli and if it does not exist, it will be created.

The problem here is that this function is called each time the get_app_dir function is executed (during qiime info for exemple), but in Apptainer, every folders are only writable at build time. After that, the container is frozen in read-only. In Docker, that is not the case, the image is static, but the container in itself is not read only and can be written at any time (so no problem in that case).

That would explain why I can launch qiime info during build (which display the correct folder /opt/conda/envs/qiime2-amplicon-2024.10/var/q2cli, but cannot do so during runtime since it's frozen and it fallback on /home/qiime/q2cli and then crash because it cannot create the directory since it's also read-only in Apptainer.

As a temporary fix, I copied the /opt/conda/envs/qiime2-amplicon-2024.10/var/q2cli to /home/qiime/q2cli, but I don't know if it could negatively affect qiime during runtime.

Do you have any inputs ?

As a possible solution, would it be possible to modify the get_app_dir functions (and other similar functions) to only check if a folder is writable when it need to be modify (if only read is needed, do a check if folder is accessible) ?

1 Like
17

Hi @Patg13,

Well that makes really great sense. Thanks for taking the deep dive on it @Patg13!

That should be totally fine for now. We'll see about modifying things on our end to avoid this behavior on a read-only system.

Tracking issue: CLI help cache lookup fails when conda environment goes to read-only. · Issue #356 · qiime2/q2cli · GitHub

1 Like
18

That's great, thank you so much for this !
Looking forward to the next version.

19

This topic was automatically closed 31 days after the last reply. New replies are no longer allowed.