2023.9 Docker container missing plugins?

cjfields · January 12, 2024, 6:20pm

I ran into an odd issue when running QIIME2 through Docker using the latest release (2023.9). I should note this was pulled down using singularity, so it may have something to do with this though I haven't had issues in the past:

singularity pull docker://quay.io/qiime2/amplicon:2023.9

This pulled down the amplicon version as a .sif file. amplicon_2023.9.sif. So far, anything under qiime tools works fine, including importing sequence data and custom taxonomy, e.g.,

$ singularity exec $SINGULARITY_CACHEDIR/amplicon_2023.9.sif qiime tools import \
--input-path asvs.md5.nochim.merged.fna \
--output-path asvs.qza \
--type 'FeatureData[Sequence]'

Imported asvs.md5.nochim.merged.fna as DNASequencesDirectoryFormat to asvs.qza

However it fails when I try to use the feature classifier, in this case BLAST:

singularity exec $SINGULARITY_CACHEDIR/amplicon_2023.9.sif qiime feature-classifier classify-consensus-blast \
--i-query asvs.qza \
--i-reference-taxonomy COI-5P.full.taxonomy.qza \
--i-reference-reads COI-5P.full.sequence.qza \
--o-classification asvs.classification.qza \
--o-search-result asvs.blast6.qza

This results in the following error:

Traceback (most recent call last):
  File "/opt/conda/envs/qiime2-amplicon-2023.9/bin/qiime", line 11, in <module>
    sys.exit(qiime())
  File "/opt/conda/envs/qiime2-amplicon-2023.9/lib/python3.8/site-packages/click/core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
  File "/opt/conda/envs/qiime2-amplicon-2023.9/lib/python3.8/site-packages/click/core.py", line 1078, in main
    rv = self.invoke(ctx)
  File "/opt/conda/envs/qiime2-amplicon-2023.9/lib/python3.8/site-packages/click/core.py", line 1682, in invoke
    cmd_name, cmd, args = self.resolve_command(ctx, args)
  File "/opt/conda/envs/qiime2-amplicon-2023.9/lib/python3.8/site-packages/click/core.py", line 1729, in resolve_command
    cmd = self.get_command(ctx, cmd_name)
  File "/opt/conda/envs/qiime2-amplicon-2023.9/lib/python3.8/site-packages/q2cli/commands.py", line 100, in get_command
    plugin = self._plugin_lookup[name]
  File "/opt/conda/envs/qiime2-amplicon-2023.9/lib/python3.8/site-packages/q2cli/commands.py", line 76, in _plugin_lookup
    import q2cli.core.cache
  File "/opt/conda/envs/qiime2-amplicon-2023.9/lib/python3.8/site-packages/q2cli/core/cache.py", line 285, in <module>
    CACHE = DeploymentCache()
  File "/opt/conda/envs/qiime2-amplicon-2023.9/lib/python3.8/site-packages/q2cli/core/cache.py", line 58, in __init__
    self._cache_dir = self._get_cache_dir()
  File "/opt/conda/envs/qiime2-amplicon-2023.9/lib/python3.8/site-packages/q2cli/core/cache.py", line 85, in _get_cache_dir
    os.makedirs(cache_dir, exist_ok=True)
  File "/opt/conda/envs/qiime2-amplicon-2023.9/lib/python3.8/os.py", line 213, in makedirs
    makedirs(head, exist_ok=exist_ok)
  File "/opt/conda/envs/qiime2-amplicon-2023.9/lib/python3.8/os.py", line 213, in makedirs
    makedirs(head, exist_ok=exist_ok)
  File "/opt/conda/envs/qiime2-amplicon-2023.9/lib/python3.8/os.py", line 223, in makedirs
    mkdir(name, mode)
PermissionError: [Errno 13] Permission denied: '/home/qiime2'

It turns out, when I simply call qiime, this also fails with the same error. I can try downgrading to an older version, but I wanted to ask whether others are seeing this using Docker directly?

cjfields · January 12, 2024, 7:08pm

To add some context, qiime tools works with this:

$ singularity exec $SINGULARITY_CACHEDIR/amplicon_2023.9.sif qiime tool
Usage: qiime tools [OPTIONS] COMMAND [ARGS]...

  Tools for working with QIIME 2 files.

Options:
  --help      Show this message and exit.

Commands:
  cache-create              Create an empty cache at the given location.
  cache-fetch               Fetches an artifact out of a cache into a .qza.
  cache-garbage-collection  Runs garbage collection on the cache at the
                            specified location.
  cache-remove              Removes a given key from a cache.
  cache-status              Checks the status of the cache.
  cache-store               Stores a .qza in the cache under a key.
  cast-metadata             Designate metadata column types.
  citations                 Print citations for a QIIME 2 result.
  export                    Export data from a QIIME 2 Artifact or a
                            Visualization
  extract                   Extract a QIIME 2 Artifact or Visualization
                            archive.
  import                    Import data into a new QIIME 2 Artifact.
  inspect-metadata          Inspect columns available in metadata.
  list-formats              List the available formats.
  list-types                List the available semantic types.
  peek                      Take a peek at a QIIME 2 Artifact or
                            Visualization.
  replay-citations          Reports all citations from a QIIME 2 Artifact...
  replay-provenance         Replay provenance from a QIIME 2 Artifact...
  replay-supplement         Produces a zipfile package of useful...
  validate                  Validate data in a QIIME 2 Artifact.
  view                      View a QIIME 2 Visualization.

Based on the trace, maybe it has something to do with the cache location?

Oddant1 · January 12, 2024, 8:36pm

@cjfields The key part of that error is:

PermissionError: [Errno 13] Permission denied: '/home/qiime2'

This issue is happening when QIIME 2 attempts to cache your current deployment. This next bit may get a bit technical and unnecessary, but for your own understanding. The reason qiime tools xyz is working is because the qiime tools commands are built into QIIME 2. All of the plugins and all of the plugin commands (like qiime feature-classifier classify-consensus-blast) are dynamically generated based on what plugins you have installed.

QIIME 2 basically catalogs all installed plugins and commands and caches them. This cache is different from the cache referenced by cache-create and other such cache commands. That cache is storing artifacts this cache is storing the commands that are a part of your current QIIME 2 deployment and the only command that directly references it is qiime dev refresh-cache which prompts QIIME 2 to regenerate this cache when your deployment changes.

QIIME 2 is trying to create this cache in a directory that is under the /home/qiime2/ directory, but it does not have permission to write to the /home/qiime2/ directory.

Can you please run ls -al /home/qiime2/ and post the output here? This will allow me to view the permissions of the directory in question and the contents of it. I want this information to ensure I don't tell you to do anything silly in correcting this error.

cjfields · January 12, 2024, 9:04pm

But can it create a cache under the /home/qiime2 directory while in a non-sandboxed Docker container?

Here is the directory:

$ singularity exec $SINGULARITY_CACHEDIR/amplicon_2023.9.sif ls -la /home/qiime2/
total 1
drwxrwxrwx. 6 root     root     107 Oct 31 17:01 .
drwxr-xr-x. 1 cjfields cjfields  60 Jan 12 14:54 ..
-rwxrwxrwx. 1 root     root      56 Oct 31 17:01 .bashrc
drwxrwxrwx. 4 root     root      46 Oct 31 17:00 .cache
drwxrwsrwx. 2 root     root      39 Oct 31 16:49 .conda
-rwxrwxrwx. 1 root     root     220 Oct 31 16:49 .wget-hsts
drwxrwxrwx. 2 root     root       3 Oct 31 17:00 matplotlib
drwxrwxrwx. 3 root     root      28 Oct 31 17:01 q2cli

I suspect it has something to do with our cluster and Singularity configuration, as running this works but only if I run it from the login node (of course which is not normally permitted). The above error therefore seems to only occur when running from a worker node.

Oddant1 · January 12, 2024, 9:33pm

When you are running QIIME 2 from a "normal" conda install as opposed to through Docker it creates the cache inside of the conda enviroment's directory not directly under a home/qiime2 directory.

You will most likely need to discuss these permissions issues with the admin team of the cluster you are using. Tell them you need write access to the directory specified in the error message from a worker node while running your docker container through singularity.

I'm not intimately familiar with how singularity works, but you might try running just qiime dev refresh-cache on a login node where you have permissions (this is NOT an expensive operation at all should use less than half a gig of RAM, but I'm not sure the extent of what you are allowed to run on a login node so exercise your own judgement there) then if that works see if you can run your command as desired on a worker node.

cjfields · January 12, 2024, 9:47pm

We have a separate login node from standard users, so have a bit more flexibility. However we also don't want to overutilize the node if possible.

Running from the login node, we can't refresh the cache b/c /home/qiime2 is in the container:

$ singularity exec $SINGULARITY_CACHEDIR/amplicon_2023.9.sif qiime dev refresh-cache
QIIME is caching your current deployment for improved performance. This may take a few moments and should only happen once per deployment.
Matplotlib created a temporary config/cache directory at /tmp/matplotlib-mttbd6he because the default path (/home/qiime2/matplotlib) is not a writable directory; it is highly recommended to set the MPLCONFIGDIR environment variable to a writable directory, in particular to speed up the import of Matplotlib and to better support multiprocessing.
Traceback (most recent call last):
  File "/opt/conda/envs/qiime2-amplicon-2023.9/bin/qiime", line 11, in <module>
    sys.exit(qiime())
  File "/opt/conda/envs/qiime2-amplicon-2023.9/lib/python3.8/site-packages/click/core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
  File "/opt/conda/envs/qiime2-amplicon-2023.9/lib/python3.8/site-packages/click/core.py", line 1078, in main
    rv = self.invoke(ctx)
  File "/opt/conda/envs/qiime2-amplicon-2023.9/lib/python3.8/site-packages/click/core.py", line 1688, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/opt/conda/envs/qiime2-amplicon-2023.9/lib/python3.8/site-packages/click/core.py", line 1688, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/opt/conda/envs/qiime2-amplicon-2023.9/lib/python3.8/site-packages/click/core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/opt/conda/envs/qiime2-amplicon-2023.9/lib/python3.8/site-packages/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
  File "/opt/conda/envs/qiime2-amplicon-2023.9/lib/python3.8/site-packages/q2cli/builtin/dev.py", line 34, in refresh_cache
    q2cli.core.cache.CACHE.refresh()
  File "/opt/conda/envs/qiime2-amplicon-2023.9/lib/python3.8/site-packages/q2cli/core/cache.py", line 76, in refresh
    self._state = self._get_cached_state(refresh=True)
  File "/opt/conda/envs/qiime2-amplicon-2023.9/lib/python3.8/site-packages/q2cli/core/cache.py", line 103, in _get_cached_state
    self._cache_current_state(current_requirements)
  File "/opt/conda/envs/qiime2-amplicon-2023.9/lib/python3.8/site-packages/q2cli/core/cache.py", line 218, in _cache_current_state
    with open(path, 'w') as fh:
OSError: [Errno 30] Read-only file system: '/home/qiime2/q2cli/cache/state.json'

I can try a sandboxed version of the container to see what happens.

Oddant1 · January 12, 2024, 9:53pm

Ok yeah that makes sense. I would try the sanboxed container and if that doesn't work talk to the admin team about the permissions.

cjfields · January 12, 2024, 10:57pm

@Oddant1 we sorted it out, there is a conflict of the /home directory in the container vs the /home on the cluster. So the workaround is to not bind the host file system (default is to bind it); then we rebind /home elsewhere:

singularity exec --no-mount hostfs --bind /home:/newhome $SINGULARITY_CACHEDIR/amplicon_2023.9.sif qiime

My feeling is this will need to be tweaked to get it working precisely, but just wanted to make sure to leave this here in case others run into it.

system · February 13, 2024, 4:57am

This topic was automatically closed 31 days after the last reply. New replies are no longer allowed.