still default behavior after setting TMPDIR

Hi,
I experienced difficulties trying to assign TMPDIR to a large disk, similar to this post.

In brief, setting TMPDIR did not change the default behavior. I tried to debug and found that tempfile.gettempdir() in cache.py did not return TMPDIR.

  (qiime2-amplicon-2024.10) yc@yc:/mnt/research/microbiota/analysis/202504_LSG$ echo $TMPDIR 
  /mnt/tmp
  (qiime2-amplicon-2024.10) yc@yc:/mnt/research/microbiota/analysis/202504_LSG$ python
  Python 3.10.14 | packaged by conda-forge | (main, Mar 20 2024, 12:45:18) [GCC 12.3.0] on linux
  Type "help", "copyright", "credits" or "license" for more information.
  >>> import tempfile
  >>> tempfile.gettempdir()
  '/tmp'
  >>>

According to some post, tempfile.gettempdir() will validate directory assigned by TMPDIR. If the validation process failed, it will fall back to default.

I am not sure if it is the key problem causing the behavior but would ask for help whether this problem could be resolved.

Many thanks

YC

1 Like

Hi @b87401116,

Thanks for reaching out! Here are a few suggestions:

  1. Make sure you're setting TMPDIR as an environment variable not just a shell variable (TMPDIR="/mnt/tmp" sets it as a shell variable). You'll need export TMPDIR="/mnt/tmp".
  2. You'll also need to make sure /mnt/tmp exists (python doesn't like it if the dir you set as your tmp doesn't exist).
  3. You'll also need to make sure you have write access to /mnt/tmp/.

Hope this helps! Let us know if you run into any further issues.

Cheers :lizard:

1 Like

Hi @lizgehret,
Thank you for the reply. The computer is now running on a dada2 process (I modified cache.py to bypass this problem...) and I am not sure when it will finish (D3 now :laughing:). I will make sure of these details after completing the current running process.

Thank you!

2 Likes

Hi, @lizgehret,

I am back now and resumed the original cache.py.

  1. Make sure you're setting TMPDIR as an environment variable not just a shell variable (TMPDIR="/mnt/tmp" sets it as a shell variable). You'll need export TMPDIR="/mnt/tmp".

I used export TMPDIR="/mnt/tmp".

  1. You'll also need to make sure /mnt/tmp exists (python doesn't like it if the dir you set as your tmp doesn't exist).
  2. You'll also need to make sure you have write access to /mnt/tmp/

I manually created /mnt/tmp by mkdir /mnt/tmp.
Here comes the error message.

Traceback (most recent call last):
  File "/mnt/mambaforge/envs/qiime2-amplicon-2024.10/bin/qiime", line 11, in <module>
    sys.exit(qiime())
  File "/mnt/mambaforge/envs/qiime2-amplicon-2024.10/lib/python3.10/site-packages/click/core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
  File "/mnt/mambaforge/envs/qiime2-amplicon-2024.10/lib/python3.10/site-packages/click/core.py", line 1078, in main
    rv = self.invoke(ctx)
  File "/mnt/mambaforge/envs/qiime2-amplicon-2024.10/lib/python3.10/site-packages/click/core.py", line 1688, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/mnt/mambaforge/envs/qiime2-amplicon-2024.10/lib/python3.10/site-packages/click/core.py", line 1686, in invoke
    sub_ctx = cmd.make_context(cmd_name, args, parent=ctx)
  File "/mnt/mambaforge/envs/qiime2-amplicon-2024.10/lib/python3.10/site-packages/click/core.py", line 943, in make_context
    self.parse_args(ctx, args)
  File "/mnt/mambaforge/envs/qiime2-amplicon-2024.10/lib/python3.10/site-packages/q2cli/click/command.py", line 60, in parse_args
    value, args = param.handle_parse_result(ctx, opts, args)
  File "/mnt/mambaforge/envs/qiime2-amplicon-2024.10/lib/python3.10/site-packages/click/core.py", line 2400, in handle_parse_result
    value = self.process_value(ctx, value)
  File "/mnt/mambaforge/envs/qiime2-amplicon-2024.10/lib/python3.10/site-packages/q2cli/click/option.py", line 181, in process_value
    return super().process_value(ctx, value)
  File "/mnt/mambaforge/envs/qiime2-amplicon-2024.10/lib/python3.10/site-packages/click/core.py", line 2356, in process_value
    value = self.type_cast_value(ctx, value)
  File "/mnt/mambaforge/envs/qiime2-amplicon-2024.10/lib/python3.10/site-packages/q2cli/click/option.py", line 297, in type_cast_value
    value = super().type_cast_value(ctx, value)
  File "/mnt/mambaforge/envs/qiime2-amplicon-2024.10/lib/python3.10/site-packages/click/core.py", line 2344, in type_cast_value
    return convert(value)
  File "/mnt/mambaforge/envs/qiime2-amplicon-2024.10/lib/python3.10/site-packages/click/core.py", line 2316, in convert
    return self.type(value, param=self, ctx=ctx)
  File "/mnt/mambaforge/envs/qiime2-amplicon-2024.10/lib/python3.10/site-packages/click/types.py", line 83, in __call__
    return self.convert(value, param, ctx)
  File "/mnt/mambaforge/envs/qiime2-amplicon-2024.10/lib/python3.10/site-packages/q2cli/click/type.py", line 70, in convert
    with get_used_artifact_cache():
  File "/mnt/mambaforge/envs/qiime2-amplicon-2024.10/lib/python3.10/site-packages/q2cli/core/artifact_cache_global.py", line 81, in get_used_artifact_cache
    return Cache() if _USED_ARTIFACT_CACHE is None else _USED_ARTIFACT_CACHE
  File "/mnt/mambaforge/envs/qiime2-amplicon-2024.10/lib/python3.10/site-packages/qiime2/core/cache.py", line 417, in __init__
    self.__init(path=path, process_pool_lifespan=process_pool_lifespan)
  File "/mnt/mambaforge/envs/qiime2-amplicon-2024.10/lib/python3.10/site-packages/qiime2/core/cache.py", line 421, in __init
    temp_cache_path = pathlib.Path(_get_temp_path())
  File "/mnt/mambaforge/envs/qiime2-amplicon-2024.10/lib/python3.10/site-packages/qiime2/core/cache.py", line 158, in _get_temp_path
    raise ValueError(f"Directory '{cache_dir}' already exists without "
ValueError: Directory '/mnt/tmp/qiime2' already exists without proper permissions '0o41777' set. Current permissions are '0o40777.' This most likely means something other than QIIME 2 created the directory '/mnt/tmp/qiime2' or QIIME 2 failed between creating '/mnt/tmp/qiime2' and setting permissions on it.

According to the message, /mnt/tmp/qiime2 was already created by QIIME 2 (I only created /mnt/tmp) without setting proper permission while encountering the error.

I also noticed, in the source file of cache.py

"""
The cache is used to store unzipped data on disk in a predictable and user
controlled location. This allows us to skip constantly zipping and unzipping
large amounts of data and taking up CPU time when storage space is not an
issue. It also allows us to know exactly what data has been created and where.

By default, a cache will be created under $TMPDIR/qiime2/$USER and all
intermediate data created by QIIME 2 as it executes will be written into that
directory. This means QIIME 2 reserves usage of the $TMPDIR/qiime2 directory.
The user may also specify a new location to be used in place of this default
directory. This location must meet a few criteria.

**1.** It must be writable from any and all locations the QIIME 2 command
intending to use it will be running. This means that in an HPC context, the
location specified for the cache must be writable from the node QIIME 2 will be
executing on.

**2.** It must either not exist or already be a cache. The first time a
directory is specified to be used as a cache, it should not exist. QIIME 2 will
create a cache structure on disk at that location. Any existing directory you
attempt to use as a cache should have been created as a cache by QIIME 2.

"""

According to item 2, QIIME 2 accepts (or requires?) non-exisiting directory. However, setting TMPDIR in such case (not manually creating /mnt/tmp) failed to change the default behavior.

Looking forward to your help~

hi, @lizgehret,
I think I partially figured out the major problem. This is likely to occur due to filesystem. Previously I assigned TMPDIR to a filesystem of NTFS. After change the TMPDIR to another filesystem of ext4 (also manually created a directory pointed by TMPDIR), QIIME 2 was able to create qiime2 subdirectory with correct permission and is currently using the directory!

(base) yc@yc:/mnt/ex4_partition/yc/tmp$ ls -alh
total 24K
drwxrwxr-x  5 yc yc 4.0K Apr 12 03:35 .
drwxr-xr-x 22 yc yc 4.0K Apr 12 03:31 ..
drwx------  2 yc yc 4.0K Apr 12 03:35 Rtmpmfkng8
drwxrwxrwt  3 yc yc 4.0K Apr 12 03:33 qiime2
-rw-------  1 yc yc 1.2K Apr 12 03:35 qiime2-q2cli-err-pbxao4xl.log
drwx------  6 yc yc 4.0K Apr 12 03:35 tmprl7q0uu2

According to chatGPT (so please still have some doubt on this), directory in NTFS filesystem was not able to be set for appropriate permission. I tested and confirmed that chmod 1666 will do nothing on setting permission without an error message.

Hope this will help!

3 Likes

Great troubleshooting @b87401116,

I would say ChatGPT has that basically right. NTFS is a pretty different concept from many Unix based filesystems, so it probably shouldn't surprise us that these permissions are over-fit and break on NTFS.

1 Like