TMPDIR and QIIME2 2022.8

Hello,

It looks like there was a change in behavior for the TMPDIR variable between QIIME 2 2022.2 and 2022.8. Identical command sets do not work across versions. This was first discovered by the standard microsetta processing pipeline that is largely unchanged for the past few versions of QIIME 2.

On our shared user environment, our admin has strongly advised we set TMPDIR to point to our scratch space as our /tmp is small (and not fast). Doing so with QIIME 2 2022.8 yields an exception:

# set a temporary directory
(qiime2-2022.8) 15:26:07 (dtmcdonald@b2-006):10317@main$ export TMPDIR=$PANFS/tmp

# run a QIIME 2 command
(qiime2-2022.8) 15:26:10 (dtmcdonald@b2-006):10317@main$ qiime diversity beta --i-table raw.minfeat.mindepth.even.biom.qza --o-distance-matrix foobar.qza --p-metric braycurtis
Plugin error from diversity:

  [('/panfs/dtmcdonald/tmp/qiime2/dtmcdonald/data/8b14075f-ce90-4492-bed0-efa14e2a1be1/provenance/metadata.yaml', '/panfs/dtmcdonald/tmp/qiime2-provenance-5hqj1mr1/artifacts/8b14075f-ce90-4492-bed0-efa14e2a1be1/metadata.yaml', "[Errno 13] Permission denied: '/panfs/dtmcdonald/tmp/qiime2-provenance-5hqj1mr1/artifacts/8b14075f-ce90-4492-bed0-efa14e2a1be1/metadata.yaml'"), ('/panfs/dtmcdonald/tmp/qiime2/dtmcdonald/data/8b14075f-ce90-4492-bed0-efa14e2a1be1/provenance/VERSION', '/panfs/dtmcdonald/tmp/qiime2-provenance-5hqj1mr1/artifacts/8b14075f-ce90-4492-bed0-efa14e2a1be1/VERSION', "[Errno 13] Permission denied: '/panfs/dtmcdonald/tmp/qiime2-provenance-5hqj1mr1/artifacts/8b14075f-ce90-4492-bed0-efa14e2a1be1/VERSION'"), ('/panfs/dtmcdonald/tmp/qiime2/dtmcdonald/data/8b14075f-ce90-4492-bed0-efa14e2a1be1/provenance/citations.bib', '/panfs/dtmcdonald/tmp/qiime2-provenance-5hqj1mr1/artifacts/8b14075f-ce90-4492-bed0-efa14e2a1be1/citations.bib', "[Errno 13] Permission denied: '/panfs/dtmcdonald/tmp/qiime2-provenance-5hqj1mr1/artifacts/8b14075f-ce90-4492-bed0-efa14e2a1be1/citations.bib'"), ('/panfs/dtmcdonald/tmp/qiime2/dtmcdonald/data/8b14075f-ce90-4492-bed0-efa14e2a1be1/provenance/action/action.yaml', '/panfs/dtmcdonald/tmp/qiime2-provenance-5hqj1mr1/artifacts/8b14075f-ce90-4492-bed0-efa14e2a1be1/action/action.yaml', "[Errno 13] Permission denied: '/panfs/dtmcdonald/tmp/qiime2-provenance-5hqj1mr1/artifacts/8b14075f-ce90-4492-bed0-efa14e2a1be1/action/action.yaml'"), (<DirEntry 'action'>, '/panfs/dtmcdonald/tmp/qiime2-provenance-5hqj1mr1/artifacts/8b14075f-ce90-4492-bed0-efa14e2a1be1/action', "[Errno 13] Permission denied: '/panfs/dtmcdonald/tmp/qiime2-provenance-5hqj1mr1/artifacts/8b14075f-ce90-4492-bed0-efa14e2a1be1/action'"), ('/panfs/dtmcdonald/tmp/qiime2/dtmcdonald/data/8b14075f-ce90-4492-bed0-efa14e2a1be1/provenance', '/panfs/dtmcdonald/tmp/qiime2-provenance-5hqj1mr1/artifacts/8b14075f-ce90-4492-bed0-efa14e2a1be1', "[Errno 13] Permission denied: '/panfs/dtmcdonald/tmp/qiime2-provenance-5hqj1mr1/artifacts/8b14075f-ce90-4492-bed0-efa14e2a1be1'")]

Debug info has been saved to /panfs/dtmcdonald/tmp/qiime2-q2cli-err-almmlvfj.log

# the permissions denied statements are I believe a red herring as I am able to  write to `$TMPDIR`, and notably, QIIME's debug info is saved there and accessible:

(qiime2-2022.8) 15:27:10 (dtmcdonald@b2-006):10317@main$ touch $TMPDIR/example
(qiime2-2022.8) 15:27:24 (dtmcdonald@b2-006):10317@main$ cat  /panfs/dtmcdonald/tmp/qiime2-q2cli-err-almmlvfj.log
Traceback (most recent call last):
  File "/home/mcdonadt/miniconda3/envs/qiime2-2022.8/lib/python3.8/site-packages/q2cli/commands.py", line 339, in __call__
    results = action(**arguments)
  File "<decorator-gen-359>", line 2, in beta
  File "/home/mcdonadt/miniconda3/envs/qiime2-2022.8/lib/python3.8/site-packages/qiime2/sdk/action.py", line 210, in bound_callable
    provenance.add_input(name, artifact)
  File "/home/mcdonadt/miniconda3/envs/qiime2-2022.8/lib/python3.8/site-packages/qiime2/core/archive/provenance.py", line 425, in add_input
    self.inputs[name] = self.add_ancestor(input)
  File "/home/mcdonadt/miniconda3/envs/qiime2-2022.8/lib/python3.8/site-packages/qiime2/core/archive/provenance.py", line 167, in add_ancestor
    shutil.copytree(
  File "/home/mcdonadt/miniconda3/envs/qiime2-2022.8/lib/python3.8/shutil.py", line 557, in copytree
    return _copytree(entries=entries, src=src, dst=dst, symlinks=symlinks,
  File "/home/mcdonadt/miniconda3/envs/qiime2-2022.8/lib/python3.8/shutil.py", line 513, in _copytree
    raise Error(errors)
shutil.Error: [('/panfs/dtmcdonald/tmp/qiime2/dtmcdonald/data/8b14075f-ce90-4492-bed0-efa14e2a1be1/provenance/metadata.yaml', '/panfs/dtmcdonald/tmp/qiime2-provenance-5hqj1mr1/artifacts/8b14075f-ce90-4492-bed0-efa14e2a1be1/metadata.yaml', "[Errno 13] Permission denied: '/panfs/dtmcdonald/tmp/qiime2-provenance-5hqj1mr1/artifacts/8b14075f-ce90-4492-bed0-efa14e2a1be1/metadata.yaml'"), ('/panfs/dtmcdonald/tmp/qiime2/dtmcdonald/data/8b14075f-ce90-4492-bed0-efa14e2a1be1/provenance/VERSION', '/panfs/dtmcdonald/tmp/qiime2-provenance-5hqj1mr1/artifacts/8b14075f-ce90-4492-bed0-efa14e2a1be1/VERSION', "[Errno 13] Permission denied: '/panfs/dtmcdonald/tmp/qiime2-provenance-5hqj1mr1/artifacts/8b14075f-ce90-4492-bed0-efa14e2a1be1/VERSION'"), ('/panfs/dtmcdonald/tmp/qiime2/dtmcdonald/data/8b14075f-ce90-4492-bed0-efa14e2a1be1/provenance/citations.bib', '/panfs/dtmcdonald/tmp/qiime2-provenance-5hqj1mr1/artifacts/8b14075f-ce90-4492-bed0-efa14e2a1be1/citations.bib', "[Errno 13] Permission denied: '/panfs/dtmcdonald/tmp/qiime2-provenance-5hqj1mr1/artifacts/8b14075f-ce90-4492-bed0-efa14e2a1be1/citations.bib'"), ('/panfs/dtmcdonald/tmp/qiime2/dtmcdonald/data/8b14075f-ce90-4492-bed0-efa14e2a1be1/provenance/action/action.yaml', '/panfs/dtmcdonald/tmp/qiime2-provenance-5hqj1mr1/artifacts/8b14075f-ce90-4492-bed0-efa14e2a1be1/action/action.yaml', "[Errno 13] Permission denied: '/panfs/dtmcdonald/tmp/qiime2-provenance-5hqj1mr1/artifacts/8b14075f-ce90-4492-bed0-efa14e2a1be1/action/action.yaml'"), (<DirEntry 'action'>, '/panfs/dtmcdonald/tmp/qiime2-provenance-5hqj1mr1/artifacts/8b14075f-ce90-4492-bed0-efa14e2a1be1/action', "[Errno 13] Permission denied: '/panfs/dtmcdonald/tmp/qiime2-provenance-5hqj1mr1/artifacts/8b14075f-ce90-4492-bed0-efa14e2a1be1/action'"), ('/panfs/dtmcdonald/tmp/qiime2/dtmcdonald/data/8b14075f-ce90-4492-bed0-efa14e2a1be1/provenance', '/panfs/dtmcdonald/tmp/qiime2-provenance-5hqj1mr1/artifacts/8b14075f-ce90-4492-bed0-efa14e2a1be1', "[Errno 13] Permission denied: '/panfs/dtmcdonald/tmp/qiime2-provenance-5hqj1mr1/artifacts/8b14075f-ce90-4492-bed0-efa14e2a1be1'")]

This error seems to be associated with the filesystem itself as setting TMPDIR=$HOME/foo works. That is not a solution though in our environment. Our scratch space is based off Panfs and I suspect there is something unusual going with that specific type of filesystem.

Any ideas?

Best,
Daniel

1 Like

For posterity most of the conversation is happening in:

https://forum.qiime2.org/t/multiple-processes-and-qiime-2-2022-8/

These are technically unrelated issues as best we can tell however.
So no need to merge topics, and we'll follow up with our solution for this issue in this thread once we have one.

1 Like

This topic was automatically closed 31 days after the last reply. New replies are no longer allowed.