Multiple processes and QIIME 2 2022.8

Hello,

In 2022.2 and prior versions, it was possible to have multiple QIIME processes use the same input file. This doesn't seem to be the case with 2022.8, and it appears that processes may conflict with each other.

First, we'll start 8 processes using the exact same input each time and wait for completion.

(qiime2-2022.8) 15:45:01 ([email protected]):[email protected]$ for i in $(seq 1 8); do qiime diversity alpha --i-table raw.minfeat.mindepth.even.biom.qza --p-metric shannon --o-alpha-diversity foobar-${i} &
> done
[1] 37097
[2] 37098
[3] 37099
[4] 37100
[5] 37101
[6] 37102
[7] 37103
[8] 37104
(qiime2-2022.8) 15:45:07 ([email protected]):[email protected]$ wait

This is where it gets perplexing. A single command completes successfully, while the rest produce an error saying the input file isn't a valid QIIME 2 artifact:

(qiime2-2022.8) 15:45:07 ([email protected]):[email protected]$ wait
Plugin error from diversity:

  [Errno 2] No such file or directory: '/tmp/qiime2/dtmcdonald/data/8b14075f-ce90-4492-bed0-efa14e2a1be1/provenance'

Debug info has been saved to /tmp/qiime2-q2cli-err-slxm04lb.log
Plugin error from diversity:

  [Errno 2] No such file or directory: '/tmp/qiime2/dtmcdonald/data/8b14075f-ce90-4492-bed0-efa14e2a1be1/provenance'

Debug info has been saved to /tmp/qiime2-q2cli-err-2_33nn_9.log
Plugin error from diversity:

  [Errno 2] No such file or directory: '/tmp/qiime2/dtmcdonald/data/8b14075f-ce90-4492-bed0-efa14e2a1be1/provenance'

Debug info has been saved to /tmp/qiime2-q2cli-err-6s1o9sy4.log
Plugin error from diversity:

  [Errno 2] No such file or directory: '/tmp/qiime2/dtmcdonald/data/8b14075f-ce90-4492-bed0-efa14e2a1be1/provenance'

Debug info has been saved to /tmp/qiime2-q2cli-err-7fq3ncs1.log
Usage: qiime diversity alpha [OPTIONS]

  Computes a user-specified alpha diversity metric for all samples in a
  feature table.

Inputs:
  --i-table ARTIFACT FeatureTable[Frequency | RelativeFrequency |
    PresenceAbsence]   The feature table containing the samples for which
                       alpha diversity should be computed.          [required]
Parameters:
  --p-metric TEXT Choices('ace', 'berger_parker_d', 'brillouin_d', 'chao1',
    'chao1_ci', 'dominance', 'doubles', 'enspie', 'esty_ci', 'fisher_alpha',
    'gini_index', 'goods_coverage', 'heip_e', 'kempton_taylor_q',
    'lladser_pe', 'margalef', 'mcintosh_d', 'mcintosh_e', 'menhinick',
    'michaelis_menten_fit', 'observed_features', 'osd', 'pielou_e', 'robbins',
    'shannon', 'simpson', 'simpson_e', 'singles', 'strong')
                       The alpha diversity metric to be computed. Information
                       about specific metrics is available at
                       https://data.qiime2.org/a_diversity_metrics  [required]
Outputs:
  --o-alpha-diversity ARTIFACT SampleData[AlphaDiversity]
                       Vector containing per-sample alpha diversities.
                                                                    [required]
Miscellaneous:
  --output-dir PATH    Output unspecified results to a directory
  --verbose / --quiet  Display verbose output to stdout and/or stderr during
                       execution of this action. Or silence output if
                       execution is successful (silence is golden).
  --example-data PATH  Write example data and exit.
  --citations          Show citations and exit.
  --help               Show this message and exit.

                    There was a problem with the command:                     
 (1/1) Invalid value for '--i-table': 'raw.minfeat.mindepth.even.biom.qza' is
  not a QIIME 2 Artifact (.qza)
Usage: qiime diversity alpha [OPTIONS]

  Computes a user-specified alpha diversity metric for all samples in a
  feature table.

Inputs:
  --i-table ARTIFACT FeatureTable[Frequency | RelativeFrequency |
    PresenceAbsence]   The feature table containing the samples for which
                       alpha diversity should be computed.          [required]
Parameters:
  --p-metric TEXT Choices('ace', 'berger_parker_d', 'brillouin_d', 'chao1',
    'chao1_ci', 'dominance', 'doubles', 'enspie', 'esty_ci', 'fisher_alpha',
    'gini_index', 'goods_coverage', 'heip_e', 'kempton_taylor_q',
    'lladser_pe', 'margalef', 'mcintosh_d', 'mcintosh_e', 'menhinick',
    'michaelis_menten_fit', 'observed_features', 'osd', 'pielou_e', 'robbins',
    'shannon', 'simpson', 'simpson_e', 'singles', 'strong')
                       The alpha diversity metric to be computed. Information
                       about specific metrics is available at
                       https://data.qiime2.org/a_diversity_metrics  [required]
Outputs:
  --o-alpha-diversity ARTIFACT SampleData[AlphaDiversity]
                       Vector containing per-sample alpha diversities.
                                                                    [required]
Miscellaneous:
  --output-dir PATH    Output unspecified results to a directory
  --verbose / --quiet  Display verbose output to stdout and/or stderr during
                       execution of this action. Or silence output if
                       execution is successful (silence is golden).
  --example-data PATH  Write example data and exit.
  --citations          Show citations and exit.
  --help               Show this message and exit.

                    There was a problem with the command:                     
 (1/1) Invalid value for '--i-table': 'raw.minfeat.mindepth.even.biom.qza' is
  not a QIIME 2 Artifact (.qza)
Usage: qiime diversity alpha [OPTIONS]

  Computes a user-specified alpha diversity metric for all samples in a
  feature table.

Inputs:
  --i-table ARTIFACT FeatureTable[Frequency | RelativeFrequency |
    PresenceAbsence]   The feature table containing the samples for which
                       alpha diversity should be computed.          [required]
Parameters:
  --p-metric TEXT Choices('ace', 'berger_parker_d', 'brillouin_d', 'chao1',
    'chao1_ci', 'dominance', 'doubles', 'enspie', 'esty_ci', 'fisher_alpha',
    'gini_index', 'goods_coverage', 'heip_e', 'kempton_taylor_q',
    'lladser_pe', 'margalef', 'mcintosh_d', 'mcintosh_e', 'menhinick',
    'michaelis_menten_fit', 'observed_features', 'osd', 'pielou_e', 'robbins',
    'shannon', 'simpson', 'simpson_e', 'singles', 'strong')
                       The alpha diversity metric to be computed. Information
                       about specific metrics is available at
                       https://data.qiime2.org/a_diversity_metrics  [required]
Outputs:
  --o-alpha-diversity ARTIFACT SampleData[AlphaDiversity]
                       Vector containing per-sample alpha diversities.
                                                                    [required]
Miscellaneous:
  --output-dir PATH    Output unspecified results to a directory
  --verbose / --quiet  Display verbose output to stdout and/or stderr during
                       execution of this action. Or silence output if
                       execution is successful (silence is golden).
  --example-data PATH  Write example data and exit.
  --citations          Show citations and exit.
  --help               Show this message and exit.

                    There was a problem with the command:                     
 (1/1) Invalid value for '--i-table': 'raw.minfeat.mindepth.even.biom.qza' is
  not a QIIME 2 Artifact (.qza)
[1]   Exit 1                  qiime diversity alpha --i-table raw.minfeat.mindepth.even.biom.qza --p-metric shannon --o-alpha-diversity foobar-${i}
[2]   Exit 1                  qiime diversity alpha --i-table raw.minfeat.mindepth.even.biom.qza --p-metric shannon --o-alpha-diversity foobar-${i}
[3]   Exit 1                  qiime diversity alpha --i-table raw.minfeat.mindepth.even.biom.qza --p-metric shannon --o-alpha-diversity foobar-${i}
[4]   Exit 1                  qiime diversity alpha --i-table raw.minfeat.mindepth.even.biom.qza --p-metric shannon --o-alpha-diversity foobar-${i}
[8]+  Exit 1                  qiime diversity alpha --i-table raw.minfeat.mindepth.even.biom.qza --p-metric shannon --o-alpha-diversity foobar-${i}
Saved SampleData[AlphaDiversity] to: foobar-5.qza
[5]   Done                    qiime diversity alpha --i-table raw.minfeat.mindepth.even.biom.qza --p-metric shannon --o-alpha-diversity foobar-${i}
[6]-  Exit 1                  qiime diversity alpha --i-table raw.minfeat.mindepth.even.biom.qza --p-metric shannon --o-alpha-diversity foobar-${i}
[7]+  Exit 1                  qiime diversity alpha --i-table raw.minfeat.mindepth.even.biom.qza --p-metric shannon --o-alpha-diversity foobar-${i}

And one of the failures debug info:

(qiime2-2022.8) 15:45:22 ([email protected]):[email protected]$ cat /tmp/qiime2-q2cli-err-slxm04lb.log
Traceback (most recent call last):
  File "/home/mcdonadt/miniconda3/envs/qiime2-2022.8/lib/python3.8/site-packages/q2cli/commands.py", line 339, in __call__
    results = action(**arguments)
  File "<decorator-gen-367>", line 2, in alpha
  File "/home/mcdonadt/miniconda3/envs/qiime2-2022.8/lib/python3.8/site-packages/qiime2/sdk/action.py", line 210, in bound_callable
    provenance.add_input(name, artifact)
  File "/home/mcdonadt/miniconda3/envs/qiime2-2022.8/lib/python3.8/site-packages/qiime2/core/archive/provenance.py", line 425, in add_input
    self.inputs[name] = self.add_ancestor(input)
  File "/home/mcdonadt/miniconda3/envs/qiime2-2022.8/lib/python3.8/site-packages/qiime2/core/archive/provenance.py", line 167, in add_ancestor
    shutil.copytree(
  File "/home/mcdonadt/miniconda3/envs/qiime2-2022.8/lib/python3.8/shutil.py", line 555, in copytree
    with os.scandir(src) as itr:
FileNotFoundError: [Errno 2] No such file or directory: '/tmp/qiime2/dtmcdonald/data/8b14075f-ce90-4492-bed0-efa14e2a1be1/provenance'

Any ideas?

Best,
Daniel

1 Like

Hello @wasade, just to be sure in regards to this and TMPDIR and QIIME2 2022.8, are you running on the latest patch of QIIME 2 2022.8, 2022.8.3? I am unable to reproduce this error on the latest patch using the table.qza from moving pictures.

1 Like

Hi @Oddant1, it looks like 2022.8.0:

(qiime2-2022.8) 16:15:55 ([email protected]):[email protected]$ qiime --version
q2cli version 2022.8.0
Run `qiime info` for more version details.

Sorry @wasade, can you run qiime info and check the QIIME 2 version in that info dump? qiime --version shows the version of the cli which wasn't patched so should be 2022.8.0.

Ah, thanks :slight_smile: Seems to be 2022.8.3

$ qiime info
System versions
Python version: 3.8.13
QIIME 2 release: 2022.8
QIIME 2 version: 2022.8.3
q2cli version: 2022.8.0

Installed plugins
alignment: 2022.8.0
composition: 2022.8.0
cutadapt: 2022.8.0
dada2: 2022.8.0
deblur: 2022.8.0
demux: 2022.8.0
diversity: 2022.8.0
diversity-lib: 2022.8.0
emperor: 2022.8.0
feature-classifier: 2022.8.0
feature-table: 2022.8.1
fragment-insertion: 2022.8.0
gneiss: 2022.8.0
longitudinal: 2022.8.0
metadata: 2022.8.0
phylogeny: 2022.8.0
quality-control: 2022.8.0
quality-filter: 2022.8.0
sample-classifier: 2022.8.0
taxa: 2022.8.0
types: 2022.8.0
vsearch: 2022.8.0

Application config directory
/home/mcdonadt/miniconda3/envs/qiime2-2022.8/var/q2cli

Getting help
To get help with QIIME 2, visit https://qiime2.org

Yep, that looks correct. We are still unable to produce the error locally. Can you try doing a dev install of the framework with the function qiime2.core.util.set_permissions turned into a no-op and see if that changes anything for either of your issues? We are wondering if the issue is in some way related to how your file system handles permissions.

EDIT : We were able to finally reproduce this issue, we aren't sure about the other one though.

1 Like

Would it still help to do the noop?

I think so, at least in regards to the PanFS system.

That seemed to work

Our admin was surprised to hear about permissions issues specific to panfs so very curious to know what you find out!

(qiime2-2022.8.3) 16:53:45 ([email protected]):[email protected]$ export TMPDIR=$PANFS/tmp
(qiime2-2022.8.3) 16:53:48 ([email protected]):[email protected]$ qiime diversity beta --i-table raw.minfeat.mindepth.even.biom.qza --o-distan
ce-matrix foobar.qza --p-metric braycurtis
Saved DistanceMatrix to: foobar.qza

Can you run that a few more times to be sure? We're fighting some stochastic behavior reproducing this thread, and fear the other thread may have similar issues.

1 Like

Running 10x. FWIW I haven't observed stochastic behavior on our end, so I wonder if also related to load.

1 Like

If it's just the permission setting in PanFS for the other one, then that suggests it's returning something other than FileNotFound, (FileNotFound is expected sometimes) and which is probably pretty easy to work around (for example, maybe we just have to pass on permission error as well).

The reasoning for setting permissions at all was to prevent poorly behaved plugins from ruining long-lived artifacts once the Cache is fully released. So we set everything in the Cache's data/ directory containing artifacts to read-only when it's not being actively removed.

1 Like

10x runs all worked. I need to close for the night, are there other suggested precise edits to try tomorrow?

1 Like

No, that's super helpful, thank you @wasade! We'll keep pondering on our end.

1 Like

Just an update, we believe we've solved this issue with the following changes:

@wasade, if you were able to double check that the latest master branch is working as expected, that would be very helpful.

(The issues with PanFS remain unsolved as of yet, but we're still working on it.)

1 Like

Are there instructions for installing master? I recall installing dev in the past was not totally straight forward but maybe that's changed? Earliest I can do this is Wednesday though

Best,
Daniel

1 Like

No need to install a dev environment necessarily, the current master should be perfectly compatible with the current release.

To install, run the usual git clone and then make dev (in the root with some active environment).

1 Like

Okay cool, thank you!