Multiple processes and QIIME 2 2022.8

Technical Support
24

No go, but got a different and puzzling result where one of the run processes claimed the input was invalid when the other seven runs completed without issue. I pulled qiime2 master at approximately 3p PT

Best,
Daniel

(qiime2-2022.8) 15:10:59 (dtmcdonald@b2-012):10317@main$ for i in `seq 1 8`; do qiime diversity alpha --i-table raw.minfeat.mindepth.even.biom.qza --o-alpha-diversity foobarbaz-${i}.qza --p-metric shannon & done
[1] 36809
[2] 36810
[3] 36811
[4] 36812
[5] 36813
[6] 36814
[7] 36815
[8] 36816
(qiime2-2022.8) 15:11:01 (dtmcdonald@b2-012):10317@main$ wait
Usage: qiime diversity alpha [OPTIONS]

  Computes a user-specified alpha diversity metric for all samples in a
  feature table.

Inputs:
  --i-table ARTIFACT FeatureTable[Frequency | RelativeFrequency |
    PresenceAbsence]   The feature table containing the samples for which
                       alpha diversity should be computed.          [required]
Parameters:
  --p-metric TEXT Choices('ace', 'berger_parker_d', 'brillouin_d', 'chao1',
    'chao1_ci', 'dominance', 'doubles', 'enspie', 'esty_ci', 'fisher_alpha',
    'gini_index', 'goods_coverage', 'heip_e', 'kempton_taylor_q',
    'lladser_pe', 'margalef', 'mcintosh_d', 'mcintosh_e', 'menhinick',
    'michaelis_menten_fit', 'observed_features', 'osd', 'pielou_e', 'robbins',
    'shannon', 'simpson', 'simpson_e', 'singles', 'strong')
                       The alpha diversity metric to be computed. Information
                       about specific metrics is available at
                       https://data.qiime2.org/a_diversity_metrics  [required]
Outputs:
  --o-alpha-diversity ARTIFACT SampleData[AlphaDiversity]
                       Vector containing per-sample alpha diversities.
                                                                    [required]
Miscellaneous:
  --output-dir PATH    Output unspecified results to a directory
  --verbose / --quiet  Display verbose output to stdout and/or stderr during
                       execution of this action. Or silence output if
                       execution is successful (silence is golden).
  --example-data PATH  Write example data and exit.
  --citations          Show citations and exit.
  --help               Show this message and exit.

                    There was a problem with the command:                     
 (1/1) Invalid value for '--i-table': 'raw.minfeat.mindepth.even.biom.qza' is
  not a QIIME 2 Artifact (.qza)
Saved SampleData[AlphaDiversity] to: foobarbaz-1.qza
Saved SampleData[AlphaDiversity] to: foobarbaz-8.qza
Saved SampleData[AlphaDiversity] to: foobarbaz-3.qza
Saved SampleData[AlphaDiversity] to: foobarbaz-4.qza
Saved SampleData[AlphaDiversity] to: foobarbaz-7.qza
[1]   Done                    qiime diversity alpha --i-table raw.minfeat.mindepth.even.biom.qza --o-alpha-diversity foobarbaz-${i}.qza --p-metric shannon
[5]   Exit 1                  qiime diversity alpha --i-table raw.minfeat.mindepth.even.biom.qza --o-alpha-diversity foobarbaz-${i}.qza --p-metric shannon
[8]+  Done                    qiime diversity alpha --i-table raw.minfeat.mindepth.even.biom.qza --o-alpha-diversity foobarbaz-${i}.qza --p-metric shannon
Saved SampleData[AlphaDiversity] to: foobarbaz-6.qza
Saved SampleData[AlphaDiversity] to: foobarbaz-2.qza
[2]   Done                    qiime diversity alpha --i-table raw.minfeat.mindepth.even.biom.qza --o-alpha-diversity foobarbaz-${i}.qza --p-metric shannon
[3]   Done                    qiime diversity alpha --i-table raw.minfeat.mindepth.even.biom.qza --o-alpha-diversity foobarbaz-${i}.qza --p-metric shannon
[4]   Done                    qiime diversity alpha --i-table raw.minfeat.mindepth.even.biom.qza --o-alpha-diversity foobarbaz-${i}.qza --p-metric shannon
[6]-  Done                    qiime diversity alpha --i-table raw.minfeat.mindepth.even.biom.qza --o-alpha-diversity foobarbaz-${i}.qza --p-metric shannon
[7]+  Done                    qiime diversity alpha --i-table raw.minfeat.mindepth.even.biom.qza --o-alpha-diversity foobarbaz-${i}.qza --p-metric shannon
25

...I just attempted to recreate this and have not been successful. However, the initial failure from execution through a loop is nonetheless a surprise

26

Thanks @wasade! It sounds like we may have found 95% of the issue then, with some as-of-yet unknown other-problem. Before, only 1 of these would have succeeded. It seems like there must still be an unguarded critical section somewhere, but now with a much smaller window than before.

It's a shame that error is basically the worst one in QIIME 2, it means virtually anything went wrong while the framework was loading that artifact.

1 Like
27

Another update, we're able to reproduce this (rarely) as well and are investigating further.

2 Likes
28

This topic was automatically closed 31 days after the last reply. New replies are no longer allowed.

29

Hey @ebolyen,

@antgonza and I were just chatting about this, and the Panfs issue. Do you know what the status of those items are?

Best,
Daniel

30

Hey @wasade, yes we should have fixed this in the current release 2022.11.

For the multiple processes:

This issue has been corrected with some better locking behavior, and initializing data destined for the cache to a temporary sub directory until it is consistent, at which point we rename in a single locked operation. This does mean that competing processes using the same UUID may do redundant work, but that's just kind of the way of things, especially since there's no guarantee any given process will succeed until it has finished.

For PanFS:

Unfortunately the best we could do was remove the read-only protection on artifact caches until we have time to figure out a better approach.

Right now I'm expecting QIIME 2 will implement it's own copytree/shutil operations which don't attempt to copy extended metadata and other often un-used components that aren't well supported on the more obscure filesystems. No ETA on that, but it shouldn't matter in the interim as we've stopped being clever about protecting anything.

2 Likes
31

Super awesome, thank you so much!

1 Like