Hello,
In 2022.2 and prior versions, it was possible to have multiple QIIME processes use the same input file. This doesn't seem to be the case with 2022.8, and it appears that processes may conflict with each other.
First, we'll start 8 processes using the exact same input each time and wait for completion.
(qiime2-2022.8) 15:45:01 (dtmcdonald@b2-001):10317@main$ for i in $(seq 1 8); do qiime diversity alpha --i-table raw.minfeat.mindepth.even.biom.qza --p-metric shannon --o-alpha-diversity foobar-${i} &
> done
[1] 37097
[2] 37098
[3] 37099
[4] 37100
[5] 37101
[6] 37102
[7] 37103
[8] 37104
(qiime2-2022.8) 15:45:07 (dtmcdonald@b2-001):10317@main$ wait
This is where it gets perplexing. A single command completes successfully, while the rest produce an error saying the input file isn't a valid QIIME 2 artifact:
(qiime2-2022.8) 15:45:07 (dtmcdonald@b2-001):10317@main$ wait
Plugin error from diversity:
[Errno 2] No such file or directory: '/tmp/qiime2/dtmcdonald/data/8b14075f-ce90-4492-bed0-efa14e2a1be1/provenance'
Debug info has been saved to /tmp/qiime2-q2cli-err-slxm04lb.log
Plugin error from diversity:
[Errno 2] No such file or directory: '/tmp/qiime2/dtmcdonald/data/8b14075f-ce90-4492-bed0-efa14e2a1be1/provenance'
Debug info has been saved to /tmp/qiime2-q2cli-err-2_33nn_9.log
Plugin error from diversity:
[Errno 2] No such file or directory: '/tmp/qiime2/dtmcdonald/data/8b14075f-ce90-4492-bed0-efa14e2a1be1/provenance'
Debug info has been saved to /tmp/qiime2-q2cli-err-6s1o9sy4.log
Plugin error from diversity:
[Errno 2] No such file or directory: '/tmp/qiime2/dtmcdonald/data/8b14075f-ce90-4492-bed0-efa14e2a1be1/provenance'
Debug info has been saved to /tmp/qiime2-q2cli-err-7fq3ncs1.log
Usage: qiime diversity alpha [OPTIONS]
Computes a user-specified alpha diversity metric for all samples in a
feature table.
Inputs:
--i-table ARTIFACT FeatureTable[Frequency | RelativeFrequency |
PresenceAbsence] The feature table containing the samples for which
alpha diversity should be computed. [required]
Parameters:
--p-metric TEXT Choices('ace', 'berger_parker_d', 'brillouin_d', 'chao1',
'chao1_ci', 'dominance', 'doubles', 'enspie', 'esty_ci', 'fisher_alpha',
'gini_index', 'goods_coverage', 'heip_e', 'kempton_taylor_q',
'lladser_pe', 'margalef', 'mcintosh_d', 'mcintosh_e', 'menhinick',
'michaelis_menten_fit', 'observed_features', 'osd', 'pielou_e', 'robbins',
'shannon', 'simpson', 'simpson_e', 'singles', 'strong')
The alpha diversity metric to be computed. Information
about specific metrics is available at
https://data.qiime2.org/a_diversity_metrics [required]
Outputs:
--o-alpha-diversity ARTIFACT SampleData[AlphaDiversity]
Vector containing per-sample alpha diversities.
[required]
Miscellaneous:
--output-dir PATH Output unspecified results to a directory
--verbose / --quiet Display verbose output to stdout and/or stderr during
execution of this action. Or silence output if
execution is successful (silence is golden).
--example-data PATH Write example data and exit.
--citations Show citations and exit.
--help Show this message and exit.
There was a problem with the command:
(1/1) Invalid value for '--i-table': 'raw.minfeat.mindepth.even.biom.qza' is
not a QIIME 2 Artifact (.qza)
Usage: qiime diversity alpha [OPTIONS]
Computes a user-specified alpha diversity metric for all samples in a
feature table.
Inputs:
--i-table ARTIFACT FeatureTable[Frequency | RelativeFrequency |
PresenceAbsence] The feature table containing the samples for which
alpha diversity should be computed. [required]
Parameters:
--p-metric TEXT Choices('ace', 'berger_parker_d', 'brillouin_d', 'chao1',
'chao1_ci', 'dominance', 'doubles', 'enspie', 'esty_ci', 'fisher_alpha',
'gini_index', 'goods_coverage', 'heip_e', 'kempton_taylor_q',
'lladser_pe', 'margalef', 'mcintosh_d', 'mcintosh_e', 'menhinick',
'michaelis_menten_fit', 'observed_features', 'osd', 'pielou_e', 'robbins',
'shannon', 'simpson', 'simpson_e', 'singles', 'strong')
The alpha diversity metric to be computed. Information
about specific metrics is available at
https://data.qiime2.org/a_diversity_metrics [required]
Outputs:
--o-alpha-diversity ARTIFACT SampleData[AlphaDiversity]
Vector containing per-sample alpha diversities.
[required]
Miscellaneous:
--output-dir PATH Output unspecified results to a directory
--verbose / --quiet Display verbose output to stdout and/or stderr during
execution of this action. Or silence output if
execution is successful (silence is golden).
--example-data PATH Write example data and exit.
--citations Show citations and exit.
--help Show this message and exit.
There was a problem with the command:
(1/1) Invalid value for '--i-table': 'raw.minfeat.mindepth.even.biom.qza' is
not a QIIME 2 Artifact (.qza)
Usage: qiime diversity alpha [OPTIONS]
Computes a user-specified alpha diversity metric for all samples in a
feature table.
Inputs:
--i-table ARTIFACT FeatureTable[Frequency | RelativeFrequency |
PresenceAbsence] The feature table containing the samples for which
alpha diversity should be computed. [required]
Parameters:
--p-metric TEXT Choices('ace', 'berger_parker_d', 'brillouin_d', 'chao1',
'chao1_ci', 'dominance', 'doubles', 'enspie', 'esty_ci', 'fisher_alpha',
'gini_index', 'goods_coverage', 'heip_e', 'kempton_taylor_q',
'lladser_pe', 'margalef', 'mcintosh_d', 'mcintosh_e', 'menhinick',
'michaelis_menten_fit', 'observed_features', 'osd', 'pielou_e', 'robbins',
'shannon', 'simpson', 'simpson_e', 'singles', 'strong')
The alpha diversity metric to be computed. Information
about specific metrics is available at
https://data.qiime2.org/a_diversity_metrics [required]
Outputs:
--o-alpha-diversity ARTIFACT SampleData[AlphaDiversity]
Vector containing per-sample alpha diversities.
[required]
Miscellaneous:
--output-dir PATH Output unspecified results to a directory
--verbose / --quiet Display verbose output to stdout and/or stderr during
execution of this action. Or silence output if
execution is successful (silence is golden).
--example-data PATH Write example data and exit.
--citations Show citations and exit.
--help Show this message and exit.
There was a problem with the command:
(1/1) Invalid value for '--i-table': 'raw.minfeat.mindepth.even.biom.qza' is
not a QIIME 2 Artifact (.qza)
[1] Exit 1 qiime diversity alpha --i-table raw.minfeat.mindepth.even.biom.qza --p-metric shannon --o-alpha-diversity foobar-${i}
[2] Exit 1 qiime diversity alpha --i-table raw.minfeat.mindepth.even.biom.qza --p-metric shannon --o-alpha-diversity foobar-${i}
[3] Exit 1 qiime diversity alpha --i-table raw.minfeat.mindepth.even.biom.qza --p-metric shannon --o-alpha-diversity foobar-${i}
[4] Exit 1 qiime diversity alpha --i-table raw.minfeat.mindepth.even.biom.qza --p-metric shannon --o-alpha-diversity foobar-${i}
[8]+ Exit 1 qiime diversity alpha --i-table raw.minfeat.mindepth.even.biom.qza --p-metric shannon --o-alpha-diversity foobar-${i}
Saved SampleData[AlphaDiversity] to: foobar-5.qza
[5] Done qiime diversity alpha --i-table raw.minfeat.mindepth.even.biom.qza --p-metric shannon --o-alpha-diversity foobar-${i}
[6]- Exit 1 qiime diversity alpha --i-table raw.minfeat.mindepth.even.biom.qza --p-metric shannon --o-alpha-diversity foobar-${i}
[7]+ Exit 1 qiime diversity alpha --i-table raw.minfeat.mindepth.even.biom.qza --p-metric shannon --o-alpha-diversity foobar-${i}
And one of the failures debug info:
(qiime2-2022.8) 15:45:22 (dtmcdonald@b2-001):10317@main$ cat /tmp/qiime2-q2cli-err-slxm04lb.log
Traceback (most recent call last):
File "/home/mcdonadt/miniconda3/envs/qiime2-2022.8/lib/python3.8/site-packages/q2cli/commands.py", line 339, in __call__
results = action(**arguments)
File "<decorator-gen-367>", line 2, in alpha
File "/home/mcdonadt/miniconda3/envs/qiime2-2022.8/lib/python3.8/site-packages/qiime2/sdk/action.py", line 210, in bound_callable
provenance.add_input(name, artifact)
File "/home/mcdonadt/miniconda3/envs/qiime2-2022.8/lib/python3.8/site-packages/qiime2/core/archive/provenance.py", line 425, in add_input
self.inputs[name] = self.add_ancestor(input)
File "/home/mcdonadt/miniconda3/envs/qiime2-2022.8/lib/python3.8/site-packages/qiime2/core/archive/provenance.py", line 167, in add_ancestor
shutil.copytree(
File "/home/mcdonadt/miniconda3/envs/qiime2-2022.8/lib/python3.8/shutil.py", line 555, in copytree
with os.scandir(src) as itr:
FileNotFoundError: [Errno 2] No such file or directory: '/tmp/qiime2/dtmcdonald/data/8b14075f-ce90-4492-bed0-efa14e2a1be1/provenance'
Any ideas?
Best,
Daniel
Seems to be