q2-boots and Parsl testing

[colinbrislawn (Colin J Brislawn)]

When running boots with --parallel, I get an error

! qiime boots core-metrics \
 --i-table     results_paper3_2024_1/dada2_f240_r240/table.qza \
 --i-phylogeny results_paper3_2024_1/dada2_f240_r240/sepp_tree.qza \
 --m-metadata-file haberg-metadata-cache.qza \
 --p-sampling-depth 5000 --p-n 10 \
 --parallel \
 --p-replacement \
 --output-dir results_paper3_2024_1/dada2_f240_r240/boots_withreplacement_10at5k
# 2.5 mins single thread, 

Plugin error from boots:
cannot copy tree '/tmp/qiime2-provenance-2s3t2ri8': not a directory
Debug info has been saved to /tmp/qiime2-q2cli-err-jm5py2mw.log

Here are no folder with that name in my temp:

(qiime2-amplicon-2024.5) colin@cymis-2:~/github-cymis/barnes-haberg$ ll -t /tmp/ | grep qiime2
-rw-------   1 colin colin   6096 Sep  5 19:09 qiime2-q2cli-err-73iul3k_.log
-rw-------   1 colin colin   6096 Sep  5 19:08 qiime2-q2cli-err-jm5py2mw.log
-rw-------   1 colin colin   6096 Sep  5 18:49 qiime2-q2cli-err-ehzqa0so.log
-rw-------   1 colin colin   6096 Sep  5 18:39 qiime2-q2cli-err-xy03pqb5.log
drwx------   4 colin colin   4096 Aug 29 17:43 qiime2-provenance-4whwmsh3/
drwx------   4 colin colin   4096 Aug 29 17:32 qiime2-provenance-37uv38kf/

When I remove the --parallel argument it works fine!

This is my first time using parsl and I am using the default config.

Watching top, I can see it running many processes, so I think the issue is after calculations while writing the output

[gregcaporaso]

@colinbrislawn, it looks like you're running in a 2024.5 release environment. Is that right? Would you mind installing q2-boots according to the 2024.10 development version instructions? We updated the default parsl config in the development version.

Also, can you tell us a little more about the system that you're running this on? Single node/computer, or a multi-node system?

[colinbrislawn (Colin J Brislawn)]

System versions
Python version: 3.9.19
QIIME 2 release: 2024.10
QIIME 2 version: 2024.10.0.dev0+12.g3581493
q2cli version: 2024.10.0.dev0+5.g76b8d1e

Installed plugins
alignment: 2024.10.0.dev0
boots: 2024.5.beta+2.g96b7692
composition: 2024.10.0.dev0
cutadapt: 2024.10.0.dev0
dada2: 2024.10.0.dev0+2.g7d89c3b
deblur: 2024.10.0.dev0
demux: 2024.10.0.dev0+2.gc13cafb
diversity: 2024.10.0.dev0+2.gc81c7a0
diversity-lib: 2024.10.0.dev0+1.gbd7e767
emperor: 2024.10.0.dev0
...

cat ~/miniconda3/envs/q2-boots-2024.10.dev/etc/qiime2_config.toml 
[parsl]
strategy = "None"

[[parsl.executors]]
class = "ThreadPoolExecutor"
label = "tpool"
max_threads = 7

[[parsl.executors]]
class = "HighThroughputExecutor"
label = "default"
max_workers = 7

[parsl.executors.provider]
class = "LocalProvider"

currently crashing code block:

! rm -rf results_paper3_2024_1/dada2_f240_r240/bootstrap_100at5k
! qiime boots core-metrics \
 --i-table     results_paper3_2024_1/dada2_f240_r240/table.qza \
 --i-phylogeny results_paper3_2024_1/dada2_f240_r240/sepp_tree.qza \
 --m-metadata-file haberg-metadata-cache.qza \
 --p-sampling-depth 5000 --p-n 100 \
 --parallel \
 --p-replacement \
 --output-dir results_paper3_2024_1/dada2_f240_r240/bootstrap_100at5k

Computer: Cloud VM, 16 GB memory, 4 cores w/ 8 threads, linux-x86

[cherman2 (Chloe Herman)]

I actually think this is the same as the issue michal and I are having with Kraken.

This is a known parsl issue. A work around is described here:

@Oddant1

@misialq yeah this is an issue that has been cropping up more and more. I know where it's coming from, but I'm not sure why it suddenly started happening every single time (this did make it much easier to track down what was happening though). I'm aiming to have it entirely resolved in the next release.

There is a work around that should be easy for you to implement. It involves cracking open the version of the framework you're using and making a small code change.

Go here and just put a return None at the top of that method. This will leave some qiime2-provenance-<random string> dirs littered around your temp dir. The problem is that the lifecycles of these directories aren't being tracked properly when running with multiple processes.

[Oddant1 (Anthony Simard)]

@cherman2 was correct about this

[colinbrislawn (Colin J Brislawn)]

Github cross ref

Re nest parallelize

Thank you all for your help. And so fast, too!