# The flufl.lock problem is encountered during parallel multitasking **URL:** \ **Category:** Technical Support\ **Tags:** cache\ **Created:** [July 16, 2025, 5:04am UTC](https://forum.qiime2.org/t/the-flufl-lock-problem-is-encountered-during-parallel-multitasking/33450 "2025-07-16T05:04:15Z")\ **Posts on this page:** 18\ **Page:** 1
**Author:** ![sqzhou](https://forum.qiime2.org/letter_avatar_proxy/v4/letter/s/6f9a4e/32.png) [@sqzhou](https://forum.qiime2.org/u/sqzhou)\ **Post date:** [July 16, 2025, 5:04am UTC](https://forum.qiime2.org/t/the-flufl-lock-problem-is-encountered-during-parallel-multitasking/33450/1 "2025-07-16T05:04:16Z")
When I have a large number of parallel tasks at the same time, the following error will be reported: About 1000 parallel samples,Taking one sample as an example, run the command as follows: qiime tools import --type 'SampleData[PairedEndSequencesWithQuality]' --input-path Process/A5\_1/A5\_1\_list.tsv --output-path Process/A5\_1/A5\_1\_demux.qza --input-format PairedEndFastqManifestPhred33V2 qiime cutadapt trim-paired --i-demultiplexed-sequences Process/A5\_1/A5\_1\_demux.qza --p-minimum-length 100 --p-overlap 17 --p-front-f CCTAGGTGNTTAWGCAG --p-front-r GATGACHAACCTAATCC --o-trimmed-sequences Process/A5\_1/A5\_1\_demux-trimmed.qza --verbose qiime vsearch merge-pairs --i-demultiplexed-seqs Process/A5\_1/A5\_1\_demux-trimmed.qza --o-merged-sequences Process/A5\_1/A5\_1\_merged.qza --o-unmerged-sequences Process/A5\_1/A5\_1\_unmerged.qza qiime quality-filter q-score --i-demux Process/A5\_1/A5\_1\_merged.qza --o-filtered-sequences Process/A5\_1/A5\_1\_filtered.qza --o-filter-stats Process/A5\_1/A5\_1\_filter\_stats.qza --p-min-quality 20 --p-quality-window 100 --p-max-ambiguous 5 qiime tools export --input-path Process/A5\_1/A5\_1\_filtered.qza --output-path Process/A5\_1/A5\_1\_demux-filtered-export qiime demux summarize --i-data Process/A5\_1/A5\_1\_filtered.qza --o-visualization Process/A5\_1/A5\_1\_Clean-data-length.qzv /public1/Softwares/vsearch-2.3.4/bin/vsearch --uchime\_ref Process/A5\_1/A5\_1\_demux-filtered-export/A5\_1\_L001\_R1\_001.fasta --db /mnt/dgfs/database/BLAST\_db/RDP\_GOLD/rdp\_gold.fa --nonchimeras Process/A5\_1/A5\_1\_demux-filtered-export/A5\_1\_no\_chimera.fasta --threads 4 \> Process/A5\_1/A5\_1\_demux-filtered-export/A5\_1\_no\_chimera.fasta.log 2\>&1 During the whole process, the following error messages may be triggered when running one of the steps (each step may be possible): Traceback (most recent call last): File "/opt/conda/envs/qiime2-amplicon-2024.10/lib/python3.10/site-packages/q2cli/builtin/tools.py", line 852, in _import artifact = qiime2.sdk.Artifact.import\_data( File "/opt/conda/envs/qiime2-amplicon-2024.10/lib/python3.10/site-packages/qiime2/sdk/result.py", line 345, in import\_data provenance\_capture = archive.ImportProvenanceCapture(format_, md5sums) File "/opt/conda/envs/qiime2-amplicon-2024.10/lib/python3.10/site-packages/qiime2/core/archive/provenance.py", line 525, in **init** super(). **init** () File "/opt/conda/envs/qiime2-amplicon-2024.10/lib/python3.10/site-packages/qiime2/core/archive/provenance.py", line 296, in **init** self.\_build\_paths() File "/opt/conda/envs/qiime2-amplicon-2024.10/lib/python3.10/site-packages/qiime2/core/archive/provenance.py", line 303, in \_build\_paths self.path = qiime2.core.path.ProvenancePath() File "/opt/conda/envs/qiime2-amplicon-2024.10/lib/python3.10/site-packages/qiime2/core/path.py", line 138, in **new** cache = get\_cache() File "/opt/conda/envs/qiime2-amplicon-2024.10/lib/python3.10/site-packages/qiime2/core/cache.py", line 113, in get\_cache \_CACHE.temp\_cache = Cache() File "/opt/conda/envs/qiime2-amplicon-2024.10/lib/python3.10/site-packages/qiime2/core/cache.py", line 417, in **init** self.\_\_init(path=path, process\_pool\_lifespan=process\_pool\_lifespan) File "/opt/conda/envs/qiime2-amplicon-2024.10/lib/python3.10/site-packages/qiime2/core/cache.py", line 457, in \_\_init with self.lock: File "/opt/conda/envs/qiime2-amplicon-2024.10/lib/python3.10/site-packages/qiime2/core/cache.py", line 304, in **exit** self.flufl\_lock.unlock() File "/opt/conda/envs/qiime2-amplicon-2024.10/lib/python3.10/site-packages/flufl/lock/\_lockfile.py", line 417, in unlock raise NotLockedError('Already unlocked') flufl.lock.\_lockfile.NotLockedError: Already unlocked This problem is prone to occur when running large sample sizes, and it may not occur in all samples. When running around 1000 parallel samples, this error may occur in a small subset of samples, and sometimes the exception may not be displayed after re running. ---
**Author:** ![ebolyen](https://forum.qiime2.org/user_avatar/forum.qiime2.org/ebolyen/32/11_2.png) [@ebolyen](https://forum.qiime2.org/u/ebolyen)\ **Post date:** [July 16, 2025, 4:39pm UTC](https://forum.qiime2.org/t/the-flufl-lock-problem-is-encountered-during-parallel-multitasking/33450/2 "2025-07-16T16:39:05Z")
Hi @sqzhou, Thank you for the report. We've run into this situation internally a few times and have been trying to get a workable solution going. We nearly had a fix in this last release, but ran into issues with our Mac runners and had to postpone it. Basically what is occurring is we set a lock on the cache (in your case I think it must be the default temp cache) so that the bookkeeping of references is kept consistent. This cache has a timeout of 10 minutes after which a process is permitted to break the lock and take over. Once the original process comes back to the lock, it sees it was broken and does not posses the lock and raises the error you see. Our 10 minute timeout was intended to be orders of magnitude longer than we needed, however there is one operation, garbage collection, which can take quite a bit longer when there are many thousands of artifacts in a cache. This is almost certainly what is happening here. The solution would be to set up multiple caches and partition your work into different caches. This is in no way convenient or desired, but it is a workaround for the moment. I think you can also set `--no-recycle` to reduce the amount of references we keep around, which is mostly what slows down garbage collection. Hopefully soon we'll have a more robust answer here. Ideally garbage collection and reference tracking would only take seconds to perform, so we've got a ways to go before this works like we want it to. ---
**Author:** ![ebolyen](https://forum.qiime2.org/user_avatar/forum.qiime2.org/ebolyen/32/11_2.png) [@ebolyen](https://forum.qiime2.org/u/ebolyen)\ **Post date:** [July 16, 2025, 4:56pm UTC](https://forum.qiime2.org/t/the-flufl-lock-problem-is-encountered-during-parallel-multitasking/33450/3 "2025-07-16T16:56:59Z")
@Oddant1 has just pointed out that it looks like this is occurring prior to garbage collection, so we're investigating further as we're somehow spending 10 minutes in a relatively benign section of the cache setup. ---
**Author:** ![sqzhou](https://forum.qiime2.org/letter_avatar_proxy/v4/letter/s/6f9a4e/32.png) [@sqzhou](https://forum.qiime2.org/u/sqzhou)\ **Post date:** [July 18, 2025, 5:06am UTC](https://forum.qiime2.org/t/the-flufl-lock-problem-is-encountered-during-parallel-multitasking/33450/4 "2025-07-18T05:06:45Z")
Hi ebolyen, Thank you very much for your reply and possible solutions. I hope that this problem can be solved as soon as possible. In my many tests, this situation seems to be triggered when a large number of parallel tasks are running, but a large number of computing resources are required when running a large sample size. Ordinary macOS users and Windows users cannot provide such a huge amount of computing power. Therefore, I think it is possible to solve this problem for Linux users first. Thank you again for your reply. ---
**Author:** ![Oddant1](https://forum.qiime2.org/user_avatar/forum.qiime2.org/oddant1/32/17130_2.png) [@Oddant1](https://forum.qiime2.org/u/Oddant1)\ **Post date:** [July 21, 2025, 6:12pm UTC](https://forum.qiime2.org/t/the-flufl-lock-problem-is-encountered-during-parallel-multitasking/33450/5 "2025-07-21T18:12:05Z")
Hello @sqzhou, A few questions that might help us figure out what's going on. Can you please describe your compute environment? It sounds like you are using a Linux HPC cluster, but can you please describe the flavor of Linux and the filesystem you are using? We have had issues previously with some non-standard filesystem types. You say you are running a large number of parallel tasks, but I'm not seeing any parallelization within any of the commands you are running. Do you mean you are running the above commands many different times on different data all at once on the same system? Finally, can you please run the command `ls //qiime2/` and post the results here? Thank you P.S. Some additional context in case you're interested. As @ebolyen said, we only allow a given process to hold the lock for 10 minutes before we allow another process to come in and break the lock. This is to ensure that if a process dies while holding the lock and doesn't release it, another process will still be able to take over the lock instead of the dead process holding it until someone goes and manually removes the lock file. If process A is holding the lock long enough for it to time out and process B comes and takes the lock, you will see this error when process A tries to release the lock. In practice, we had only ever seen this under very specific circumstances related to running garbage collection on a large cache (think several terabytes) on a slow file system; however, given where your traceback indicates the error is occurring, it appears as though the process is somehow spending 10 minutes on an operation that should only involve reading a few small files. ---
**Author:** ![sqzhou](https://forum.qiime2.org/letter_avatar_proxy/v4/letter/s/6f9a4e/32.png) [@sqzhou](https://forum.qiime2.org/u/sqzhou)\ **Post date:** [July 22, 2025, 6:25am UTC](https://forum.qiime2.org/t/the-flufl-lock-problem-is-encountered-during-parallel-multitasking/33450/6 "2025-07-22T06:25:07Z")
Hello @Oddant1 ， Yes, I use a large computing cluster for work, and I use a container environment for analysis. The QIMME2 version information is 2024.10, and the container source is the official Docker image (docker pull quay. io/QIIME2/example: 2024.10). After downloading, the image is converted to a sif container for subsequent analysis. The Linux version information is Rocky Linux 8.10 (Green Observer). The above does not provide all the commands for parallel tasks. Yes, I run the above commands on a thousand samples at the same time, and only one sample command is taken as an example. The information of ls qiime2/MetaSeq is as follows: ![image](https://forum-qiime2-org.s3.dualstack.us-west-2.amazonaws.com/original/3X/4/6/46f1ae87a4dc03e35aa67eb90457414e1e84bb36.png) Thank you for your help. ---
**Author:** ![sqzhou](https://forum.qiime2.org/letter_avatar_proxy/v4/letter/s/6f9a4e/32.png) [@sqzhou](https://forum.qiime2.org/u/sqzhou)\ **Post date:** [July 22, 2025, 6:25am UTC](https://forum.qiime2.org/t/the-flufl-lock-problem-is-encountered-during-parallel-multitasking/33450/7 "2025-07-22T06:25:17Z")
As you said, some of these steps involve relatively small amount of calculation, and locks should not be held for a long time. I also have some other ideas about whether it is possible that the number of tasks is too large, and the process of generating the tmp file is not random enough, which leads to repeated tmp names and error messages; Or the task has a queuing time. When the tmp file path has been generated after the task is delivered, but the task is in the queue due to insufficient computing resources. If the time exceeds 10 minutes, the lock will be released and other tasks will take over. ---
**Author:** ![colinvwood](https://forum.qiime2.org/user_avatar/forum.qiime2.org/colinvwood/32/17303_2.png) [@colinvwood](https://forum.qiime2.org/u/colinvwood)\ **Post date:** [July 22, 2025, 5:43pm UTC](https://forum.qiime2.org/t/the-flufl-lock-problem-is-encountered-during-parallel-multitasking/33450/8 "2025-07-22T17:43:36Z")
---
**Author:** ![Oddant1](https://forum.qiime2.org/user_avatar/forum.qiime2.org/oddant1/32/17130_2.png) [@Oddant1](https://forum.qiime2.org/u/Oddant1)\ **Post date:** [July 22, 2025, 7:15pm UTC](https://forum.qiime2.org/t/the-flufl-lock-problem-is-encountered-during-parallel-multitasking/33450/9 "2025-07-22T19:15:27Z")
We agree it most likely has something to do with queueing. It doesn't seem possible that the actual operations being conducted during the locking period you are seeing the error in are taking 10 minutes to complete. One thing you can do to resolve this issue on your end is to assign each workflow for each sample a different tmpdir. Then they won't be fighting over locking the same directory. If you are deploying these workflows in a job array or similar (which I would assume you are because deploying 1000 workflows manually does not sound like fun), then this should be easy enough to do programmatically by exporting the TMPDIR environment variable to a different location for each job. I would recommend setting the TMPDIRs to different subdirectories of the default TMPDIR. Please let me know if you have any trouble with this We have some ideas for how to resolve or at least mitigate this issue going forward in future QIIME 2 releases. ---
**Author:** ![Oddant1](https://forum.qiime2.org/user_avatar/forum.qiime2.org/oddant1/32/17130_2.png) [@Oddant1](https://forum.qiime2.org/u/Oddant1)\ **Post date:** [July 22, 2025, 7:15pm UTC](https://forum.qiime2.org/t/the-flufl-lock-problem-is-encountered-during-parallel-multitasking/33450/10 "2025-07-22T19:15:33Z")
---
**Author:** ![colinbrislawn](https://forum.qiime2.org/user_avatar/forum.qiime2.org/colinbrislawn/32/6221_2.png) [@colinbrislawn](https://forum.qiime2.org/u/colinbrislawn)\ **Post date:** [July 23, 2025, 4:45am UTC](https://forum.qiime2.org/t/the-flufl-lock-problem-is-encountered-during-parallel-multitasking/33450/11 "2025-07-23T04:45:32Z")
Thank you for the report and the support from the devs. I'm hoping to test a Dockerized deployment of Qiime2 soon so I appreciate everyone else who is already using it. 🐳 📦 :qiime2: ---
**Author:** ![sqzhou](https://forum.qiime2.org/letter_avatar_proxy/v4/letter/s/6f9a4e/32.png) [@sqzhou](https://forum.qiime2.org/u/sqzhou)\ **Post date:** [July 24, 2025, 6:53am UTC](https://forum.qiime2.org/t/the-flufl-lock-problem-is-encountered-during-parallel-multitasking/33450/12 "2025-07-24T06:53:00Z")
Well, I would be very grateful if you could provide a specific code example for each workflow of each sample to assign different tmpdir. ---
**Author:** ![Oddant1](https://forum.qiime2.org/user_avatar/forum.qiime2.org/oddant1/32/17130_2.png) [@Oddant1](https://forum.qiime2.org/u/Oddant1)\ **Post date:** [July 24, 2025, 4:43pm UTC](https://forum.qiime2.org/t/the-flufl-lock-problem-is-encountered-during-parallel-multitasking/33450/13 "2025-07-24T16:43:54Z")
To give you a specific example, I will need to see how you're deploying the jobs. In general, you will need to put the following at the top of your bash script. `export TMPDIR=/path/to/your/temp/directory` In your case, I would probably do something like: `export TMPDIR=//` This should cause each job to look at a different directory instead of all fighting over the same one. How exactly to accomplish this depends on how you're deploying the jobs, but I would assume the sample name, the `A5_1` part of the example you sent, is being parameterized somehow. You can also most likely mitigate the error by not running all of the sequences at once and doing smaller batches instead. ---
**Author:** ![sqzhou](https://forum.qiime2.org/letter_avatar_proxy/v4/letter/s/6f9a4e/32.png) [@sqzhou](https://forum.qiime2.org/u/sqzhou)\ **Post date:** [July 25, 2025, 4:54pm UTC](https://forum.qiime2.org/t/the-flufl-lock-problem-is-encountered-during-parallel-multitasking/33450/14 "2025-07-25T16:54:09Z")
My current running mode is as follows, taking A\_2 sample as an example: srun -p all -c 1 singularity exec --bind /public1 qiime2\_2024.10.1.sif perl process\_qc.pl -s A\_2 -m 100 -l 17 -r1 CCTACGGGNGGCWGCAG -r2 GACTACHVGGGTWTCTAATCC -q 20 -w 100 -a 5 Then through process\_qc.pl, the specific execution of each small step is as follows: my ($sample,$minimum\_length,$overlap,$adapter1,$adapter2,$min\_quality,$quality\_window,$max\_ambiguous) sub prosessRun{ `qiime tools import --type 'SampleData[PairedEndSequencesWithQuality]' --input-path Process/$sample/$sample\_list.tsv --output-path Process/$sample/$sample\_demux.qza --input-format PairedEndFastqManifestPhred33V2`; ``` `qiime cutadapt trim-paired --i-demultiplexed-sequences Process/$sample/$sample\_demux.qza --p-minimum-length $minimum_length --p-overlap $overlap --p-front-f $adapter1 --p-front-r $adapter2 --o-trimmed-sequences Process/$sample/$sample\_demux-trimmed.qza --verbose > Process/$sample/$sample\_cutadapt.log 2>&1`; `qiime vsearch merge-pairs --i-demultiplexed-seqs Process/$sample/$sample\_demux-trimmed.qza --o-merged-sequences Process/$sample/$sample\_merged.qza --o-unmerged-sequences Process/$sample/$sample\_unmerged.qza`; `qiime quality-filter q-score --i-demux Process/$sample/$sample\_merged.qza --o-filtered-sequences Process/$sample/$sample\_filtered.qza --o-filter-stats Process/$sample/$sample\_filter_stats.qza --p-min-quality $min_quality --p-quality-window $quality_window --p-max-ambiguous $max_ambiguous`; `qiime tools export --input-path Process/$sample/$sample\_filtered.qza --output-path Process/$sample/$sample\_demux-filtered-export`; `qiime demux summarize --i-data Process/$sample/$sample\_filtered.qza --o-visualization Process/$sample/$sample\_Clean-data-length.qzv`; ``` } In this case, it seems ineffective for me to specify export TMPDIR in the first line of each sample shell script. Maybe I should take each step apart and run it directly, for example: srun -p all -c 1 singularity exec --bind /public1 qiime2\_2024.10.1.sif qiime tools import --type 'SampleData[PairedEndSequencesWithQuality]' --input-path Process/A\_2/A\_2\_list.tsv --output-path Process/A\_2/A\_2\_demux.qza --input-format PairedEndFastqManifestPhred33V2 ---
**Author:** ![cherman2](https://forum.qiime2.org/user_avatar/forum.qiime2.org/cherman2/32/17129_2.png) [@cherman2](https://forum.qiime2.org/u/cherman2)\ **Post date:** [July 28, 2025, 5:04pm UTC](https://forum.qiime2.org/t/the-flufl-lock-problem-is-encountered-during-parallel-multitasking/33450/15 "2025-07-28T17:04:14Z")
---
**Author:** ![Oddant1](https://forum.qiime2.org/user_avatar/forum.qiime2.org/oddant1/32/17130_2.png) [@Oddant1](https://forum.qiime2.org/u/Oddant1)\ **Post date:** [July 28, 2025, 5:41pm UTC](https://forum.qiime2.org/t/the-flufl-lock-problem-is-encountered-during-parallel-multitasking/33450/16 "2025-07-28T17:41:59Z")
I believe you can put: `export TMPDIR=/tmp/$sample` at the top of your `subprocessRun` and achieve the desired result. ---
**Author:** ![Oddant1](https://forum.qiime2.org/user_avatar/forum.qiime2.org/oddant1/32/17130_2.png) [@Oddant1](https://forum.qiime2.org/u/Oddant1)\ **Post date:** [July 28, 2025, 5:42pm UTC](https://forum.qiime2.org/t/the-flufl-lock-problem-is-encountered-during-parallel-multitasking/33450/17 "2025-07-28T17:42:08Z")
---
**Author:** ![system](https://forum-qiime2-org.s3.dualstack.us-west-2.amazonaws.com/original/3X/2/1/21af5fe23cb6f4579467c66a9ed94e55274ca7bd.svg) [@system](https://forum.qiime2.org/u/system)\ **Post date:** [August 28, 2025, 11:42pm UTC](https://forum.qiime2.org/t/the-flufl-lock-problem-is-encountered-during-parallel-multitasking/33450/18 "2025-08-28T23:42:32Z")
This topic was automatically closed 31 days after the last reply. New replies are no longer allowed.