The flufl.lock problem is encountered during parallel multitasking

Technical Support
,
1

When I have a large number of parallel tasks at the same time, the following error will be reported:

About 1000 parallel samples,Taking one sample as an example, run the command as follows:

qiime tools import --type 'SampleData[PairedEndSequencesWithQuality]' --input-path Process/A5_1/A5_1_list.tsv --output-path Process/A5_1/A5_1_demux.qza --input-format PairedEndFastqManifestPhred33V2

qiime cutadapt trim-paired --i-demultiplexed-sequences Process/A5_1/A5_1_demux.qza --p-minimum-length 100 --p-overlap 17 --p-front-f CCTAGGTGNTTAWGCAG --p-front-r GATGACHAACCTAATCC --o-trimmed-sequences Process/A5_1/A5_1_demux-trimmed.qza --verbose

qiime vsearch merge-pairs --i-demultiplexed-seqs Process/A5_1/A5_1_demux-trimmed.qza --o-merged-sequences Process/A5_1/A5_1_merged.qza --o-unmerged-sequences Process/A5_1/A5_1_unmerged.qza

qiime quality-filter q-score --i-demux Process/A5_1/A5_1_merged.qza --o-filtered-sequences Process/A5_1/A5_1_filtered.qza --o-filter-stats Process/A5_1/A5_1_filter_stats.qza --p-min-quality 20 --p-quality-window 100 --p-max-ambiguous 5

qiime tools export --input-path Process/A5_1/A5_1_filtered.qza --output-path Process/A5_1/A5_1_demux-filtered-export

qiime demux summarize --i-data Process/A5_1/A5_1_filtered.qza --o-visualization Process/A5_1/A5_1_Clean-data-length.qzv

/public1/Softwares/vsearch-2.3.4/bin/vsearch --uchime_ref Process/A5_1/A5_1_demux-filtered-export/A5_1_L001_R1_001.fasta --db /mnt/dgfs/database/BLAST_db/RDP_GOLD/rdp_gold.fa --nonchimeras Process/A5_1/A5_1_demux-filtered-export/A5_1_no_chimera.fasta --threads 4 > Process/A5_1/A5_1_demux-filtered-export/A5_1_no_chimera.fasta.log 2>&1

During the whole process, the following error messages may be triggered when running one of the steps (each step may be possible):
Traceback (most recent call last):
File "/opt/conda/envs/qiime2-amplicon-2024.10/lib/python3.10/site-packages/q2cli/builtin/tools.py", line 852, in import
artifact = qiime2.sdk.Artifact.import_data(
File "/opt/conda/envs/qiime2-amplicon-2024.10/lib/python3.10/site-packages/qiime2/sdk/result.py", line 345, in import_data
provenance_capture = archive.ImportProvenanceCapture(format, md5sums)
File "/opt/conda/envs/qiime2-amplicon-2024.10/lib/python3.10/site-packages/qiime2/core/archive/provenance.py", line 525, in init
super().init()
File "/opt/conda/envs/qiime2-amplicon-2024.10/lib/python3.10/site-packages/qiime2/core/archive/provenance.py", line 296, in init
self._build_paths()
File "/opt/conda/envs/qiime2-amplicon-2024.10/lib/python3.10/site-packages/qiime2/core/archive/provenance.py", line 303, in _build_paths
self.path = qiime2.core.path.ProvenancePath()
File "/opt/conda/envs/qiime2-amplicon-2024.10/lib/python3.10/site-packages/qiime2/core/path.py", line 138, in new
cache = get_cache()
File "/opt/conda/envs/qiime2-amplicon-2024.10/lib/python3.10/site-packages/qiime2/core/cache.py", line 113, in get_cache
_CACHE.temp_cache = Cache()
File "/opt/conda/envs/qiime2-amplicon-2024.10/lib/python3.10/site-packages/qiime2/core/cache.py", line 417, in init
self.__init(path=path, process_pool_lifespan=process_pool_lifespan)
File "/opt/conda/envs/qiime2-amplicon-2024.10/lib/python3.10/site-packages/qiime2/core/cache.py", line 457, in __init
with self.lock:
File "/opt/conda/envs/qiime2-amplicon-2024.10/lib/python3.10/site-packages/qiime2/core/cache.py", line 304, in exit
self.flufl_lock.unlock()
File "/opt/conda/envs/qiime2-amplicon-2024.10/lib/python3.10/site-packages/flufl/lock/_lockfile.py", line 417, in unlock
raise NotLockedError('Already unlocked')
flufl.lock._lockfile.NotLockedError: Already unlocked

This problem is prone to occur when running large sample sizes, and it may not occur in all samples. When running around 1000 parallel samples, this error may occur in a small subset of samples, and sometimes the exception may not be displayed after re running.

1 Like
2

Hi @sqzhou,

Thank you for the report. We've run into this situation internally a few times and have been trying to get a workable solution going. We nearly had a fix in this last release, but ran into issues with our Mac runners and had to postpone it.

Basically what is occurring is we set a lock on the cache (in your case I think it must be the default temp cache) so that the bookkeeping of references is kept consistent. This cache has a timeout of 10 minutes after which a process is permitted to break the lock and take over. Once the original process comes back to the lock, it sees it was broken and does not posses the lock and raises the error you see.

Our 10 minute timeout was intended to be orders of magnitude longer than we needed, however there is one operation, garbage collection, which can take quite a bit longer when there are many thousands of artifacts in a cache. This is almost certainly what is happening here. The solution would be to set up multiple caches and partition your work into different caches. This is in no way convenient or desired, but it is a workaround for the moment. I think you can also set --no-recycle to reduce the amount of references we keep around, which is mostly what slows down garbage collection.

Hopefully soon we'll have a more robust answer here. Ideally garbage collection and reference tracking would only take seconds to perform, so we've got a ways to go before this works like we want it to.

2 Likes
3

@Oddant1 has just pointed out that it looks like this is occurring prior to garbage collection, so we're investigating further as we're somehow spending 10 minutes in a relatively benign section of the cache setup.

2 Likes
4

Hi ebolyen,

Thank you very much for your reply and possible solutions. I hope that this problem can be solved as soon as possible. In my many tests, this situation seems to be triggered when a large number of parallel tasks are running, but a large number of computing resources are required when running a large sample size. Ordinary macOS users and Windows users cannot provide such a huge amount of computing power. Therefore, I think it is possible to solve this problem for Linux users first. Thank you again for your reply.

5

Hello @sqzhou,

A few questions that might help us figure out what's going on.

Can you please describe your compute environment? It sounds like you are using a Linux HPC cluster, but can you please describe the flavor of Linux and the filesystem you are using? We have had issues previously with some non-standard filesystem types.

You say you are running a large number of parallel tasks, but I'm not seeing any parallelization within any of the commands you are running. Do you mean you are running the above commands many different times on different data all at once on the same system?

Finally, can you please run the command ls /<your temp dir>/qiime2/<your username> and post the results here?

Thank you




P.S. Some additional context in case you're interested. As @ebolyen said, we only allow a given process to hold the lock for 10 minutes before we allow another process to come in and break the lock. This is to ensure that if a process dies while holding the lock and doesn't release it, another process will still be able to take over the lock instead of the dead process holding it until someone goes and manually removes the lock file.

If process A is holding the lock long enough for it to time out and process B comes and takes the lock, you will see this error when process A tries to release the lock. In practice, we had only ever seen this under very specific circumstances related to running garbage collection on a large cache (think several terabytes) on a slow file system; however, given where your traceback indicates the error is occurring, it appears as though the process is somehow spending 10 minutes on an operation that should only involve reading a few small files.

2 Likes
6

Hello @Oddant1 ，
Yes, I use a large computing cluster for work, and I use a container environment for analysis. The QIMME2 version information is 2024.10, and the container source is the official Docker image (docker pull quay. io/QIIME2/example: 2024.10). After downloading, the image is converted to a sif container for subsequent analysis.
The Linux version information is Rocky Linux 8.10 (Green Observer). The above does not provide all the commands for parallel tasks. Yes, I run the above commands on a thousand samples at the same time, and only one sample command is taken as an example.
The information of ls qiime2/MetaSeq is as follows:

image
image1599×264 19.6 KB

Thank you for your help.

2 Likes
7

As you said, some of these steps involve relatively small amount of calculation, and locks should not be held for a long time. I also have some other ideas about whether it is possible that the number of tasks is too large, and the process of generating the tmp file is not random enough, which leads to repeated tmp names and error messages; Or the task has a queuing time. When the tmp file path has been generated after the task is delivered, but the task is in the queue due to insufficient computing resources. If the time exceeds 10 minutes, the lock will be released and other tasks will take over.

3 Likes
9

We agree it most likely has something to do with queueing. It doesn't seem possible that the actual operations being conducted during the locking period you are seeing the error in are taking 10 minutes to complete.

One thing you can do to resolve this issue on your end is to assign each workflow for each sample a different tmpdir. Then they won't be fighting over locking the same directory. If you are deploying these workflows in a job array or similar (which I would assume you are because deploying 1000 workflows manually does not sound like fun), then this should be easy enough to do programmatically by exporting the TMPDIR environment variable to a different location for each job. I would recommend setting the TMPDIRs to different subdirectories of the default TMPDIR. Please let me know if you have any trouble with this

We have some ideas for how to resolve or at least mitigate this issue going forward in future QIIME 2 releases.

1 Like
11

Thank you for the report and the support from the devs.

I'm hoping to test a Dockerized deployment of Qiime2 soon so I appreciate everyone else who is already using it.

:spouting_whale: :package: :qiime2:

12

Well, I would be very grateful if you could provide a specific code example for each workflow of each sample to assign different tmpdir.

13

To give you a specific example, I will need to see how you're deploying the jobs. In general, you will need to put the following at the top of your bash script.

export TMPDIR=/path/to/your/temp/directory

In your case, I would probably do something like:

export TMPDIR=/<current tmp dir>/<sample_name>

This should cause each job to look at a different directory instead of all fighting over the same one.

How exactly to accomplish this depends on how you're deploying the jobs, but I would assume the sample name, the A5_1 part of the example you sent, is being parameterized somehow.

You can also most likely mitigate the error by not running all of the sequences at once and doing smaller batches instead.

14

My current running mode is as follows, taking A_2 sample as an example:
srun -p all -c 1 singularity exec --bind /public1 qiime2_2024.10.1.sif perl process_qc.pl -s A_2 -m 100 -l 17 -r1 CCTACGGGNGGCWGCAG -r2 GACTACHVGGGTWTCTAATCC -q 20 -w 100 -a 5
Then through process_qc.pl, the specific execution of each small step is as follows:

my ($sample,$minimum_length,$overlap,$adapter1,$adapter2,$min_quality,$quality_window,$max_ambiguous)

sub prosessRun{
qiime tools import --type 'SampleData[PairedEndSequencesWithQuality]' --input-path Process/$sample/$sample\_list.tsv --output-path Process/$sample/$sample\_demux.qza --input-format PairedEndFastqManifestPhred33V2;

`qiime cutadapt trim-paired --i-demultiplexed-sequences Process/$sample/$sample\_demux.qza --p-minimum-length $minimum_length --p-overlap $overlap --p-front-f $adapter1 --p-front-r $adapter2 --o-trimmed-sequences Process/$sample/$sample\_demux-trimmed.qza --verbose > Process/$sample/$sample\_cutadapt.log 2>&1`;

`qiime vsearch merge-pairs --i-demultiplexed-seqs Process/$sample/$sample\_demux-trimmed.qza --o-merged-sequences Process/$sample/$sample\_merged.qza --o-unmerged-sequences Process/$sample/$sample\_unmerged.qza`;

`qiime quality-filter q-score --i-demux Process/$sample/$sample\_merged.qza --o-filtered-sequences Process/$sample/$sample\_filtered.qza --o-filter-stats Process/$sample/$sample\_filter_stats.qza --p-min-quality $min_quality --p-quality-window $quality_window --p-max-ambiguous $max_ambiguous`;

`qiime tools export --input-path Process/$sample/$sample\_filtered.qza --output-path Process/$sample/$sample\_demux-filtered-export`;

`qiime demux summarize --i-data Process/$sample/$sample\_filtered.qza --o-visualization Process/$sample/$sample\_Clean-data-length.qzv`;

}

In this case, it seems ineffective for me to specify export TMPDIR in the first line of each sample shell script. Maybe I should take each step apart and run it directly, for example:

srun -p all -c 1 singularity exec --bind /public1 qiime2_2024.10.1.sif qiime tools import --type 'SampleData[PairedEndSequencesWithQuality]' --input-path Process/A_2/A_2_list.tsv --output-path Process/A_2/A_2_demux.qza --input-format PairedEndFastqManifestPhred33V2