Hi,
I am experiencing this error
[Errno 1] Operation not permitted: '/beegfs/scratch/tmp/qiime2-archive-e9lx9icb/03768735-9eeb-476e-a5ea-5a095c808e05/data/feature-table.biom' -> '/beegfs/scratch/tmp/qiime2-archive-15lwjjmz/d0094ed6-6608-48fd-b9d8-37eb9bcffe52/data/feature-table.biom'")]
Could you please help me?
May it be related to permissions to write in the specified temporary folders?
Thanks
Hi @MichelaRiba! What version of QIIME 2 are you using? We attempted to address this back in the 2021.4 release, but perhaps there is still something wrong here? If you aren't using 2021.4 or newer, please update and try again. Thanks!
Hi,
thanks a lot for the quick feedback. I have just installed
2021.8;
I am evaluating if the could be something related to one sample with very low number of sequences.
I am thinking of excluding that sample and go on.
I will tell you
Michela
I,
I have tried and excluded one sample with low number of sequences in the hypothesis that could influence the execution, however I still have problems,
I report hereafter the error I get:
Plugin error from vsearch:
[('/beegfs/scratch/tmp/qiime2-archive-vc4bl7qs/80cfb6ec-e7d0-4ebb-bf94-280d3af5a001/data/feature-table.biom', '/beegfs/scratch/tmp/qiime2-archive-tig60mkf/39192f1d-de4b-47b1-b243-26a62bc0266c/data/feature-table.biom', "[Errno 1] Operation not permitted: '/beegfs/scratch/tmp/qiime2-archive-vc4bl7qs/80cfb6ec-e7d0-4ebb-bf94-280d3af5a001/data/feature-table.biom' -> '/beegfs/scratch/tmp/qiime2-archive-tig60mkf/39192f1d-de4b-47b1-b243-26a62bc0266c/data/feature-table.biom'")]
running external command line application. This may print messages to stdout and/or stderr.
The command being run is below. This command cannot be manually re-run as it will depend on temporary files that no longer exist.Command: vsearch --usearch_global /beegfs/scratch/tmp/tmpegylvue1 --id 0.85 --db /beegfs/scratch/tmp/qiime2-archive-8fawjhsd/2f45e966-178d-4468-901f-5a8de00d8134/data/dna-sequences.fasta --uc /beegfs/scratch/tmp/tmpldr6o3b0 --strand plus --qmask none --notmatched /beegfs/scratch/tmp/tmpilvon15x --threads 1 --minseqlength 1 --fasta_width 0
vsearch v2.7.0_linux_x86_64, 377.4GB RAM, 72 cores
GitHub - torognes/vsearch: Versatile open-source tool for microbiome analysisReading file /beegfs/scratch/tmp/qiime2-archive-8fawjhsd/2f45e966-178d-4468-901f-5a8de00d8134/data/dna-sequences.fasta 100%
7276798 nt in 5088 seqs, min 1258, max 2353, avg 1430
Masking 100%
Counting k-mers 100%
Creating k-mer index 100%
Searching 100%
Matching query sequences: 1148 of 1528 (75.13%)
Running external command line application. This may print messages to stdout and/or stderr.
The command being run is below. This command cannot be manually re-run as it will depend on temporary files that no longer exist.Command: vsearch --sortbysize /beegfs/scratch/tmp/tmpilvon15x --xsize --output /beegfs/scratch/tmp/q2-DNAFASTAFormat-2uf2ow7n --minseqlength 1 --fasta_width 0
vsearch v2.7.0_linux_x86_64, 377.4GB RAM, 72 cores
GitHub - torognes/vsearch: Versatile open-source tool for microbiome analysisReading file /beegfs/scratch/tmp/tmpilvon15x 100%
159010 nt in 380 seqs, min 301, max 596, avg 418
Getting sizes 100%
Sorting 100%
Median abundance: 1
Writing output 100%
Running external command line application. This may print messages to stdout and/or stderr.
The command being run is below. This command cannot be manually re-run as it will depend on temporary files that no longer exist.Command: vsearch --cluster_size /beegfs/scratch/tmp/tmpyhsw0iyg --id 0.85 --centroids /beegfs/scratch/tmp/q2-DNAFASTAFormat-b3cbkj4f --uc /beegfs/scratch/tmp/tmp_zq_5m8b --qmask none --xsize --threads 1 --minseqlength 1 --fasta_width 0
vsearch v2.7.0_linux_x86_64, 377.4GB RAM, 72 cores
GitHub - torognes/vsearch: Versatile open-source tool for microbiome analysisReading file /beegfs/scratch/tmp/tmpyhsw0iyg 100%
159010 nt in 380 seqs, min 301, max 596, avg 418
Sorting by abundance 100%
Counting k-mers 100%
Clustering 100%
Sorting clusters 100%
Writing clusters 100%
Clusters: 256 Size min 1, max 9, avg 1.5
Singletons: 190, 50.0% of seqs, 74.2% of clusters
Traceback (most recent call last):
File "/home/riba.michela/.conda/envs/Miqiime2-2021.8/lib/python3.8/site-packages/q2cli/commands.py", line 329, in call
results = action(**arguments)
File "", line 2, in cluster_features_open_reference
File "/home/riba.michela/.conda/envs/Miqiime2-2021.8/lib/python3.8/site-packages/qiime2/sdk/action.py", line 245, in bound_callable
outputs = self.callable_executor(scope, callable_args,
File "/home/riba.michela/.conda/envs/Miqiime2-2021.8/lib/python3.8/site-packages/qiime2/sdk/action.py", line 513, in callable_executor
aliased_result = output._alias(prov)
File "/home/riba.michela/.conda/envs/Miqiime2-2021.8/lib/python3.8/site-packages/qiime2/sdk/result.py", line 199, in _alias
alias._archiver = archive.Archiver.from_data(
File "/home/riba.michela/.conda/envs/Miqiime2-2021.8/lib/python3.8/site-packages/qiime2/core/archive/archiver.py", line 316, in from_data
Format.write(rec, type, format, data_initializer, provenance_capture)
File "/home/riba.michela/.conda/envs/Miqiime2-2021.8/lib/python3.8/site-packages/qiime2/core/archive/format/v5.py", line 20, in write
super().write(archive_record, type, format, data_initializer,
File "/home/riba.michela/.conda/envs/Miqiime2-2021.8/lib/python3.8/site-packages/qiime2/core/archive/format/v1.py", line 18, in write
super().write(archive_record, type, format, data_initializer,
File "/home/riba.michela/.conda/envs/Miqiime2-2021.8/lib/python3.8/site-packages/qiime2/core/archive/format/v0.py", line 62, in write
data_initializer(data_dir)
File "/home/riba.michela/.conda/envs/Miqiime2-2021.8/lib/python3.8/site-packages/qiime2/sdk/result.py", line 194, in clone_original
shutil.copytree(str(self._archiver.data_dir), str(into),
File "/home/riba.michela/.conda/envs/Miqiime2-2021.8/lib/python3.8/shutil.py", line 557, in copytree
return _copytree(entries=entries, src=src, dst=dst, symlinks=symlinks,
File "/home/riba.michela/.conda/envs/Miqiime2-2021.8/lib/python3.8/shutil.py", line 513, in _copytree
raise Error(errors)
shutil.Error: [('/beegfs/scratch/tmp/qiime2-archive-vc4bl7qs/80cfb6ec-e7d0-4ebb-bf94-280d3af5a001/data/feature-table.biom', '/beegfs/scratch/tmp/qiime2-archive-tig60mkf/39192f1d-de4b-47b1-b243-26a62bc0266c/data/feature-table.biom', "[Errno 1] Operation not permitted: '/beegfs/scratch/tmp/qiime2-archive-vc4bl7qs/80cfb6ec-e7d0-4ebb-bf94-280d3af5a001/data/feature-table.biom' -> '/beegfs/scratch/tmp/qiime2-archive-tig60mkf/39192f1d-de4b-47b1-b243-26a62bc0266c/data/feature-table.biom'")]
Hello,
I'm a MichelaRiba Collegue. Most probably the issue is related to the creation of hard links in the _alias() method (line 189 in the file /home/riba.michela/.conda/envs/Miqiime2-2021.8/lib/python3.8/site-packages/qiime2/sdk/result.py).
189 def _alias(self, provenance_capture): 190 def clone_original(into): 191 # directory is empty, this function is meant to fix that, so we 192 # can rmdir so that copytree is happy 193 into.rmdir() 194 shutil.copytree(str(self._archiver.data_dir), str(into), 195 copy_function=os.link) # Use hardlinks
Indeed, the issue is that the run is conduced on beegfs filesystem, which does not support hard links (BeeGFS Documentation 7.4.2 — BeeGFS Documentation 7.4.2). When the run was performed on other filesystem, everything went smoothly.
I was wondering if it's possible to manage this exepction with a try/except to manage the case where the creation of a hard link is not possible but that's just an idea.
I hope this is helpful
Ilario
Hi @itagliaf, thanks for sharing - we had previously tested our earlier fixes on newer versions of BeeGFS, which would explain why we didn't run into this specific error path. We have a work-in-progress for unifying all of the different ways we can shuffle inodes around:
In the meantime, using a different filesystem is a great alternative, if possible. Thanks!
Hi @thermokarst,
that's great news.
Up to now we are using the local /tmp of each computational node just by exporting TMP_DIR and TMPDIR and it's manageble as a workaround.
Thank you for your time and effort
This topic was automatically closed 31 days after the last reply. New replies are no longer allowed.