Plugin error from diversity [Errno 28] No space left on device

User Support
1

Hello, I ran the following code to get the alpha rarefaction curve results.

my code：

(qiime2-2023.5) [m@localhost alpha]$ nohup \

qiime diversity alpha-rarefaction
--i-table 3.filtered_DADA2_table-run32mym1.qza
--p-max-depth 43000
--p-min-depth 1000
--p-steps 5000
--m-metadata-file 0.run32mym-metadata-sample-5.tsv
--o-visualization 3.filtered_DADA2_alpha-rarefaction-run32mym43000.qzv
--verbose &

But the following error message appears.

Plugin error from diversity:

[Errno 28] No space left on device: '/tmp/qiime2-archive-x8ar25cs/3d2fa27c-fac2-4c3b-9276-e8ced2fd1e22/provenance'

Debug info has been saved to /tmp/qiime2-q2cli-err-os5d2pjl.log
Traceback (most recent call last):
File "/miniconda3/envs/qiime2-2023.5/lib/python3.8/site-packages/q2cli/commands.py", line 468, in call
results = action(**arguments)
File "", line 2, in alpha_rarefaction
File "/miniconda3/envs/qiime2-2023.5/lib/python3.8/site-packages/qiime2/sdk/action.py", line 274, in bound_callable
outputs = self.callable_executor(
File "/miniconda3/envs/qiime2-2023.5/lib/python3.8/site-packages/qiime2/sdk/action.py", line 558, in callable_executor
ret_val = self._callable(output_dir=temp_dir, **view_args)
File "/miniconda3/envs/qiime2-2023.5/lib/python3.8/site-packages/q2_diversity/_alpha/_visualizer.py", line 375, in alpha_rarefaction
data = _compute_rarefaction_data(table, min_depth, max_depth,
File "/miniconda3/envs/qiime2-2023.5/lib/python3.8/site-packages/q2_diversity/_alpha/_visualizer.py", line 310, in _compute_rarefaction_data
vector, = alpha(table=rt, metric=metric)
File "/miniconda3/envs/qiime2-2023.5/lib/python3.8/site-packages/qiime2/sdk/context.py", line 140, in deferred_action
return action_obj._bind(
File "", line 2, in alpha
File "/miniconda3/envs/qiime2-2023.5/lib/python3.8/site-packages/qiime2/sdk/action.py", line 274, in bound_callable
outputs = self.callable_executor(
File "/miniconda3/envs/qiime2-2023.5/lib/python3.8/site-packages/qiime2/sdk/action.py", line 590, in callable_executor
outputs = self._callable(scope.ctx, **view_args)
File "/miniconda3/envs/qiime2-2023.5/lib/python3.8/site-packages/q2_diversity/_alpha/_pipeline.py", line 25, in alpha
vector, = action(table=table)
File "/miniconda3/envs/qiime2-2023.5/lib/python3.8/site-packages/qiime2/sdk/context.py", line 140, in deferred_action
return action_obj._bind(
File "", line 2, in observed_features
File "/miniconda3/envs/qiime2-2023.5/lib/python3.8/site-packages/qiime2/sdk/action.py", line 271, in bound_callable
self.signature.transform_and_add_callable_args_to_prov(
File "/miniconda3/envs/qiime2-2023.5/lib/python3.8/site-packages/qiime2/core/type/signature.py", line 390, in transform_and_add_callable_args_to_prov
self._transform_and_add_input_to_prov(
File "/miniconda3/envs/qiime2-2023.5/lib/python3.8/site-packages/qiime2/core/type/signature.py", line 405, in _transform_and_add_input_to_prov
provenance.add_input(name, _input)
File "/miniconda3/envs/qiime2-2023.5/lib/python3.8/site-packages/qiime2/core/archive/provenance.py", line 448, in add_input
self.inputs[name] = self.add_ancestor(input)
File "/miniconda3/envs/qiime2-2023.5/lib/python3.8/site-packages/qiime2/core/archive/provenance.py", line 177, in add_ancestor
shutil.copytree(str(grandcestor), str(destination))
File "/miniconda3/envs/qiime2-2023.5/lib/python3.8/shutil.py", line 557, in copytree
return _copytree(entries=entries, src=src, dst=dst, symlinks=symlinks,
File "/miniconda3/envs/qiime2-2023.5/lib/python3.8/shutil.py", line 513, in _copytree
raise Error(errors)
shutil.Error: [('/tmp/qiime2/mym/data/b98298fe-846a-4a67-a0ce-a4887485c992/provenance/artifacts/b0308b9b-deeb-4f59-84b6-c790ccd69946/action/action.yaml', '/tmp/qiime2-provenance-fozqa8bf/artifacts/b0308b9b-deeb-4f59-84b6-c790ccd69946/action/action.yaml', "[Errno 28] No space left on device: '/tmp/qiime2/mym/data/b98298fe-846a-4a67-a0ce-a4887485c992/provenance/artifacts/b0308b9b-deeb-4f59-84b6-c790ccd69946/action/action.yaml' -> '/tmp/qiime2-provenance-fozqa8bf/artifacts/b0308b9b-deeb-4f59-84b6-c790ccd69946/action/action.yaml'"), ('/tmp/qiime2/mym/data/b98298fe-846a-4a67-a0ce-a4887485c992/provenance/artifacts/b0308b9b-deeb-4f59-84b6-c790ccd69946/citations.bib', '/tmp/qiime2-provenance-fozqa8bf/artifacts/b0308b9b-deeb-4f59-84b6-c790ccd69946/citations.bib', "[Errno 28] No space left on device: '/tmp/qiime2/mym/data/b98298fe-846a-4a67-a0ce-a4887485c992/provenance/artifacts/b0308b9b-deeb-4f59-84b6-c790ccd69946/citations.bib' -> '/tmp/qiime2-provenance-fozqa8bf/artifacts/b0308b9b-deeb-4f59-84b6-c790ccd69946/citations.bib'"), ('/tmp/qiime2/mym/data/b98298fe-846a-4a67-a0ce-a4887485c992/provenance/artifacts/b0308b9b-deeb-4f59-84b6-c790ccd69946/metadata.yaml', '/tmp/qiime2-provenance-fozqa8bf/artifacts/b0308b9b-deeb-4f59-84b6-c790ccd69946/metadata.yaml', "[Errno 28] No space left on device: '/tmp/qiime2/mym/data/b98298fe-846a-4a67-a0ce-a4887485c992/provenance/artifacts/b0308b9b-deeb-4f59-84b6-c790ccd69946/metadata.yaml' -> '/tmp/qiime2-provenance-fozqa8bf/artifacts/b0308b9b-deeb-4f59-84b6-c790ccd69946/metadata.yaml'")]

Plugin error from diversity:

[('/tmp/qiime2/mym/data/b98298fe-846a-4a67-a0ce-a4887485c992/provenance/artifacts/b0308b9b-deeb-4f59-84b6-c790ccd69946/action/action.yaml', '/tmp/qiime2-provenance-fozqa8bf/artifacts/b0308b9b-deeb-4f59-84b6-c790ccd69946/action/action.yaml', "[Errno 28] No space left on device: '/tmp/qiime2/mym/data/b98298fe-846a-4a67-a0ce-a4887485c992/provenance/artifacts/b0308b9b-deeb-4f59-84b6-c790ccd69946/action/action.yaml' -> '/tmp/qiime2-provenance-fozqa8bf/artifacts/b0308b9b-deeb-4f59-84b6-c790ccd69946/action/action.yaml'"), ('/tmp/qiime2/mym/data/b98298fe-846a-4a67-a0ce-a4887485c992/provenance/artifacts/b0308b9b-deeb-4f59-84b6-c790ccd69946/citations.bib', '/tmp/qiime2-provenance-fozqa8bf/artifacts/b0308b9b-deeb-4f59-84b6-c790ccd69946/citations.bib', "[Errno 28] No space left on device: '/tmp/qiime2/mym/data/b98298fe-846a-4a67-a0ce-a4887485c992/provenance/artifacts/b0308b9b-deeb-4f59-84b6-c790ccd69946/citations.bib' -> '/tmp/qiime2-provenance-fozqa8bf/artifacts/b0308b9b-deeb-4f59-84b6-c790ccd69946/citations.bib'"), ('/tmp/qiime2/mym/data/b98298fe-846a-4a67-a0ce-a4887485c992/provenance/artifacts/b0308b9b-deeb-4f59-84b6-c790ccd69946/metadata.yaml', '/tmp/qiime2-provenance-fozqa8bf/artifacts/b0308b9b-deeb-4f59-84b6-c790ccd69946/metadata.yaml', "[Errno 28] No space left on device: '/tmp/qiime2/mym/data/b98298fe-846a-4a67-a0ce-a4887485c992/provenance/artifacts/b0308b9b-deeb-4f59-84b6-c790ccd69946/metadata.yaml' -> '/tmp/qiime2-provenance-fozqa8bf/artifacts/b0308b9b-deeb-4f59-84b6-c790ccd69946/metadata.yaml'")]

See above for debug info.

May I ask if anyone has encountered this situation? Looking forward to any possible solutions? Thank you very much!！

2

Hello and welcome to the forum!

My mentor once told me, that reading errors may help to solve some issues! For example, the error from your topic says that you run out of space:

So, you need to free some space on your machine, or run it on another machine with bigger storage, or export temp directory to a larger drive.

Best,

3

hello，Thank you for your reply！
The linux server I'm using looks like it has space remaining as shown here：

%Cpu(s): 14.9 us, 0.1 sy, 0.0 ni, 85.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
KiB Mem : 52678390+total, 1160928 free, 48718809+used, 38434868 buff/cache
KiB Swap: 33554428 total, 7884156 free, 25670272 used. 37716016 avail Mem

Is there anything I can do to free up some space? Thank you very much！
I tried reinstalling a new version of qiime2, but it didn't work.

Best,

1 Like
4

You are running out of the storage (not RAM) - plugin can't write temporal files.
Sometimes servers have separate faster drives for temporal files. You can export your temp directory to another drive with enough of storage. For example, in my case I run out of space on the drive for temporal files, provided by the server, so I exported my temp directory to the working drive that is slower but has a lot of space (WS is my working directory):

cd $WS

mkdir -p TEMP
export TMPDIR=$WS/TEMP
2 Likes
5

Thanks for the advice and code!

6

Hello，

I tried setting the temporary memory path as per the code.

(qiime2-2023.5) [mym@localhost alpha]$ export TMPDIR=WS/TEMP
(qiime2-2023.5) [mym@localhost alpha] echo {TMPDIR:-/tmp}
/TEMP
(qiime2-2023.5) [mym@localhost alpha] qiime diversity alpha-rarefaction --i-table 3.filtered_DADA2_table-run32mym1.qza --p-max-depth 43000 --p-min-depth 1000 --p-steps 5000 --m-metadata-file 0.run32mym-metadata-sample.tsv --o-visualization 3.filtered_DADA2_alpha-rarefaction-run32mym43000.qzv --verbose

But still this error is reported.

OSError: [Errno 28] No space left on device

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/miniconda3/envs/qiime2-2023.5/lib/python3.8/site-packages/qiime2/core/archive/archiver.py", line 412, in from_data
process_alias, data_path = cache._rename_to_data(uuid, path)
File "/miniconda3/envs/qiime2-2023.5/lib/python3.8/site-packages/qiime2/core/cache.py", line 1079, in _rename_to_data
with self.lock:
File "/miniconda3/envs/qiime2-2023.5/lib/python3.8/site-packages/qiime2/core/cache.py", line 292, in enter
self.flufl_lock.lock()
File "/miniconda3/envs/qiime2-2023.5/lib/python3.8/site-packages/flufl/lock/_lockfile.py", line 328, in lock
self._write()
File "/miniconda3/envs/qiime2-2023.5/lib/python3.8/site-packages/flufl/lock/_lockfile.py", line 493, in _write
fp.write(self._claimfile)
OSError: [Errno 28] No space left on device

During handling of the above exception, another exception occurred:

OSError: [Errno 28] No space left on device

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/miniconda3/envs/qiime2-2023.5/lib/python3.8/site-packages/q2cli/commands.py", line 468, in call
results = action(**arguments)
File "", line 2, in alpha_rarefaction
File "/miniconda3/envs/qiime2-2023.5/lib/python3.8/site-packages/qiime2/sdk/action.py", line 274, in bound_callable
outputs = self.callable_executor(
File "/miniconda3/envs/qiime2-2023.5/lib/python3.8/site-packages/qiime2/sdk/action.py", line 558, in callable_executor
ret_val = self._callable(output_dir=temp_dir, **view_args)
File "/miniconda3/envs/qiime2-2023.5/lib/python3.8/site-packages/q2_diversity/_alpha/_visualizer.py", line 375, in alpha_rarefaction
data = _compute_rarefaction_data(table, min_depth, max_depth,
File "/miniconda3/envs/qiime2-2023.5/lib/python3.8/site-packages/q2_diversity/_alpha/_visualizer.py", line 310, in _compute_rarefaction_data
vector, = alpha(table=rt, metric=metric)
File "/miniconda3/envs/qiime2-2023.5/lib/python3.8/site-packages/qiime2/sdk/context.py", line 140, in deferred_action
return action_obj._bind(
File "", line 2, in alpha
File "/miniconda3/envs/qiime2-2023.5/lib/python3.8/site-packages/qiime2/sdk/action.py", line 274, in bound_callable
outputs = self.callable_executor(
File "/miniconda3/envs/qiime2-2023.5/lib/python3.8/site-packages/qiime2/sdk/action.py", line 590, in callable_executor
outputs = self._callable(scope.ctx, **view_args)
File "/miniconda3/envs/qiime2-2023.5/lib/python3.8/site-packages/q2_diversity/_alpha/_pipeline.py", line 25, in alpha
vector, = action(table=table)
File "/miniconda3/envs/qiime2-2023.5/lib/python3.8/site-packages/qiime2/sdk/context.py", line 140, in deferred_action
return action_obj._bind(
File "", line 2, in shannon_entropy
File "/miniconda3/envs/qiime2-2023.5/lib/python3.8/site-packages/qiime2/sdk/action.py", line 274, in bound_callable
outputs = self.callable_executor(
File "/miniconda3/envs/qiime2-2023.5/lib/python3.8/site-packages/qiime2/sdk/action.py", line 526, in callable_executor
self.signature.coerce_given_outputs(output_views, output_types,
File "/miniconda3/envs/qiime2-2023.5/lib/python3.8/site-packages/qiime2/core/type/signature.py", line 487, in coerce_given_outputs
output = self._create_output_artifact(
File "/miniconda3/envs/qiime2-2023.5/lib/python3.8/site-packages/qiime2/core/type/signature.py", line 511, in _create_output_artifact
artifact = qiime2.sdk.Artifact._from_view(
File "/miniconda3/envs/qiime2-2023.5/lib/python3.8/site-packages/qiime2/sdk/result.py", line 362, in _from_view
artifact._archiver = archive.Archiver.from_data(
File "/miniconda3/envs/qiime2-2023.5/lib/python3.8/site-packages/qiime2/core/archive/archiver.py", line 421, in from_data
cls._destroy_temp_path(uuid)
File "/miniconda3/envs/qiime2-2023.5/lib/python3.8/site-packages/qiime2/core/archive/archiver.py", line 301, in _destroy_temp_path
cache.process_pool.remove(str(process_alias))
File "/miniconda3/envs/qiime2-2023.5/lib/python3.8/site-packages/qiime2/core/cache.py", line 1633, in remove
with self.cache.lock:
File "/miniconda3/envs/qiime2-2023.5/lib/python3.8/site-packages/qiime2/core/cache.py", line 292, in enter
self.flufl_lock.lock()
File "/miniconda3/envs/qiime2-2023.5/lib/python3.8/site-packages/flufl/lock/_lockfile.py", line 328, in lock
self._write()
File "/miniconda3/envs/qiime2-2023.5/lib/python3.8/site-packages/flufl/lock/_lockfile.py", line 493, in _write
fp.write(self._claimfile)
OSError: [Errno 28] No space left on device

Plugin error from diversity:

[Errno 28] No space left on device

See above for debug info.

I have the following basic linux server information at runtime and it looks like a lot of space.

top - 13:14:39 up 81 days, 4:57, 2 users, load average: 3.00, 3.02, 2.96
Tasks: 447 total, 4 running, 443 sleeping, 0 stopped, 0 zombie
%Cpu(s): 7.5 us, 0.0 sy, 0.0 ni, 92.5 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
KiB Mem : 52678390+total, 51735027+free, 4489676 used, 4943952 buff/cache
KiB Swap: 33554428 total, 33204368 free, 350060 used. 52071513+avail Mem

Is there any solution for this situation please? Is it possible that there is a problem with some plugin for qiime2?

Looking forward for suggestions. Thanks a lot!

Best,

7

Hello!
Was the temporary folder WS/TEMP created in your drive? WS is hjust a name of my working space, you don't need it.

cd your_working_space_abs_path

mkdir -p TEMP
export TMPDIR=your_working_space_abs_path/TEMP

Or export temp directory to another drive with larger storage capacity.

Best,

1 Like
8

Hello,
Thank you very much for the code you gave, this problem is solved!!
I ran it with all the files and the run path set in the TEMP folder and got the qzv file after 3 days of running.
Thanks again!
Best,

9

This topic was automatically closed 31 days after the last reply. New replies are no longer allowed.