Plugin error from feature-classifier: [Errno 28] No space left on device (on a virtual machine)

Rosie · April 23, 2021, 12:12pm

Hello everyone!

I have a problem with the feature classifier. I know that it has been already discussed in the qiime2 forum, but I still have not find the solution.
I am working on a virtual machine from Azure with a PC that has 110G RAM (92G available) and 8 CPU.
Since my qiime2 environment is installed in the PC, when I run the script

qiime feature-classifier fit-classifier-naive-bayes --i-reference-reads silva_99_515F806R.seqs.qza --i-reference-taxonomy silva_99_consensus_taxonomy.qza --o-classifier silva_99_515F806R_classifier.qza

I am getting a plugin error, saying:

[Errno 28] No space left on device
/home/ale/anaconda3/envs/qiime2-2021.2/lib/python3.6/site-packages/q2_feature_classifier/classifier.py:102: UserWarning: The TaxonomicClassifier artifact that results from this method was trained using scikit-learn version 0.23.1. It cannot be used with other versions of scikit-learn. (While the classifier may complete successfully, the results will be unreliable.)
warnings.warn(warning, UserWarning)
Traceback (most recent call last):
File "/home/ale/anaconda3/envs/qiime2-2021.2/lib/python3.6/site-packages/q2_feature_classifier/_taxonomic_classifier.py", line 86, in _2
tar.add(fn, os.path.basename(fn))
File "/home/ale/anaconda3/envs/qiime2-2021.2/lib/python3.6/tarfile.py", line 1954, in add
self.addfile(tarinfo, f)
File "/home/ale/anaconda3/envs/qiime2-2021.2/lib/python3.6/tarfile.py", line 1982, in addfile
copyfileobj(fileobj, self.fileobj, tarinfo.size, bufsize=bufsize)
File "/home/ale/anaconda3/envs/qiime2-2021.2/lib/python3.6/tarfile.py", line 252, in copyfileobj
dst.write(buf)
OSError: [Errno 28] No space left on device

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/home/ale/anaconda3/envs/qiime2-2021.2/lib/python3.6/site-packages/q2cli/commands.py", line 329, in call
results = action(**arguments)
File "", line 2, in fit_classifier_naive_bayes
File "/home/ale/anaconda3/envs/qiime2-2021.2/lib/python3.6/site-packages/qiime2/sdk/action.py", line 245, in bound_callable
output_types, provenance)
File "/home/ale/anaconda3/envs/qiime2-2021.2/lib/python3.6/site-packages/qiime2/sdk/action.py", line 418, in callable_executor
spec.qiime_type, output_view, spec.view_type, prov)
File "/home/ale/anaconda3/envs/qiime2-2021.2/lib/python3.6/site-packages/qiime2/sdk/result.py", line 267, in _from_view
result = transformation(view, validate_level)
File "/home/ale/anaconda3/envs/qiime2-2021.2/lib/python3.6/site-packages/qiime2/core/transform.py", line 70, in transformation
new_view = transformer(view)
File "/home/ale/anaconda3/envs/qiime2-2021.2/lib/python3.6/site-packages/q2_feature_classifier/_taxonomic_classifier.py", line 87, in _2
os.unlink(fn)
File "/home/ale/anaconda3/envs/qiime2-2021.2/lib/python3.6/tarfile.py", line 2441, in exit
self.fileobj.close()
OSError: [Errno 28] No space left on device

I initially thought that temporary files needed to be located elsewhere, so I tried with:

mkdir home/ale/mpodars-new-temporary-directory
export TMPDIR=' home/ale/mpodars-new-temporary-directory'

without any success.
Any suggestions?
Thanks a lot for your help.
All the best,

Rosangela

colinbrislawn · April 23, 2021, 2:51pm

Hello Rosangela

I like your idea of moving the temp file to resolve the disk space issue. Unfortunately, that temp file looks like it’s still residing on the disk of your Azure virtual machine, and that disk is full.

What is the name / size / make and model of the VM you are using? All the Azure VMs are listed here, and hopefully we can find out how much disk space your VM has, and maybe we can find a VM with enough disk so you can run this command!

Colin

Rosie · April 23, 2021, 4:19pm

Hello Colin!

Thanks a lot for your quick reply.
The Virtual Machine I am using is a H8m_Promo Standard (8 cpu, 112 GiB, Disk SKU size 1024 GiB) with Linux as OS.
I guess I just need more space, but how much more?
Best,

Rosie

colinbrislawn · April 23, 2021, 9:23pm

Hello Rosie,

Looks like that type of instance has 1TB of local temp storage!

According to the Azure docs, that disk is located at /dev/sdb. So try this:

mkdir /dev/sdb/qiime2-temp
export TMPDIR='/dev/sdb/qiime2-temp'

Let me us know how it goes!
Colin

Rosie · April 24, 2021, 1:20pm

Hello Colin,

unfortunately, it didn't work:

mkdir: cannot create directory ‘/dev/sdb/qiime2-temp’: Not a directory

I have already tried to move temporary files to another directory that seems to have more space (/dev/sdb1), as shown here:

> Filesystem      Size  Used Avail Use% Mounted on
> udev             56G     0   56G   0% /dev
> tmpfs            12G  720K   12G   1% /run
> /dev/sda1        29G   21G  8.5G  71% /
> tmpfs            56G     0   56G   0% /dev/shm
> tmpfs           5.0M     0  5.0M   0% /run/lock
> tmpfs            56G     0   56G   0% /sys/fs/cgroup
> /dev/sda15      105M  6.1M   99M   6% /boot/efi
> /dev/sdb1      1007G   77M  956G   1% /mnt
> tmpfs            12G     0   12G   0% /run/user/1000

but, again, it didn't work (same error).
I really don't know how to solve this problem, I am thinking about changing my VM with another one with more space, do you think 2TB are enough?
Thanks a lot for your help.
Best,

Rosie

colinbrislawn · April 24, 2021, 4:53pm

Hello Rosie,

Oh, I made a linux mistake

The temp folder should be made in the Mount location, not the Filesystem location. So while /dev/sdb1 is the filesystem, we should be looking at /mnt where that drive is mounted.

Try this:

mkdir /mnt/qiime2-temp
export TMPDIR='/mnt/qiime2-temp'

There’s 956G free space there, so hopefully you will not have to pay more for a different instance!

Colin

PS. Great detective work finding /dev/sdb1/ instead of /dev/sdb/ and posting the result of df! I wrapped your result in a code block using ``` so we can see the columns better

Rosie · April 26, 2021, 11:56am

Hello Colin,

Thank you so much for your advice.
I have tried to move temp folder in mnt (I needed to force it under sudo), but it is still not working

/home/ale/anaconda3/envs/qiime2-2021.2/lib/python3.6/site-packages/q2_feature_classifier/classifier.py:102: UserWarning: The TaxonomicClassifier artifact that results from this method was trained using scikit-learn version 0.23.1. It cannot be used with other versions of scikit-learn. (While the classifier may complete successfully, the results will be unreliable.)
warnings.warn(warning, UserWarning)
Traceback (most recent call last):
File "/home/ale/anaconda3/envs/qiime2-2021.2/lib/python3.6/site-packages/q2cli/commands.py", line 329, in call
results = action(**arguments)
File "", line 2, in fit_classifier_naive_bayes
File "/home/ale/anaconda3/envs/qiime2-2021.2/lib/python3.6/site-packages/qiime2/sdk/action.py", line 245, in bound_callable
output_types, provenance)
File "/home/ale/anaconda3/envs/qiime2-2021.2/lib/python3.6/site-packages/qiime2/sdk/action.py", line 418, in callable_executor
spec.qiime_type, output_view, spec.view_type, prov)
File "/home/ale/anaconda3/envs/qiime2-2021.2/lib/python3.6/site-packages/qiime2/sdk/result.py", line 267, in _from_view
result = transformation(view, validate_level)
File "/home/ale/anaconda3/envs/qiime2-2021.2/lib/python3.6/site-packages/qiime2/core/transform.py", line 70, in transformation
new_view = transformer(view)
File "/home/ale/anaconda3/envs/qiime2-2021.2/lib/python3.6/site-packages/q2_feature_classifier/_taxonomic_classifier.py", line 86, in _2
tar.add(fn, os.path.basename(fn))
File "/home/ale/anaconda3/envs/qiime2-2021.2/lib/python3.6/tarfile.py", line 1954, in add
self.addfile(tarinfo, f)
File "/home/ale/anaconda3/envs/qiime2-2021.2/lib/python3.6/tarfile.py", line 1982, in addfile
copyfileobj(fileobj, self.fileobj, tarinfo.size, bufsize=bufsize)
File "/home/ale/anaconda3/envs/qiime2-2021.2/lib/python3.6/tarfile.py", line 252, in copyfileobj
dst.write(buf)
OSError: [Errno 28] No space left on device

It is still using tmp as the temp folder, it doesn't let me use the space I have in mnt.
I guess the one and only solution could be changing VM with more space. The fact is: how much is enough?
Maybe placing the analysis in a place without sudo permission (no ‘root’), as described here?
I am stuck on this and I really don't know what else to do.
Thanks again for your kind help.
Best,

Rosie

colinbrislawn · April 26, 2021, 3:03pm

Hello Rosie,

Interesting...

It looks like the actual training step (in the TMPDIR) succeeded - it appears to be failing while saving the results in your home directory.

Well said!

Well, your current home directory is here
> /dev/sda1 29G 21G 8.5G 71% /
and that's only 30 GB, so upgrading to 300 GB should give you plenty of space for both your input data and your classification results!

The easiest way to do this is to resize your system disk for your current VM, so it's larger than the 30 GB default.

Let us know how well this works for you!
Colin

Rosie · April 30, 2021, 9:22am

Hello Colin,

I have resized my VM and it works!!!
Thanks a lot for your help!
Best,

Rosie

system · May 31, 2021, 3:23pm

This topic was automatically closed 31 days after the last reply. New replies are no longer allowed.