Silva Classifier Issue and Cach clearing

[Carlmed00 (Carl Med)]

Good day!

I was trying out qiime2 using both Virtualbox and a computer running on Ubuntu. Tried out the moving pictures tutorial using the provided files and some sequence files I have as well. I have two questions regarding this:

1. In the Taxonomical Analysis, the moving pictures used Greengenes as classifier. I tried and it worked well. However, when I tried it using silva instead, it runs quite some time until I get an error (picture attached in this thread). I am supposing that this is an issue regarding my memory, however it seems that it only occurs if I use Silva. I am unsure what I might have done wrong since it works with Greengenes and the only part I alter in the script is the classifier file.
   
   
2. While I run the taxonomical classification script using Silva, I notice that it eats up a large chunk of file space. Since it generates an error, there is no output file. I am unsure how to clear that space up. Seems like it is not stored in my cache. I am just worried in my future analysis that it'll just eat up spaces that I wouldn't know where to locate and clear.

My system setting for Virtual box is 8gig memory and 5 processors while my ubuntu computer have 6gig memory and 4processors.

Hoping for some support.

Thank you very much in advance!

[Nicholas_Bokulich]

Welcome to the forum, @Carlmed00!

@Carlmed00

However, when I tried it using silva instead, it runs quite some time until I get an error (picture attached in this thread). I am supposing that this is an issue regarding my memory, however it seems that it only occurs if I use Silva. I am unsure what I might have done wrong since it works with Greengenes and the only part I alter in the script is the classifier file.

You are correct, that's a memory error. SILVA is ~4X as large as Greengenes so takes much more memory and time to use. See this post for some options to reduce memory load:

Feature-classifiers error

This is a memory error; essentially you do not have enough memory to open the SILVA classifier on your computer. This is low for the SILVA classifier — it will often take up to 32GB+ if left to its own devices! There’s a couple of things you might be able to do. First, you can use the --p-reads-per-batch parameter (e.g., set to 1000 or 2000) to reduce the number of reads classified at a time. However, it looks like you do not have enough memory to load the SILVA classifier, not the reads, s…

@Carlmed00

1. While I run the taxonomical classification script using Silva, I notice that it eats up a large chunk of file space. Since it generates an error, there is no output file. I am unsure how to clear that space up. Seems like it is not stored in my cache. I am just worried in my future analysis that it’ll just eat up spaces that I wouldn’t know where to locate and clear.

Any temp files should automatically clear when the job complete but let us know if you're still experiencing a problem.

I hope that helps!

[Carlmed00 (Carl Med)]

Thanks for the tip on lowering memory load!

As for the disk space, still not clearing for both. I have been monitoring the disk consumption especially when running silva on both virtual machine and my ubuntu computer. It consumes more than 10gb before it gives me the memory error. I track manually searching folders but im failing to locate. The space eaten is still there. Is the silva gza unpacked somewhere?

[thermokarst]

Hi @Carlmed00 -

Are you monitoring the disk space on the host OS (I'm assuming Windows), or the guest OS (the QIIME 2 VM), or both?

If you're monitoring the disk usage from the host OS, you'll never see the size of the VM decrease on its own - Virtualbox will just continue to grow (to a limit), unless you do something like what is described in this post:

How to compact VirtualBox's VDI file size?

@Carlmed00

It consumes more than 10gb before it gives me the memory error

It consumes more than 10 GB of disk space? Just making sure I follow what you're saying, since Memory is not the same thing as Disk Size, but both use bytes as a description of their capactiy.

Keep us posted.

[Carlmed00 (Carl Med)]

Thank you very much!

The ubuntu machine seems to store it somewhere. The disc space does’t go down after a failed script run. I tried checking folders one by one and apparently, the numbers do not add up to a very significant degree.