Memory and Classification for 18S

[fhermanto96 (Feri Hermanto)]

Dear @Nicholas_Bokulich,
Is there a guarantee that I will solve the problem related to memory error during classification step when I run the classify-sklearn in 32 GB of memory?
I have some problems as previous posts (Silva memory error), and I have no idea whether I need more powerful computer (and I think I still can’t afford it) or there is some classifiers for 18S sequences other than Silva.
Thank you

[fhermanto96 (Feri Hermanto)]

Dear Experts

Do you have some suggestion regarding the classifier for 18S sequences other than Silva? I've tried to run classification using Silva, and I got the same error message as discussed before:

Silva memory error

Sounds like you are left with one of two options: Use a less memory-intensive classifier, like greengenes as @timanix advises:

Unfortunately, I have no more resource to upgrade my computer into higher memory to deal with the silva classifier (My CPU is equipped with Ryzen 3 3200G and 16 GB memory). It will be so helpful if there is someone who can help me with this problem.
Thank you very much!

[SoilRotifer (Mike Robeson)]

Hi @fhermanto96, Have you tried try one of the newer SILVA 138 classifiers here:

SILVA 138 Classifiers

This pipeline has been vastly improved via the new RESCRIPt plugin. Which you can check out via the link below. We hope the process of constructing your own reference sequence database (e.g. SILVA) will be far less onerous.

Click here to see the original documentation

I just wanted to let everyone be aware that I've hobbled together a simple <a href="https://github.com/mikerobeson/make_SILVA_db">pipeline</a> for constructing classifiers based …

They should have a smaller memory footprint... especially the versions of the classifiers which were made without the species label (only goes down to genus).

-Mike

[Nicholas_Bokulich]

@fhermanto96

My CPU is equipped with Ryzen 3 3200G and 16 GB memory

Did you see the reads-per-batch parameter suggestions elsewhere in the forum? Reducing this will greatly reduce memory demands. You have plenty of memory! (For comparison, I routinely run SILVA classifiers on an 8GB RAM laptop but adjusting reads-per-batch)

good luck!

[fhermanto96 (Feri Hermanto)]

Dear @Nicholas_Bokulich, I’ve tried to adjust p-reads-per-batch into 50, but still found the same problem. Is there another suggestion? I already used Silva 132 full-length and 515F/806R region of sequences and still face the same problem.

[fhermanto96 (Feri Hermanto)]

Dear @SoilRotifer, thanks for your suggestion. I will try it first then I’ll report it whether it works or not. Hopefully this one will solve my problems

[fhermanto96 (Feri Hermanto)]

Dear @SoilRotifer, I would like to thanks for your help, particularly by providing Silva classifier with smaller memory footprint. My problem solved now and can be directed to further analysis.
Thank you