An error: Plugin error from feature-classifier

Hi! When I used q2-feature-classifier to train a classifier, I found an error: Unable to allocate array with shape (20000, 5391) and data type int64. Does anyone know the causation and how to solve this problem? Thanks very much!

1 Like

Hello @Zhanzhan,

Welcome to the Qiime 2 forums! :qiime2:

The "Unable to allocate array" error means you don't have enough memory / RAM on your computer or in your virtual box to train the classifier.

Training a taxonomic classifier takes a lot of RAM! How much RAM / memory do you have now? How large is your database?

Colin

2 Likes

Hello, @colinbrislawn!
Thank you for your reply! I used greegene database: 99_otus.fasta and 99_otu_taxonomy.txt to train a classifier. Here is my code to produce a V4 ref-seqs.qza: qiime feature-classifier extract-reads \

--i-sequences 99_otus.qza
--p-f-primer GTGTGYCAGCMGCCGCGGTAA
--p-r-primer CCGGACTACNVGGGTWTCTAAT
--p-min-length 100
--p-max-length 400
--o-reads ref-seqs.qza
and then I wanted to use ref-seqs.qza and ref-taxonomy.qza(99_otu_taxonomy.txt) to produce a classifier, but failed. My code:
qiime feature-classifier fit-classifier-naive-bayes
--i-reference-reads ref-seqs.qza
--i-reference-taxonomy ref-taxonomy.qza
--o-classifier classifier.qza
Plugin error from feature-classifier:
Unable to allocate array with shape (20000, 5391) and data type int64

My ram is so small and it's just 2G. And I want to update this device. Could you please give me advice about how much ram memory I need for training a classifier like this?
Ps:my biggest dataset is about 20G,130 samples, and this demands how much ram memory and how much disk memory? Thank you very much!:grin:

1 Like

Hello again,

I've edited your post a little bit. When you do a quote with > you need one extra line after to show the next text starts a new paragraph.


Back to Qiime 2 --> :fast_forward: :qiime2:

Thank you for telling me more. How large (GB or MB) is 99_otus.fasta? That is the main limitation with training a classifier.

My ram is so small and it’s just 2G. And I want to update this device. Could you please give me advice about how much ram memory I need for training a classifier like this?

Memory usage is hard to estimate because it depends on the size and complexity of the database. :man_shrugging:

If you only have a small amount of memory, the best option is to use a pre-trained classifier. We got some good ones here: Data resources — QIIME 2 2023.9.2 documentation

Training a classifier takes a lot of RAM, but analyzing samples does not need as much RAM. If you could find a computer or VM with 8 GB of RAM should work well.

Thank you! :slight_smile:
Colin

1 Like

@colinbrislawn :star_struck::star_struck:Thank you for your advice very much!Thanks for your editing!
The 99_otus.fasta is 270MB and after extraction and gzip, the 99-v4-ref-seqs.qza is about 8.82MB. The 99_otu_taxonomy.txt is 20.43MB and after importing qiime2, the ref-taxonomy.qza is 2.49MB. And how much ram memory does this demand? Thank you!:smiling_face_with_three_hearts:

1 Like

Because 99_otus.fasta is only 270MB, I think 2 GB of RAM would be enough… but it looks like you do need more. Could you find a computer with 8 GB and try that?

Have you tried using any of the pre-trained classifiers?

Colin

@colinbrislawn Thank you Colin! I want to rent a 8GB ram ECS to analyze it and I used a V4 pre-classifier to perform taxonomy annotation and I got the results. But I found no significant differential abundance between two groups. Thank you again for your sincere advice!

1 Like

Thank you so much!

Feel free to open a new thread if you have more questions.

Colin

@colinbrislawn Thank you!

A post was split to a new topic: Virtual Box. Memory allocation error