An error: Plugin error from feature-classifier

Hi! When I used q2-feature-classifier to train a classifier, I found an error: Unable to allocate array with shape (20000, 5391) and data type int64. Does anyone know the causation and how to solve this problem? Thanks very much!

1 Like

Hello @Zhanzhan,

Welcome to the Qiime 2 forums! :qiime2:

The “Unable to allocate array” error means you don’t have enough memory / RAM on your computer or in your virtual box to train the classifier.

Training a taxonomic classifier takes a lot of RAM! How much RAM / memory do you have now? How large is your database?

Colin

2 Likes

Hello, @colinbrislawn!
Thank you for your reply! I used greegene database: 99_otus.fasta and 99_otu_taxonomy.txt to train a classifier. Here is my code to produce a V4 ref-seqs.qza: qiime feature-classifier extract-reads \

–i-sequences 99_otus.qza
–p-f-primer GTGTGYCAGCMGCCGCGGTAA
–p-r-primer CCGGACTACNVGGGTWTCTAAT
–p-min-length 100
–p-max-length 400
–o-reads ref-seqs.qza
and then I wanted to use ref-seqs.qza and ref-taxonomy.qza(99_otu_taxonomy.txt) to produce a classifier, but failed. My code:
qiime feature-classifier fit-classifier-naive-bayes
–i-reference-reads ref-seqs.qza
–i-reference-taxonomy ref-taxonomy.qza
–o-classifier classifier.qza
Plugin error from feature-classifier:
Unable to allocate array with shape (20000, 5391) and data type int64

My ram is so small and it’s just 2G. And I want to update this device. Could you please give me advice about how much ram memory I need for training a classifier like this?
Ps:my biggest dataset is about 20G,130 samples, and this demands how much ram memory and how much disk memory? Thank you very much!:grin:

1 Like

Hello again,

I’ve edited your post little bit. When you do a quote with starting with > you need one extra line between

the quote

and the line after the quote.

the quote
and the line after the quote.

Oh no! Let’s fix that.

the quote

and the line after the quote are now fixed!


Back to Qiime 2 --> :fast_forward: :qiime2:

Thank you for telling me more. How large (GB or MB) is 99_otus.fasta? That is the main limitation with training a classifier.

My ram is so small and it’s just 2G. And I want to update this device. Could you please give me advice about how much ram memory I need for training a classifier like this?

Memory usage is hard to estimate, because it depends on the size and complexity of the database. :man_shrugging:

If you only have a small amount of memory, the best option is to use a pre-trained classifier. We got some good ones here: https://docs.qiime2.org/2019.7/data-resources/
(Greengenes has not been updated in 5 years, while silva is actively maintained. Try silva :+1: )

Training a classifier takes a lot of RAM, but analyzing samples does not need as much RAM. If you could find a computer or VM with 8 GB of RAM should work well.

Thank you! :slight_smile:
Colin

1 Like

@colinbrislawn :star_struck::star_struck:Thank you for your advice very much!Thanks for your editing!
The 99_otus.fasta is 270MB and after extraction and gzip, the 99-v4-ref-seqs.qza is about 8.82MB. The 99_otu_taxonomy.txt is 20.43MB and after importing qiime2, the ref-taxonomy.qza is 2.49MB. And how much ram memory does this demand? Thank you!:smiling_face_with_three_hearts:

1 Like

Because 99_otus.fasta is only 270MB, I think 2 GB of RAM would be enough… but it looks like you do need more. Could you find a computer with 8 GB and try that?

Have you tried using any of the pre-trained classifiers?

Colin

@colinbrislawn Thank you Colin! I want to rent a 8GB ram ECS to analyze it and I used a V4 pre-classifier to perform taxonomy annotation and I got the results. But I found no significant differential abundance between two groups. Thank you again for your sincere advice!

1 Like

Thank you so much!

Feel free to open a new thread if you have more questions.

Colin

@colinbrislawn Thank you!

A post was split to a new topic: Virtual Box. Memory allocation error