Unable to allocate array with shape (74619, 8192) and data type float64

[jose_gacia (Jomagrax)]

Hi everyone,

Im working with qiime2 2019.7 in conda, and Im trying to train my classifier as explained in the tutorial

I have extracted the reference reads and taxonomy from the lattest version of SILVA database, for V3-V4 region of the 16S

qiime tools import  \ 
--type 'FeatureData[Taxonomy]'   \
--input-format HeaderlessTSVTaxonomyFormat   \
--input-path ../SILVA_132_QIIME_release/taxonomy/16S_only/99/taxonomy_all_levels.txt  \ 
--output-path ref-taxonomy.qza

qiime tools import \
--type 'FeatureData[Sequence]' \
--input-path  ../SILVA_132_QIIME_release/rep_set/rep_set_16S_only/99/silva_132_99_18S.fna \
--output-path 99_otus.qza

 qiime feature-classifier extract-reads   \
--i-sequences 99_otus.qza   \
--p-f-primer  CCTAYGGGRBGCASCAG \
--p-r-primer GGACTACNNGGGTATCTAAT \
--p-min-length 100   \
--p-max-length  450  \
--o-reads ref-seqs.qza

So finally when I trie to train the classifier

 qiime feature-classifier fit-classifier-naive-bayes \
      --i-reference-reads ref-seqs.qza \
      --i-reference-taxonomy ref-taxonomy.qza \
      --o-classifier classifier.qza

I get this error message.

Unable to allocate array with shape (74619, 8192) and data type float64

I find it odd, since as I understand It, It seems to requier 4Gb of memory, and Im working with 32Gb

Any ideas?

Thanks in advance

[timanix (Timur Yergaliyev)]

Hi, it's requiring 4 gb for a file on which error occurred, but definitely it's not the whole memory you need. Silva database requires a lot of RAMy, so maybe 32 is not enough. I got such error once on the laptop with 32 gb and used stronger machine to train classifier. But you always can download pretrained classifier from the forum for your region and database

[jose_gacia (Jomagrax)]

Hi, thanks for the reply. I have alrready used the pre-trained classifier. But I wanted to know how much would the results change with a trained one.

[ChrisKeefe]

A worthy pursuit, @jose_gacia! Like @timanix, suggested, this is an out-of-memory error. Silva's a big ol ! If you're working on a virtual machine, make sure you've allocated enough RAM to the VM. If not, you might need to consider using an institutional compute cluster, or renting a server. Setting up an ubuntu instance isn't too hard, and can be quite cheap.

Best of luck!

[jose_gacia (Jomagrax)]

Thanks for the reply @ChrisKeefe!
Im working on my company's computer with 32Gb of RAM, I thougth It could be enough. I was wondering If there´s a way of enhacing the performance/use of memory

[Nicholas_Bokulich]

@jose_gacia

I was wondering If there´s a way of enhacing the performance/use of memory

Yes, please see the many existing topics on the forum that discuss troubleshooting memory errors. E.g., see this topic:

Feature-classifiers error

This is a memory error; essentially you do not have enough memory to open the SILVA classifier on your computer. This is low for the SILVA classifier — it will often take up to 32GB+ if left to its own devices! There's a couple of things you might be able to do. First, you can use the --p-reads-per-batch parameter (e.g., set to 1000 or 2000) to reduce the number of reads classified at a time. However, it looks like you do not have enough memory to load the SILVA classifier, not the reads, s…

Good luck!