Im working with qiime2 2019.7 in conda, and Im trying to train my classifier as explained in the
I have extracted the reference reads and taxonomy from the lattest version of SILVA database, for V3-V4 region of the 16S
qiime tools import \
--type 'FeatureData[Taxonomy]' \
--input-format HeaderlessTSVTaxonomyFormat \
--input-path ../SILVA_132_QIIME_release/taxonomy/16S_only/99/taxonomy_all_levels.txt \
qiime tools import \
--type 'FeatureData[Sequence]' \
--input-path ../SILVA_132_QIIME_release/rep_set/rep_set_16S_only/99/silva_132_99_18S.fna \
qiime feature-classifier extract-reads \
--i-sequences 99_otus.qza \
--p-f-primer CCTAYGGGRBGCASCAG \
--p-r-primer GGACTACNNGGGTATCTAAT \
--p-min-length 100 \
--p-max-length 450 \
So finally when I trie to train the classifier
qiime feature-classifier fit-classifier-naive-bayes \
--i-reference-reads ref-seqs.qza \
--i-reference-taxonomy ref-taxonomy.qza \
I get this error message.
Unable to allocate array with shape (74619, 8192) and data type float64
I find it odd, since as I understand It, It seems to requier 4Gb of memory, and Im working with 32Gb
Thanks in advance
Hi, it’s requiring 4 gb for a file on which error occurred, but definitely it’s not the whole memory you need. Silva database requires a lot of RAMy, so maybe 32 is not enough. I got such error once on the laptop with 32 gb and used stronger machine to train classifier. But you always can download pretrained classifier from the forum for your region and database
Hi, thanks for the reply. I have alrready used the pre-trained classifier. But I wanted to know how much would the results change with a trained one.
A worthy pursuit,
@jose_gacia! Like @timanix, suggested, this is an out-of-memory error. Silva’s a big ol ! If you’re working on a virtual machine, make sure you’ve allocated enough RAM to the VM. If not, you might need to consider using an institutional compute cluster, or renting a server. Setting up an ubuntu instance isn’t too hard, and can be quite cheap.
Best of luck!
Thanks for the reply
Im working on my company’s computer with 32Gb of RAM, I thougth It could be enough. I was wondering If there´s a way of enhacing the performance/use of memory
Yes, please see the many existing topics on the forum that discuss troubleshooting memory errors. E.g., see this topic:
This is a memory error; essentially you do not have enough memory to open the SILVA classifier on your computer.
This is low for the SILVA classifier — it will often take up to 32GB+ if left to its own devices!
There’s a couple of things you might be able to do.
First, you can use the --p-reads-per-batch parameter (e.g., set to 1000 or 2000) to reduce the number of reads classified at a time. However, it looks like you do not have enough memory to load the SILVA classifier, not the reads, s…
This topic was automatically closed 31 days after the last reply. New replies are no longer allowed.