Plugin error from feature-classifier when running Silva-132

Hi all,
I’m a relatively new Qiime user running version 2019.7 and I’m currently having a problem using Silva 132 for taxonomic analysis. I’m using the Moving Pictures tutorial as a guideline, but keep receiving a plugin error. I downloaded the Silva package directly from the qiime2 data resources page for the version that I am using, and have made sure that my sklearn package is up to date.

Here is the code that I have been entering:

qiime feature-classifier classify-sklearn
–verbose
–i-classifier silva-132-99-515-806-nb-classifier.qza
–i-reads rep-seqs.qza
–o-classification taxonomy.qza

And here is the error that I have been getting back:

Traceback (most recent call last):
File “/home/lab/miniconda3/envs/qiime2-2019.7/lib/python3.6/site-packages/q2cli/commands.py”, line 327, in call
results = action(**arguments)
File “</home/lab/miniconda3/envs/qiime2-2019.7/lib/python3.6/site-packages/decorator.py:decorator-gen-347>”, line 2, in classify_sklearn
File “/home/lab/miniconda3/envs/qiime2-2019.7/lib/python3.6/site-packages/qiime2/sdk/action.py”, line 240, in bound_callable
output_types, provenance)
File “/home/lab/miniconda3/envs/qiime2-2019.7/lib/python3.6/site-packages/qiime2/sdk/action.py”, line 383, in callable_executor
output_views = self._callable(**view_args)
File “/home/lab/miniconda3/envs/qiime2-2019.7/lib/python3.6/site-packages/q2_feature_classifier/classifier.py”, line 215, in classify_sklearn
reads, classifier, read_orientation=read_orientation)
File “/home/lab/miniconda3/envs/qiime2-2019.7/lib/python3.6/site-packages/q2_feature_classifier/classifier.py”, line 170, in _autodetect_orientation
result = list(zip(*predict(first_n_reads, classifier, confidence=0.)))
File “/home/lab/miniconda3/envs/qiime2-2019.7/lib/python3.6/site-packages/q2_feature_classifier/_skl.py”, line 45, in predict
for chunk in _chunks(reads, chunk_size)) for m in c)
File “/home/lab/miniconda3/envs/qiime2-2019.7/lib/python3.6/site-packages/joblib/parallel.py”, line 921, in call
if self.dispatch_one_batch(iterator):
File “/home/lab/miniconda3/envs/qiime2-2019.7/lib/python3.6/site-packages/joblib/parallel.py”, line 759, in dispatch_one_batch
self._dispatch(tasks)
File “/home/lab/miniconda3/envs/qiime2-2019.7/lib/python3.6/site-packages/joblib/parallel.py”, line 716, in _dispatch
job = self._backend.apply_async(batch, callback=cb)
File “/home/lab/miniconda3/envs/qiime2-2019.7/lib/python3.6/site-packages/joblib/_parallel_backends.py”, line 182, in apply_async
result = ImmediateResult(func)
File “/home/lab/miniconda3/envs/qiime2-2019.7/lib/python3.6/site-packages/joblib/_parallel_backends.py”, line 549, in init
self.results = batch()
File “/home/lab/miniconda3/envs/qiime2-2019.7/lib/python3.6/site-packages/joblib/parallel.py”, line 225, in call
for func, args, kwargs in self.items]
File “/home/lab/miniconda3/envs/qiime2-2019.7/lib/python3.6/site-packages/joblib/parallel.py”, line 225, in
for func, args, kwargs in self.items]
File “/home/lab/miniconda3/envs/qiime2-2019.7/lib/python3.6/site-packages/q2_feature_classifier/_skl.py”, line 52, in _predict_chunk
return _predict_chunk_with_conf(pipeline, separator, confidence, chunk)
File “/home/lab/miniconda3/envs/qiime2-2019.7/lib/python3.6/site-packages/q2_feature_classifier/_skl.py”, line 66, in _predict_chunk_with_conf
prob_pos = pipeline.predict_proba(X)
File “/home/lab/miniconda3/envs/qiime2-2019.7/lib/python3.6/site-packages/sklearn/utils/metaestimators.py”, line 116, in
out = lambda *args, **kwargs: self.fn(obj, *args, **kwargs)
File “/home/lab/miniconda3/envs/qiime2-2019.7/lib/python3.6/site-packages/sklearn/pipeline.py”, line 474, in predict_proba
return self.steps[-1][-1].predict_proba(Xt)
File “/home/lab/miniconda3/envs/qiime2-2019.7/lib/python3.6/site-packages/sklearn/naive_bayes.py”, line 103, in predict_proba
return np.exp(self.predict_log_proba(X))
File “/home/lab/miniconda3/envs/qiime2-2019.7/lib/python3.6/site-packages/sklearn/naive_bayes.py”, line 83, in predict_log_proba
jll = self._joint_log_likelihood(X)
File “/home/lab/miniconda3/envs/qiime2-2019.7/lib/python3.6/site-packages/sklearn/naive_bayes.py”, line 737, in joint_log_likelihood
return (safe_sparse_dot(X, self.feature_log_prob
.T) +
File “/home/lab/miniconda3/envs/qiime2-2019.7/lib/python3.6/site-packages/sklearn/utils/extmath.py”, line 137, in safe_sparse_dot
ret = a * b
File “/home/lab/miniconda3/envs/qiime2-2019.7/lib/python3.6/site-packages/scipy/sparse/base.py”, line 472, in mul
return self._mul_multivector(other)
File “/home/lab/miniconda3/envs/qiime2-2019.7/lib/python3.6/site-packages/scipy/sparse/compressed.py”, line 487, in _mul_multivector
other.ravel(), result.ravel())
numpy.core._exceptions.MemoryError: Unable to allocate array with shape (796852224,) and data type float64

Plugin error from feature-classifier:

Unable to allocate array with shape (796852224,) and data type float64

See above for debug info.

Please let me know if there is any way to solve this problem and move forward with the taxonomic analysis.

Hello @hustads,

Welcome to Qiime 2 forms! :qiime2:

Thanks for posting your full error. I think the key line is right here:

numpy.core._exceptions.MemoryError: Unable to allocate array with shape (796852224,) and data type float64

This means that the command needs more memory / RAM to complete successfully. :ram:
Are you running this on a laptop, server, or VM? Do you know how much memory / RAM you have on the system?

Colin

Hi Colin,

Thanks for replying so quickly! We are using a laptop to run this program and from what I can tell, it has 15.6 GB of memory available. How much do you think that this program would need to successfully run?

1 Like

@hustads we have seen SILVA classifiers require up to 32 GB to run, but there are several tricks to get around this — search the forum archive for "MemoryError" to see some troubleshooting advice.

Good luck!

2 Likes

A post was split to a new topic: Memory and Classification for ITS