Feature-classifiers error

User Support
1

Hi,

I’m having difficulties with the feature-classifier plugin, in qiime 2019.7. When I run the following command:

qiime feature-classifier classify-sklearn
–i-classifier silva-132-99-nb-classifier.qza
–i-reads rep-seqs.qza
–p-pre-dispatch ‘n_jobs’
–o-classification taxonomy.qza
–verbose

I receive this error:
Traceback (most recent call last):
File “/home/qiime2/miniconda/envs/qiime2-2019.7/lib/python3.6/site-packages/q2cli/commands.py”, line 327, in call
results = action(**arguments)
File “</home/qiime2/miniconda/envs/qiime2-2019.7/lib/python3.6/site-packages/decorator.py:decorator-gen-347>”, line 2, in classify_sklearn
File “/home/qiime2/miniconda/envs/qiime2-2019.7/lib/python3.6/site-packages/qiime2/sdk/action.py”, line 229, in bound_callable
spec.view_type, recorder)
File “/home/qiime2/miniconda/envs/qiime2-2019.7/lib/python3.6/site-packages/qiime2/sdk/result.py”, line 289, in _view
result = transformation(self._archiver.data_dir)
File “/home/qiime2/miniconda/envs/qiime2-2019.7/lib/python3.6/site-packages/qiime2/core/transform.py”, line 70, in transformation
new_view = transformer(view)
File “/home/qiime2/miniconda/envs/qiime2-2019.7/lib/python3.6/site-packages/q2_feature_classifier/_taxonomic_classifier.py”, line 72, in _1
pipeline = joblib.load(os.path.join(dirname, ‘sklearn_pipeline.pkl’))
File “/home/qiime2/miniconda/envs/qiime2-2019.7/lib/python3.6/site-packages/joblib/numpy_pickle.py”, line 598, in load
obj = _unpickle(fobj, filename, mmap_mode)
File “/home/qiime2/miniconda/envs/qiime2-2019.7/lib/python3.6/site-packages/joblib/numpy_pickle.py”, line 526, in _unpickle
obj = unpickler.load()
File “/home/qiime2/miniconda/envs/qiime2-2019.7/lib/python3.6/pickle.py”, line 1050, in load
dispatchkey[0]
File “/home/qiime2/miniconda/envs/qiime2-2019.7/lib/python3.6/site-packages/joblib/numpy_pickle.py”, line 352, in load_build
self.stack.append(array_wrapper.read(self))
File “/home/qiime2/miniconda/envs/qiime2-2019.7/lib/python3.6/site-packages/joblib/numpy_pickle.py”, line 195, in read
array = self.read_array(unpickler)
File “/home/qiime2/miniconda/envs/qiime2-2019.7/lib/python3.6/site-packages/joblib/numpy_pickle.py”, line 141, in read_array
array = unpickler.np.empty(count, dtype=self.dtype)
numpy.core._exceptions.MemoryError: Unable to allocate array with shape (895606784,) and data type float64

Plugin error from feature-classifier:

Unable to allocate array with shape (895606784,) and data type float64

I’ve tried this with and without the --p-pre-dispatch ‘n_jobs’ \ but the result is always the same. The VM currently has around 8gb ram allocated to it, and is on a dynamic disk, with the host having around 100gb storage on the drive

Thanks!

1 Like
2

This is a memory error; essentially you do not have enough memory to open the SILVA classifier on your computer.

This is low for the SILVA classifier — it will often take up to 32GB+ if left to its own devices!

There's a couple of things you might be able to do.

First, you can use the --p-reads-per-batch parameter (e.g., set to 1000 or 2000) to reduce the number of reads classified at a time. However, it looks like you do not have enough memory to load the SILVA classifier, not the reads, so this might not help.

The other option is to use the greengenes classifier, which is much smaller and hence less memory-intensive to load.

the only other option will be to find a more powerful machine to run this on, e.g., an computing cluster, AWS.

Good luck!

3 Likes
3

Hi

The other option is to use the greengenes classifier, which is much smaller and hence less memory-intensive to load

I tried this and it worked!! Thanks so much!!

1 Like
4

This topic was automatically closed 31 days after the last reply. New replies are no longer allowed.