Error running feature-classifier classify-sklearn (memory related)

[Nicholas_Bokulich]

Welcome to the forum @matrs, and thanks for asking about this/troubleshooting:

@matrs

Edit: When I run it with less cores and the classification finished without errors, I could see that those almost 20 GB per job were a peak, which occurs when multiple jobs get started (which isn’t the beginning of the run), then every job stabilized its use of memory around 13-14 GB and ended that way, so I always got the error at the beginning of the multi-core part of the run.

Yes, this behavior is (unfortunately) expected — the reason is that each core is storing a separate copy of the classifier in memory, hence the rapid memory consumption at the start of each job. The solution is as you have discovered: either reduce the number of jobs or increase the memory available to each job (if possible).

The SILVA database is also quite large and consumes a great deal of memory. Using Greengenes or another smaller database will require around 4X less memory.