Classify-samples `--p-n-jobs`: high cpu load

[nick-youngblut]

I’m running qiime sample-classifier classify-samples with --p-n-jobs set to 30. Initially during the job run, the CPU load is 30 as expected. However, the CPU load jumps to ~200 later in the run. For part of classify-samples, are subprocesses calling more subprocess or something? I’m just wondering why the CPU load jumps so high.

Server OS: Ubuntu 16.04.3

$ qiime info
System versions
Python version: 3.5.4
QIIME 2 release: 2017.10
QIIME 2 version: 2017.10.0
q2cli version: 2017.10.0

Installed plugins
alignment 2017.10.0
composition 2017.10.0
dada2 2017.10.0
deblur 2017.10.0
demux 2017.10.0
diversity 2017.10.0
emperor 2017.10.0
feature-classifier 2017.10.0
feature-table 2017.10.0
gneiss 2017.10.0
longitudinal 2017.10.0
metadata 2017.10.0
phylogeny 2017.10.0
quality-control 2017.10.0
quality-filter 2017.10.0
sample-classifier 2017.10.0
taxa 2017.10.0
types 2017.10.0
vsearch 2017.10.0

[Nicholas_Bokulich]

Thanks for reporting, @nick-youngblut!

It sounds like maybe a hidden subprocess is being called somewhere — I am not 100% certain but have put in a ticket to get this checked out.

Could you also please share the command that you are using?

[nick-youngblut]

Thanks for looking into it! Here’s the general command that caused the large CPU load:

qiime sample-classifier classify-samples \
  --i-table {feature_table} \
  --m-metadata-file {metadata} \
  --m-metadata-category {category} \
  --p-optimize-feature-selection \
  --p-parameter-tuning \
  --p-estimator RandomForestClassifier \
  --p-cv 5 \
  --p-n-estimators 500 \
  --p-n-jobs 30 \
  --o-visualization {output}