Plugin error from diversity: error with ssu computing phylogenetic diversity metrics

I am running into an issue calculating core diversity metrics.

Here is my current configuration:
System versions
Python version: 3.8.13
QIIME 2 release: 2022.8
QIIME 2 version: 2022.8.3
q2cli version: 2022.8.0

Installed plugins
alignment: 2022.8.0
composition: 2022.8.0
cutadapt: 2022.8.0
dada2: 2022.8.0
deblur: 2022.8.0
demux: 2022.8.0
diversity: 2022.8.0
diversity-lib: 2022.8.0
emperor: 2022.8.0
feature-classifier: 2022.8.0
feature-table: 2022.8.1
fragment-insertion: 2022.8.0
gneiss: 2022.8.0
longitudinal: 2022.8.0
metadata: 2022.8.0
phylogeny: 2022.8.0
quality-control: 2022.8.0
quality-filter: 2022.8.0
sample-classifier: 2022.8.0
taxa: 2022.8.0
types: 2022.8.0
vsearch: 2022.8.0

I originally ran the following command:

qiime diversity core-metrics-phylogenetic \
  --i-phylogeny rooted-tree-dada2.qza \
  --i-table table-dada2.qza \
  --p-sampling-depth 5260 \
  --m-metadata-file ../16s.its.9.26.2022.Metadata.Hamilton_corrected_data.tab \
  --output-dir core-metrics-results \
  --verbose

Here is the verbose output:

/home/vinson/miniconda3/envs/qiime2-2022.8/lib/python3.8/site-packages/sklearn/metrics/pairwise.py:1776: DataConversionWarning: Data was converted to boolean for metric jaccard
warnings.warn(msg, DataConversionWarning)
Running external command line application. This may print messages to stdout and/or stderr.
The command being run is below. This command cannot be manually re-run as it will depend on temporary files that no longer exist.

Command:

faithpd -i /tmp/qiime2/vinson/data/c4c939d9-94a7-472d-91d4-7ab17408b769/data/feature-table.biom -t /tmp/qiime2/vinson/data/64121c21-c122-40f5-8391-9359c55a7e09/data/tree.nwk -o /tmp/q2-AlphaDiversityFormat-bejy74cw

Running external command line application. This may print messages to stdout and/or stderr.
The command being run is below. This command cannot be manually re-run as it will depend on temporary files that no longer exist.

Command:

ssu -i /tmp/qiime2/vinson/data/c4c939d9-94a7-472d-91d4-7ab17408b769/data/feature-table.biom -t /tmp/qiime2/vinson/data/64121c21-c122-40f5-8391-9359c55a7e09/data/tree.nwk -m unweighted -o /tmp/q2-LSMatFormat-6aa_twx3

Current file: /opt/conda/conda-bld/unifrac-binaries_1659558422263/work/src/unifrac_cmp.cpp
function: _ZN6su_acc21UnifracUnweightedTaskIdE4_runEjPKd
line: 558
This file was compiled: -ta=tesla:cc35,cc50,cc60,cc60,cc70,cc75,cc80,cc80
Traceback (most recent call last):
File "/home/vinson/miniconda3/envs/qiime2-2022.8/lib/python3.8/site-packages/q2cli/commands.py", line 339, in call
results = action(**arguments)
File "", line 2, in core_metrics_phylogenetic
File "/home/vinson/miniconda3/envs/qiime2-2022.8/lib/python3.8/site-packages/qiime2/sdk/action.py", line 234, in bound_callable
outputs = self.callable_executor(scope, callable_args,
File "/home/vinson/miniconda3/envs/qiime2-2022.8/lib/python3.8/site-packages/qiime2/sdk/action.py", line 475, in callable_executor
outputs = self._callable(scope.ctx, **view_args)
File "/home/vinson/miniconda3/envs/qiime2-2022.8/lib/python3.8/site-packages/q2_diversity/_core_metrics.py", line 66, in core_metrics_phylogenetic
dms += unweighted_unifrac(table=cr.rarefied_table, phylogeny=phylogeny,
File "", line 2, in unweighted_unifrac
File "/home/vinson/miniconda3/envs/qiime2-2022.8/lib/python3.8/site-packages/qiime2/sdk/action.py", line 234, in bound_callable
outputs = self.callable_executor(scope, callable_args,
File "/home/vinson/miniconda3/envs/qiime2-2022.8/lib/python3.8/site-packages/qiime2/sdk/action.py", line 381, in callable_executor
output_views = self._callable(**view_args)
File "", line 2, in unweighted_unifrac
File "/home/vinson/miniconda3/envs/qiime2-2022.8/lib/python3.8/site-packages/q2_diversity_lib/_util.py", line 69, in _disallow_empty_tables
return wrapped_function(*args, **kwargs)
File "", line 2, in unweighted_unifrac
File "/home/vinson/miniconda3/envs/qiime2-2022.8/lib/python3.8/site-packages/q2_diversity_lib/_util.py", line 112, in _validate_requested_cpus
return wrapped_function(*bound_arguments.args, **bound_arguments.kwargs)
File "/home/vinson/miniconda3/envs/qiime2-2022.8/lib/python3.8/site-packages/q2_diversity_lib/beta.py", line 221, in unweighted_unifrac
_omp_cmd_wrapper(threads, cmd)
File "/home/vinson/miniconda3/envs/qiime2-2022.8/lib/python3.8/site-packages/q2_diversity_lib/_util.py", line 128, in _omp_cmd_wrapper
return _run_external_cmd(cmd, verbose=verbose, env=env)
File "/home/vinson/miniconda3/envs/qiime2-2022.8/lib/python3.8/site-packages/q2_diversity_lib/_util.py", line 122, in _run_external_cmd
return subprocess.run(cmd, check=True, env=env)
File "/home/vinson/miniconda3/envs/qiime2-2022.8/lib/python3.8/subprocess.py", line 516, in run
raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['ssu', '-i', '/tmp/qiime2/vinson/data/c4c939d9-94a7-472d-91d4-7ab17408b769/data/feature-table.biom', '-t', '/tmp/qiime2/vinson/data/64121c21-c122-40f5-8391-9359c55a7e09/data/tree.nwk', '-m', 'unweighted', '-o', '/tmp/q2-LSMatFormat-6aa_twx3']' returned non-zero exit status 1.

Plugin error from diversity:

Command '['ssu', '-i', '/tmp/qiime2/vinson/data/c4c939d9-94a7-472d-91d4-7ab17408b769/data/feature-table.biom', '-t', '/tmp/qiime2/vinson/data/64121c21-c122-40f5-8391-9359c55a7e09/data/tree.nwk', '-m', 'unweighted', '-o', '/tmp/q2-LSMatFormat-6aa_twx3']' returned non-zero exit status 1.

See above for debug info.

I then decided to run several alpha and beta-diversity metrics individually. There were no problems with qiime diversity alpha or alpha-phylogenetic distances (observed_features, faith_pd, simpson_e). There were no problems with qiime diversity beta (braycurtis - once I removed samples with zero features). However, I did run into a problem when trying to run qiime diversity beta-phylogenetic (unweighted unifrac).

Here is the command:

qiime diversity beta-phylogenetic \
  --i-table table-dada2.qza \
  --i-phylogeny rooted-tree-dada2.qza \
  --p-metric unweighted_unifrac \
  --o-distance-matrix diversity-metrics/unweighted_unifrac_distance_matrix.qza

Here is the error:

Command '['ssu', '-i', '/tmp/qiime2/vinson/data/17621874-0a08-46c4-9593-cda227972ecc/data/feature-table.biom', '-t', '/tmp/qiime2/vinson/data/64121c21-c122-40f5-8391-9359c55a7e09/data/tree.nwk', '-m', 'unweighted', '-o', '/tmp/q2-LSMatFormat-twh939bi']' returned non-zero exit status 1.
Debug info has been saved to /tmp/qiime2-q2cli-err-4losvsrf.log

Not verbose, but looks the same as the previous error when calculating all metrics.

SO, the next thing I checked was if ssu was installed and running properly. It is. There is no version information for ssu, but here is the usage to confirm I have it installed:

usage: ssu -i -o <out.dm> -m [METHOD] -t [-n threads] [-a alpha] [-f] [--vaw]
[--mode [MODE]] [--start starting-stripe] [--stop stopping-stripe] [--partial-pattern ]
[--n-partials number_of_partitions] [--report-bare] [--format|-r out-mode]

So, I converted my feature table to BIOM and tree (qza) to newick with qiime before running ssu.

mkdir biom

qiime tools export \
  --input-path table-dada2.qza \
  --output-path biom

mkdir rooted-tree-dada2-nwk

qiime tools export \
  --input-path rooted-tree-dada2.qza \
  --output-path rooted-tree-dada2-nwk

mkdir ssu.output

ssu -i biom/feature-table.biom -t rooted-tree-dada2-nwk/tree.nwk -m unweighted -o /home/vinson/qiime2analyses/qiime.phragmites/16s.its.9.26.2022/Q1.analyses/ssu.output/ssu.out.dm 

It seems to have run fine, given the output:

Current file: /opt/conda/conda-bld/unifrac-binaries_1659558422263/work/src/unifrac_cmp.cpp
function: _ZN6su_acc21UnifracUnweightedTaskIdE4_runEjPKd
line: 558
This file was compiled: -ta=tesla:cc35,cc50,cc60,cc60,cc70,cc75,cc80,cc80

However, I can't find a distance matrix anywhere.

Furthermore, I am not sure where to go from here.

I saw another posting about this at plugin error from diversity - going through tutorial, but it was closed without resolution because the OP did not respond, I think.

Thanks for any help.

Hi @vincent,

What operating system, and processor type, are you using? Could you also please provide the output of conda list | grep unifrac?

Thanks,
Daniel

1 Like

Hi @vincent,

We just traced what is most likely going on. Briefly, UniFrac is optimized for GPUs. What we suspect is the codebase is detecting a viable GPU on your system, but the GPU is unavailable for compute, causing UniFrac to fail. It appears the detection mechanism is not robust enough, and we have opened an issue to track this. We expect it to be resolved in the next release of QIIME 2.

In the meantime, use of the GPU can be disabled by setting an environmental variable. Can you try running:

$ export UNIFRAC_USE_GPU=N

...and then rerun the call to UniFrac?

All the best,
Daniel

3 Likes

Update: I reran qiime diversity core-metrics-phylogenetic with success after disabling the GPU.

Incidentally, on a pretty large dataset the calculations were fast, so I am not sure what use cases would necessitate GPU-enabled analyses.

Thanks again,
V

Thanks, Daniel. Sorry for the slow reply. I guess I do not have the appropriate settings to be notified when there is a response posted.

Quick update: I switched to a different linux box (same OS) and upgraded to qiime2-2022.11. The same error message is returned.

One other thing related to your response above. I know I need to upgrade my GPU (for another project to do adaptive sequencing with a Minion). Just in case it is a CUDA capability issue, I have a NVIDIA NVS 310 with a CUDA capability of 2.1

lSb_release -a
Distributor ID: Ubuntu
Description: Ubuntu 18.04.6 LTS
Release: 18.04
Codename: bionic

conda list | grep unifrac

unifrac 1.1.1 py38h17adfb0_1 bioconda
unifrac-binaries 1.1.1 h15a0faf_4 bioconda

Thanks again for your response. I will rerun with the GPU disabled and report the result.

Regards,
V.

1 Like

Wonderful, thank you! I suspect the GPU disable should work

Best,
Daniel

Thanks, Daniel. Disabling the GPU did work and it runs pretty quickly to boot.

Regards,
V

1 Like

Ah, that's great to hear!!