Diversity: The table does not appear to be completely represented by the phylogeny.

Kinga_Zielinska · January 24, 2024, 9:00pm

Hi, I'm trying to calculate beta diversity (unifrac), but I keep getting the following error: "The table does not appear to be completely represented by the phylogeny.". Specifically, the "--verbose" output produces:

Traceback (most recent call last):
File "/home/kaz/miniconda3/envs/qiime2/lib/python3.6/site-packages/q2cli/commands.py", line 329, in call
results = action(**arguments)
File "", line 2, in beta_phylogenetic
File "/home/kaz/miniconda3/envs/qiime2/lib/python3.6/site-packages/qiime2/sdk/action.py", line 245, in bound_callable
output_types, provenance)
File "/home/kaz/miniconda3/envs/qiime2/lib/python3.6/site-packages/qiime2/sdk/action.py", line 484, in callable_executor
outputs = self._callable(scope.ctx, **view_args)
File "/home/kaz/miniconda3/envs/qiime2/lib/python3.6/site-packages/q2_diversity/_beta/_pipeline.py", line 32, in beta_phylogenetic
bypass_tips=bypass_tips)
File "", line 2, in unweighted_unifrac
File "/home/kaz/miniconda3/envs/qiime2/lib/python3.6/site-packages/qiime2/sdk/action.py", line 245, in bound_callable
output_types, provenance)
File "/home/kaz/miniconda3/envs/qiime2/lib/python3.6/site-packages/qiime2/sdk/action.py", line 390, in callable_executor
output_views = self._callable(**view_args)
File "", line 2, in unweighted_unifrac
File "/home/kaz/miniconda3/envs/qiime2/lib/python3.6/site-packages/q2_diversity_lib/_util.py", line 49, in _disallow_empty_tables
return wrapped_function(*args, **kwargs)
File "", line 2, in unweighted_unifrac
File "/home/kaz/miniconda3/envs/qiime2/lib/python3.6/site-packages/q2_diversity_lib/_util.py", line 92, in _validate_requested_cpus
return wrapped_function(*bound_arguments.args, **bound_arguments.kwargs)
File "/home/kaz/miniconda3/envs/qiime2/lib/python3.6/site-packages/q2_diversity_lib/beta.py", line 159, in unweighted_unifrac
variance_adjusted=False, bypass_tips=bypass_tips)
File "/home/kaz/miniconda3/envs/qiime2/lib/python3.6/site-packages/unifrac/_methods.py", line 103, in unweighted
variance_adjusted, 1.0, bypass_tips, threads)
File "unifrac/_api.pyx", line 86, in unifrac._api.ssu
ValueError: The table does not appear to be completely represented by the phylogeny.

Plugin error from diversity:

The table does not appear to be completely represented by the phylogeny.

See above for debug info.

I tried the "qiime phylogeny filter-table" to overlap the tree and my input matrices but the output file is empty.
I modified the tree manually based on my table, and vice versa. I left only species level (with and without "s__"), I also tried keeping all taxonomic levels. Still no results.
The tree is downloaded from the GTDB (version 214) and imported to QIIME2 as a tree in a newick format, *qza.

I would greatly appreciate your help!

SoilRotifer · January 24, 2024, 10:38pm

In order to make use of UniFrac, or any phylogenetic-based diversity metric, the features in your feature-table must also be present in your tree. So, unless you somehow inserted your sequences into the GTDB tree, this will not work. There are a couple of approaches you can use to construct a phylogeny which are already built into QIIME 2:

Construct a de novo phylogeny with your features.
Run fragment insertion. You can use either the Greengenes or SILVA SEPP files located here.

Kinga_Zielinska · January 25, 2024, 7:08am

Thank you for your response!
What I did not add is that this is whole metagenome sequencing, not 16S, and I do not have representative sequences.
The taxonomy was assigned using GTDB-Tk, and GTDB allows you to download the tree that they're using. When I manually compare my matrix and the GTDB tree in the *tree format before importing it as *qza (I do it to be able to open it in Notepad), many of the features, if not all, are present in the tree.

SoilRotifer · January 25, 2024, 2:04pm

Ideally, UniFrac is meant to work with amplicon sequences, not shotgun metagenomic data. Though if the feature-IDs in your feature-table match the IDs in the tree I'd think it'd be possible to get this to work. As long as it is a standard feature table and newick tree.

Can you share your feature-table and tree files? You can send me a private DM with a link to the files if that is easiest.

Kinga_Zielinska · January 25, 2024, 2:18pm

Thank you! Just sent a private message.

SoilRotifer · January 25, 2024, 3:33pm

Hi @Kinga_Zielinska,

The issue is that the IDs in the phylogeny are not an exact match to the IDs in the feature table. For example, the tree tip labels are in the form of: GB_GCA_017411805.1. These tip labels must match the OTU ID / feature-id of the feature table. Unfortunately, the OTU IDs in the feature-table are in the form of CAG-390 sp000437015. Which do not exist in the tree.

Although the the label CAG-390 does exist within the tree as a node in the form of g__CAG-390:

You'll have to revisit how you can generate files compatible for phylogenetic-based diversity analyses. Again, the IDs in the feature table must match the tip labels of the tree.

Perhaps others on the forum have suggestions?

-Cheers!

Kinga_Zielinska · January 25, 2024, 3:42pm

Ok, that makes sense. So if I understand it correctly, will it work if I change my species names ("s__" etc) to the GB_GCA_* notation right?

SoilRotifer · January 25, 2024, 4:14pm

As long as you have a way of mapping the IDs in your feature-table to the IDs in the tree.

Kinga_Zielinska · January 26, 2024, 4:22pm

I performed the mapping but still same error I sent you a test file in a DM.

SoilRotifer · January 26, 2024, 4:55pm

Not sure what error you are seeing. Please provide all commands you've run to import the data and run beta diversity.

I did notice that I had to modify your input file by replacing name with Sample ID in the header. Then I ran the following commands to import into QIIME 2:

biom convert \
    -i Illumina_merged_output_all_shared_samename_s_mapped_v02.txt \
    -o Illumina_merged_output_all_shared_samename_s_mapped_v02.biom \
    --table-type="OTU table" \
    --to-hdf5

qiime tools import \
    --input-path Illumina_merged_output_all_shared_samename_s_mapped_v02.biom \
    --type 'FeatureTable[RelativeFrequency]' \
    --output-path Illumina_merged_output_all_shared_samename_s_mapped_v02.qza

Kinga_Zielinska · January 27, 2024, 6:42pm

I imported the txt matrix to QIIME 2 in the same way as you (plus now changed "name" to "Sample ID"). When I type the following command:

qiime phylogeny filter-table --i-table Illumina_merged_output_all_shared_samename_s_mapped_vo2.qza --i-tree bac120_r214.qza --o-filtered-table Illumina_merged_output_all_shared_samename_s_mapped_filtered.qza

the output is empty.
When I'm trying to calculate beta diversity:

qiime diversity beta-phylogenetic --i-table Illumina_merged_output_all_shared_samename_s_mapped_vo2.qza --i-phylogen
y bac120_r214.qza --p-metric weighted_unifrac --o-distance-matrix Illumina_merged_output_all_shared_samename_s_mapped_beta_unifrac.qza

I'm getting the following error:

Plugin error from diversity:

  The table does not appear to be completely represented by the phylogeny.

Debug info has been saved to /tmp/qiime2-q2cli-err-6jqyxzum.log

and the log file says:

Traceback (most recent call last):
  File "/home/kaz/miniconda3/envs/qiime2/lib/python3.6/site-packages/q2cli/commands.py", line 329, in __call__
    results = action(**arguments)
  File "<decorator-gen-399>", line 2, in beta_phylogenetic
  File "/home/kaz/miniconda3/envs/qiime2/lib/python3.6/site-packages/qiime2/sdk/action.py", line 245, in bound_callable
    output_types, provenance)
  File "/home/kaz/miniconda3/envs/qiime2/lib/python3.6/site-packages/qiime2/sdk/action.py", line 484, in _callable_executor_
    outputs = self._callable(scope.ctx, **view_args)
  File "/home/kaz/miniconda3/envs/qiime2/lib/python3.6/site-packages/q2_diversity/_beta/_pipeline.py", line 32, in beta_phylogenetic
    bypass_tips=bypass_tips)
  File "<decorator-gen-554>", line 2, in weighted_unifrac
  File "/home/kaz/miniconda3/envs/qiime2/lib/python3.6/site-packages/qiime2/sdk/action.py", line 245, in bound_callable
    output_types, provenance)
  File "/home/kaz/miniconda3/envs/qiime2/lib/python3.6/site-packages/qiime2/sdk/action.py", line 390, in _callable_executor_
    output_views = self._callable(**view_args)
  File "<decorator-gen-397>", line 2, in weighted_unifrac
  File "/home/kaz/miniconda3/envs/qiime2/lib/python3.6/site-packages/q2_diversity_lib/_util.py", line 49, in _disallow_empty_tables
    return wrapped_function(*args, **kwargs)
  File "<decorator-gen-396>", line 2, in weighted_unifrac
  File "/home/kaz/miniconda3/envs/qiime2/lib/python3.6/site-packages/q2_diversity_lib/_util.py", line 92, in _validate_requested_cpus
    return wrapped_function(*bound_arguments.args, **bound_arguments.kwargs)
  File "/home/kaz/miniconda3/envs/qiime2/lib/python3.6/site-packages/q2_diversity_lib/beta.py", line 170, in weighted_unifrac
    bypass_tips=bypass_tips)
  File "/home/kaz/miniconda3/envs/qiime2/lib/python3.6/site-packages/unifrac/_methods.py", line 212, in weighted_unnormalized
    variance_adjusted, 1.0, bypass_tips, threads)
  File "unifrac/_api.pyx", line 86, in unifrac._api.ssu
ValueError: The table does not appear to be completely represented by the phylogeny.

SoilRotifer · January 29, 2024, 12:23am

Hi @Kinga_Zielinska,

ValueError: The table does not appear to be completely represented by the phylogeny.

This error is the same error message you've encountered initially. Like I said:

In order to make use of UniFrac, or any phylogenetic-based diversity metric, the features in your feature-table must also be present in your tree.

Again, this message is telling you that you have feature IDs in your table that are not present in your tree. You need to make sure each feature ID in your table is present in your tree.

Kinga_Zielinska · January 30, 2024, 7:43am

You're right - I still had a few missing IDs left! All worked now - thank you!!

system · March 1, 2024, 1:43pm

This topic was automatically closed 31 days after the last reply. New replies are no longer allowed.