Error when running Ghost tree

Hi,
I'm having the same problem when I use the diversity plugin. I am analyzing ITS readings that particularly do not show problems to merge, but the use of ghost-tree sounds very interesting and I would like to compare its diversity results with a FastTree generated tree.

I've tried the Method 2 from the tutorial and I managed to make a ghost-tree with Silva 138 and UNITE 2020-02-20 release but I couldn't root it neither with qiime phylogeny midpoint-root nor with midpoint.root function on r ('phytools' package).

The former gives the following error:

Traceback (most recent call last):
File "/home/qiime/miniconda2/envs/qiime2-2018.8/lib/python3.5/site-packages/q2cli/commands.py", line 274, in call
results = action(**arguments)
File "", line 2, in midpoint_root
File "/home/qiime/miniconda2/envs/qiime2-2018.8/lib/python3.5/site-packages/qiime2/sdk/action.py", line 199, in bound_callable
self.signature.check_types(**user_input)
File "/home/qiime/miniconda2/envs/qiime2-2018.8/lib/python3.5/site-packages/qiime2/core/type/signature.py", line 299, in check_types
"subtype of %r." % (name, spec.qiime_type))

  Plugin error from phylogeny:

  Argument to parameter 'tree' is not a subtype of Phylogeny[Unrooted].

The latter:

  Error in double(nm * nm) : vector size cannot be NA
  In addition: Warning message:
  In nm * nm : NAs produced by integer overflow

Here is the artifact of my first ghost-tree: ghost-tree-foundation-allignment-80-otus.qza (1.2 MB) (also happens at 90% and 100% similarity. I'd like to compare those too)

I tried re-importing the tree as suggested in this post but didn't work.

Finally, I decided to generate the ghost tree using a pre- and non-pre rooted foundation .nwk tree with the following command:
qiime ghost-tree scaffold-hybrid-tree-foundation-tree
> --i-otu-map extensions_otu_map_80.qza
> --i-extension-taxonomy sh_taxonomy_qiime_ver8_dynamic_s_04.02.2020.qza
> --i-extension-sequences sh_refs_qiime_ver8_dynamic_s_04.02.2020.qza
> --i-foundation-tree tree-tax_slv_ssu_138.qza
> --i-foundation-taxonomy taxonomy_foundation.qza
> --o-ghost-tree ghost-tree-foundation-tree-80-otus.qza

but sandly I received the following error:

Traceback (most recent call last):
File "/home/qiime/miniconda2/envs/qiime2-2018.8/lib/python3.5/site-packages/q2cli/commands.py", line 274, in call
results = action(**arguments)
File "", line 2, in scaffold_hybrid_tree_foundation_tree
File "/home/qiime/miniconda2/envs/qiime2-2018.8/lib/python3.5/site-packages/qiime2/sdk/action.py", line 231, in bound_callable
output_types, provenance)
File "/home/qiime/miniconda2/envs/qiime2-2018.8/lib/python3.5/site-packages/qiime2/sdk/action.py", line 362, in callable_executor
output_views = self._callable(**view_args)
File "/slowdata/smcgreig/q2-ghost-tree/q2_ghost_tree/_scaffold_hybrid_tree_foundation_tree.py", line 49, in scaffold_hybrid_tree_foundation_tree
foundation_taxonomy_fh)[0]
File "/home/qiime/miniconda2/envs/qiime2-2018.8/lib/python3.5/site-packages/ghosttree/scaffold/hybridtree.py", line 116, in extensions_onto_foundation
graft_letter, foundation_taxonomy)
File "/home/qiime/miniconda2/envs/qiime2-2018.8/lib/python3.5/site-packages/ghosttree/scaffold/hybridtree.py", line 275, in _make_nr_foundation_newick
sheared_tree = foundation_tree.shear(foundation_unique_accessions)
File "/home/qiime/miniconda2/envs/qiime2-2018.8/lib/python3.5/site-packages/skbio/tree/_tree.py", line 468, in shear
node.parent.remove(node)
AttributeError: 'NoneType' object has no attribute 'remove'

Plugin error from ghost-tree:

'NoneType' object has no attribute 'remove'

Here are the artifacts used there: extensions_otu_map_80.qza (790.4 KB) sh_refs_qiime_ver8_dynamic_s_04.02.2020.qza (7.6 MB) sh_taxonomy_qiime_ver8_dynamic_s_04.02.2020.qza (1.4 MB) taxonomy_foundation.qza (117.9 KB) tree-tax_slv_ssu_138.qza (28.6 KB) .

I'm no more than a very beginner in bioinformatics and I ran out of ideas. I hope I've been clear enough.
Thank you very much for any help and thank you for developing this interesting tool. I hope the errors are caused by a mistake I've done.

2 Likes

Hello @MatiasF_H,

Welcome to the forums! :qiime2:

Thanks for posting your full log files, along with your .qza files. I think you are running the right commands, but let's see if @Jennifer_Fouquier has any recommendations. :deciduous_tree: :evergreen_tree:

Colin
P.S. I've moved your post to a new thread so we can track it better.

Hi @colinbrislawn,
thanks for the welcome and taking care of the post.
Looking forward to hearing from @Jennifer_Fouquier

M

2 Likes

Hi there! Thanks for the detailed write up. Sorry you’re having trouble. I will try to get to this in a few days but there is a chance I will not be able to get to this until after Tuesday because I have a talk and other things to prepare for that require a lot of time. :grinning:

-Jennifer

Hi Jennifer,
thank you very much for your time.
I just want to point out that I skipped the filter step of SILVA alignments for the creation of the ghost-tree I uploaded above, but unfortunately that wasn't the problem. Here is the new ghost-tree artifact: ghost-tree-foundation-allignment-80-otus.qza (1.2 MB)

Best,
Matias

1 Like

Hi Matias,

I'm so sorry for the delays. In between the national emergencies related to COVID-19, I had a few talks to give this month for my PhD updates. I hope you are doing well!

I got the same error you got with the original files you provided. I extracted the foundation tree and it looks like it doesn't have any information about distance.

Screen Shot 2020-04-01 at 2.45.37 PM

I'm not sure that this tree makes sense to use as a foundation tree. I could be wrong though.... but the point of having a tree is to have the information about relationships in the form of a numerical distance which this tree doesn't give. Let me know if I'm missing something here.

I'm confused about your last reply. If you have the ghost-tree mentioned there, the same commands are not working for you to get other ghost trees?

Thanks!

1 Like

P.S. Although I can see that it is tempting, you really should not align ITS sequences to themselves unless you're only looking at within one genus for some reason. Anything can be aligned because these software tools will attempt to align whatever you give to them, but it doesn't make biological sense to align ITS sequences because they have too many mutations to make high quality alignments. Garbage in, garbage out :slight_smile: If you cannot get ghost tree to work, you would be better off using Bray Curtis or Binary Jaccard metrics to look at the differences in your samples rather than a phylogenetic metric with no meaning. I attached my paper where I used those non-phylogenetic distance metrics before I published ghost-tree. It's usually a pay only article (bummer) so I attached it here. Just to show that I did not use FastTree on my ITS seqs. :grinning: ina.12279.pdf (1.4 MB)

1 Like

Hi Jennifer,
sorry for the delay and thank you very much for taking some of your time in this hard situation!
I haven’t had time to return to this yet, but I will try to give a closer look to my foundation tree as soon as possible and post here any conclusion and the script I’m using, hopefully I has something to do with it.
Also, thanks for sharing your paper, I’m sure it’s good guiding material.

Matias

Hello again,
the good new is that I got the pre-built ghost-tree to work and I like how the results look like:

Unifrac-FastTree


Unifrac-GhostTree

It also got reflected in the stat results :slightly_smiling_face:

However, I couldn't manage to create my own ghost-tree with most updated databases of Unite and Silva.
In the first attemp using QIIME 2018.8, I tryed to create a ghost tree using foundation aligned sequences instead of a tree in order to save some time (at the end it didn't take that much as I thought). That's why it didn’t have any information about distance as you pointed out.
This time, I tryed again with a newer version of QIIME (2019.7) but having the same problems. I also tryed using a foundation tree instead of foundation aligment. Here's the scrip with the files I used:

#Import a SILVA DB and transform from RNA to DNA upon import
qiime tools import \
  --input-path SILVA_138_SSURef_NR99_tax_silva_full_align_trunc.fasta \
  --type FeatureData[AlignedSequence] \
  --input-format AlignedRNAFASTAFormat \
  --output-path SILVA_138_SSURef_NR99_tax_silva_full_align_trunc.qza

#Silva Taxonomy File
qiime tools import \
--input-path tax_slv_ssu_138.txt \
--type SilvaTaxonomy \
--output-path tax_slv_ssu_138.qza \
--input-format SilvaTaxonomyFormat

#Silva Accession ID Map
qiime tools import \
--input-path tax_slv_ssu_138.acc_taxid \
--type SilvaAccession \
--output-path tax_slv_ssu_138.acc_taxid.qza \
--input-format SilvaAccessionFormat

#Extract fungi only from Silva
qiime ghost-tree extract-fungi \
--i-aligned-silva-file SILVA_138_SSURef_NR99_tax_silva_full_align_trunc.qza \
--i-accession-file tax_slv_ssu_138.acc_taxid.qza \
--i-taxonomy-file tax_slv_ssu_138.qza \
--o-aligned-seqs silva_fungi_only_full_aligned_138.qza

#Filter alignment positions
qiime ghost-tree filter-alignment-positions \
--i-aligned-sequences-file silva_fungi_only_full_aligned_138.qza \
--p-maximum-gap-frequency 0.9 \
--p-maximum-position-entropy 0.8 \
--o-aligned-seqs silva_fungi_only_full_aligned_138_FILTERED.qza

#Import a UNITE ITS sequence database:
qiime tools import \
--input-path sh_refs_qiime_ver8_dynamic_s_04.02.2020.fasta \
--type FeatureData[Sequence] \
--output-path sh_refs_qiime_ver8_dynamic_s_04.02.2020.qza

#Import a UNITE ITS taxonomy:
qiime tools import \
--input-path sh_taxonomy_qiime_ver8_dynamic_s_04.02.2020.txt \
--type FeatureData[Taxonomy] \
--input-format HeaderlessTSVTaxonomyFormat \
--output-path sh_taxonomy_qiime_ver8_dynamic_s_04.02.2020.qza

#Group your rep seqs at 90% similarity
qiime ghost-tree extensions-cluster \
--i-extension-sequences sh_refs_qiime_ver8_dynamic_s_04.02.2020.qza \
--p-similarity-threshold 0.90 \
--o-otu-map extensions_otu_map_90.qza

extensions_otu_map_90.qza (898.8 KB)
sh_taxonomy_qiime_ver8_dynamic_s_04.02.2020.qza (1.4 MB)
sh_refs_qiime_ver8_dynamic_s_04.02.2020.qza (7.6 MB)
silva_fungi_only_full_aligned_138_FILTERED.qza (430.5 KB)

#Create a ghost tree using aligned sequences instead of a tree as your foundation
qiime ghost-tree scaffold-hybrid-tree-foundation-alignment \
--i-otu-map extensions_otu_map_90.qza \
--i-extension-taxonomy sh_taxonomy_qiime_ver8_dynamic_s_04.02.2020.qza \
--i-extension-sequences sh_refs_qiime_ver8_dynamic_s_04.02.2020.qza \
--i-foundation-alignment silva_fungi_only_full_aligned_138_FILTERED.qza \
--o-ghost-tree ghost-tree-foundation-allignment-90-otus.qza

qiime phylogeny midpoint-root \
--i-tree ghost-tree-foundation-allignment-90-otus.qza \
--o-rooted-tree rooted-ghost-tree-foundation-allignment-90-otus.qza

#FAILED ROOTING  AS IN FIRST ATTEMPT:
 (1/1) Invalid value for "--i-tree": Expected an artifact of at least type
  Phylogeny[Unrooted]. An artifact of type Phylogeny[Rooted] was provided.

#CREATE A TREE WITH FOUNDATION ALIGNMENTS
qiime phylogeny fasttree \
  --i-alignment silva_fungi_only_full_aligned_138_FILTERED.qza \
  --o-tree unrooted-foundation-tree.qza
qiime phylogeny midpoint-root \
  --i-tree unrooted-foundation-tree.qza \
  --o-rooted-tree rooted-foundation-tree.qza

#IMPORTING THE SAME TAXONOMY FILE AS FeatureData[Taxonomy] AS ghost-tree REQUIRED
    qiime tools import \
    --input-path tax_slv_ssu_138.txt \
    --type FeatureData[Taxonomy] \
    --input-format HeaderlessTSVTaxonomyFormat \
    --output-path foundation-taxonomy.qza

foundation-taxonomy.qza (119.1 KB)
rooted-foundation-tree.qza (242.4 KB)

    qiime ghost-tree scaffold-hybrid-tree-foundation-tree \
    --i-otu-map extensions_otu_map_90.qza \
    --i-extension-taxonomy sh_taxonomy_qiime_ver8_dynamic_s_04.02.2020.qza \
    --i-extension-sequences sh_refs_qiime_ver8_dynamic_s_04.02.2020.qza \
    --i-foundation-tree rooted-foundation-tree.qza \
    --i-foundation-taxonomy foundation-taxonomy.qza \
    --o-ghost-tree ghost-tree-foundation-tree-90-otus-class-level-graft-points.qza \
    --p-graft-level c

    #FAILED CREATING GHOST-TREE
Traceback (most recent call last):
  File "/home/smcgreig/miniconda2/envs/qiime2_19.7/lib/python3.6/site-packages/q2cli/commands.py", line 327, in __call__
    results = action(**arguments)
  File "</home/smcgreig/miniconda2/envs/qiime2_19.7/lib/python3.6/site-packages/decorator.py:decorator-gen-483>", line 2, in scaffold_hybrid_tree_foundation_tree
  File "/home/smcgreig/miniconda2/envs/qiime2_19.7/lib/python3.6/site-packages/qiime2/sdk/action.py", line 240, in bound_callable
    output_types, provenance)
  File "/home/smcgreig/miniconda2/envs/qiime2_19.7/lib/python3.6/site-packages/qiime2/sdk/action.py", line 383, in _callable_executor_
    output_views = self._callable(**view_args)
  File "/home/smcgreig/q2-ghost-tree/q2_ghost_tree/_scaffold_hybrid_tree_foundation_tree.py", line 49, in scaffold_hybrid_tree_foundation_tree
    foundation_taxonomy_fh)[0]
  File "/home/smcgreig/miniconda2/envs/qiime2_19.7/lib/python3.6/site-packages/ghosttree/scaffold/hybridtree.py", line 116, in extensions_onto_foundation
    graft_letter, foundation_taxonomy)
  File "/home/smcgreig/miniconda2/envs/qiime2_19.7/lib/python3.6/site-packages/ghosttree/scaffold/hybridtree.py", line 275, in _make_nr_foundation_newick
    sheared_tree = foundation_tree.shear(foundation_unique_accessions)
  File "/home/smcgreig/miniconda2/envs/qiime2_19.7/lib/python3.6/site-packages/skbio/tree/_tree.py", line 459, in shear
    node.parent.remove(node)
AttributeError: 'NoneType' object has no attribute 'remove'

Plugin error from ghost-tree:

  'NoneType' object has no attribute 'remove'

See above for debug info.

unrooted-foundation-tree.qza (239.9 KB)

    #TRYING WITH THE UNROOTED FOUNDATION TREE
    qiime ghost-tree scaffold-hybrid-tree-foundation-tree \
    --i-otu-map extensions_otu_map_90.qza \
    --i-extension-taxonomy sh_taxonomy_qiime_ver8_dynamic_s_04.02.2020.qza \
    --i-extension-sequences sh_refs_qiime_ver8_dynamic_s_04.02.2020.qza \
    --i-foundation-tree unrooted-foundation-tree.qza \
    --i-foundation-taxonomy foundation-taxonomy.qza \
    --o-ghost-tree ghost-tree-foundation-tree-90-otus-class-level-graft-points.qza \
    --p-graft-level c

    #FAILED CREATING GHOST-TREE

Sorry for the long posts and I would totally understand if this can not be addressed due to the current situation. As I said I'm quite hapy with the results obtained using a pre-built ghost-tree and I guess that I wouldn't change drastically my results using newer verisons of the databases.

Best wishes and take care,
Matias

2 Likes