Import frequency tables from DADA2 to qiime2

Hi! I am trying to import dada2 table (following the tutorial in its website) to qiime2, and nothing works :frowning:
hope someone could give a better solution.

I firstly converted freq_table.txt (it was seqtab.nochim transposed) to hdf5 biom table format (freq_table.biom).
then with qiime import tools, convert it to an artifact for qiime2 with this command
qiime tools import --input-path freq_table.biom --type "FeatureTable[Frequency]" --output-path freq_table

I did the same with rep_set.nochimeras.OTUs.fasta (
qiime tools import --input-path rep_set.nochimeras.OTUs.fasta --output-path rep_seqs-table --type "FeatureData[Sequence]"
and Generate the tree for phylogenetic diversity analyses.

Then, I try to run the command qiime diversity core-metrics, it gave this error message.

(qiime2) [labsani@labsani-HP-ProBook-470-G1 200-160]$ qiime diversity core-metrics --i-phylogeny rooted-tree.qza --i-table freq_table.qza --p-sampling-depth 55624 --output-dir cm55624
Traceback (most recent call last):
  File "/home/labsani/miniconda3/envs/qiime2/bin/qiime", line 6, in <module>
    sys.exit(q2cli.__main__.qiime())
  File "/home/labsani/miniconda3/envs/qiime2/lib/python3.5/site-packages/click/core.py", line 716, in __call__
    return self.main(*args, **kwargs)
  File "/home/labsani/miniconda3/envs/qiime2/lib/python3.5/site-packages/click/core.py", line 696, in main
    rv = self.invoke(ctx)
  File "/home/labsani/miniconda3/envs/qiime2/lib/python3.5/site-packages/click/core.py", line 1060, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/labsani/miniconda3/envs/qiime2/lib/python3.5/site-packages/click/core.py", line 1060, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/labsani/miniconda3/envs/qiime2/lib/python3.5/site-packages/click/core.py", line 889, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/labsani/miniconda3/envs/qiime2/lib/python3.5/site-packages/click/core.py", line 534, in invoke
    return callback(*args, **kwargs)
  File "/home/labsani/miniconda3/envs/qiime2/lib/python3.5/site-packages/q2cli-0.0.6-py3.5.egg/q2cli/commands.py", line 210, in __call__
  File "<decorator-gen-234>", line 2, in core_metrics
  File "/home/labsani/miniconda3/envs/qiime2/lib/python3.5/site-packages/qiime-2.0.6-py3.5.egg/qiime/core/callable.py", line 227, in callable_wrapper
    output_types, provenance)
  File "/home/labsani/miniconda3/envs/qiime2/lib/python3.5/site-packages/qiime-2.0.6-py3.5.egg/qiime/core/callable.py", line 328, in _callable_executor_
    output_views = callable(**view_args)
  File "<string>", line 26, in core_metrics
  File "/home/labsani/miniconda3/envs/qiime2/lib/python3.5/site-packages/q2_diversity-0.0.6-py3.5.egg/q2_diversity/_alpha/_method.py", line 44, in alpha_phylogenetic
  File "/home/labsani/miniconda3/envs/qiime2/lib/python3.5/site-packages/skbio/diversity/_driver.py", line 170, in alpha_diversity
    counts, otu_ids, tree, validate, single_sample=False)
  File "/home/labsani/miniconda3/envs/qiime2/lib/python3.5/site-packages/skbio/diversity/alpha/_faith_pd.py", line 136, in _setup_faith_pd
    _validate_otu_ids_and_tree(counts[0], otu_ids, tree)
  File "/home/labsani/miniconda3/envs/qiime2/lib/python3.5/site-packages/skbio/diversity/_util.py", line 106, in _validate_otu_ids_and_tree
    " ".join(missing_tip_names)))
skbio.tree._exception.MissingNodeError: All ``otu_ids`` must be present as tip names in ``tree``. ``otu_ids`` not corresponding to tip names (n=971): OTU_370 OTU_145 OTU_846 OTU_539 OTU_944 OTU_200 OTU_470 OTU_205 OTU_819 OTU_853 OTU_829 OTU_940 OTU_103 OTU_237 OTU_264 OTU_973 OTU_533 OTU_355 OTU_343 OTU_363 OTU_949 OTU_759 OTU_588 OTU_739 OTU_872 OTU_762 OTU_662 OTU_92 OTU_868 OTU_315 OTU_531 OTU_27 OTU_858 OTU_675 OTU_571 OTU_771 OTU_451 OTU_78 OTU_63 OTU_52 OTU_891 OTU_397 OTU_442 OTU_406 OTU_305 OTU_419 OTU_309 OTU_915 OTU_895 OTU_970 OTU_542 OTU_263 OTU_257 OTU_864 OTU_589 OTU_719 OTU_354 OTU_273 OTU_20 OTU_130 OTU_921 OTU_478 OTU_960 OTU_398 OTU_296 OTU_91 OTU_192 OTU_848 OTU_15 OTU_559 OTU_179 OTU_875 OTU_881 OTU_534 OTU_285 OTU_266 OTU_964 OTU_520 OTU_688 OTU_597 OTU_232 OTU_458 OTU_898 OTU_258 OTU_371 OTU_678 OTU_906 OTU_368 OTU_823 OTU_209 OTU_789 OTU_802 OTU_365 OTU_543 OTU_639 OTU_56 OTU_244 OTU_48 OTU_77 OTU_161 OTU_447 OTU_380 OTU_854 OTU_694 OTU_149 OTU_943 OTU_268 OTU_182 OTU_299 OTU_885 OTU_961 OTU_776 OTU_839 OTU_653 OTU_918 OTU_832 OTU_81 OTU_602 OTU_967 OTU_874 OTU_8 OTU_219 OTU_345 OTU_54 OTU_49 OTU_624 OTU_903 OTU_625 OTU_393 OTU_279 OTU_98 OTU_911 OTU_723 OTU_408 OTU_367 OTU_251 OTU_19 OTU_908 OTU_689 OTU_213 OTU_745 OTU_210 OTU_157 OTU_724 OTU_959 OTU_38 OTU_247 OTU_191 OTU_506 OTU_265 OTU_805 OTU_456 OTU_657 OTU_847 OTU_880 OTU_269 OTU_515 OTU_732 OTU_238 OTU_699 OTU_167 OTU_786 OTU_746 OTU_217 OTU_859 OTU_55 OTU_443 OTU_513 OTU_704 OTU_754 OTU_391 OTU_204 OTU_837 OTU_978 OTU_508 OTU_472 OTU_930 OTU_738 OTU_95 OTU_256 OTU_640 OTU_133 OTU_884 OTU_93 OTU_88 OTU_267 OTU_979 OTU_412 OTU_528 OTU_74 OTU_121 OTU_471 OTU_125 OTU_525 OTU_208 OTU_538 OTU_275 OTU_926 OTU_937 OTU_431 OTU_626 OTU_344 OTU_935 OTU_289 OTU_716 OTU_3 OTU_430 OTU_33 OTU_609 OTU_659 OTU_246 OTU_171 OTU_830 OTU_198 OTU_32 OTU_293 OTU_413 OTU_399 OTU_907 OTU_831 OTU_556 OTU_600 OTU_68 OTU_429 OTU_298 OTU_580 OTU_582 OTU_41 OTU_425 OTU_593 OTU_796 OTU_685 OTU_26 OTU_420 OTU_128 OTU_968 OTU_766 OTU_410 OTU_339 OTU_127 OTU_455 OTU_183 OTU_564 OTU_417 OTU_79 OTU_690 OTU_385 OTU_105 OTU_433 OTU_173 OTU_616 OTU_714 OTU_155 OTU_446 OTU_302 OTU_548 OTU_598 OTU_825 OTU_512 OTU_203 OTU_567 OTU_774 OTU_5 OTU_43 OTU_629 OTU_16 OTU_236 OTU_168 OTU_599 OTU_952 OTU_278 OTU_654 OTU_428 OTU_554 OTU_971 OTU_498 OTU_311 OTU_622 OTU_517 OTU_710 OTU_312 OTU_107 OTU_36 OTU_223 OTU_156 OTU_924 OTU_335 OTU_769 OTU_282 OTU_190 OTU_113 OTU_778 OTU_876 OTU_503 OTU_194 OTU_494 OTU_426 OTU_222 OTU_845 OTU_349 OTU_642 OTU_611 OTU_120 OTU_353 OTU_369 OTU_330 OTU_382 OTU_172 OTU_310 OTU_901 OTU_813 OTU_701 OTU_529 OTU_842 OTU_184 OTU_744 OTU_322 OTU_124 OTU_755 OTU_466 OTU_96 OTU_734 OTU_607 OTU_772 OTU_454 OTU_462 OTU_692 OTU_592 OTU_199 OTU_932 OTU_303 OTU_596 OTU_577 OTU_80 OTU_482 OTU_216 OTU_812 OTU_933 OTU_474 OTU_764 OTU_958 OTU_329 OTU_527 OTU_253 OTU_788 OTU_672 OTU_324 OTU_366 OTU_1 OTU_917 OTU_387 OTU_294 OTU_888 OTU_763 OTU_806 OTU_211 OTU_252 OTU_697 OTU_863 OTU_185 OTU_475 OTU_945 OTU_75 OTU_620 OTU_396 OTU_601 OTU_480 OTU_45 OTU_469 OTU_507 OTU_535 OTU_546 OTU_637 OTU_750 OTU_667 OTU_108 OTU_146 OTU_817 OTU_914 OTU_735 OTU_700 OTU_768 OTU_245 OTU_197 OTU_621 OTU_818 OTU_843 OTU_569 OTU_486 OTU_177 OTU_619 OTU_392 OTU_117 OTU_284 OTU_473 OTU_614 OTU_11 OTU_740 OTU_207 OTU_705 OTU_742 OTU_334 OTU_180 OTU_931 OTU_941 OTU_591 OTU_336 OTU_544 OTU_414 OTU_665 OTU_76 OTU_154 OTU_394 OTU_46 OTU_671 OTU_563 OTU_214 OTU_636 OTU_623 OTU_375 OTU_782 OTU_364 OTU_709 OTU_405 OTU_465 OTU_633 OTU_791 OTU_286 OTU_501 OTU_175 OTU_308 OTU_481 OTU_821 OTU_85 OTU_39 OTU_23 OTU_877 OTU_110 OTU_758 OTU_730 OTU_461 OTU_713 OTU_886 OTU_887 OTU_106 OTU_373 OTU_202 OTU_963 OTU_668 OTU_927 OTU_321 OTU_797 OTU_416 OTU_139 OTU_913 OTU_316 OTU_785 OTU_553 OTU_909 OTU_628 OTU_712 OTU_338 OTU_502 OTU_318 OTU_83 OTU_249 OTU_174 OTU_757 OTU_972 OTU_557 OTU_218 OTU_99 OTU_144 OTU_753 OTU_261 OTU_53 OTU_243 OTU_484 OTU_70 OTU_728 OTU_890 OTU_82 OTU_581 OTU_37 OTU_905 OTU_632 OTU_47 OTU_670 OTU_647 OTU_291 OTU_733 OTU_195 OTU_69 OTU_566 OTU_160 OTU_231 OTU_337 OTU_545 OTU_651 OTU_119 OTU_452 OTU_18 OTU_288 OTU_674 OTU_946 OTU_947 OTU_233 OTU_347 OTU_361 OTU_404 OTU_572 OTU_14 OTU_388 OTU_225 OTU_612 OTU_300 OTU_568 OTU_485 OTU_595 OTU_307 OTU_586 OTU_348 OTU_434 OTU_715 OTU_896 OTU_975 OTU_811 OTU_59 OTU_66 OTU_794 OTU_28 OTU_104 OTU_444 OTU_826 OTU_779 OTU_698 OTU_227 OTU_44 OTU_22 OTU_491 OTU_660 OTU_115 OTU_893 OTU_381 OTU_7 OTU_680 OTU_643 OTU_151 OTU_129 OTU_631 OTU_476 OTU_815 OTU_957 OTU_729 OTU_460 OTU_403 OTU_761 OTU_807 OTU_851 OTU_165 OTU_500 OTU_112 OTU_170 OTU_814 OTU_725 OTU_441 OTU_295 OTU_61 OTU_438 OTU_435 OTU_965 OTU_21 OTU_90 OTU_346 OTU_248 OTU_606 OTU_239 OTU_810 OTU_374 OTU_325 OTU_143 OTU_449 OTU_922 OTU_109 OTU_916 OTU_297 OTU_902 OTU_558 OTU_467 OTU_673 OTU_457 OTU_594 OTU_317 OTU_64 OTU_587 OTU_804 OTU_953 OTU_574 OTU_358 OTU_722 OTU_242 OTU_519 OTU_855 OTU_681 OTU_835 OTU_840 OTU_135 OTU_360 OTU_497 OTU_547 OTU_850 OTU_411 OTU_551 OTU_861 OTU_71 OTU_682 OTU_866 OTU_956 OTU_319 OTU_448 OTU_362 OTU_240 OTU_942 OTU_440 OTU_292 OTU_94 OTU_900 OTU_824 OTU_407 OTU_856 OTU_439 OTU_630 OTU_189 OTU_147 OTU_206 OTU_820 OTU_590 OTU_798 OTU_241 OTU_841 OTU_126 OTU_857 OTU_934 OTU_111 OTU_706 OTU_328 OTU_153 OTU_703 OTU_645 OTU_883 OTU_928 OTU_510 OTU_314 OTU_254 OTU_372 OTU_516 OTU_879 OTU_67 OTU_164 OTU_499 OTU_65 OTU_718 OTU_57 OTU_468 OTU_663 OTU_60 OTU_84 OTU_751 OTU_635 OTU_463 OTU_379 OTU_100 OTU_390 OTU_720 OTU_780 OTU_123 OTU_777 OTU_138 OTU_162 OTU_422 OTU_166 OTU_152 OTU_676 OTU_384 OTU_641 OTU_711 OTU_320 OTU_708 OTU_912 OTU_450 OTU_140 OTU_948 OTU_13 OTU_962 OTU_4 OTU_141 OTU_923 OTU_869 OTU_748 OTU_504 OTU_17 OTU_646 OTU_415 OTU_809 OTU_610 OTU_573 OTU_9 OTU_137 OTU_976 OTU_201 OTU_652 OTU_644 OTU_351 OTU_677 OTU_377 OTU_658 OTU_784 OTU_731 OTU_378 OTU_58 OTU_920 OTU_565 OTU_341 OTU_304 OTU_432 OTU_792 OTU_427 OTU_765 OTU_323 OTU_800 OTU_181 OTU_801 OTU_950 OTU_487 OTU_276 OTU_383 OTU_386 OTU_395 OTU_707 OTU_97 OTU_488 OTU_376 OTU_767 OTU_691 OTU_743 OTU_250 OTU_272 OTU_212 OTU_521 OTU_760 OTU_936 OTU_634 OTU_50 OTU_865 OTU_122 OTU_492 OTU_696 OTU_726 OTU_522 OTU_882 OTU_356 OTU_148 OTU_352 OTU_274 OTU_747 OTU_186 OTU_277 OTU_87 OTU_89 OTU_524 OTU_224 OTU_136 OTU_873 OTU_532 OTU_132 OTU_799 OTU_585 OTU_684 OTU_878 OTU_560 OTU_781 OTU_904 OTU_40 OTU_727 OTU_340 OTU_30 OTU_142 OTU_229 OTU_134 OTU_401 OTU_118 OTU_327 OTU_479 OTU_255 OTU_116 OTU_808 OTU_131 OTU_683 OTU_666 OTU_687 OTU_196 OTU_62 OTU_862 OTU_290 OTU_530 OTU_737 OTU_230 OTU_836 OTU_176 OTU_389 OTU_270 OTU_437 OTU_342 OTU_178 OTU_514 OTU_549 OTU_910 OTU_459 OTU_693 OTU_576 OTU_951 OTU_695 OTU_163 OTU_505 OTU_803 OTU_158 OTU_226 OTU_849 OTU_518 OTU_897 OTU_477 OTU_939 OTU_828 OTU_24 OTU_29 OTU_301 OTU_550 OTU_669 OTU_537 OTU_661 OTU_445 OTU_424 OTU_679 OTU_894 OTU_955 OTU_721 OTU_795 OTU_870 OTU_235 OTU_664 OTU_615 OTU_326 OTU_2 OTU_816 OTU_169 OTU_42 OTU_570 OTU_287 OTU_228 OTU_31 OTU_526 OTU_756 OTU_357 OTU_790 OTU_421 OTU_834 OTU_648 OTU_313 OTU_331 OTU_833 OTU_359 OTU_496 OTU_402 OTU_752 OTU_262 OTU_575 OTU_101 OTU_51 OTU_686 OTU_603 OTU_552 OTU_649 OTU_511 OTU_844 OTU_72 OTU_736 OTU_741 OTU_215 OTU_271 OTU_10 OTU_892 OTU_827 OTU_34 OTU_793 OTU_423 OTU_613 OTU_899 OTU_234 OTU_259 OTU_540 OTU_493 OTU_114 OTU_464 OTU_187 OTU_749 OTU_608 OTU_280 OTU_333 OTU_787 OTU_12 OTU_489 OTU_702 OTU_332 OTU_860 OTU_561 OTU_523 OTU_627 OTU_954 OTU_938 OTU_73 OTU_578 OTU_281 OTU_584 OTU_221 OTU_541 OTU_483 OTU_618 OTU_260 OTU_605 OTU_617 OTU_579 OTU_220 OTU_25 OTU_555 OTU_6 OTU_773 OTU_509 OTU_150 OTU_102 OTU_919 OTU_418 OTU_193 OTU_871 OTU_35 OTU_159 OTU_770 OTU_852 OTU_889 OTU_350 OTU_436 OTU_717 OTU_409 OTU_867 OTU_656 OTU_655 OTU_638 OTU_495 OTU_929 OTU_283 OTU_583 OTU_536 OTU_400 OTU_188 OTU_490 OTU_604 OTU_453 OTU_783 OTU_306 OTU_562 OTU_86 OTU_838

Apparently, both .qza inputs have the semantic type needed. Both the rooted-tree.qza and freq_table.qza have 979 OTUs. I think that the problem was in importing the tables, but not sure.

Any help would much be appreciated!
Thankss!

Hi @m-ocejo, the error message indicates that 971 of the feature IDs in freq_table.qza didn’t have a matching tip name in your phylogenetic tree (rooted-tree.qza ). Each feature ID in your feature table must have a corresponding tip in the phylogenetic tree.

Can you please post the commands you ran to generate the rooted phylogenetic tree?

1 Like

sure! I followed the “Moving Pictures” Tutorial of qiime2:
Generate a tree for phylogenetic diversity analyses

qiime alignment mafft --i-sequences rep-seqs-table.qza --o-alignment aligned-rep-seqs
qiime alignment mask --i-alignment aligned-rep-seqs.qza --o-masked-alignment masked-aligned-rep-seqs
qiime phylogeny fasttree --i-alignment masked-aligned-rep-seqs.qza --o-tree unrooted-tree
qiime phylogeny midpoint-root  --i-tree unrooted-tree.qza --o-rooted-tree rooted-tree

I explored the tree and all the OTUs from the freq_table.biom seemed to be present in the tree.

Hi @m-ocejo,
The error message lists the feature ids that are not present in the tree - for example, OTU_370 seems to be one of those. Can you confirm whether that is in your tree?

It looks like the sequences file that you’re starting with has had chimera filtering applied to it. Have the features that were identified as chimeras also been removed from the table?

There is a method available, qiime phylogeny filter-table, that will remove features from the table if they are not present as tips in the tree. This would solve the issue that you’re having, but I think you should first figure out why those features aren’t in the tree just to make sure that there isn’t something going on that you’re not aware of.

If you can confirm that the feature ID OTU_370 is present in the file rep_set.nochimeras.OTUs.fasta, could you send me that file and freq_table.biom and I’ll try to reproduce the issue on my end.

Hi Greg,
Thx for the reply. For the rep-set.nochimeras.otus.fasta, I used the seqtab.nochim of DADA2 and run UniquesToFasta in R.
After the alignments, I visualized the nwk tree file and found all the OTUS from the biom table.

I am sending you the artifacts used for the core-metrics input to your email. My idea was to use a sample depth excluding the lowest read sample.

Thanks!!

Hi @m-ocejo, could you email me the files you imported into qiime as well? I’ll be able to take a look tomorrow morning.

OK @gregcaporaso !
Sent to your email. :slight_smile:

Hi @m-ocejo,
The issue here actually comes down to a requirement on the newick file format (which gets used internally in QIIME) that the FastTree program isn’t handling by defualt. The underscore character that you’re using in your OTU ids (i.e., _ as in OTU_123) gets interpreted as a space in the newick format, so the feature ids in the tree that you’re generating end up looking like OTU 123. (You can find a description of this in the scikit-bio documentation here, but it’s not something that QIIME users are expected to have to deal with, so we’ll sort out a better way of detecting this and handling it.)

The solution for now is to replace the underscore characters in your feature ids. I replaced these with dashes (i.e., -, so a feature id would now look like OTU-123), and was able to successfully run your core-metrics command. I’ll email you the files that worked for me.

I created an issue to track this so we can have a better solution in the future. I think you should be good to go now. Can you let me know if this now works for you?

2 Likes

Hi @gregcaporaso,
Yes! It solved the issue! Many thanks!

btw,
Happy Christmas & Happy New Year! :grinning:

2 Likes