Training a large Custom Database

[SoilRotifer (Mike Robeson)]

Hi @URB, welcome to !

@URB

I am running on my organization’s HPC high_mem node for the past 4 days. I am worried, how much time will it take to run the complete custom DB with 3320193 reads.

That is an awful lot of reference reads. I'd suggest dereplicating the reference data to make the database much smaller. In fact, this is a great task for RESCRIPt, you can find more details here:

Processing, filtering, and evaluating the SILVA database (and other reference sequence data) with RESCRIPt

Please consider this tutorial a living document, which may change based upon community feedback and ongoing plugin development. RESCRIPt RESCRIPt (REference Sequence annotation and CuRatIon Pipeline) is a python package and QIIME 2 plugin for formatting, managing, and manipulating sequence reference databases. This package was designed for compiling, manipulating, and evaluating sequence reference databases from SILVA, NCBI, Greengenes, GTDB, and other sources, and for construc…

In particular read the part about Dereplication of sequences and taxonomy. If you know what amplicon region you are using, then you can shrink the reference database even further by extracting the amplicon region from your reference database, then dereplicate that output. This will save you a lot of memory and run time.

-Mike