# Training a large Custom Database

**URL:** <https://forum.qiime2.org/t/training-a-large-custom-database/19440>\
**Category:** Technical Support\
**Created:** [May 3, 2021, 3:41pm UTC](https://forum.qiime2.org/t/training-a-large-custom-database/19440 "2021-05-03T15:41:34Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![URB](https://forum.qiime2.org/letter_avatar_proxy/v4/letter/u/b5e925/32.png) [@URB](https://forum.qiime2.org/u/URB)\
**Post date:** [May 3, 2021, 3:41pm UTC](https://forum.qiime2.org/t/training-a-large-custom-database/19440/1 "2021-05-03T15:41:34Z")

</div>

Hello,

I have created a custom DB, which includes curated 16S from SILVA and RDP Bacteria and Archaea reads and the fungal ITS UNITE reads. I curated them separately and extracted common and unique reads based on family nomenclature and removed those which do not have proper nomenclature. The purpose of doing this is then the chances of getting false positives will be less.

I have formatted the fasta and taxonomy file, and ran the classifier on the individual Archaea (SILVA + RDP reads) and UNITE reads successfully.

But, when I am running the combined Bacterial reads from SILVA and RDP (2615591) for testing purposes. It is taking too long for the classifier to complete. Also, In the Archaea samples I have combined RDP and SILVA reads and they ran properly, so formatting should not be an issue.

I am running on my organization's HPC high\_mem node for the past 4 days. I am worried, how much time will it take to run the complete custom DB with 3320193 reads.

Do you have any suggestions to get it done on time or at least know if it is running properly and if this is a feasible way of doing it?

following is the command I used:  
#!/bin/bash -l

#$ -cwd  
#$ -q highmem.q  
module load qiime2/latest

qiime feature-classifier fit-classifier-naive-bayes   
--i-reference-reads Bac\_all.qza   
--i-reference-taxonomy Bac\_all\_tax.qza   
--o-classifier Bac\_all\_classifier.qza![Screenshot (46)|690x388]

 ![image](https://forum-qiime2-org.s3.dualstack.us-west-2.amazonaws.com/original/2X/2/2e2430b8a2a357662623d33c6f9fb4e85a02e3ae.png)  
 ![image](https://forum-qiime2-org.s3.dualstack.us-west-2.amazonaws.com/original/2X/3/3c4fce0192ffd5006897187227a19441737fb2d1.png)

---

<div class="post-metadata">

**Author:** ![SoilRotifer](https://forum.qiime2.org/user_avatar/forum.qiime2.org/soilrotifer/32/21071_2.png) [@SoilRotifer](https://forum.qiime2.org/u/SoilRotifer)\
**Post date:** [May 3, 2021, 4:12pm UTC](https://forum.qiime2.org/t/training-a-large-custom-database/19440/2 "2021-05-03T16:12:17Z")

</div>

Hi @URB, welcome to :qiime2:!

> [@URB](#):
>
> I am running on my organization’s HPC high\_mem node for the past 4 days. I am worried, how much time will it take to run the complete custom DB with 3320193 reads.

That is an awful lot of reference reads. I'd suggest dereplicating the reference data to make the database much smaller. In fact, this is a great task for RESCRIPt, you can find more details here:

> [@Processing, filtering, and evaluating the SILVA database (and other reference sequence data) with RESCRIPt](https://forum.qiime2.org/t/processing-filtering-and-evaluating-the-silva-database-and-other-reference-sequence-data-with-rescript/15494):
>
> construction Please consider this tutorial a living document, which may change based upon community feedback and ongoing plugin development. RESCRIPt [RESCRIPt](https://github.com/bokulich-lab/RESCRIPt) (REference Sequence annotation and CuRatIon Pipeline) is a python package and QIIME 2 plugin for formatting, managing, and manipulating sequence reference databases. This package was designed for compiling, manipulating, and evaluating sequence reference databases from SILVA, NCBI, Greengenes, GTDB, and other sources, and for construc…

In particular read the part about [Dereplication of sequences and taxonomy](https://forum.qiime2.org/t/processing-filtering-and-evaluating-the-silva-database-and-other-reference-sequence-data-with-rescript/15494#heading--fifth-header). If you know what amplicon region you are using, then you can shrink the reference database even further by extracting the amplicon region from your reference database, then dereplicate that output. This will save you a lot of memory and run time.

-Mike

---

<div class="post-metadata">

**Author:** ![URB](https://forum.qiime2.org/letter_avatar_proxy/v4/letter/u/b5e925/32.png) [@URB](https://forum.qiime2.org/u/URB)\
**Post date:** [May 10, 2021, 7:48pm UTC](https://forum.qiime2.org/t/training-a-large-custom-database/19440/3 "2021-05-10T19:48:51Z")

</div>

Hello Mike, I tried to download the minimal Rescript environment using conda on HPC, but I got the following error.

Preparing transaction: done  
Verifying transaction: done  
Executing transaction: failed

ERROR conda.core.link:\_execute(502): An error occurred while installing package 'conda-forge::async\_generator-1. 10-py\_0'.  
FileNotFoundError(2, "No such file or directory: '/scixxx/home-pure/xxx/anaconda3/envs/rescript/bin/python3.6' ")  
Attempting to roll back.

Rolling back transaction: done

FileNotFoundError(2, "No such file or directory: '/scixxx/home-pure/xxx/anaconda3/envs/rescript/bin/python3.6' ")

Any suggestions or any alternative way to install Rescript.

Thank you - URB

---

<div class="post-metadata">

**Author:** ![SoilRotifer](https://forum.qiime2.org/user_avatar/forum.qiime2.org/soilrotifer/32/21071_2.png) [@SoilRotifer](https://forum.qiime2.org/u/SoilRotifer)\
**Post date:** [May 10, 2021, 8:11pm UTC](https://forum.qiime2.org/t/training-a-large-custom-database/19440/4 "2021-05-10T20:11:58Z")

</div>

It looks like you are trying to install into the main HPC conda system? If so, you'll likely _not_ be able to do this unless you have file permissions, and will have to ask the IT admins.

You can try installing conda into your user account, or at least allowing the HPC conda setup to read and use your own conda environments. This usually requires running a `conda init` command. I'd ask your IT admins for help on this.

I just double-checked both install options for RESCRIPt, and both are working as intended.

-Mike

---

<div class="post-metadata">

**Author:** ![system](https://forum-qiime2-org.s3.dualstack.us-west-2.amazonaws.com/original/3X/2/1/21af5fe23cb6f4579467c66a9ed94e55274ca7bd.svg) [@system](https://forum.qiime2.org/u/system)\
**Post date:** [June 11, 2021, 2:12am UTC](https://forum.qiime2.org/t/training-a-large-custom-database/19440/5 "2021-06-11T02:12:49Z")

</div>

This topic was automatically closed 31 days after the last reply. New replies are no longer allowed.
