Questions about the SILVA 132 QIIME-compatible database, and using it for taxonomic classification of 16S rRNA amplicons

[SoilRotifer (Mike Robeson)]

@KQUB

I'm trying to understand which files in the SILVA 132 QIIME-compatible database ... and also how those files were constructed from the raw SILVA data. (If anyone's wondering, I know that 132 isn't the most recent release of SILVA, but it's the release I happen to be interested in.)

Great to see someone dive into the weeds!

@KQUB

It also contains a Silva_132_notes.txt file, which supposedly explains all of these subdirectories and their files, but I'm quite new to bioinformatics, so a lot of the explanations are over my head.

As one of the contributors to this pipeline, let me see if I can help you out.

@KQUB

And because I'm only interested in 16S sequences, I assume I need the rep_set/rep_set_16S_only directory and the taxonomy/16S_only directory. Is this correct?

Correct. However, I'd recommend keeping all of the reference sequences, including the eukaryotes, in the files. It allows you to more robustly identify off-target sequences, i.e. non-bacterial and non-archaeal sequences. Often the different versions of the database, i.e. 16S only, was more for practical reasons to reduce the memory and storage footprint of the reference database.

@KQUB

Each of the four rep_set subdirectories contains a single file: silva_132_90_16S.fna, silva_132_94_16S.fna, silva_132_97_16S.fna and silva_132_99_16S.fna, respectively.

If I understand correctly, the creation of these .fna files is primarily described in the Filtering raw fasta file, creation of representative sequence files section of the Silva_132_notes.txt file.

Correct.

@KQUB

In preparing the SILVA 132 QIIME-compatible database, full 16S and 18S rRNA sequences ...

Looks like you have it all figured out.

@KQUB

... If this is roughly accurate, then are the .fna files in the subdirectories of rep_set/rep_set_16S_only basically lists of unique representative sequences from OTUs generated by clustering the processed SILVA sequences at different percentages of sequence identity?

Yep!

@KQUB

... @Nicholas_Bokulich seems to recommend the use of 99% OTUs in this forum post, so it seems like I should use the silva_132_99_16S.fna as a source of reference sequences for training the classifier. But is this okay even if I want to classify ASVs derived from DADA2 output?

Yes. the 99% helps removes some extra noise in the reference data, and reduce the size of the data set. We currently (see below) prefer to use the SILVA NR99 dataset for the reasons outlined here.

@KQUB

Also, why must the SILVA sequences be clustered at all? Why not just use the sequences as they are?

We'll come back to this. We have a treat for you at the end.
But, great question! You get a !
Again, traditionally, clustering was a way to remove noisy sequence data, and reduce the size of the reference set to be used for taxonomic classification. Back in the day, many researchers did not have access to good computing resources that had the memory and cpu power to construct classifiers.

@KQUB

In other words, the taxonomy strings have been removed from the sequences. Is there a reason for this?

Just to reduce the file size. Once we have the taxonomy file there is no need to keep that redundant information.

@KQUB

the names of the representative OTU sequences are somehow derived from the raw data (which they seem to be),

Correct. The original old code referenced in the README is here. A description on the labeling format is available here:

Why all taxonomic levels on BarPlot have 'D' letter?

I can likely provide some insight here as I am one of the contributors that helped to format SILVA database for QIIME. The D_X__ convention was chosen to be as much of a unique and “safe” text string as possible, considering many of the bizarre taxonomy text annotations within the SILVA reference database. That is, it was meant as a quick fix to be able to search and parse these taxonomy strings. The ‘D’ was a way of annotating the “taxonomic Depth”. At the time some of the code was written, th…

However, we have since taken a different approach to parsing taxonomy. See the end of this post.

@KQUB

... the representative OTU sequence named >AB302407.1.2962 in the rep_set/rep_set_16S_only/99/silva_132_99_16S.fna file could perhaps represent several raw sequences (if I'm not mistaken), and maybe not all of those raw sequences had the exact same taxonomic classification in the raw data. Is that about right?

Yep.

@KQUB

I think I understand the difference – assuming the word 'cluster' here corresponds to the same process of generating OTUs by which the rep_set subdirectories were prepared. Does it?

Here, the "cluster" is the set of sequences that fall into an OTU. That is, a representative sequence. We are forming the consensus / majority taxonomy by taking into account all of the lineages that fall within that OTU / representative sequence, and collapsing the taxonomy that is, hopefully, a good representation of all the sequences that fall into that cluster.

@KQUB

Apologies for the ungodly length of this post, and the potentially very noob questions. Perhaps this kind of 'thinking out loud' post will be somehow useful to other noobs in the future. Anyway, big thanks to the :qiime2: team for developing this amazing tool and for maintaining this very useful forum.

Not a problem. Not many have dug deep into the process of generating these files. I for one appreciate your interest! Thank you!

Okay... so that bit of I was promising... Instead of re-working your way through that old pipeline, you can simply make use RESCRIPt, to make your own SILVA reference database, even for version 132! You can work through the tutorials and curate the reference data the way you'd like. That is you can simply run this command:

qiime rescript get-silva-data \
    --p-version 132 \
    --p-target SSURef_NR99 \
    --p-include-species-labels \
    --output-dir silva-132

and then follow the rest of the SILVA tutorial. I hope you'll find that it is superior to what we've done in the past. For example, how we parse taxonomy, etc... Although we provide a few ready-to-use files and classifiers, you can certainly go ahead and make your own files, curated the way you'd like . The goal of RESCRIPt is to make life a little easier for those of us interested in constructing and curating our own little piece of reference database heaven.

As a little history... the old SILVA parsing code, that I linked above, was eventually updated to this, and then found a home in RESCRIPt.

Take it for a spin and let us know how it works out.