This is a more general question. Now that we are switching completely to QIIME2, i did one comparison between 97% OTU clustering in QIIME 1 and 99% ESV clustering in QIIME 2/deblur.
The dataset is a fairly standard MiSeq 2x250 bp run with >100 bp overlap.
I compared two experiments by
reads surviving the pipeline
raw OTUs/ESVs identified by sample.
I found that QIIME2 accepts more reads, while identifying much less ESVs than OTUs. This was surprising because i would expect that 99% clustering would increase diversity over 97% thresholds.
There was also a good, but not a perfect correlation between OTUs and ESVs identified, meaning that the causes for the diversity deflation do not apply in the same way all the time.
If needed i will later add the parameters set for the pipelines. For now, i would like to know if this is a known issue and what the possible reasons could be.
Thank you very much.
Your results are fairly consistent with expectations and past observations. OTU clustering does a poor job of weeding out sequence errors (or to be more correct, it does not, it just clusters sequences hopefully collapsing some of the rarer errors into hopefully correct centroids). Denoising methods (such as used in QIIME 2) actually remove and/or correct those erroneous sequences, leading to fewer spurious observations. In QIIME 1 we recommended using stringent filtering settings to weed out spurious OTUs; without that, alpha diversity estimates will be vastly overinflated (see the article for details).
Because denoising is removing those spurious reads that otherwise just get clustered into rare but likely incorrect OTUs. Check out the dada2 and deblur original publications for benchmarked examples.
The lack of perfect correlation does not surprise me, since denoising methods are combining filtering as well as (in the case of dada2) error correction, thereby dereplicating reads into fewer ASVs as well as weeding out noise.
Not an issue, an expectation! If you follow the stringent OTU filtering recommendations for qiime1 linked above, you will find somewhat more similar results between OTUs + ASVs.
Here's a ream of other forum reading that might interest you (both past answers to this same question as well as feedback that relates to your specific workflow, esp. the use of clustering after denoising):
