I conducted 16S Illumina sequencing on some environmental samples and wanted to predict metabolic pathways associated with these communities using the QIIME2 PICRUSt2 plugin. I followed the QIIME2 picrust2 tutorial (https://github.com/picrust/picrust2/wiki/q2-picrust2-Tutorial).
I wanted to make sure my workflow looks okay - I can’t tell how accurate the predictions are, and the QIIME2 picrust2 plugin doesn’t seem to calculate the confidence associated with pathway predictions. This is my workflow.
Firstly I removed ASVs with <100 sequences per sample to focus on the predominant taxa present in samples (less likely to be sequencing mistakes).
#Then I ran PICRUSt2 using the MP HSP method and an NSTI value of 2 on all samples
qiime picrust2 full-pipeline --i-table /projects/ar20/control-filtered-table-100ASVs.qza --i-seq /projects/ar20/dada2/representative_sequences.qza --output-dir /projects/ar20/all-soil-100ASVs-picrust2 --p-hsp-method mp --p-max-nsti 2 –verbose
The only pathways I paid any attention to were those inferred to be present in multiple biological replicates of a treatment (>50%), which would increase the probability that the predicted pathway is probably accurate and not just something random.
Supposedly the nearest sequence taxon index (NSTI) is meant to be most important in ensuring Picrust’s predictive accuracy, and the accuracy of PICRUSt generally decreases with increasing NSTI. I used a nearest sequence taxon index (NSTI) of 2 as per the tutorial (default). Although I wasn’t sure what else I could do to ensure that the accuracy of predictions was sound.
No confidence values are output for pathway predictions unfortunately. You might be thinking of the “coverage” values output by HUMAnN2, which I initially did implement, but did not find to be useful for determining confidence. The best answer I think is that you should not have high confidence in reconstructed pathways from microbiome data in general, and especially not for predicted metagenome data. Not everyone agrees with that, but it’s very difficult to prove that pathway abundances are accurately inferred, even for shotgun metagenomics data.
Nonetheless, you could look at the concordance between a subset of your samples based on metagenomics sequencing compared to the predicted metagenomes (see: https://github.com/picrust/picrust2/wiki/Validation-with-paired-metagenomes). The challenge unfortunately even if you had this data is that it’s very difficult to prove that the predictions match the actual metagenomes “well enough” to yield interpretable signals.
To be more specific - there are a few points where noise can be introduced. First, this is predicted data so the underlying gene family calls could be incorrect in ASVs that are present in all samples. If these gene calls result in a pathway being called then looking at pathways found across most samples would still include noisy data. The NSTI cut-off is a decent method for major outliers, but it doesn’t ensure sound predictions. If you take a look at the manuscript you can see that the predictions perform only moderately better than the “Null”, which is totally random data. This means that it’s hard to be confident in any single predicted function and only that the overall profile will correlate better with real data.
To make matters more confusing, we also showed in our paper that differential abundance tests based on actual metagenomes and predicted metagenomes can result in starkly different results even if the profiles themselves are well correlated. It’s difficult to disentangle what exactly the issues are here, because there is no clear gold-standard to compare to. If you use different shotgun metagenomics pipelines for generating the profiles you also typically get very different results. This is a massive issue that a lot of people are worried about and I think reflects the difficult of analyzing microbiome data in general.
One final issue is that pathway reconstruction is especially difficult to validate. We were able to show that predicted pathways correlate better with pathways reconstructed from metagenomics data than predicted by chance, but that doesn’t mean the pathway reconstruction approach is correct. There are a few different methods for reconstructing pathway abundances from genome annotations (even for individual reference genomes) and it’s controversial what the best workflow should be. It’s especially difficult to prove that pathway reconstructions are working well across a metagenome, because you can’t really do experiments to confirm pathway activity in an easy way like you might for a cultured bacteria.
The default pathway reconstruction method in PICRUSt2 is the same in HUMAnN2 which is that pathway reconstructions are based on the community-wide abundances of genes (like considering genes as though they were all freely interacting). This is the standard way to analyze this data type, but I prefer personally to look at predictions for individual genomes. You can do this with the standalone version with the --per_sequence_contrib option (it takes longer to run), but it’s not in the QIIME2 plugin unfortunately.
Hopefully that was helpful and not too discouraging!
Thank you for this comprehensive response, Gavin. I will attempt to use the original Picrust2 --per_sequence_contrib option. Unfortunately I don’t have corresponding metagenomics data to complement the 16S data, so I can’t check for concordance with Picrust2 predicted metagenomes.
Other than that - my takeaway is that, with just 16S data, it seems there’s not much else I can really do to improve the accuracy of the predictions from my end. Therefore, I will make sure to be VERY tentative when drawing any conclusions from my Picrust analyses, and will only leave this as a small section in my manuscript. I have made sure to cite your bioRxiv and Nature 2020 papers in reference to Picrust’s limitations.