I am working to interpret the q-values for alpha group significance tests. I found this post that explains that the test is Benjamini Hochberg, but I'm unclear what the chosen FDR is to calculate the critical value. The SciPy page for multiple testing says "e.g. 0.1". Is this the default and was it used in Q2?
Additionally (from my understanding of BH) you find the largest p-value smaller than the critical value (while sorted) and anything above it (i.e. smaller) is considered significant. Should I be resorting the resulting pairwise tables on p-value (smallest to largest) and comparing the q-value to find which comparisons are significant? I've attached an output from observed features calculation to try an explain what I'm referring to.
QIIME 2 using the statsmodels multicompare under the hood which by default sets their p-value to 0.05. However, I’m not sure how many people rely on that sorted critical value; the underlying code simply pulls out the adjusted p-values and ignores the automatically calculated threshold.
My personal inclination is to be somewhat conservative if the univariate test is your final alpha diversity test; I tend to be a bit more permissive when I’m using it for metadata selection to feed into an adjusted model.
So, I’m not sure if the sorting will buy you anything personally, I guess it depends on how you want to work with the data and how you plan to go forward.
Thank you for your response! I’m still a little confused; sorry if this is super simple and I’m spacing. The group significance test is generally my final alpha diversity test. Therefore, do I need to be comparing the adjusted p-value number to 0.05 to determine if the pairwise test is significant?
You want to compare to your critical value. I tend to send mine apriori at 0.05 based on historical traditions and the fact that most people assume I’m going to set it there anyway.
I don’t know what your right p-value is, but you would compare the pairwise adjusted number to whatever p-value you set. I will say you haven’t done a ton of comparisons here, and you probably don’t have a lot of samples, so maybe this should factor in?
Perfect, I had been comparing the q-value to 0.05 (my standard alpha value as well) before I started questioning myself.
I’m unsure how to control for sample size/number of comparisons in the test. Do you have recommendations? Or are you suggesting to make the alpha value more/less stringent than 0.05?