Hi there,
I wanted to double check what the p-value significance level setting is for qiime2 PERMANOVA tests (999 default permutations) / Bray-Curtis dissimilarity calculations. Is it p<0.05 or p<0.01?
Hi @skose82!
The answer is neither. We prefer to only report values, we don't set rejection criteria (or if for some reason it is necessary, you'll find a parameter on the action to configure it), that's up to you!
Here's a potentially relevant (although old and from a different field) critique on the matter that might be of interest:
6 Likes
Ah so, how can one tell if, for example the output below between two groups, is a significant difference given the p-value?
|Group 1|Group 2|Sample size|Permutations|pseudo-F|p-value|q-value|
|Y|X|10|999|3.696546533|0.014|0.014|
1 Like
Unfortunately it depends on what you want significant to mean in the first place (see my linked paper for a discussion on the various interpretations/goals here).
It also depends on whether or not it's reasonable to expect X and Y to differ. We're asking what is the probability [edit] of our data if [/edit] X and Y have similar measures of center and dispersion (generally we might infer that they come from the same distribution if they did, but it's not required).
So for this test, we're making a statement like P( data | X \sim Y ) = 0.014 . That doesn't mean that X and Y are different, just that data which did come from the same distribution (simplifying) would only be as extreme as this some proportion of the time. It also doesn't explain why there's a difference, should we choose to reject, which is usually the more important part.
In any case, a super typical value in our field is still \alpha=0.05 , so going by that alone, we could say the groups Y and X appear to be significantly different.
I would report the q-value as it is more robust to the problem of multiple-testing (it has essentially the same interpretation as the p-value, but controls for false-discovery (or family-wise error if you are into that sort of thing)).
So I would say something like:
X and Y were found to be significantly different under
PERMANOVA (999 permutations, p=0.014, q=0.014 )
Now if you have done many such comparisons across multiple actions, you will need the original p-values to perform the FDR correction yourself across all of the tests and comparisons you may have done (as the q-values will be different).
There's no convenient way to do that in QIIME 2 at the moment, so you'll want to do it via excel or some other tool.
EDIT: updated the example statement to include permutations (thanks @jwdebelius!)
5 Likes
Thank you, Evan for this detailed explanation and help! I really appreciate it and I will certainly read the linked paper.
1 Like
This topic was automatically closed 31 days after the last reply. New replies are no longer allowed.