Understanding beta-group-significance PERMANOVA results

HI @MaestSi,

Permanova is a permutative test. You’re asking whether the data is more extreme that a random rearrangement of the data (which lets you escape some of the assumptions of a tradition test statitistics). The limitation here is that the p-value is then the observed probability of finding something more extreme. For the sake of having a p-value that is not 0, we actually calculate this as \frac{n_{\textrm{more extreme}} + 1}{n_{\textrm{permutations}}}.

So, if I have 999 permutations and nothing I can find is more extreme, then my p-value is \frac{0 + 1}{1 + 999} = 0.001. You can decrease the p-value by adjusting the number of permtuations (for example, a the minimum p-value from 9999 permutations is 0.0001. However, there is some sense that this is its own kind of p-hacking. It’s also why, when you report a permutative p-value, you need to report the number of permutations you performed. p=0.01 for 9999 permutations is very different from p=0.01 for 99 permutations.

Best,
Justine

10 Likes