Permanova analysis for distances

Dear Qiimers!
I have a question about permanova analysis.
Permanova may be used both to test multiple groups together and in the pairwise mode.
So, if I want to perform a post hoc analysis after permanova, is it an appropriate to perform it like this:

  1. Permanova(a,b,c)
  2. If p value of Permanova(a,b,c) < 0.05:
    2.1 Permanova(a,b)
    2.2 Permanova(a,c)
    2.3 Permanova(b,c)
    2.4 Correction of all p-values by Benjamini–Hochberg procedure

Is it as it is implemented in Qiime2, or I am missing something important?

Hi Timur,

Yep, qiime diversity beta-group-significance does the test you describe for beta diversity and qiime diversity alpha-group-significance does this for alpha.

There are great examples in the Parkinson’s Mouse Tutorial :mouse: that you may have discovered already. In those examples, the pairwise post-hoc tests are shown in ‘Pairwise permanova results’ section at the very bottom of the output .qzv file.

Colin

4 Likes

Hi, thank you a lot for your reply!
Yeah, I already performed permanova in Qiime2 for beta diversity tests. For my dataset, it is necessary to perform multiple filtration steps on beta distances. And I also have a lot of columns in a metadata file, so I ended up with a lot of files produced by Qiime2. So I decided to run permanova directly in Jupyter notebook, but had some doubts regarding implementing permanova pairwise tests. I did it like I described above and want to make sure that it is a right way to do it. I also checked it by comparing corrected values I received and produced by Qiime2 and they are the same.

In apha diversity, Kruskal Wallis is applied for multiple comparison. I read on the Qiime2 forum, that Kruskal Wallis test also was used for pairwise comparisons. But I had a discussion with colleagues and also read on some forums that after Kruskal Wallis test Dunn, Wilkoxon or Mann Whitney U tests should be used for posthoc test with corrections. I performed Dunn test as a post hoc with fdr corrections by BH procedure and results are a little bit different with Qiime2 outputs. Mostly some significant q values from Qiime2 were lost by comparison with Kruskal Wallis test, followed by Dunn pairwise comparisons.

1 Like

Yeah there are lots of choices for stat tests and multiple testing corrections. I am not a statistician so I will not give bad advice. :speak_no_evil:

Do we have a resident statistician on the forms? @.moderators

Colin

1 Like

Im not a formal statistician. I’m guessing about the loop:

  1. Test if there’s an association in the whole variable
  2. If there’s a difference, then do post-hoc testing.

I assume you’re possibly doing posthoc correction in the first? I think determining the correct number of tests to do post-hoc correction with in the second is an important question. (I dont have the right answer).

To me, one caveat with correction for permanova (and something to note in general!) is that your p-value is capped: there is a minumum of 1/(1 + num permutations). So, for 999 permutations, you can have a p-value of 0.001. This means that you couldn’t find anything more extreme in those 999 permutations, but not address anything else. To me (personally), this is a special case: we don’t know how extreme the actual data is, we just know it’s more extreme. So, I tend to be less inclined to correct here. I also think there’s a correction method that penalizes this permutation less.

Best,
Justine

3 Likes

Hi Justine,
Thank you for your reply.
Yes, I am performing corrections on the whole variable with 999 permutations, even if I am not interested in certain pairs, since reducing amount of testing pairs may affect correction. Only after it I am slicing resulted dataframe to show only desired pairs. I only filter out the samples for which no treatment was applied and that do not belong to the control group.

I am also curious why in alpha diversity comparisons Kruskal-Wallis test was implemented for pairwise comparisons as a post hoc after multiple comparisons by the first Kruskal Wallis test. Usually after Kruskal Wallis Dunn, Wilkoxon or Mann Whitney U tests are applied as a post hoc. I am not very competent in statistics, so I am trying to understand differences and advantages of implemented tests.

Best,
Timur

Hi @timanix,

That all. makes sense; I do the same thing.

I think the motivation for re-using the kruskal wallis is to fold. First, for two groups, the kruskal wallis p-value and Mann Whitney U p-value are identical (sort of like an ANOVA on two groups collapses to a t-test). Im not sure of the implementation motivation, it may also be that the implementation with a single test is easier.

Best,
Justine

2 Likes

Hi Justine!
As always, thanks for excellent explanations :clap: :clap: :clap:.

Best regards,
Timur

2 Likes