Evident: Effect size and power calculations for microbiome community data

Evident: Community Tutorial

Evident is a tool for calculating the effect sizes of sample groupings microbiome data and performing statistical power analysis. With Evident, you can easily explore columns in your metadata that are highly associated with differences in microbial communities. These effect sizes can, in turn, be used to perform power analysis for variable levels of significance and number of observations.

We hope that researchers will use Evident to more carefully design their sequencing experiments to maximize power while minimizing wasted resources. You can look at an existing dataset, determine the effect size of the sample groupings, and calculate the number of samples required to reject the null hypothesis at a priori defined values of statistical significance and power given this effect size.

Evident can operate on either

  • Univariate data (e.g. alpha diversity, log-ratios, etc.)
  • Multivariate data (e.g. beta diversity distance matrix)

See our preprint on bioRxiv for more details. While we do provide a QIIME 2 interface (hence this post!), we also provide a standalone Python package that is more full featured, including an interactive web app for effect size exploration and power analysis (see the README).

Installing

pip install evident

Example Usage - Power Analysis

We can use a QIIME 2 SampleData[AlphaDiversity] Artifact as input. We pass this object in as a --m-sample-metadata-file in addition to our sample metadata. This is possible because the SampleData[AlphaDiversity] can be interpreted as Metadata and merged with all files passed to --m-sample-metadata-file. We provide the --p-data-column with the column containing the diversity values and --p-group-column with the column containing the groupings of interest.

We want to evaluate the statistical power at:

  • Significance levels of 0.01, 0.05, 0.1
  • Total observations from 10 to 100 in intervals of 10

We can pass these values to Evident separated by spaces. Notice that we use the seq UNIX command to generate the possible sample sizes.

qiime evident univariate-power-analysis \
    --m-sample-metadata-file metadata.qza \
    --m-sample-metadata-file faith_pd.qza \
    --p-data-column faith_pd \
    --p-group-column classification \
    --p-alpha 0.01 0.05 0.1 \
    --p-total-observations $(seq 10 10 100) \
    --o-power-analysis-results results.faithpd.qza

We can transform these results into a Visualization with

qiime evident visualize-results \
    --i-results results.faithpd.qza \
    --o-visualization results.faithpd.qzv

This creates a table that you can view directly at QIIME 2 View.

Alternatively, you can create a power curve from these results using

qiime evident plot-power-curve \
    --i-power-analysis-results results.faithpd.qza \
    --p-target-power 0.8 \
    --p-style alpha \
    --o-visualization curve.qzv

Example Usage - Effect Sizes

We can also use Evident to generate effect sizes for multiple categorical groupings at once. This process can be parallelized for efficiency on high-performance computing environments. In this example, we want to calculate the effect size of alpha diversity differences among samples in three categories: classification, sex, & cd_behavior.

For binary categories, Evident calculates Cohen's d. For multi-class (>2) categories, Evident calculates Cohen's f.

qiime evident univariate-effect-size-by-category \
    --m-sample-metadata-file metadata.qza \
    --m-sample-metadata-file faith_pd.qza \
    --p-data-column faith_pd \
    --p-n-jobs 3 \
    --p-group-columns classification sex cd_behavior \
    --o-effect-size-results alpha_effect_sizes.qza

Additionally, you can specify that you want to calculate all pairwise comparisons of groups with more than two factors. This will calculate Cohen's d for all possible combinations of levels. For example, if you had groups high-fat diet, low-fat diet, and control diet, you could evaluate the effect sizes of alpha diversity differences between the following comparisons:

  • High-fat diet vs. low-fat diet
  • High-fat diet vs. control diet
  • Low-fat diet vs. control diet
qiime evident univariate-effect-size-by-category \
    --m-sample-metadata-file metadata.qza \
    --m-sample-metadata-file faith_pd.qza \
    --p-data-column faith_pd \
    --p-pairwise \
    --p-n-jobs 3 \
    --p-group-columns classification sex cd_behavior \
    --o-effect-size-results alpha_effect_sizes.qza

While we used univariate data in this tutorial, the commands are nearly identical for multivariate distance matrices.

Please see the README on the GitHub repository for more details. Feel free to post in the Community Plugin Support category if you encounter have questions. If your issue is related to the code (e.g. a bug), you may consider opening an issue on GitHub.

6 Likes

Hi,
Thanks for building the Evident plug-in.
I am trying to use it on multivariate data, bray-curtis distance, and I am having trouble finding enough guidelines..

I tried to run this command :
qiime evident multivariate-power-analysis
--m-sample-metadata-file metadata.tsv
--m-sample-metadata-file core-metrics-results/bray_curtis_distance_matrix.qza
--p-group-column Diagnosis
--p-alpha 0.01 0.05 0.1
--p-total-observations $(seq 10 10 100)
--o-power-analysis-results results_evident_bray.qza

I get an error :

There was an issue with viewing the artifact core-metrics-results/bray_curtis_distance_matrix.qza as QIIME 2 Metadata:

Artifact <artifact: DistanceMatrix uuid: bb2c32fb-cd41-4159-8cc0-be717fe4ace2> cannot be viewed as QIIME 2 Metadata.

As I understand I cannot feed the bray curtis distance matrix under other options either to the plug-in.. Or how could I make this command to work?

Thank you in advance!

Best,
Rahel

Hi, @rahel_park

Thank you for trying Evident!

When using multivariate data you should pass in the distance matrix to the --i-data parameter rather than --m-sample-metadata-file. Please see the help message for more details.

$ qiime evident multivariate-power-analysis --help
Usage: qiime evident multivariate-power-analysis [OPTIONS]

  Use sample Multivariate data data to perform power calculations for
  desired significance level, power, or sample size.

Inputs:
  --i-data ARTIFACT      Sample distance matrix
    DistanceMatrix                                                  [required]
Parameters:
  --m-sample-metadata-file METADATA...
    (multiple            Sample metadata.
     arguments will be
     merged)                                                        [required]
  --p-group-column TEXT  Column to use for groupings.               [required]
  --p-max-levels-per-category INTEGER
                         Max number of levels in a category to keep. Any
                         categorical columns that have more than this number
                         of unique levels will not be saved, defaults to 5.
                                                                  [default: 5]
  --p-min-count-per-level INTEGER
                         Min number of samples in a given category level to
                         keep. Any levels that have fewer than this many
                         samples will not be saved, defaults to 3.
                                                                  [default: 3]
  --p-alpha PROPORTIONS...
    Range(0, 1)          Significance level.                        [optional]
  --p-power PROPORTIONS...
    Range(0, 1)          Probability of rejecting the null hypothesis given
                         that the alternative is true.              [optional]
  --p-total-observations INTEGERS...
    List[Int]            Total number of observations to consider. Groups are
                         assumed to be all the same size.           [optional]
  --p-difference NUMBERS...
    List[Float]          Difference between groups to consider. If this
                         argument is provided, evident will use this value
                         instead of calculating the mean difference. The
                         pooled standard deviation of the groups will still be
                         used. If not provided, evident will calculate the
                         difference in means automatically.         [optional]
Outputs:
  --o-power-analysis-results ARTIFACT PowerAnalysisResults
                                                                    [required]
Miscellaneous:
  --output-dir PATH      Output unspecified results to a directory
  --verbose / --quiet    Display verbose output to stdout and/or stderr
                         during execution of this action. Or silence output if
                         execution is successful (silence is golden).
  --example-data PATH    Write example data and exit.
  --citations            Show citations and exit.
  --help                 Show this message and exit.
2 Likes

Hi @gibsramen,
Thank you for the reply, indeed I somehow missed this piece of information.
One more question - how to interpret results from beta diversity when there's several category levels?

Hi, @rahel_park

This will, of course, depend on the specifics of your dataset but generally:

  • When there are several categories Evident calculates Cohen's f.
  • The power analysis assumes that a new experimental design will have equal number of samples from each of the groups. This is not always feasible or desired.
  • The beta-diversity analysis is looking at within-group pairwise distances - or how the overall variance of beta-diversity differs among groups. Notably, this will likely vary depending on the beta-diversity metric used (see xebec for a work-in-progress pipeline to benchmark different diversity metrics).