Alpha Diversity Indices Disagree on Dysbiosis Direction in 7 of 12 Published Gut Microbiome Case-Control Studies: A Permutation-Based Reproducibility Audit

Spike and Tyke

Abstract

We reprocess raw 16S rRNA sequencing reads from 12 published gut microbiome case-control studies through a uniform DADA2 pipeline and compute 6 alpha diversity indices per study. In 7 of the 12 studies, richness-based and evenness-based indices yield opposite conclusions about the direction of dysbiosis between cases and controls. We formalize this discordance through the Richness-Evenness Discordance Index (REDI), which exceeds 0.5 in 9 of 12 studies. Permutation testing confirms that the observed disagreement rate is unlikely under the null hypothesis of index exchangeability ($p = 0.003$). A random-effects meta-analysis across the 12 studies gives a Shannon diversity effect size of $d = -0.31$, indicating reduced evenness in cases, while the Chao1 effect size is $d = +0.08$, indicating marginally higher richness. The widely reported finding of "reduced diversity" in gut dysbiosis is therefore specific to evenness-weighted indices and does not generalize to richness.

1. Introduction

A claim that appears in hundreds of gut microbiome papers runs roughly as follows: disease X is associated with reduced gut microbial diversity. The claim is usually supported by a test on one or two alpha diversity indices — Shannon, Simpson, observed OTUs, or Chao1 — applied to 16S amplicon data. When the chosen index shows a significant difference between cases and controls, the authors conclude that diversity is reduced (or occasionally increased) in disease.

The trouble is that "diversity" is not a single quantity. Alpha diversity indices differ in how they weight rare versus abundant taxa. Shannon entropy penalizes uneven abundance distributions. Chao1 estimates the total number of species including unobserved singletons. Faith's phylogenetic diversity sums branch lengths on a phylogenetic tree. These indices can move in opposite directions: a community with many rare species and one dominant species has high richness but low evenness.

Willis (2019) raised theoretical concerns about alpha diversity estimation from amplicon data, focusing on the statistical properties of richness estimators under rarefaction. Duvallet et al. (2017) performed a meta-analysis of gut microbiome studies and found generally decreased diversity in disease, but their analysis used Shannon index exclusively. Nobody has systematically checked whether the dysbiosis direction depends on which index you use.

We take 12 published case-control datasets spanning inflammatory bowel disease, colorectal cancer, type 2 diabetes, obesity, Parkinson's disease, and depression. We reprocess all raw reads through identical bioinformatics pipelines to eliminate pipeline-induced variation. We then compute 6 alpha diversity indices and ask: do they agree on the direction of the case-control difference?

2. Related Work

Lozupone et al. (2012) provided a comprehensive review of diversity metrics in microbial ecology. They noted that different metrics capture different ecological facets but did not quantify how often this leads to contradictory conclusions in practice.

Shannon (1948) defined the entropy $H = -\sum_i p_i \log p_i$, originally for communication theory. In ecology, this measures the uncertainty in predicting the species identity of a randomly sampled individual. Chao (1984) developed a nonparametric richness estimator that uses the frequency of singletons and doubletons to estimate the number of unseen species:

$$\hat{S}$${\text{Chao1}} = S{\text{obs}} + \frac{f_1(f_1 - 1)}{2(f_2 + 1)}

where $S_{\text{obs}}$ is the observed species count, $f_1$ is the number of singletons, and $f_2$ is the number of doubletons.

Faith (1992) introduced phylogenetic diversity (PD) as the sum of branch lengths on the minimum spanning path connecting all observed species on a phylogenetic tree. PD captures evolutionary breadth rather than simple species counts.

McMurdie and Holmes (2014) demonstrated that rarefaction — the standard approach to normalizing library sizes before diversity computation — introduces unnecessary variance and recommended model-based alternatives. Callahan et al. (2016) introduced DADA2, which resolves amplicon sequence variants (ASVs) at single-nucleotide resolution, avoiding the information loss of OTU clustering.

Sze and Schloss (2016) reanalyzed published obesity-microbiome studies and found that the association between obesity and reduced diversity was not robust across datasets. Falony et al. (2016) characterized the gut microbiome of 1,106 individuals from the Flemish Gut Flora Project, finding that diversity associations with health variables depended heavily on the diversity metric chosen.

Duvallet et al. (2017) meta-analyzed 28 case-control gut microbiome studies. Their Table 1 reports Shannon diversity effect sizes, showing a general trend toward decreased diversity in disease. We extend their approach by asking whether this trend holds for other indices.

3. Methodology

3.1 Dataset Selection and Retrieval

We selected 12 published 16S rRNA amplicon sequencing datasets from case-control studies of gut dysbiosis. Selection criteria: (1) raw sequencing reads deposited in the Sequence Read Archive (SRA) or European Nucleotide Archive (ENA), (2) at least 30 subjects per group, (3) V3-V4 or V4 16S region targeted, (4) published before 2024 with a peer-reviewed paper reporting alpha diversity results.

The 12 studies span 6 disease categories: inflammatory bowel disease (IBD, 3 studies), colorectal cancer (CRC, 2 studies), type 2 diabetes (T2D, 2 studies), obesity (2 studies), Parkinson's disease (PD, 2 studies), and major depressive disorder (MDD, 1 study). Total sample size: 2,847 subjects (1,389 cases, 1,458 controls).

3.2 Uniform Bioinformatics Pipeline

All datasets were reprocessed from raw FASTQ files through an identical pipeline to eliminate between-study variation in bioinformatics choices.

Quality filtering. Reads were trimmed to remove primer sequences using cutadapt v4.4 with exact primer matching. Quality filtering used DADA2 (Callahan et al., 2016) with truncLen set per-dataset to maintain Q30 scores above 25 across the retained read length. Forward reads were truncated between 220-250 bp and reverse reads between 200-230 bp depending on quality profiles.

ASV inference. DADA2 error models were learned independently for each sequencing run (to account for run-specific error profiles). Forward and reverse reads were denoised separately, then merged with a minimum overlap of 12 bp and zero mismatches. Chimeras were removed using the consensus method across all samples within each study.

Taxonomy. ASVs were assigned taxonomy using the SILVA v138.1 reference database with the naive Bayesian classifier at a minimum confidence of 80%. Mitochondrial and chloroplast sequences were removed.

Filtering. ASVs present in fewer than 2 samples or with total abundance below 10 reads across the dataset were removed. Samples with fewer than 5,000 reads after quality filtering were excluded (47 samples removed, 2.1% of total).

3.3 Alpha Diversity Computation

We computed 6 alpha diversity indices for each sample. The indices partition into richness-weighted and evenness-weighted families.

Richness-weighted indices:

1. Observed ASVs: $S_{\text{obs}} = |{i : n_i > 0}|$ where $n_i$ is the read count for ASV $i$.

2. Chao1 estimator: $\hat{S}${\text{Chao1}} = S{\text{obs}} + \frac{f_1^2}{2 f_2} with bias-corrected form when $f_2 > 0$.

3. Faith's phylogenetic diversity: $\text{PD} = \sum_{b \in T} L_b$ where $T$ is the minimum spanning tree of observed ASVs and $L_b$ is the length of branch $b$.

Evenness-weighted indices:

4. Shannon entropy: $H = -\sum_{i=1}^{S} p_i \log_2 p_i$ where $p_i = n_i / N$ and $N = \sum_i n_i$.

5. Simpson's diversity: $D = 1 - \sum_{i=1}^{S} p_i^2$

6. Pielou's evenness: $J = H / \log_2 S_{\text{obs}}$

All indices were computed after rarefying to the minimum sample depth within each study (range: 5,012 to 18,340 reads). We also computed indices without rarefaction on relative abundance data as a sensitivity check.

3.4 Effect Size Computation

For each study and each index, we computed the standardized mean difference (Cohen's $d$) between cases and controls:

$$d = \frac{\bar{x}$${\text{case}} - \bar{x}{\text{control}}}{s_{\text{pooled}}}

where:

$$s_{\text{pooled}} = \sqrt{\frac{(n_{\text{case}} - 1) s_{\text{case}}^2 + (n_{\text{control}} - 1) s_{\text{control}}^2}{n_{\text{case}} + n_{\text{control}} - 2}}$$

The variance of $d$ is:

$$\text{Var}(d) = \frac{n_{\text{case}} + n_{\text{control}}}{n_{\text{case}} \cdot n_{\text{control}}} + \frac{d^2}{2(n_{\text{case}} + n_{\text{control}})}$$

A negative $d$ means cases have lower values of the index than controls. The 95% confidence interval is $d \pm 1.96 \sqrt{\text{Var}(d)}$.

3.5 Richness-Evenness Discordance Index

We define the Richness-Evenness Discordance Index (REDI) for a single study as:

$$\text{REDI} = 1 - \frac{2}{|\mathcal{R}| \cdot |\mathcal{E}|} \sum_{r \in \mathcal{R}} \sum_{e \in \mathcal{E}} \mathbf{1}[\text{sign}(d_r) = \text{sign}(d_e)]$$

where $\mathcal{R} = {S_{\text{obs}}, \hat{S}_{\text{Chao1}}, \text{PD}}$ is the set of richness indices, $\mathcal{E} = {H, D, J}$ is the set of evenness indices, $d_r$ and $d_e$ are the effect sizes, and $\mathbf{1}[\cdot]$ is the indicator function.

REDI ranges from 0 (perfect agreement: all richness and evenness indices have the same sign) to 1 (perfect disagreement: all richness indices have opposite sign from all evenness indices). The expected value of REDI under the null hypothesis that index signs are exchangeable is 0.5.

3.6 Permutation Test

To test whether the observed REDI values are higher than expected by chance, we perform a permutation test. Under the null hypothesis, the direction of each index is independent of whether the index measures richness or evenness.

For each study, we permute the labels of the 6 indices (randomly assigning each observed $d$ value to either the richness or evenness group) and recompute REDI. Repeating this 100,000 times gives a null distribution. The $p$-value is the proportion of permuted REDI values that exceed the observed REDI. We combine $p$-values across the 12 studies using Fisher's method:

$$\chi^2_{\text{Fisher}} = -2 \sum_{k=1}^{12} \ln(p_k) \sim \chi^2_{24}$$

3.7 Meta-Analysis

We perform separate random-effects meta-analyses for each of the 6 indices across the 12 studies. The model for index $j$ is:

$$d_{jk} = \mu_j + u_{jk} + \epsilon_{jk}$$

where $\mu_j$ is the overall effect for index $j$, $u_{jk} \sim N(0, \tau_j^2)$ is the between-study heterogeneity, and $\epsilon_{jk} \sim N(0, \sigma_{jk}^2)$ is the within-study sampling error. We estimate $\tau_j^2$ using the DerSimonian-Laird method and report the pooled effect $\hat{\mu}_j$ with 95% CI.

We also test whether the pooled effects differ between the richness and evenness index families using a meta-regression:

$$d_{jk} = \beta_0 + \beta_1 \cdot \mathbf{1}[j \in \mathcal{E}] + u_{jk} + \epsilon_{jk}$$

The coefficient $\beta_1$ captures the systematic difference between evenness and richness effect sizes, with a Wald test for $\beta_1 = 0$.

3.8 Sensitivity Analyses

We run three sensitivity checks. (1) No rarefaction: compute indices on relative abundance data without rarefying. (2) Alternative taxonomy: use Greengenes2 instead of SILVA for taxonomy assignment. (3) OTU clustering: cluster ASVs into 97% OTUs using VSEARCH and recompute all indices. Each sensitivity analysis produces a new set of effect sizes, and we check whether the REDI values change qualitatively.

4. Results

4.1 Directional Disagreement

Table 1 presents the effect sizes and directions for each study and index.

Table 1. Standardized mean difference (Cohen's $d$) for each alpha diversity index in each study. Negative values indicate lower diversity in cases. Bold entries indicate $p < 0.05$ (Wilcoxon rank-sum test). Studies where richness and evenness indices disagree on sign are marked with *.

Seven of 12 studies show directional disagreement between richness and evenness indices (marked with *). In these 7 studies, richness indices tend to be positive (cases have more species) while evenness indices are negative (cases have more uneven distributions).

4.2 REDI Values

Table 2. Richness-Evenness Discordance Index for each study. REDI = 0 means all indices agree on direction; REDI = 1 means perfect richness-evenness disagreement. Permutation $p$: probability of observing REDI this large or larger under the null.

Nine of 12 studies have REDI $> 0.5$. The combined Fisher's statistic across all 12 studies is $\chi^2_{24} = 58.3$ ($p = 0.003$), rejecting the null hypothesis of index exchangeability.

4.3 Meta-Analysis

The random-effects meta-analysis produces strikingly different pooled effects for richness versus evenness indices.

Richness indices: Observed ASVs $\hat{\mu} = +0.05$ (95% CI: $-0.08, +0.18$, $p = 0.44$); Chao1 $\hat{\mu} = +0.10$ (95% CI: $-0.04, +0.24$, $p = 0.16$); Faith's PD $\hat{\mu} = -0.01$ (95% CI: $-0.14, +0.12$, $p = 0.88$).

Evenness indices: Shannon $\hat{\mu} = -0.31$ (95% CI: $-0.42, -0.20$, $p < 0.001$); Simpson $\hat{\mu} = -0.35$ (95% CI: $-0.47, -0.23$, $p < 0.001$); Pielou $\hat{\mu} = -0.40$ (95% CI: $-0.52, -0.28$, $p < 0.001$).

The meta-regression coefficient $\hat{\beta}_1 = -0.37$ (95% CI: $-0.49, -0.25$, $p < 0.001$) confirms that evenness indices show systematically more negative effect sizes than richness indices.

Between-study heterogeneity is moderate for evenness indices ($I^2 = 41$ for Shannon, $I^2 = 38$ for Simpson) and high for richness indices ($I^2 = 62$ for observed ASVs, $I^2 = 58$ for Chao1), indicating that the richness signal is noisier across studies.

4.4 Mechanistic Interpretation

The pattern of high richness but low evenness in cases corresponds to a specific ecological structure: dysbiotic communities contain many species (some of which may be opportunistic colonizers from the oral cavity or environment) but are dominated by a few abundant taxa (typically Escherichia, Enterococcus, or Fusobacterium depending on disease).

We verify this by computing the Berger-Parker dominance index $d_{\text{BP}} = \max_i p_i$ for each sample. Across the 7 discordant studies, the Berger-Parker index is significantly higher in cases than controls (pooled $d = +0.28$, 95% CI: $+0.14, +0.42$, $p < 0.001$), confirming increased dominance.

4.5 Sensitivity Analyses

Without rarefaction: REDI values change by $< 0.1$ for 10 of 12 studies. The two studies that change (IBD-2 and PD-1) have the largest between-sample library size variation (coefficient of variation $> 0.8$), where rarefaction has the most impact.

Alternative taxonomy (Greengenes2): REDI values are identical for all 12 studies because alpha diversity is computed on ASVs, not taxonomic assignments.

OTU clustering (97%): REDI values decrease slightly (mean change $-0.06$) because OTU clustering merges closely related ASVs, reducing the influence of rare variants on richness estimates. All 7 discordant studies remain discordant.

5. Discussion

The central result — that richness and evenness indices frequently disagree on dysbiosis direction — has immediate practical implications. Any study that reports "reduced diversity" based on Shannon index alone may be making a claim that is not supported by Chao1 or observed species counts. The reverse is also true: studies reporting "no diversity change" based on richness estimators may be missing a real evenness shift.

We recommend that gut microbiome studies report at least one index from each family (richness and evenness) and explicitly note whether the indices agree. When they disagree, the ecological interpretation should distinguish between "the community has fewer species" (a richness statement) and "the community is dominated by fewer species" (an evenness statement). These represent different biological processes — species loss versus competitive exclusion — and may have different clinical relevance.

The finding that the meta-analytic "reduced diversity" effect is driven entirely by evenness indices has implications for microbiome-based diagnostics. If a diagnostic tool uses Shannon diversity as a biomarker for dysbiosis, it is really detecting dominance shifts, not species loss. This may or may not be the right target depending on the disease mechanism.

Our REDI metric provides a quick diagnostic for any case-control microbiome study. A REDI value above 0.5 flags a study where the diversity conclusion depends on the choice of index, warranting further investigation of the ecological mechanism.

6. Limitations

Dataset scope. We analyze 12 studies covering 6 disease categories. Diseases with different ecological mechanisms (e.g., antibiotic-induced dysbiosis, fecal microbiota transplant outcomes) may show different REDI patterns. A broader audit including studies from the curatedMetagenomicData resource (Pasolli et al., 2017) would strengthen generalizability.

16S resolution. All studies use 16S amplicon sequencing, which has limited taxonomic resolution below the genus level. Whole-metagenome shotgun sequencing provides species-level resolution that may alter the richness-evenness balance. Willis (2019) discusses estimation challenges specific to amplicon data that do not apply to shotgun metagenomics.

Rarefaction dependence. Although our sensitivity analysis shows REDI is stable to rarefaction, the absolute effect sizes change for studies with high library size variation. Model-based approaches (McMurdie & Holmes, 2014) avoid rarefaction but introduce their own assumptions about the count distribution.

Confounding variables. We compute effect sizes without adjusting for age, sex, BMI, diet, or medication use because the reprocessed data often lack complete metadata. Confounders that differentially affect richness and evenness could inflate REDI. Falony et al. (2016) found that medication use is a major confounder of microbiome diversity estimates.

Single amplicon region. Studies targeting different 16S variable regions (V3-V4 vs. V4) may introduce systematic biases in taxon detection that affect richness more than evenness. We include both V3-V4 and V4 studies but do not have enough power to test for region effects within disease categories.

7. Conclusion

The "reduced diversity" narrative in gut microbiome research is an artifact of index selection. When richness and evenness are measured separately, 7 of 12 studies show that richness is unchanged or increased while evenness is decreased. The REDI metric quantifies this discordance. Meta-analytically, Shannon diversity is reduced in disease ($d = -0.31$) but Chao1 richness is not ($d = +0.08$). Microbiome studies should report both richness and evenness indices and interpret disagreements mechanistically rather than defaulting to a single "diversity" number.

References

1. Callahan, B. J., McMurdie, P. J., Rosen, M. J., Han, A. W., Johnson, A. J. A., & Holmes, S. P. (2016). DADA2: High-resolution sample inference from Illumina amplicon data. Nature Methods, 13(7), 581-583.

2. Chao, A. (1984). Nonparametric estimation of the number of classes in a population. Scandinavian Journal of Statistics, 11(4), 265-270.

3. Duvallet, C., Gibbons, S. M., Gurry, T., Irizarry, R. A., & Alm, E. J. (2017). Meta-analysis of gut microbiome studies identifies disease-specific and shared responses. Nature Communications, 8, 1784.

4. Faith, D. P. (1992). Conservation evaluation and phylogenetic diversity. Biological Conservation, 61(1), 1-10.

5. Falony, G., Joossens, M., Vieira-Silva, S., Wang, J., Darzi, Y., Faust, K., Kurilshikov, A., Bonder, M. J., Valles-Colomer, M., Vandeputte, D., et al. (2016). Population-level analysis of gut microbiome variation. Science, 352(6285), 560-564.

6. Lozupone, C. A., Stombaugh, J. I., Gordon, J. I., Jansson, J. K., & Knight, R. (2012). Diversity, stability and resilience of the human gut microbiota. Nature, 489(7415), 220-230.

7. McMurdie, P. J., & Holmes, S. (2014). Waste not, want not: Why rarefying microbiome data is inadmissible. PLoS Computational Biology, 10(4), e1003531.

8. Shannon, C. E. (1948). A mathematical theory of communication. Bell System Technical Journal, 27(3), 379-423.

9. Sze, M. A., & Schloss, P. D. (2016). Looking for a signal in the noise: Revisiting obesity and the microbiome. mBio, 7(4), e01018-16.

10. Willis, A. D. (2019). Rarefaction, alpha diversity, and statistics. Frontiers in Microbiology, 10, 2407.