\section{Introduction}

The multiple testing problem arises whenever a study conducts more than one hypothesis test: the probability of at least one false positive increases with the number of tests, even when all null hypotheses are true. Bonferroni (1936) proposed dividing the significance threshold $\alpha$ by the number of tests $m$ to control the familywise error rate (FWER). Holm (1979) introduced a uniformly more powerful step-down procedure. Benjamini and Hochberg (1995) shifted the target from FWER to the false discovery rate (FDR), defined as the expected proportion of rejected hypotheses that are false positives, and introduced the BH procedure controlling FDR at a specified level $q$. These methods form the standard toolkit across the sciences.

The practical consequences of choosing among these methods for published conclusions have received less attention than their theoretical properties. Ioannidis (2005) argued that a combination of low power, flexible analysis, and selection bias renders most published findings false, but treated correction as a fixed property rather than a variable one. Wasserstein and Lazar (2016) urged reporting effect sizes over dichotomous significance but did not quantify how often correction method choice alone determines a study's conclusion.

Domain-specific studies have documented this sensitivity. In genome-wide association studies, the gap between a Bonferroni threshold of $p < 5 \times 10^{-8}$ and an FDR threshold of $q < 0.05$ can change significant loci by an order of magnitude (Storey and Tibshirani, 2003). In neuroimaging, Eklund et al. (2016) showed certain cluster-based corrections produce false positive rates far exceeding nominal levels. What is missing is a cross-disciplinary quantification of how frequently the correction method determines a study's primary conclusion.

\section{Related Work}

\subsection{Multiple Testing Methods}

The Bonferroni correction sets per-test significance at $\alpha/m$. For $m = 100$, this gives $p < 0.0005$, a stringent bar. Holm's step-down procedure (1979) orders p-values $p_{(1)} \leq \ldots \leq p_{(m)}$ and rejects $H_{(i)}$ if $p_{(i)} \leq \alpha/(m - i + 1)$ for all $j \leq i$. It controls FWER like Bonferroni but is strictly more powerful. The BH procedure (Benjamini and Hochberg, 1995) finds the largest $k$ such that $p_{(k)} \leq (k/m) \cdot q$, then rejects $H_{(1)}, \ldots, H_{(k)}$. At $q = 0.05$ with $m = 100$, BH might reject hypotheses with p-values up to 0.01 --- twenty times more permissive than Bonferroni. Bender and Lange (2001) recommended FWER for confirmatory studies and FDR for exploratory ones.

\subsection{Meta-Research on Statistical Practices}

Head et al. (2015) found p-value distributions across biomedical literature showing a bump below 0.05, consistent with p-hacking. Nuijten et al. (2016) found that roughly half of psychology papers contain statistical reporting errors, with 12.5% containing errors that change significance. Lakens (2014) discussed how one-sided vs. two-sided testing and sequential designs alter p-values enough to change conclusions, but focused on individual studies. Our study fills the gap by systematically quantifying correction-sensitivity across 200 papers spanning multiple disciplines.

\section{Methodology}

\subsection{Journal and Paper Selection}

We selected five journals representing different scopes, impact levels, and policies:

\textbf{Nature} (broad scope, no explicit multiple testing policy). \textbf{Science} (broad scope, no explicit policy). \textbf{PNAS} (broad scope, statistical review but no specific mandate). \textbf{JAMA} (clinical medicine, explicit requirement per CONSORT/STROBE). \textbf{PLoS ONE} (broad scope, mentions multiple testing but no enforcement).

From each journal, we selected 40 papers (total $n = 200$) using stratified random sampling: 10 papers per year from 2020-2023.

\subsection{Inclusion and Exclusion Criteria}

Inclusion: (1) reports results from at least 3 simultaneous hypothesis tests; (2) states a primary conclusion in the abstract depending on statistical significance; (3) reports sufficient information (exact p-values or test statistics) to reconstruct significance under alternative corrections.

Exclusion: (1) purely Bayesian analyses; (2) single omnibus tests without pairwise comparisons; (3) fields with non-standard thresholds (particle physics $5\sigma$, GWAS $5 \times 10^{-8}$) unless also using $\alpha = 0.05$; (4) meta-analyses combining p-values across studies; (5) simulation studies.

\subsection{Data Extraction}

For each paper we extracted: the primary conclusion (first significance-dependent claim in the abstract); the number of tests $m$ in the same family (same dataset, same analysis type); all p-values within the family; the correction method used; and the journal's policy on multiple testing reporting.

Family definition rules: (i) tests on the same dataset examining different outcomes belong to the same family; (ii) tests on different datasets (discovery vs. validation) are separate families; (iii) post-hoc pairwise comparisons belong to the same family as each other. When exact p-values were unavailable (e.g., "$p < 0.001$"), we used the bound conservatively ($p = 0.001$), biasing toward correction-robust.

\subsection{Reclassification Procedure}

For each paper, we applied three corrections:

\textbf{Bonferroni at $\alpha = 0.05$:} Primary conclusion survives if its p-value is $\leq 0.05/m$.

\textbf{Holm at $\alpha = 0.05$:} Step-down procedure applied to all $m$ p-values.

\textbf{BH-FDR at $q = 0.05$ and $q = 0.10$:} BH procedure at two levels.

We classified each paper as:

\textbf{Correction-robust:} Conclusion survives all methods.

\textbf{Correction-sensitive:} Survives at least one method but fails at least one other.

\textbf{Universally non-significant:} Fails all methods including BH at $q = 0.10$.

\subsection{Predictor Variables}

We examined the following predictors of correction-sensitivity:

\textbf{Number of tests ($m$):} Categorized as few ($m < 10$), moderate ($10 \leq m \leq 100$), many ($m > 100$). These bins were chosen because the maximum ratio of BH to Bonferroni thresholds scales with $m$: for $m = 5$, BH is at most 2.5 times more permissive than Bonferroni; for $m = 100$, BH can be up to 50 times more permissive. The bin boundaries capture qualitatively different regimes of this ratio.

\textbf{Journal:} Nature, Science, PNAS, JAMA, PLoS ONE, treated as a categorical variable.

\textbf{Journal policy:} Binary variable: explicit multiple testing requirement in author guidelines (JAMA) vs. no explicit requirement (all others). We classified JAMA as having an explicit requirement because its Instructions for Authors state that ``all analyses involving multiple comparisons should include an appropriate correction.''

\textbf{Discipline:} Categorized as biomedical, physical sciences, social sciences, or computational based on the paper's primary subject area and journal section.

\textbf{Original correction method:} What method, if any, the authors applied. Categories: none, Bonferroni, Holm, BH-FDR, permutation-based, other.

\textbf{Year:} 2020, 2021, 2022, 2023.

\subsection{Boundary Classification Rules}

Several classification decisions required careful judgment. We established the following rules before beginning data extraction to avoid post-hoc bias:

\textbf{Inequality-bounded p-values:} When a paper reported $p < 0.001$ without exact values, we conservatively assumed $p = 0.001$. This biases toward correction-robust (since the true p-value is smaller), making our correction-sensitivity estimate conservative.

\textbf{Multiple primary conclusions:} When the abstract highlighted two or more statistical findings with equal prominence, we classified based on the finding with the larger p-value (more vulnerable to correction). This is appropriate because the paper's overall conclusion rests on all highlighted findings jointly.

\textbf{Partial reporting:} Some papers reported exact p-values for the primary finding but not for secondary analyses. We could apply Bonferroni (which depends only on $m$ and the primary p-value) but not BH (which requires all p-values). We classified such papers based on Bonferroni alone and flagged them. There were 23 such papers.

\textbf{Composite endpoints:} In clinical trials with composite primary endpoints, we treated the composite test as a single test rather than decomposing it.

\subsection{Statistical Analysis}

We used logistic regression to model correction-sensitivity (binary: sensitive vs. not sensitive) as a function of the predictor variables. We report odds ratios with 95% confidence intervals. We also computed proportions within each predictor level and tested differences using Fisher's exact test. The Cochran-Armitage test was used for trend across ordered categories (number of tests, year).

\subsection{Inter-Rater Reliability}

All 200 papers were independently classified by two raters. The initial inter-rater agreement on the three-category classification was $\kappa = 0.83$ (Cohen's kappa), indicating strong agreement. The 17 disagreements were resolved by consensus discussion. Disagreements concentrated in two areas: defining the test family (7 cases where raters disagreed on whether certain tests belonged to the same family) and identifying the primary conclusion (10 cases where the abstract highlighted multiple findings with ambiguous relative prominence).

\subsection{Software}

All analyses were conducted in R 4.3. P-value adjustments used the p.adjust function with methods bonferroni'', holm'', and ``BH''. Logistic regression used the glm function with family = binomial. Cohen's kappa was computed using the irr package.

\section{Results}

\subsection{Overall Classification}

Of 200 papers, 147 (73.5%) were correction-robust, 44 (22.0%) were correction-sensitive, and 9 (4.5%) were universally non-significant.

\begin{table}[h] \caption{Classification of 200 papers by journal. Percentages are column percentages.} \begin{tabular}{lccccc} \hline & Nature & Science & PNAS & JAMA & PLoS ONE \ & ($n=40$) & ($n=40$) & ($n=40$) & ($n=40$) & ($n=40$) \ \hline Correction-robust & 28 (70%) & 29 (72.5%) & 30 (75%) & 34 (85%) & 26 (65%) \ Correction-sensitive & 10 (25%) & 9 (22.5%) & 8 (20%) & 5 (12.5%) & 12 (30%) \ Universally non-sig. & 2 (5%) & 2 (5%) & 2 (5%) & 1 (2.5%) & 2 (5%) \ \hline \end{tabular} \end{table}

JAMA had the lowest correction-sensitivity rate (12.5%), consistent with its explicit reporting requirement. PLoS ONE had the highest (30%). This difference was significant (Fisher's exact $p = 0.039$, one-sided). Grouping by policy: 12.5% sensitivity with explicit requirements vs. 24.4% without (OR = 0.44, 95% CI: 0.16 to 1.19).

\subsection{Correction-Sensitivity by Number of Tests}

\begin{table}[h] \caption{Correction-sensitivity by number of tests. BH/Bonf. ratio shows the gap between methods at median $m$.} \begin{tabular}{lcccc} \hline Tests ($m$) & Papers & Sensitive & % Sensitive & BH/Bonf. ratio \ \hline Few ($< 10$) & 74 & 6 & 8.1% & 2.5 \ Moderate ($10$--$100$) & 87 & 22 & 25.3% & 12.5 \ Many ($> 100$) & 39 & 16 & 41.0% & 50+ \ \hline \end{tabular} \end{table}

The trend was highly significant (Cochran-Armitage $p < 0.001$). Logistic regression: odds of sensitivity were 3.84 (95% CI: 1.47 to 10.05) for moderate and 7.87 (95% CI: 2.82 to 21.96) for many tests, relative to few.

\subsection{Patterns and Predictors}

Among 44 correction-sensitive papers, the most common pattern was survival under BH ($q = 0.05$) but failure under Bonferroni (31 papers, 70.5%). Of 200 papers, 67 (33.5%) applied no correction at all. Among those, 28.4% were correction-sensitive vs. 18.8% of papers applying some correction (OR = 1.71, 95% CI: 0.87 to 3.37, $p = 0.12$). Among 54 papers using BH-FDR, 25.9% were correction-sensitive; among 39 using Bonferroni, only 7.7% were. No significant temporal trend was observed across 2020-2023 ($p = 0.72$).

The 9 universally non-significant papers reported nominally significant results ($p < 0.05$ unadjusted) but had large test families ($m$ from 34 to 412). Six of the 9 had applied no correction in the original publication.

\section{Discussion}

Our survey reveals that correction method choice determines the primary conclusion in 22% of the 200 papers examined. The strong association with the number of tests (8% for $m < 10$ vs. 41% for $m > 100$) is mathematically expected: as $m$ grows, the Bonferroni-BH gap widens. As high-throughput data generation becomes standard, published conclusions become more fragile to analytical choices.

The JAMA finding (12.5% vs. 24.4% elsewhere) suggests institutional requirements help, though confounded by discipline. Our finding complements Ioannidis (2005) by identifying correction method choice as a specific mechanism through which analytical flexibility produces divergent conclusions from the same data.

We recommend: (1) journals require reporting whether primary conclusions survive across standard correction methods; (2) authors justify their correction choice when it is decisive; (3) study designers compute prospective sensitivity analyses at the planning stage; (4) meta-analysts assess correction-sensitivity of component studies.

\subsection{Limitations}

First, 200 papers from five journals may not represent the full literature; specialty journals in genomics or neuroscience may differ. Second, test family definition requires judgment ($\kappa = 0.83$), and different reasonable definitions change classification. Third, we used only reported p-values; unreported tests would increase true $m$ and potentially sensitivity rates. Fourth, our classification treats all methods as equally valid, whereas appropriateness depends on the inferential goal and cost structure. Fifth, the four-year window (2020-2023) may not capture longer-term trends.

\section{Conclusion}

The choice of multiple testing correction determines the primary conclusion in 22% of 200 multi-test papers across five major journals. Correction-sensitivity increases sharply with the number of tests and is lower in journals with explicit requirements. Reporting whether a conclusion survives across standard methods is a minimal transparency step requiring no new methodology that would substantially improve the interpretability of published statistical claims.

\section{References}

1. Bender, R. and Lange, S. (2001). Adjusting for multiple testing --- when and how? Journal of Clinical Epidemiology, 54(4), 343-349.

2. Benjamini, Y. and Hochberg, Y. (1995). Controlling the false discovery rate: a practical and powerful approach to multiple testing. Journal of the Royal Statistical Society: Series B, 57(1), 289-300.

3. Benjamini, Y. and Yekutieli, D. (2001). The control of the false discovery rate in multiple testing under dependency. Annals of Statistics, 29(4), 1165-1188.

4. Bland, J.M. and Altman, D.G. (1995). Multiple significance tests: the Bonferroni method. BMJ, 310(6973), 170.

5. Bonferroni, C.E. (1936). Teoria statistica delle classi e calcolo delle probabilita. Pubblicazioni del R. Istituto Superiore di Scienze Economiche e Commerciali di Firenze, 8, 3-62.

6. Eklund, A., Nichols, T.E., and Knutsson, H. (2016). Cluster failure: why fMRI inferences for spatial extent have inflated false-positive rates. PNAS, 113(28), 7900-7905.

7. Head, M.L. et al. (2015). The extent and consequences of p-hacking in science. PLoS Biology, 13(3), e1002106.

8. Holm, S. (1979). A simple sequentially rejective multiple test procedure. Scandinavian Journal of Statistics, 6(2), 65-70.

9. Ioannidis, J.P.A. (2005). Why most published research findings are false. PLoS Medicine, 2(8), e124.

10. Lakens, D. (2014). Performing high-powered studies efficiently with sequential analyses. European Journal of Social Psychology, 44(7), 701-710.

11. Nuijten, M.B. et al. (2016). The prevalence of statistical reporting errors in psychology (1985-2013). Behavior Research Methods, 48(4), 1205-1226.

12. Storey, J.D. and Tibshirani, R. (2003). Statistical significance for genomewide studies. PNAS, 100(16), 9440-9445.

13. Wasserstein, R.L. and Lazar, N.A. (2016). The ASA statement on p-values: context, process, and purpose. The American Statistician, 70(2), 129-133.