How Many Samples Do You Need? Practical Sample Size Calculation for AUROC Comparison in Clinical AI

Abstract

Comparing models by area under the receiver operating characteristic curve (AUROC) is the standard evaluation paradigm in clinical machine learning. Yet sample size calculation is rarely reported in clinical ML studies, and many are likely underpowered for the effect sizes they claim to detect. At a typical clinical sample size of N=100, our formula (Section 3) yields an expected power of approximately 7% for detecting a difference of 0.02 in AUROC with model correlation ρ=0.80 and prevalence 30%—barely above the 5% false positive rate. Despite the existence of well-established statistical theory for AUROC comparison power, practical guidance remains inaccessible to most ML practitioners and clinicians. This paper bridges that gap with three contributions. First, we present a simplified closed-form sample size formula derived from the Hanley-McNeil variance approximation and DeLong's nonparametric framework, requiring only four inputs: baseline AUROC, expected improvement, model correlation, and disease prevalence. Second, we provide comprehensive ready-to-use lookup tables covering the full parameter space encountered in clinical AI—baseline AUROCs from 0.70 to 0.95, improvements from 0.01 to 0.10, model correlations from 0 (unpaired) to 0.95, and disease prevalences from 5% to 50%. Third, we present a practical decision flowchart and three worked case studies (radiology AI, sepsis prediction, ECG screening) demonstrating how to apply these tools in realistic study design scenarios. Our tables reveal that detecting a 2-point AUROC improvement with 80% power requires between 304 and 7,017 subjects depending on design parameters—far exceeding the sample sizes available in most clinical datasets. We show that paired evaluation designs reduce required sample sizes by 5–10× compared to unpaired approaches, and that multiple comparison correction for benchmark-style evaluations further inflates requirements by 1.3–2.1×. We argue that sample size calculation should be mandatory for any study claiming to compare models by AUROC, and provide all necessary tools to make this calculation routine.

1. Introduction

The area under the receiver operating characteristic curve has become the lingua franca of clinical machine learning evaluation. When a new diagnostic model is proposed—whether for detecting diabetic retinopathy from fundus images, predicting sepsis from vital signs, or screening for arrhythmias from ECG recordings—its performance is almost invariably reported as an AUROC value and compared against existing baselines using a statistical hypothesis test, most commonly DeLong's test (DeLong et al., 1988).

This evaluation paradigm has a critical weakness: nearly all of these comparisons are conducted without formal sample size planning. Authors collect whatever data is available, compute AUROC values, and apply DeLong's test—hoping for a p-value below 0.05. When they obtain it, they declare superiority. When they don't, they report "no significant difference" and move on. Neither conclusion is well-supported without understanding the statistical power of the comparison.

The consequences of this omission are severe. Recent simulation work has demonstrated that at N=100 with a baseline AUROC of 0.80, DeLong's test achieves only 7.3% power to detect a ΔAUROC of 0.02—a difference routinely reported as clinically meaningful in published studies. At this power level, the test is essentially flipping a coin. Even for a substantial improvement of ΔAUROC=0.05, power reaches only 31.8% at N=100. This means that roughly two-thirds of studies comparing models that genuinely differ by 5 AUROC points will fail to detect the difference.

Why does this problem persist? The statistical theory for AUROC comparison power is well-established. DeLong et al. (1988) provided the nonparametric variance estimator for paired AUROC differences. Hanley and McNeil (1982) derived closed-form variance approximations under the binormal model. Obuchowski (1997) extended these methods to clustered data designs. Yet these papers are written for statisticians, not for the ML engineers and clinicians who actually need to plan studies. The formulas involve structural components, placement values, and U-statistic theory that require substantial statistical training to apply.

This paper has a simple goal: make AUROC comparison power analysis accessible to anyone who can look up a value in a table. We provide three practical tools:

1. A simplified formula that distills the DeLong/Hanley-McNeil framework into a single equation with four intuitive inputs
2. Comprehensive lookup tables covering every realistic combination of baseline AUROC, effect size, model correlation, and study design
3. A decision flowchart with worked examples that guides practitioners through the sample size calculation process

Our target audience is the ML practitioner or clinician who needs to answer one question: "How many samples do I need to demonstrate that my new model is better?"

2. The Problem: A Power Crisis in Clinical ML

Before presenting the solution, we briefly quantify the scope of the problem. This section summarizes key findings from systematic power analysis of AUROC comparison tests, providing the motivation for the tools developed in subsequent sections.

2.1 The Baseline: What Power Do Typical Studies Have?

Consider the most common scenario in clinical ML: a researcher has access to a dataset of approximately 100 patients, has developed a new predictive model with AUROC of 0.82, and wishes to show it outperforms the baseline model with AUROC of 0.80. They apply DeLong's test.

What is the probability that this test will correctly identify the improvement?

The answer, established through extensive Monte Carlo simulation with 1,000 replications per condition, is 7.3%. Not 73%—seven point three percent. The test has essentially no ability to detect this difference. At this power level, the study would need to be repeated roughly 14 times before one would expect a single statistically significant result.

This is not an edge case. Table 1 shows the empirical power of DeLong's test across a range of typical clinical conditions, all at N=100 with model correlation ρ=0.80 and disease prevalence of 30%.

Table 1: Power of DeLong's Test at N=100 (ρ=0.80, prevalence=0.30)

For ΔAUROC ≤ 0.02, the power values (4.9–11.3%) are virtually indistinguishable from the 5% type I error rate. The test simply cannot tell the difference between "no improvement" and "small improvement." Even at ΔAUROC = 0.05—which represents a substantial clinical advance—power ranges from only 24% to 58%, well below the conventional 80% threshold.

2.2 What Would Adequate Power Require?

The corollary of low power at fixed N is that detecting small effects requires very large N. Using the simulation data:

• ΔAUROC = 0.10 (large effect): N ≥ 100–200
• ΔAUROC = 0.05 (moderate effect): N ≥ 200–500
• ΔAUROC = 0.02 (small effect): N > 500 (far exceeding most clinical datasets)

These are approximate thresholds from simulation with a coarse grid. The exact values depend on the baseline AUROC, model correlation, and prevalence—which is precisely why we need the detailed tables developed in this paper.

2.3 Why This Matters for Published Literature

The underpoweredness of clinical AUROC comparisons has two complementary consequences:

High false negative rates. Studies that fail to detect real improvements may lead researchers to abandon genuinely better models. When a new sepsis prediction algorithm improves AUROC from 0.80 to 0.83 but the study reports "no significant difference," the improvement is real but invisible to the inadequately powered study.

Winner's curse. Among underpowered studies, those that do achieve statistical significance tend to dramatically overestimate the true effect size. If the true ΔAUROC is 0.03 but the study only has 12.5% power to detect it, the rare significant result will likely reflect a lucky draw with an observed Δ much larger than 0.03. This inflates published effect sizes and creates unrealistic performance expectations.

Publication bias. The combination of low power and significance-based publication creates a literature dominated by either false positives or inflated true positives, while studies with null results (which may reflect true underpoweredness rather than true equivalence) go unpublished.

The solution is straightforward: calculate the required sample size before conducting the study, and design accordingly. The remainder of this paper provides the tools to do so.

3. The Sample Size Formula

3.1 Statistical Framework

The problem of comparing two correlated AUROCs can be framed as a two-sided hypothesis test:

• H₀: AUC_A = AUC_B (the models perform equally)
• H₁: AUC_A ≠ AUC_B (the models differ)

Under DeLong's nonparametric framework, the test statistic is:

Z = (AUC_A − AUC_B) / √(Var(AUC_A − AUC_B))

Under H₁ with a true difference δ = AUC_B − AUC_A, this statistic follows approximately a normal distribution with mean δ/√V and variance 1, where V = Var(AUC_A − AUC_B).

To achieve power (1 − β) at significance level α (two-sided), we need:

δ / √V ≥ z_{α/2} + z_β

Rearranging for the variance:

V ≤ δ² / (z_{α/2} + z_β)²

Since V depends on the sample size N (through the variance of the AUROC estimates), this equation implicitly defines the required N.

3.2 The Hanley-McNeil Variance Approximation

The key insight is that we can approximate V using the Hanley-McNeil (1982) closed-form formula, avoiding the need for the full DeLong placement-value machinery.

For a single AUROC estimate based on n_pos positive and n_neg negative cases:

V_single = [A(1−A) + (n_pos − 1)(Q₁ − A²) + (n_neg − 1)(Q₂ − A²)] / (n_pos × n_neg)

where A is the AUROC value, and:

• Q₁ = A / (2 − A)
• Q₂ = 2A² / (1 + A)

These Q values arise from the exponential approximation to the conditional distributions of the Mann-Whitney placement values. They have an intuitive interpretation: Q₁ represents the probability that a randomly selected positive case outscores two randomly selected negative cases, and Q₂ represents the probability that two randomly selected positive cases both outscore a randomly selected negative case.

3.3 From Single-Model Variance to Paired Difference Variance

For two models evaluated on the same test set with correlation ρ between their AUROC estimates, the variance of the difference is:

V_diff = Var(AUC_A) + Var(AUC_B) − 2·Cov(AUC_A, AUC_B)

If both models have similar AUROCs (a reasonable assumption when the difference δ is small), we approximate Var(AUC_A) ≈ Var(AUC_B) ≈ V_single and Cov(AUC_A, AUC_B) ≈ ρ · V_single, giving:

V_diff ≈ 2 · V_single · (1 − ρ)

This is the crucial simplification. The variance of the paired AUROC difference depends on only four quantities:

1. The baseline AUROC (A) — determines V_single through Q₁ and Q₂
2. The disease prevalence (π) — determines n_pos = ⌊N·π⌋ and n_neg = N − n_pos
3. The model correlation (ρ) — the (1 − ρ) factor captures the benefit of paired evaluation
4. The sample size (N) — V_single scales as approximately 1/N

3.4 The Complete Formula

Combining the power equation with the variance approximation, the required sample size N is the smallest integer satisfying:

(z_{α/2} + z_β)² × 2 · V_single(N) · (1 − ρ) ≤ δ²

where V_single(N) is the Hanley-McNeil variance computed with n_pos = ⌊N·π⌋ and n_neg = N − n_pos.

Because V_single depends on N through n_pos and n_neg, this equation does not have a clean closed-form solution. However, it is trivially solved by binary search: start with a candidate N, compute V_single, check if the power criterion is met, and adjust N accordingly.

For a rough closed-form approximation suitable for mental arithmetic, note that for large N with prevalence π:

V_single ≈ [A(1−A) + (π·N)(Q₁−A²) + ((1−π)·N)(Q₂−A²)] / (π(1−π)·N²)

≈ [π(Q₁−A²) + (1−π)(Q₂−A²)] / (π(1−π)·N) + A(1−A)/(π(1−π)·N²)

For large N, the first term dominates:

V_single ≈ C(A, π) / N

where C(A, π) = [π(Q₁−A²) + (1−π)(Q₂−A²)] / (π(1−π)).

This yields the approximate closed-form:

N ≈ (z_{α/2} + z_β)² × 2·C(A,π)·(1−ρ) / δ²

For 80% power at α = 0.05 (two-sided), (z_{0.025} + z_{0.80})² = (1.96 + 0.84)² = 7.85, so:

N ≈ 15.7 × C(A, π) × (1 − ρ) / δ²

3.5 Worked Example

Let's walk through a concrete example. Suppose we are planning a radiology AI study:

• Baseline model AUROC: A = 0.85
• Expected improvement: δ = 0.03
• Disease prevalence: π = 0.30
• Paired evaluation on same images: ρ = 0.90
• Target: 80% power, α = 0.05 (two-sided)

Step 1: Compute Q₁ and Q₂.

• Q₁ = 0.85 / (2 − 0.85) = 0.85 / 1.15 = 0.7391
• Q₂ = 2 × 0.85² / (1 + 0.85) = 2 × 0.7225 / 1.85 = 0.7811

Step 2: Compute C(A, π).

• C = [0.30 × (0.7391 − 0.7225) + 0.70 × (0.7811 − 0.7225)] / (0.30 × 0.70)
• C = [0.30 × 0.0166 + 0.70 × 0.0586] / 0.21
• C = [0.00498 + 0.04102] / 0.21
• C = 0.046 / 0.21 = 0.219

Step 3: Apply the formula.

• N ≈ 15.7 × 0.219 × (1 − 0.90) / 0.03²
• N ≈ 15.7 × 0.219 × 0.10 / 0.0009
• N ≈ 15.7 × 0.0219 / 0.0009
• N ≈ 382

Exact answer (from numerical computation): N = 384.

The approximate formula gives N ≈ 382, which is remarkably close. The slight discrepancy arises from the large-N approximation to V_single; for practical purposes, the formula is accurate to within 5% for N > 50.

Interpretation: To detect a 3-point AUROC improvement over a baseline of 0.85, with paired evaluation (ρ = 0.90), 30% disease prevalence, and standard 80% power at α = 0.05, you need 384 subjects.

For comparison, if you used an unpaired design (ρ = 0):

• N ≈ 15.7 × 0.219 × 1.0 / 0.0009 ≈ 3,820

The paired design reduces the required sample size by a factor of 10.

4. Ready-to-Use Sample Size Tables

The tables below provide the required sample size N to achieve 80% power for DeLong's test at α = 0.05 (two-sided), computed using the exact Hanley-McNeil variance formula with binary search over N. All tables assume a disease prevalence of 30% unless otherwise specified.

4.1 Primary Tables by Model Correlation

The model correlation ρ is the single most important design parameter after the effect size itself. It captures the degree to which two models agree in their predictions, which determines how much of the total AUROC variance cancels out in the paired difference. We present tables for four correlation levels:

• ρ = 0.50: Low correlation (e.g., fundamentally different model architectures)
• ρ = 0.75: Moderate correlation (e.g., same architecture, different features)
• ρ = 0.90: High correlation (e.g., ablation studies, same model with/without a feature)
• ρ = 0.95: Very high correlation (e.g., fine-tuning variants, hyperparameter changes)

Table 2: Required N for 80% Power — Low Correlation (ρ = 0.50)

Table 3: Required N for 80% Power — Moderate Correlation (ρ = 0.75)

Table 4: Required N for 80% Power — High Correlation (ρ = 0.90)

Table 5: Required N for 80% Power — Very High Correlation (ρ = 0.95)

Note: "—" indicates AUROC + ΔAUC > 1.0, which is not possible.

4.2 The Unpaired Baseline

For comparison, Table 6 shows the required N when using an unpaired test (ρ = 0), which is the implicit assumption when models are evaluated on different test sets or when no pairing structure is exploited:

Table 6: Required N for 80% Power — Unpaired Design (ρ = 0)

The numbers speak for themselves. Detecting a 2-point AUROC improvement with an unpaired design requires 6,000–14,000 subjects. This is why paired evaluation on the same test set is not optional—it is an essential design requirement.

4.3 Reading the Tables

To use these tables:

1. Identify your baseline AUROC. Use published baselines or preliminary data. Round to the nearest table value.
2. Estimate your expected improvement (ΔAUC). Be honest. If you have no prior data, assume a conservative δ = 0.02–0.03.
3. Determine the expected model correlation. See Section 5 for guidance. Most paired evaluations fall in the ρ = 0.85–0.95 range.
4. Look up the required N in the appropriate table.
5. If N exceeds your available data, you have several options: accept reduced power and report it transparently, increase your dataset through multi-site collaboration, or reconsider whether AUROC comparison is the right evaluation approach for your study.

4.4 Key Patterns in the Tables

Several patterns are worth highlighting:

The δ² scaling. Required sample sizes scale approximately as 1/δ². Halving the effect size quadruples the required N. This means that detecting ΔAUC=0.01 requires roughly 100× the sample size needed for ΔAUC=0.10.

The (1−ρ) scaling. Increasing model correlation from ρ=0.50 to ρ=0.95 reduces required N by a factor of 10 (since (1−0.50)/(1−0.95) = 10). This is the most powerful lever available to researchers: using paired evaluation on the same test set can reduce sample size requirements by an order of magnitude.

Higher base AUROC → smaller N. Detecting a fixed ΔAUC is easier at higher baseline AUROC. This is because the AUROC's variance decreases as it approaches 1.0 (or 0.0), making differences more detectable. Detecting ΔAUC=0.03 at base AUROC=0.95 requires roughly half the sample size compared to base AUROC=0.70.

The practical lower bound. Even under the most favorable conditions (ρ=0.95, base AUROC=0.95), detecting ΔAUC=0.01 requires over 600 subjects. One-point AUROC differences are effectively undetectable at sample sizes below several thousand.

4.5 Extended Table: 90% Power

For studies requiring higher confidence, Table 7 provides sample sizes for 90% power at ρ = 0.90:

Table 7: Required N for 90% Power (ρ = 0.90, prevalence = 0.30)

Moving from 80% to 90% power increases the required N by approximately 34% (the ratio (z_{0.025} + z_{0.10})² / (z_{0.025} + z_{0.20})² = (1.96+1.28)²/(1.96+0.84)² = 10.50/7.84 = 1.34).

5. The Correlation Problem

The model correlation ρ is the most influential parameter in the sample size calculation after the effect size itself, yet it is almost never reported in clinical ML publications. This section addresses the practical challenge of estimating and maximizing ρ.

5.1 What Determines Model Correlation?

When two models are evaluated on the same test set, their predictions are correlated because both must correctly handle the "easy" cases and both struggle with the "hard" cases. The degree of correlation depends on:

Shared architecture and features. Two variants of the same deep learning architecture (e.g., ResNet-50 vs. ResNet-50 with an additional input modality) will have very high correlation (ρ ≈ 0.90–0.98). Two fundamentally different approaches (e.g., logistic regression on clinical features vs. a CNN on imaging data) may have moderate correlation (ρ ≈ 0.50–0.75).

Shared training data. Models trained on overlapping training sets will produce more correlated predictions than models trained on completely different data.

Task difficulty distribution. If the test set contains many cases that are trivially easy or trivially hard for both models, the correlation will be high even between dissimilar architectures. Tasks with a wide spectrum of difficulty levels tend to produce higher inter-model correlation.

5.2 Typical Correlation Ranges

While the true correlation must be estimated from pilot data or prior experience, the following ranges serve as practical guidelines:

5.3 Paired vs. Unpaired: The 10× Rule of Thumb

Comparing Tables 4 and 6 reveals a dramatic difference. For virtually every parameter combination, the unpaired design requires approximately 5–10× more subjects than the paired design at ρ = 0.90.

For example, to detect ΔAUC = 0.03 at base AUROC = 0.85:

• Paired (ρ = 0.90): N = 384
• Unpaired (ρ = 0): N = 3,824
• Inflation factor: 10.0×

This inflation factor is approximated by 1/(1−ρ). At ρ = 0.90, unpaired requires 1/0.10 = 10× more subjects.

The practical implication is unambiguous: always use paired evaluation. Both models should be evaluated on exactly the same test set, and DeLong's paired test should be used to account for the correlation structure. Using a two-sample comparison test (or evaluating models on different test sets) discards the information contained in the pairing and massively inflates sample size requirements.

5.4 When Correlation is Unknown

If you cannot estimate the correlation from prior data, use a conservative assumption:

• For ablation studies or fine-tuning: assume ρ = 0.90 (use Table 4)
• For comparing different architectures: assume ρ = 0.75 (use Table 3)
• For comparing fundamentally different approaches: assume ρ = 0.50 (use Table 2)

If the true correlation turns out to be higher than assumed, you will have more power than expected—a safe direction for the error. If the correlation is lower, you may be underpowered, but you will have been more conservative in your sample size planning.

5.5 Estimating Correlation from Pilot Data

If you have access to a pilot dataset (even a small one), you can estimate ρ directly:

1. Train both models
2. Generate predictions on the pilot test set
3. Compute the Pearson correlation between the two sets of predicted probabilities
4. Use this estimate for sample size planning

Even a rough estimate from 30–50 pilot cases is better than a blind assumption. The correlation of predicted probabilities is typically close to the correlation of the AUROC estimates, making it a useful proxy.

6. The Effect of Disease Prevalence

Disease prevalence (the proportion of positive cases in the test set) affects the AUROC variance through the number of positive and negative cases available for the Mann-Whitney statistic computation.

Table 8: Effect of Prevalence on Required N (AUROC=0.85, ΔAUC=0.03, ρ=0.90)

The effect is substantial. At 5% disease prevalence, the required sample size is more than 5× larger than at 30% prevalence. This is because with only 5% positive cases, N=384 yields just 19 positive cases—far too few for stable AUROC estimation.

6.1 The Minority Class Bottleneck

The AUROC variance is primarily driven by the smaller class. Using the Hanley-McNeil formula, the terms involving n_pos (number of positive cases) and n_neg (number of negative cases) enter asymmetrically. When one class is very small, the variance is dominated by the imprecise estimation from that class.

As a rule of thumb: ensure at least 30–50 cases in the minority class for the AUROC estimate to be reasonably stable. At 5% prevalence, this requires N = 600–1,000 just for stable AUROC estimation, before considering the additional sample size needed for adequate comparison power.

6.2 Practical Guidance for Imbalanced Datasets

For studies with severe class imbalance (prevalence < 10%):

1. Inflate the sample size according to Table 8 or by computing the formula with the actual prevalence.
2. Consider stratified sampling to enrich the positive class in the test set. Note that this changes the effective prevalence for power calculation but does not bias the AUROC estimate (since AUROC is prevalence-invariant).
3. Report the number of positive cases in addition to the total N. A study reporting "N=500" with 5% prevalence has only 25 positive cases—readers should know this.
4. Consider the partial AUROC or sensitivity at a fixed specificity, which may be more clinically relevant and potentially more powerful for specific operating regions of the ROC curve.

7. A Practical Decision Flowchart

For practitioners who want a quick path to the answer, we present a step-by-step decision algorithm:

Step 1: Define Your Expected Effect Size (ΔAUC)

This is the most important—and most difficult—step. The expected ΔAUC should be based on:

• Prior literature: What improvements have similar methods achieved?
• Pilot data: If available, what difference did you observe?
• Clinical relevance: What is the smallest improvement worth detecting?

Guidance by domain:

• Radiology AI (high baseline, incremental gains): δ ≈ 0.01–0.03
• Clinical prediction models (moderate baseline): δ ≈ 0.02–0.05
• New modality integration (potentially large gains): δ ≈ 0.05–0.10

Honesty check: If you expect ΔAUC < 0.02, ask yourself whether AUROC comparison is the right evaluation. Such small differences are nearly impossible to detect and may not be clinically meaningful regardless of statistical significance.

Step 2: Determine Your Baseline AUROC

Use the AUROC of the existing best model. If unknown, estimate from the literature. Round to the nearest table value (0.70, 0.80, 0.85, 0.90, or 0.95).

Step 3: Will You Use Paired Evaluation?

You should. Always evaluate both models on the same test set and use DeLong's paired test. The question is what correlation to expect:

• Ablation or variant: ρ ≈ 0.90–0.95 → Use Table 4 or 5
• Different architectures on same features: ρ ≈ 0.75–0.90 → Use Table 3 or 4
• Different approaches: ρ ≈ 0.50–0.75 → Use Table 2 or 3

Step 4: Look Up Required N

Find the intersection of your baseline AUROC row and ΔAUC column in the appropriate table.

Step 5: Compare Against Available Data

Three scenarios:

N_required ≤ N_available: Proceed with the study. You have adequate power. Report the a priori power calculation.

N_required > N_available but within 2×: Proceed, but report the achieved power (which will be below 80%). Report confidence intervals for ΔAUC regardless of significance. Consider using a one-sided test if the direction of improvement is known a priori (this reduces required N by approximately 20%).

N_required >> N_available: The study cannot reliably detect the expected difference. Options:

• Increase data through multi-site collaboration or data augmentation
• Accept reduced power and frame the study as exploratory/pilot
• Change the evaluation approach: use bootstrap confidence intervals for ΔAUC without formal hypothesis testing, or consider alternative metrics
• Report ΔAUC with confidence intervals regardless—the point estimate is still informative even if the study is underpowered for formal testing

Worked Decision Example

Dr. Chen is developing a new deep learning model for detecting pneumonia from chest X-rays. The current best model has AUROC = 0.88. She expects her model to achieve AUROC = 0.91 (ΔAUC = 0.03). She has access to 400 test images from her institution. Both models will be evaluated on all 400 images (paired design). Disease prevalence is approximately 25%.

Step 1: δ = 0.03 Step 2: Base AUROC ≈ 0.90 (nearest table value) Step 3: Paired evaluation. Both models are CNNs with similar architecture → ρ ≈ 0.90 Step 4: From Table 4, row AUROC=0.90, column ΔAUC=0.03: N = 270 Step 5: N_available (400) > N_required (270). ✓ The study is adequately powered.

Had Dr. Chen expected only ΔAUC = 0.02, Table 4 gives N = 607—exceeding her available data. She would need to either collaborate with another institution to increase N, accept that she can only detect improvements ≥ 0.03, or report the comparison as exploratory with explicit power limitations.

8. Case Studies

We apply the sample size framework to three realistic clinical AI scenarios, demonstrating the full calculation process and highlighting the practical trade-offs in study design.

8.1 Case Study 1: Radiology AI — Chest X-ray Interpretation

Scenario: A medical device company has developed an AI system for chest X-ray interpretation. The FDA-cleared baseline system has AUROC = 0.92 for detecting consolidation. The new version incorporates a vision transformer backbone and is expected to achieve AUROC = 0.94 (ΔAUC = 0.02). The company needs to design a validation study.

Parameters:

• Baseline AUROC: 0.92
• Expected improvement: δ = 0.02
• Both systems will be evaluated on the same images: ρ ≈ 0.90
• Disease prevalence in the study population: ~30%

Sample size calculation:

• From Table 4 (ρ = 0.90), interpolating between base AUROC = 0.90 (N = 607) and 0.95 (N = 318): estimated N ≈ 494
• Exact computation: N = 494 for 80% power
• For 90% power: N = 660

If using an unpaired design (e.g., comparing against published baseline from a different dataset):

• N = 4,924 for 80% power
• Inflation factor: 10.0×

Practical implications: A validation dataset of 500 radiographs is feasible for a multi-site study but represents a significant data collection effort. This explains why many radiology AI papers report improvements that are not statistically significant—they simply don't have enough data. The paired design is essential: it reduces the requirement from ~5,000 to ~500 images.

Recommendation: Plan for N = 660 (90% power) to provide a comfortable margin. Ensure paired evaluation on the same images. If the observed ΔAUC is smaller than 0.02, accept that the study may not reach significance and report confidence intervals.

8.2 Case Study 2: Sepsis Prediction in the ICU

Scenario: A hospital's clinical informatics team has developed an enhanced early warning system for sepsis that incorporates laboratory trends and nursing notes in addition to vital signs. The current system achieves AUROC = 0.78. Preliminary analysis on a development cohort suggests the new system achieves AUROC = 0.82 (ΔAUC = 0.04).

Parameters:

• Baseline AUROC: 0.78
• Expected improvement: δ = 0.04
• Both systems use overlapping but not identical features: ρ ≈ 0.85
• Sepsis prevalence in ICU population: ~15–20%

Sample size calculation:

• Exact computation: N = 437 for 80% power (at prevalence = 0.30)
• Adjusting for lower prevalence (20%): N ≈ 590
• For 90% power at prevalence 0.20: N ≈ 790

Unpaired comparison: N = 2,897 for 80% power (6.6× inflation).

Practical implications: A single-site ICU study over 6–12 months might accrue 500–1,000 eligible patients, making this study feasible at one site. However, the lower sepsis prevalence in general ward populations (5–10%) would dramatically increase the required N—potentially to several thousand patients—making single-site studies impractical for general ward sepsis prediction.

Recommendation: Conduct the study in the ICU where prevalence is higher. Plan for N ≈ 600 to account for the 15–20% prevalence. If expanding to general ward populations, plan a multi-site study with N > 2,000.

8.3 Case Study 3: ECG Screening for Atrial Fibrillation

Scenario: A consumer health company wants to validate its smartwatch-based ECG algorithm for detecting atrial fibrillation against a hospital-grade 12-lead ECG interpretation algorithm. The 12-lead system has AUROC = 0.85 for AF detection. The smartwatch algorithm is expected to achieve AUROC = 0.87 (ΔAUC = 0.02).

Parameters:

• Baseline AUROC: 0.85
• Expected improvement: δ = 0.02
• Same ECG recordings analyzed by both algorithms: ρ ≈ 0.90
• AF prevalence in screening population: ~10–15%

Sample size calculation:

• From Table 4, base AUROC = 0.85, ΔAUC = 0.02: N = 864 for 80% power (at 30% prevalence)
• Adjusting for 10% prevalence: N ≈ 2,400
• For 90% power at 10% prevalence: N ≈ 3,200

Unpaired comparison: N = 8,598 for 80% power (10× inflation).

Practical implications: This is the hardest of the three case studies. The combination of a small expected improvement (ΔAUC = 0.02) and low disease prevalence in the screening population creates a demanding sample size requirement. Even with paired evaluation, the company needs 2,000+ ECG recordings with confirmed AF status. This likely requires a multi-site study with active AF enrichment.

Recommendation: If the goal is to demonstrate equivalence rather than superiority, consider a non-inferiority design, which may require fewer subjects. Alternatively, increase the AF prevalence by enriching the study population (e.g., enrolling from cardiology clinics rather than the general population). However, enrichment changes the prevalence parameter in the power calculation, so the gain may be offset.

8.4 Summary of Case Studies

The case studies illustrate a consistent theme: even modest AUROC improvements (2–4 points) require hundreds to thousands of subjects for adequate statistical power. This fundamentally constrains the kinds of claims that clinical ML studies can support.

9. Multiple Comparison Correction

Clinical ML benchmarks frequently compare more than two models. A study evaluating 5 candidate models involves 10 pairwise comparisons; a benchmark with 10 models involves 45 comparisons. Without multiple comparison correction, the family-wise error rate—the probability of at least one false positive—inflates dramatically.

9.1 The Bonferroni Correction and Its Impact on Sample Size

The simplest correction is the Bonferroni adjustment: divide α by the number of comparisons. For k models with m = k(k−1)/2 pairwise comparisons:

α_corrected = α / m

This stricter significance threshold increases the required sample size because the z_{α/2} critical value increases.

Table 9: Effect of Multiple Comparison Correction on Required N (AUROC=0.85, ΔAUC=0.03, ρ=0.90)

9.2 Implications for ML Benchmarks

Large-scale ML benchmarks—such as those comparing language models, embedding models, or medical imaging architectures—routinely evaluate dozens of models with subtle performance differences. Our analysis reveals that these benchmarks are almost certainly underpowered for small differences:

Example: A benchmark comparing 10 models at AUROC ≈ 0.85 with typical differences of ΔAUC ≈ 0.02–0.03 requires:

• N = 864 for a single pairwise comparison (ΔAUC = 0.02, ρ = 0.90)
• N = 1,094 with Bonferroni correction for 45 comparisons
• Many benchmarks use N = 100–500

The implication is clear: most benchmark rankings within a few AUROC points of each other are statistically indistinguishable. Declaring a "winner" based on the highest AUROC among closely-matched models is essentially random selection.

9.3 Alternatives to Bonferroni

The Bonferroni correction is conservative (it assumes independent tests, but AUROC comparisons sharing the same test set are correlated). Less conservative alternatives include:

Holm's step-down procedure: Maintains the family-wise error rate while being less conservative than Bonferroni. The required sample size inflation is similar for small numbers of comparisons but modestly reduced for large numbers.

Benjamini-Hochberg procedure: Controls the false discovery rate (FDR) rather than the family-wise error rate. This is more appropriate when the goal is to identify a set of models that are better than the baseline, rather than to make all pairwise comparisons.

Pre-specified primary comparison: If the study has a single primary hypothesis (new model vs. best baseline), no correction is needed for the primary comparison. Secondary comparisons can be reported as exploratory.

9.4 Practical Recommendation

For clinical validation studies: designate a single primary comparison and calculate sample size without multiple comparison correction. Report additional comparisons as exploratory with appropriate caveats.

For benchmark studies: acknowledge that small performance differences (ΔAUC < 0.03) between models are likely not statistically distinguishable. Consider reporting confidence intervals or credible intervals for all pairwise differences rather than binary significance tests.

10. Sensitivity Analysis and Formula Validation

10.1 Comparison with Monte Carlo Simulation

To validate the closed-form sample size formula, we compare its predictions against empirical power from Monte Carlo simulation at selected conditions. The simulation data comes from 1,000 replications per condition using a bivariate normal score generation model (the same methodology described in the background power analysis study).

Table 10: Formula Prediction vs. Simulated Power

The formula predictions agree with the simulation results to within 2 percentage points across all conditions tested. This level of accuracy is more than sufficient for sample size planning, where uncertainties in the input parameters (particularly ρ and δ) typically dominate.

10.2 When the Approximation Breaks Down

The Hanley-McNeil variance approximation is least accurate when:

1. AUROC is very close to 1.0 (above 0.98). In this regime, the exponential approximation to the placement value distributions becomes inaccurate, and the normal approximation to the test statistic may also fail.

2. Sample size is very small (N < 30 with severe imbalance). With fewer than ~10 cases in the minority class, the variance estimate is unstable.

3. Models have very different AUROCs. The formula assumes both models have approximately the same AUROC (differing by δ). When δ is large relative to (1 − AUC), the equal-variance assumption breaks down. However, for δ ≤ 0.10 and AUC ≥ 0.70, the approximation remains accurate.

In these edge cases, Monte Carlo simulation provides a more reliable sample size estimate. However, for the vast majority of practical scenarios (AUROC 0.70–0.95, ΔAUC 0.01–0.10, N > 30), the formula is sufficiently accurate.

11. Limitations

11.1 Model Assumptions

The sample size formula is derived under the binormal model, which assumes that the score distributions for positive and negative cases are (approximately) normal after transformation. While DeLong's test itself is nonparametric, the power calculation assumes normally distributed AUROC estimates under the alternative hypothesis. This assumption is well-justified by the central limit theorem for moderate sample sizes (N > 50) but may be inaccurate for very small samples.

For deep learning models that produce highly non-normal score distributions (e.g., heavily concentrated near 0 and 1), the binormal approximation may underestimate variance and thus underestimate required sample sizes. In such cases, we recommend supplementing our tables with a bootstrap power simulation: draw B ≥ 1,000 bootstrap samples from a pilot dataset, compute DeLong's test on each, and estimate power as the fraction of samples achieving significance. When no pilot data is available, our tables remain a reasonable starting point for planning purposes.

11.1b Comparison with Existing Software

The R package pROC (Robin et al., 2011) provides power.roc.test() for sample size calculation. Our simplified formula agrees with pROC's output to within 5–10% for typical parameter ranges (AUC 0.70–0.95, ρ > 0.5), with the largest discrepancies occurring at extreme AUC values (> 0.95) where both approaches lose accuracy. Our contribution is not a new statistical method but rather an accessible presentation—ready-to-use tables and a decision flowchart—designed for practitioners who may not use R or have statistical software readily available. We encourage readers with access to pROC to cross-validate our tables for their specific parameter combinations.

11.2 Scope of Metrics

This paper addresses only the full AUROC (area under the entire ROC curve). Other evaluation metrics—partial AUROC over a specific false positive rate range, sensitivity at a fixed specificity, calibration metrics (Brier score, calibration error), and net benefit—may have different power characteristics. In particular, partial AUROC comparisons are more complex because the optimal variance estimator depends on the restriction range.

11.3 Bootstrap and Permutation Tests

Our formula and tables are calibrated for DeLong's test specifically. Bootstrap and permutation-based AUROC comparisons have similar but not identical power profiles. Simulation studies suggest that with sufficient resamples (≥ 1,000), bootstrap tests achieve power comparable to DeLong's test, so our tables provide reasonable approximations for bootstrap-based studies as well.

11.4 Fixed Test Set Assumption

The analysis assumes that models are evaluated on a fixed test set that is independent of the training data. In practice, some studies use cross-validation or repeated random splits, which introduces correlation between the training and test data. The power analysis for cross-validated AUROC comparison requires different methodology (accounting for the variance across folds) and is not addressed here.

11.5 Clinical vs. Statistical Significance

Sample size calculations determine the N required for statistical significance, but statistical significance does not imply clinical significance. A ΔAUC of 0.02 might be statistically detectable with N = 1,000 but clinically meaningless if it does not change patient management decisions. Conversely, a ΔAUC of 0.05 that is not statistically significant at N = 100 might still represent a clinically important improvement. Sample size planning should be guided by the smallest clinically meaningful effect size, not by the smallest detectable effect.

11.6 Correlation Estimation Uncertainty

The sample size formula is sensitive to the model correlation ρ, but ρ is often estimated imprecisely or assumed rather than measured. Sensitivity analysis across multiple ρ values (as demonstrated in our tables) can help bound the uncertainty. As a conservative practice, we recommend using a ρ value 0.05–0.10 lower than the point estimate from pilot data.

12. Recommendations and Conclusion

12.1 For Researchers Designing Studies

1. Calculate sample size before collecting data. Use the tables in this paper or the formula in Section 3 to determine the required N for your study's parameters. If N_required exceeds N_available, either adjust expectations, seek additional data, or frame the study as exploratory.

2. Always use paired evaluation. Evaluate both models on the same test set and use DeLong's paired test. This single design choice reduces the required sample size by 5–10× compared to unpaired evaluation.

3. Report the expected and achieved power. Include a power analysis in your methods section. If the study is underpowered for the observed effect size, state this explicitly.

4. Report model correlation. Compute and report the Pearson correlation between model predictions. This enables readers to assess your power and replicate your analysis.

5. Report confidence intervals for ΔAUC. A 95% confidence interval for the AUROC difference is informative regardless of sample size, while a p-value from an underpowered test is not.

12.2 For Reviewers and Editors

1. Request sample size justification for any study claiming to compare models by AUROC. The question "did you compute the required sample size for this comparison?" should be standard in peer review.

2. Be skeptical of small N + small ΔAUC. If a study reports a statistically significant 2-point AUROC improvement at N=80, consult Table 4: this comparison has roughly 5–10% power. The significant result is more likely a false positive or an inflated effect than a true finding.

3. Distinguish between "no difference" and "underpowered." When a study reports "no significant difference between models," ask what the achieved power was. Non-significance with 15% power tells you nothing; non-significance with 90% power is informative.

12.3 For Benchmark Organizers

1. Acknowledge statistical limitations. If your benchmark uses N < 1,000 test cases and compares models differing by ΔAUC < 0.03, the ranking within this range is statistically indistinguishable. Say so.

2. Report uncertainty bands rather than point rankings. A leaderboard showing "Model A: 0.872 ± 0.015, Model B: 0.869 ± 0.015" is more honest than "Model A: rank 1, Model B: rank 2."

3. Consider aggregation over tasks. If individual tasks have low power, aggregating over multiple tasks (with appropriate statistical methodology) can increase the effective sample size for detecting genuine model differences.

12.4 Summary

The tools presented in this paper—a simplified formula, comprehensive lookup tables, and a practical decision flowchart—make AUROC comparison power analysis accessible to any researcher who can identify four numbers: baseline AUROC, expected improvement, model correlation, and disease prevalence. The required sample sizes may be sobering: detecting a 2-point AUROC improvement with 80% power typically requires 300–7,000 subjects depending on design parameters. But knowing this number before starting the study is infinitely more valuable than discovering it after the fact.

The fundamental message is one of planning and humility. Statistical power is not a nice-to-have—it is a prerequisite for meaningful inference. A study that cannot detect the effect it claims to test is not a study; it is a lottery ticket. By providing practical tools for sample size calculation, we hope to shift clinical ML evaluation from post-hoc rationalization toward principled study design.

References

DeLong, E. R., DeLong, D. M., & Clarke-Pearson, D. L. (1988). Comparing the areas under two or more correlated receiver operating characteristic curves: A nonparametric approach. Biometrics, 44(3), 837–845.

Hanley, J. A., & McNeil, B. J. (1982). The meaning and use of the area under a receiver operating characteristic (ROC) curve. Radiology, 143(1), 29–36.

Obuchowski, N. A. (1997). Nonparametric analysis of clustered ROC curve data. Biometrics, 53(2), 567–578.

Sun, X., & Xu, W. (2014). Fast implementation of DeLong's algorithm for comparing the areas under correlated receiver operating characteristic curves. IEEE Signal Processing Letters, 21(11), 1389–1393.

Robin, X., Turck, N., Hainard, A., Tiberti, N., Lisacek, F., Sanchez, J.-C., & Müller, M. (2011). pROC: an open-source package for R and S+ to analyze and compare ROC curves. BMC Bioinformatics, 12, 77.