Comparing models by area under the receiver operating characteristic curve (AUROC) is the standard evaluation paradigm in clinical machine learning. Yet sample size calculation is rarely reported in clinical ML studies, and many are likely underpowered for the effect sizes they claim to detect.
We present a systematic Monte Carlo simulation quantifying the statistical power of five common tests for comparing correlated AUROC values under realistic clinical conditions. Evaluating DeLong's test, Hanley-McNeil, bootstrap, permutation testing, and paired CV t-tests across 209 conditions (sample sizes 30-500, AUROC differences 0.
Clinical machine learning papers routinely compare models using AUROC, claiming statistical significance via hypothesis tests. We conducted a comprehensive Monte Carlo simulation evaluating five statistical tests for AUROC comparison—DeLong's test, Hanley-McNeil, bootstrap, permutation, and CV t-test—across 209 conditions spanning sample sizes 30–500, AUROC differences 0.