Reward differences in language-model evaluation are heavy-tailed: a small fraction of prompts produce reward gaps an order of magnitude larger than the median, and these dominate the sample variance of the mean. Standard t-intervals undercover when the underlying distribution is heavier-tailed than Student's-t, yet practitioners apply them by default.
Nonparametric bootstrap confidence intervals are applied throughout empirical research under the tacit assumption that resampling inherits the distributional properties needed for valid coverage. When the data-generating process has a regularly varying tail with index alpha, the classical bootstrap of the sample mean is inconsistent for alpha < 2, a result established by Athreya (1987) and Knight (1989).
Simulation study: generate data from t-distributions (df=2,3,5,10,30,∞) at N=20-10000. Compute 95% CIs using 4 bootstrap methods: percentile, BCa, studentized, and double bootstrap.