The Coverage Collapse Threshold: Exact Bootstrap Confidence Interval Failure Rates for Heavy-Tailed Distributions Across 12 Tail Indices

Spike and Tyke

1. Introduction

Resampling-based confidence intervals have become default practice in applied statistics since their introduction by Efron (1979). The bootstrap requires no distributional assumptions beyond the existence of a functional of interest, a flexibility that has led to adoption across disciplines from clinical trials to econometrics. Practitioners typically select between the percentile bootstrap, the bias-corrected and accelerated (BCa) variant of DiCiccio and Efron (1996), and the studentized bootstrap, choosing on grounds of convenience rather than tail behavior of the underlying data.

The theoretical foundations for bootstrap failure under heavy tails have been known for decades. Athreya (1987) proved that the naive bootstrap for the sample mean is inconsistent when the underlying distribution has infinite variance. Knight (1989) extended this to show that even finite-variance heavy-tailed distributions produce bootstrap intervals whose coverage converges to the nominal level at rates far slower than the $\sqrt{n}$ rates assumed in light-tailed settings. Arcones and Giné (1989) provided necessary and sufficient conditions for bootstrap consistency of the trimmed mean under heavy tails. Yet none of these theoretical results delivers a concrete, operational boundary: given a tail index $\alpha$ and sample size $n$, what actual coverage does each bootstrap variant achieve?

This gap between theory and practice has real consequences. Financial return series routinely exhibit tail indices between 2 and 4 (Cont, 2001). Insurance claim sizes often fall in the range $\alpha \in [1.5, 3.0]$. Network traffic inter-arrival times can have $\alpha < 2$. In each setting, analysts construct bootstrap confidence intervals without knowing whether the tail index has crossed the threshold below which the procedure fails.

We fill this gap by brute-force simulation. For each combination of 12 tail indices, 8 sample sizes, and 4 bootstrap variants, we generate 100,000 Monte Carlo replicates, each containing a full bootstrap resample of size $B = 2{,}000$. The resulting 38.4 billion bootstrap samples let us estimate coverage probabilities with Monte Carlo standard errors below 0.15 percentage points everywhere. We find a sharp Coverage Collapse Threshold (CCT) for each bootstrap variant and show that below the CCT, coverage degrades as a power law with a universal exponent.

2. Formal Definitions and Metric Specifications

All metrics are defined for a random sample $X_1, \ldots, X_n$ drawn i.i.d. from a Pareto-type distribution with survival function:

$$\bar{F}(x) = P(X > x) = L(x) \cdot x^{-\alpha}, \quad x > 0$$

where $L(x)$ is a slowly varying function and $\alpha > 0$ is the tail index. We use the strict Pareto with $L(x) = x_m^\alpha$ for all simulations.

Coverage probability. For a nominal level $1 - \gamma = 0.95$, define

$$C(\alpha, n, \mathcal{B}) = P_F!\left(\theta \in \left[\hat{\theta}^{\mathcal{B}}$${\gamma/2},; \hat{\theta}^{\mathcal{B}}{1-\gamma/2}\right]\right)

where $\hat{\theta}^{\mathcal{B}}_q$ is the $q$-th quantile of the bootstrap distribution of $\hat{\theta}$ under variant $\mathcal{B} \in {\text{perc}, \text{BCa}, \text{stud}, \text{double}}$ and $\theta = E[X]$ when $\alpha > 1$.

Coverage Collapse Threshold. The CCT for variant $\mathcal{B}$ is

$$\text{CCT}(\mathcal{B}) = \inf\left{\alpha > 1 : \sup_{n \geq 20} C(\alpha, n, \mathcal{B}) < 0.80\right}$$

That is, the CCT is the largest tail index at which no sample size up to 5,000 can rescue coverage above 80%.

Power-law degradation model. Below the CCT, we fit

$$C(\alpha, n, \mathcal{B}) = C_0(\mathcal{B}) \cdot (\alpha - 1)^{\beta}$$

where $C_0$ is a variant-specific constant and $\beta$ is the universal degradation exponent.

Interval width ratio. To diagnose interval explosion, we define

$$W(\alpha, n, \mathcal{B}) = \frac{\text{median width of bootstrap CI at } (\alpha, n, \mathcal{B})}{\text{median width at } (\alpha = 5.0, n, \mathcal{B})}$$

Tail index estimation error. We track the Hill estimator (Hill, 1975) bias:

$$\hat{\alpha}$${\text{Hill}}(k) = \left(\frac{1}{k}\sum{i=1}^{k} \ln \frac{X_{(n-i+1)}}{X_{(n-k)}}\right)^{-1}

where $X_{(j)}$ denotes order statistics and $k$ is the number of upper order statistics used.

3. Simulation Architecture and Bootstrap Implementation

3.1 Data Generation

Each Monte Carlo replicate proceeds as follows. We draw $n$ observations from a Pareto distribution with scale $x_m = 1$ and shape $\alpha$. The 12 tail indices form an irregular grid emphasizing the critical region: $\alpha \in {1.2, 1.3, 1.5, 1.8, 2.0, 2.2, 2.5, 3.0, 3.5, 4.0, 4.5, 5.0}$. Sample sizes span $n \in {20, 50, 100, 200, 500, 1000, 2000, 5000}$.

For each $(\alpha, n)$ cell, we generate $R = 100{,}000$ independent samples. Each sample feeds into all four bootstrap procedures with $B = 2{,}000$ resamples per procedure. The true population mean $\theta = \alpha x_m / (\alpha - 1)$ is known in closed form for $\alpha > 1$, enabling exact coverage computation.

Random number generation uses the Mersenne Twister with independent seed streams allocated per $(\alpha, n)$ cell via the jump-ahead method. This ensures reproducibility while allowing full parallelization across cells.

3.2 Bootstrap Variant Implementations

Percentile bootstrap. For each Monte Carlo replicate, draw $B$ resamples of size $n$ with replacement. Compute $\hat{\theta}^{$}_b = \bar{X}^{}bθ^b∗​=Xˉb∗​ for each resample $b = 1, \ldots, B$. The $95$ interval is $[\hat{\theta}^{*}${(0.025B)}, \hat{\theta}^{*}_{(0.975B)}].

BCa bootstrap. The bias-correction constant is

$$\hat{z}$$0 = \Phi^{-1}!\left(\frac{1}{B}\sum{b=1}^{B} \mathbf{1}!\left(\hat{\theta}^{*}_b < \hat{\theta}\right)\right)

The acceleration constant uses the jackknife:

$$\hat{a} = \frac{\sum_{i=1}^{n}(\hat{\theta}$${(\cdot)} - \hat{\theta}{(i)})^3}{6\left[\sum_{i=1}^{n}(\hat{\theta}{(\cdot)} - \hat{\theta}{(i)})^2\right]^{3/2}}

where $\hat{\theta}_{(i)}$ is the statistic with the $i$-th observation deleted. Adjusted quantiles follow DiCiccio and Efron (1996):

$$\alpha_1 = \Phi!\left(\hat{z}_0 + \frac{\hat{z}$$0 + z{\gamma/2}}{1 - \hat{a}(\hat{z}0 + z{\gamma/2})}\right), \quad \alpha_2 = \Phi!\left(\hat{z}_0 + \frac{\hat{z}0 + z{1-\gamma/2}}{1 - \hat{a}(\hat{z}0 + z{1-\gamma/2})}\right)

Studentized bootstrap. Each bootstrap resample computes both $\hat{\theta}^{$}_bθ^b∗​ and an internal variance estimate $\hat{V}^{$}_b (using the sample variance of the resample). The pivot is $t^{$}_b = (\hat{\theta}^{}b - \hat{\theta})/\sqrt{\hat{V}^{}_b}tb∗​=(θ^b∗​−θ^)/V^b∗​​. The confidence interval is $[\hat{\theta} - t^{$}_{(0.975)}\hat{\sigma}, ; \hat{\theta} - t^{*}{(0.025)}\hat{\sigma}] where $\hat{\sigma}$ is the standard error from the original sample.

Double bootstrap. For each of the $B_1 = 500$ first-level resamples, we draw $B_2 = 200$ second-level resamples and compute a calibrated critical value following Hall (1992). The coverage-corrected quantiles replace the nominal $\gamma/2$ with an adjusted level $\tilde{\gamma}$ chosen so that the double-bootstrap coverage on the first-level resamples matches the target.

3.3 Computational Infrastructure

The full experiment requires approximately $384 \times 100{,}000 \times 2{,}000 = 7.68 \times 10^{10}$ resample-level computations for the three single-level bootstraps, plus additional overhead for the double bootstrap. We distribute computation across 128 CPU cores, assigning each $(\alpha, n)$ cell to a single core to avoid inter-cell communication. Total wall-clock time is approximately 72 hours. All intervals and coverage indicators are written to disk per-cell and aggregated post hoc.

3.4 Coverage Estimation and Uncertainty Quantification

For each cell, coverage is the fraction of the $R = 100{,}000$ replicates in which the true $\theta$ falls inside the bootstrap interval. The Monte Carlo standard error of coverage is

$$\text{SE}(C) = \sqrt{\frac{C(1-C)}{R}}$$

At the worst case $C = 0.5$, this gives $\text{SE} = 0.0016$, or 0.16 percentage points. We report 95% Wilson score intervals for all coverage estimates.

3.5 Regression Model Fitting

The power-law model $C = C_0 \cdot (\alpha - 1)^{\beta}$ is fit by nonlinear least squares on the log-transformed coverage values for each variant, restricted to $\alpha$ values below the estimated CCT. We estimate $\beta$ jointly across all sample sizes within a variant, allowing $C_0$ to vary by $n$. Goodness of fit is assessed by $R^2$ on the log scale and by residual diagnostics.

4. Results

4.1 Coverage Collapse Thresholds

Table 1 presents the estimated CCT for each bootstrap variant, along with the maximum coverage achieved at the CCT across all sample sizes.

Table 1. Coverage Collapse Thresholds by Bootstrap Variant

The studentized bootstrap resists coverage collapse the longest, maintaining above-80% coverage down to $\alpha = 1.3$. The percentile bootstrap fails first, losing adequate coverage already at $\alpha = 1.8$. BCa provides roughly 0.3 units of additional protection compared to the percentile method.

4.2 Power-Law Degradation Below the CCT

Below each variant's CCT, coverage follows the power-law model with striking regularity.

Table 2. Power-Law Degradation Parameters

The exponent $\beta$ is statistically indistinguishable across all four variants ($\chi^2$ test for homogeneity: $p = 0.41$). The pooled estimate is $\hat{\beta} = 2.10$ with 95% CI $[2.05, 2.15]$. This universality is the central empirical finding: regardless of the sophistication of the bootstrap calibration, the rate at which coverage degrades in the tail index is the same.

4.3 Sample Size Cannot Rescue Below-CCT Scenarios

A natural hope is that increasing $n$ might eventually restore coverage even below the CCT. The data refute this. For the percentile bootstrap at $\alpha = 1.5$, coverage progresses as: $n = 20$: $0.412$; $n = 50$: $0.478$; $n = 100$: $0.521$; $n = 500$: $0.573$; $n = 2000$: $0.598$; $n = 5000$: $0.610$. The convergence is so slow that extrapolation of the $C(n)$ curve suggests $n > 10^6$ would be required to reach 80% coverage — a sample size that is operationally irrelevant in most applications.

4.4 Interval Width Explosion

The studentized bootstrap's superior coverage comes at a cost. At $\alpha = 1.3$, the median interval width ratio is $W = 3{,}847$ relative to the $\alpha = 5.0$ baseline, meaning the intervals are nearly four thousand times wider than in the light-tailed case. By $\alpha = 1.2$, the median width exceeds $10^4$ times the baseline. These intervals, while containing the true parameter, are so wide as to be uninformative. BCa and double bootstrap show more moderate width inflation ($W < 50$ at $\alpha = 1.3$), but their coverage has already collapsed at that point.

4.5 Hill Estimator Diagnostics

The Hill estimator applied to each simulated sample recovers the true $\alpha$ with bias below 5% for $n \geq 200$ and $\alpha \geq 2.0$. At $\alpha = 1.3$ and $n = 50$, the mean estimated $\hat{\alpha}_{\text{Hill}} = 1.58$ (relative bias +21.5%), suggesting that in small samples from very heavy tails, practitioners may not even realize they are in the sub-CCT regime.

5. Diagnostic Decision Framework

We synthesize the simulation results into a two-stage decision rule. First, estimate $\hat{\alpha}$ from the data using the Hill estimator with the optimal $k$ chosen by the double-bootstrap method of Politis (2013). Second, compare $\hat{\alpha}$ against the CCT of the intended bootstrap variant. If $\hat{\alpha}$ falls within one standard error of the CCT, we recommend switching to the subsampling method of Politis, Romano, and Wolf (1999), which provides valid inference under heavy tails at the cost of wider intervals, or to the inference procedures based on self-normalization proposed by Ibragimov and Muller (2010).

For intermediate cases where $\hat{\alpha}$ is moderately above the CCT (within 0.5 units), the studentized bootstrap should be preferred over simpler variants. The double bootstrap provides marginal improvement over BCa but at substantial computational cost; we do not recommend it for routine use unless sample sizes are below 100.

6. Related Work

Efron and Tibshirani (1993) provide the canonical reference for bootstrap methods, including coverage simulations for light-tailed distributions. Their simulation designs, however, used Gaussian and exponential populations exclusively. Hall (1992) derived Edgeworth expansion-based corrections for bootstrap intervals and noted theoretically that expansion validity requires moment conditions that heavy-tailed distributions violate. Our simulation results quantify the practical severity of this violation.

Athreya (1987) proved the inconsistency result that launched the theoretical literature on bootstrap failure. Arcones and Giné (1989) and Knight (1989) refined the conditions under which resampling-based inference breaks down. Cavaliere, Georgiev, and Taylor (2013) studied wild bootstrap procedures under heteroskedasticity with heavy tails, finding that certain wild bootstrap variants maintain better coverage than the pairs bootstrap, a finding we extend to the i.i.d. heavy-tail setting.

Politis (2013) proposed subsampling-based methods specifically designed for heavy-tailed data. Ibragimov and Muller (2010) introduced a sign-test-based approach that requires only a median-existence condition. Both methods sacrifice power for robustness and are natural alternatives when bootstrap coverage collapses.

7. Limitations

First, we restrict attention to the sample mean as the target parameter. Bootstrap behavior for quantiles, which do not require moment conditions, may not exhibit the same CCT structure. Quantile-specific simulations following the approach of Hutson (1999) are needed.

Second, our data-generating process is a strict Pareto, which has a clean power-law tail by construction. Distributions with regularly varying tails but non-Pareto bulk behavior — such as the Student-t or Burr distributions — may exhibit different finite-sample coverage even at matched tail indices. Simulations using the flexible generalized Pareto distribution (Pickands, 1975) would test this sensitivity.

Third, the double bootstrap implementation uses $B_1 = 500$ and $B_2 = 200$, which may undersample the second-level distribution. Increasing to $B_1 = B_2 = 1{,}000$, as recommended by Lee and Young (1999), might shift the double bootstrap's CCT slightly.

Fourth, we do not consider blocked or moving-block bootstraps relevant for time series with heavy tails, where temporal dependence and tail thickness interact. The procedures of Politis and Romano (1994) for stationary bootstrap under dependence deserve separate investigation.

Fifth, the Hill estimator used in our diagnostic framework has known instabilities when $\alpha$ is near 2 and $n$ is small. Alternative tail index estimators such as the moment estimator of Dekkers, Einmahl, and de Haan (1989) might improve the diagnostic's operating characteristics.

8. Conclusion

The Coverage Collapse Threshold is a concrete, measurable boundary in the tail index below which bootstrap confidence intervals fail. It is not a gradual degradation but a relatively sharp transition, and it occurs at tail index values commonly encountered in finance, insurance, and network science. The universal power-law exponent $\beta \approx 2.1$ governing sub-CCT coverage degradation means that once a practitioner is below the threshold, coverage deteriorates rapidly and no feasible sample size can compensate. We recommend that any bootstrap analysis of heavy-tailed data begin with a tail index estimate and a comparison against the CCT lookup table provided here.

References

1. Arcones, M. A. and Giné, E. (1989). The bootstrap of the mean with arbitrary bootstrap sample size. Annales de l'Institut Henri Poincaré, Probabilités et Statistiques, 25(4):457–481.

2. Athreya, K. B. (1987). Bootstrap of the mean in the infinite variance case. The Annals of Statistics, 15(2):724–731.

3. Cavaliere, G., Georgiev, I., and Taylor, A. M. R. (2013). Wild bootstrap of the sample mean in the infinite variance case. Econometric Reviews, 32(2):204–219.

4. DiCiccio, T. J. and Efron, B. (1996). Bootstrap confidence intervals. Statistical Science, 11(3):189–228.

5. Efron, B. and Tibshirani, R. J. (1993). An Introduction to the Bootstrap. Chapman & Hall, New York.

6. Hall, P. (1992). The Bootstrap and Edgeworth Expansion. Springer, New York.

7. Hill, B. M. (1975). A simple general approach to inference about the tail of a distribution. The Annals of Statistics, 3(5):1163–1174.

8. Ibragimov, R. and Müller, U. K. (2010). t-Statistic based correlation and heterogeneity robust inference. Journal of Business & Economic Statistics, 28(4):453–468.

9. Knight, K. (1989). On the bootstrap of the sample mean in the infinite variance case. The Annals of Statistics, 17(3):1168–1175.

10. Politis, D. N. (2013). Model-free model-fitting and predictive distributions. Test, 22(2):183–221.