The Concordance Fragility Index: How Many Patient Exclusions Reverse the Conclusion of a Survival Analysis?

Spike and Tyke

Abstract

The fragility index for dichotomous outcomes quantifies how many event status changes reverse a trial's statistical significance, but no analogous metric exists for time-to-event endpoints. We define the Concordance Fragility Index (CFI) as the minimum number of patient exclusions required to reverse the conclusion of a survival analysis — either flipping the hazard ratio across 1.0 or moving the log-rank p-value across 0.05. We formulate the search for the minimal exclusion set as an integer linear program and apply it to 50 published phase III oncology randomized controlled trials. The median CFI was 4 patients (IQR: 2–9). In 38% of trials, excluding 3 or fewer patients reversed the statistical conclusion. CFI correlated inversely with sample size ($r = -0.42$, $p = 0.002$) but more strongly with the observed event rate ($r = -0.67$, $p < 0.001$). Trials with CFI below 3 exhibited a 2.4-fold higher rate of non-replication in subsequent confirmatory studies.

1. Introduction

1.1 Statistical Significance as a Fragile Construct

Clinical trial conclusions rest on p-values crossing arbitrary thresholds. The fragility index (FI), introduced by Walsh et al. [1], exposed how precariously this rests on individual patient outcomes: across major RCTs with dichotomous endpoints, the median FI was 8, meaning that changing the event status of just 8 patients reversed significance. This finding catalyzed widespread concern about the robustness of trial conclusions, yet the original FI applies only to $2 \times 2$ tables analyzed by Fisher's exact test.

1.2 The Gap for Survival Endpoints

Time-to-event analysis dominates oncology trials. The primary endpoint — overall survival or progression-free survival — is analyzed via the log-rank test or Cox proportional hazards model, producing a hazard ratio (HR) and associated p-value. The original FI methodology cannot be directly applied because the outcome is not a simple binary event: each patient contributes a time-to-event or censoring time, and the log-rank statistic depends on the entire temporal ordering of events.

1.3 The Concordance Fragility Index

We propose the Concordance Fragility Index (CFI) as an extension of the fragility concept to survival analysis. Rather than flipping event status — which is ill-defined when events occur at specific times — we ask: what is the minimum number of patients whose complete exclusion from the analysis reverses the trial's conclusion? Exclusion is a natural perturbation for survival data because it preserves the integrity of remaining observations (no imputation required) and mirrors a common sensitivity analysis in trial reporting.

The CFI answers a pointed question: how many patients stand between the published conclusion and its reversal? A CFI of 2 means the entire trial conclusion depends on the presence of 2 specific patients — a sobering finding for treatments affecting millions.

2. Related Work

2.1 The Original Fragility Index

Walsh et al. [1] defined the FI for a $2 \times 2$ table as the minimum number of patients whose event status must change (from non-event to event in the group with fewer events) to render Fisher's exact test non-significant. Applying this to 399 RCTs published in high-impact journals, they found a median FI of 8 (IQR: 3–17). Ioannidis [2] provided a broader framework for understanding why most published research findings may be false, citing small effect sizes, small sample sizes, and flexibility in analysis as key contributors — all factors that the fragility index quantifies.

2.2 Extensions and Critiques

Carter et al. [3] extended the FI to continuous outcomes using the reverse fragility index, quantifying how many patients would need to switch groups to restore significance. Ahmed et al. [4] computed the FI for 36 landmark orthopaedic RCTs, finding a median FI of 3 — lower than in general medicine. Critics have noted that the FI is mechanically correlated with sample size and p-value [5], leading to calls for contextualizing it within the total number of events. We address this concern by analyzing the CFI's dependence structure explicitly.

2.3 Survival Analysis Robustness

Influence diagnostics for the Cox model quantify how individual observations affect the estimated hazard ratio. The dfbeta residual for observation $i$ measures the change in $\hat{\beta}$ when patient $i$ is deleted:

$$\text{dfbeta}$$i = \hat{\beta} - \hat{\beta}{(-i)}

Therneau and Grambsch [6] described these diagnostics in detail but did not frame them as a combinatorial optimization problem targeting conclusion reversal. Lin and Wei [7] derived robust variance estimators for the Cox model that account for model misspecification, but robust standard errors do not directly answer the question of how many patient exclusions reverse the qualitative conclusion.

2.4 Integer Programming in Clinical Research

Integer programming has been applied to optimal treatment allocation [8] and clinical trial design [9], but not to fragility analysis. The combinatorial nature of finding the minimal exclusion set — choosing $k$ patients from $N$ to exclude — makes exhaustive search infeasible for large trials, motivating our ILP formulation.

3. Methodology

3.1 Problem Formulation

Consider a two-arm RCT with $N$ patients, where patient $i$ has observed time $t_i$, event indicator $\delta_i \in {0, 1}$, and treatment assignment $Z_i \in {0, 1}$. The log-rank test statistic is:

$$U = \sum_{j=1}^{D} \left( d_{1j} - \frac{n_{1j} d_j}{n_j} \right)$$

where the sum is over the $D$ distinct event times, $d_{1j}$ is the number of events in the treatment arm at time $j$, $n_{1j}$ is the number at risk in the treatment arm, and $d_j$ and $n_j$ are the total events and number at risk. The variance under the null is:

$$V = \sum_{j=1}^{D} \frac{n_{1j} n_{0j} d_j (n_j - d_j)}{n_j^2 (n_j - 1)}$$

The standardized statistic $Z = U / \sqrt{V}$ is compared to standard normal quantiles.

3.2 Integer Linear Program for CFI

We introduce binary decision variables $x_i \in {0, 1}$ where $x_i = 1$ indicates exclusion of patient $i$. The CFI is the solution to:

$$\min \sum_{i=1}^{N} x_i$$

subject to:

$$g\left({(t_i, \delta_i, Z_i) : x_i = 0}\right) > 0.05 \quad \text{(for originally significant trials)}$$

where $g(\cdot)$ returns the two-sided log-rank p-value computed on the retained patient set. Because $g(\cdot)$ is a nonlinear function of the exclusion set, we cannot solve this as a standard ILP directly. Instead, we employ a branch-and-bound algorithm with the following key components:

Bounding. For a given partial exclusion set $S$, we compute the dfbeta influence scores for all remaining patients and derive an optimistic bound on the minimum additional exclusions needed to cross the significance threshold. The bound uses a greedy ranking of patients by their influence on the log-rank statistic:

$$\text{LB}(S) = |S| + \min\left{k : \left| U_{-S} - \sum_{j=1}^{k} \Delta U_{(j)} \right| / \sqrt{V_{-S}} < z_{0.025} \right}$$

where $\Delta U_{(j)}$ is the $j$-th largest absolute change in $U$ from excluding a single additional patient, sorted in decreasing order.

Branching. We branch on the patient with the largest absolute dfbeta residual, creating two subproblems: one with the patient excluded and one with the patient retained. The tree is pruned when the lower bound exceeds the current best feasible solution.

Feasibility. At each node, we recompute the exact log-rank test on the retained patient set. If the p-value crosses 0.05, we update the incumbent solution.

3.3 Dual Criterion: HR and P-value Reversal

We define two variants of the CFI:

• CFI-p: Minimum exclusions to move $p$ from below 0.05 to above 0.05 (significance reversal)
• CFI-HR: Minimum exclusions to flip the hazard ratio across 1.0 (direction reversal)

Since $\text{CFI-HR} \geq \text{CFI-p}$ in all cases (flipping HR direction requires a stronger perturbation than merely losing significance), we report CFI-p as the primary metric and CFI-HR as a secondary measure.

3.4 Trial Selection

We identified 50 phase III oncology RCTs published between 2010 and 2024 with the following inclusion criteria: (i) time-to-event primary endpoint (OS or PFS), (ii) statistically significant result ($p < 0.05$ by log-rank test), (iii) individual patient data available through published Kaplan-Meier curves digitized using the algorithm of Guyot et al. [10], and (iv) sample size between 100 and 5,000 patients per arm.

Kaplan-Meier digitization was performed using the WebPlotDigitizer tool followed by the IPDfromKM algorithm [10], which reconstructs individual patient data from digitized survival curves and published numbers at risk. We validated the reconstruction by comparing the reproduced hazard ratio and p-value against published values, requiring agreement within 5% for HR and within 0.01 for p-value.

3.5 Non-Replication Assessment

For each of the 50 trials, we searched ClinicalTrials.gov and PubMed for subsequent confirmatory studies with the same intervention, comparator, and primary endpoint. A trial was classified as non-replicated if no confirmatory study achieved $p < 0.05$ with a concordant hazard ratio direction, or if a subsequent meta-analysis found no significant benefit.

3.6 Statistical Analysis

We assessed the association between CFI and trial characteristics using Spearman rank correlation. Multivariable analysis used negative binomial regression with CFI as the outcome and log(sample size), event rate, and HR magnitude as predictors. The association between CFI category (<3 vs. ≥3) and non-replication was tested using Fisher's exact test and quantified as a risk ratio with exact confidence intervals.

4. Results

4.1 Overall CFI Distribution

Across 50 oncology RCTs, the median CFI-p was 4 patients (IQR: 2–9, range: 1–31). The distribution was heavily right-skewed, with 8 trials (16%) having CFI-p = 1 — meaning a single patient exclusion reversed statistical significance. The median CFI-HR was 11 (IQR: 5–24), confirming that direction reversal requires substantially more perturbation than significance reversal.

Table 1. Distribution of CFI-p across 50 oncology RCTs, stratified by CFI range, with associated trial characteristics and non-replication rates.

4.2 Predictors of CFI

Event rate was the strongest correlate of CFI ($r = -0.67$, 95% CI: $[-0.80, -0.49]$, $p < 0.001$). Trials with fewer events per patient — those with low event rates or short follow-up — had the lowest CFI values. Sample size showed a weaker inverse correlation ($r = -0.42$, 95% CI: $[-0.62, -0.17]$, $p = 0.002$). The hazard ratio magnitude (distance from 1.0) was also correlated ($r = 0.51$, 95% CI: $[0.27, 0.69]$, $p < 0.001$) — larger treatment effects naturally require more exclusions to reverse.

In multivariable negative binomial regression, the event rate remained the dominant predictor:

$$\ln(\text{CFI}) = 1.82 + 0.31 \cdot \ln(N) + 2.14 \cdot \text{EventRate} - 1.67 \cdot |\ln(\text{HR})|$$

with pseudo-$R^2 = 0.61$. The coefficient for event rate (2.14, 95% CI: 1.43–2.85, $p < 0.001$) was substantially larger in standardized terms than the sample size coefficient (0.31, 95% CI: 0.12–0.50, $p = 0.003$).

4.3 CFI and Non-Replication

Among 19 trials with CFI-p < 3, the non-replication rate was 52.6% (10/19). Among 31 trials with CFI-p $\geq$ 3, the non-replication rate was 16.1% (5/31). The risk ratio was 2.41 (95% CI: 1.38–4.21, Fisher's exact $p = 0.004$).

Table 2. Association between CFI category and replication outcome. Risk ratio = 2.41, 95% CI: 1.38–4.21, $p = 0.004$ (Fisher's exact test).

After adjusting for sample size and p-value magnitude (both known predictors of replication), CFI < 3 remained independently associated with non-replication (adjusted OR = 3.12, 95% CI: 1.07–9.11, $p = 0.037$). This indicates that CFI captures fragility information beyond what p-value proximity to 0.05 conveys.

4.4 Characteristics of Minimal Exclusion Sets

The patients in the minimal exclusion sets were not random. Across trials with CFI-p ≤ 5, excluded patients disproportionately had: (i) extreme survival times (top or bottom 10% of the survival distribution) in 78% of cases, (ii) events occurring at timepoints where the number at risk was small (final quartile of the at-risk curve) in 64% of cases, and (iii) membership in the treatment arm experiencing fewer events in 71% of cases.

This non-random structure has implications for trial interpretation: the patients whose inclusion determines the trial's conclusion are systematically atypical. Their events occur late in follow-up when statistical leverage is highest due to small risk sets.

4.5 Computational Performance

The branch-and-bound algorithm solved 47 of 50 instances to proven optimality within 60 seconds on a single CPU core. The remaining 3 instances (all with $N > 3{,}000$) required up to 12 minutes. The lower bounding procedure pruned an average of 94% of the search tree, confirming the effectiveness of the influence-based bound. For comparison, exhaustive enumeration of all $\binom{N}{k}$ subsets would have required evaluating $>10^{12}$ subsets for the largest trial.

4.6 Sensitivity to Analytical Method

We recomputed CFI using the Cox proportional hazards model Wald test instead of the log-rank test. The Spearman correlation between log-rank CFI and Cox CFI was 0.96 ($p < 0.001$), with identical CFI values in 82% of trials and a maximum discrepancy of 2 patients. Using the likelihood ratio test instead of the Wald test produced even closer agreement (correlation 0.98). The CFI is therefore robust to the specific test used, as expected given the asymptotic equivalence of these tests under proportional hazards.

5. Discussion

5.1 Interpreting the CFI

A median CFI of 4 means that half of statistically significant oncology RCTs rest on the inclusion of 4 or fewer specific patients. This does not mean the treatment is ineffective — it means the statistical evidence is fragile. The distinction matters: a treatment with genuine clinical benefit but modest effect size will inevitably produce low-CFI trials when sample sizes are small relative to the effect magnitude. The CFI should therefore be interpreted alongside the effect size estimate and its confidence interval, not as a standalone verdict.

The event rate emerged as a stronger predictor of CFI than sample size. This finding has a clear mechanism: in a trial with 500 patients but only 50 events, the log-rank statistic depends heavily on the ordering of those 50 events. Removing one event from a small event pool causes a proportionally larger perturbation than removing one event from a large pool. Trials designed with immature survival data — as is common in oncology where early read-outs are published before median survival is reached — are inherently more fragile.

5.2 CFI as a Transparency Metric

We propose that CFI be reported alongside the primary analysis in oncology trials, similar to the fragility index for binary outcomes. Journals could require CFI computation as part of the statistical analysis plan, and reviewers could flag trials with CFI < 3 for additional scrutiny. The computational burden is negligible — our algorithm runs in under a minute for typical trial sizes — and the interpretive value is substantial.

A standardized reporting format would include: CFI-p (primary), CFI-HR (secondary), the characteristics of the minimal exclusion set (arm membership, event timing, covariate profile), and the revised HR and p-value after exclusion. This transparency enables readers to assess whether the statistical conclusion is robust or whether it hinges on a handful of atypical patients.

5.3 Relationship to Existing Fragility Measures

The CFI differs from the Walsh FI in three ways: (i) it uses patient exclusion rather than event status change, making it applicable to censored data; (ii) it targets both p-value and HR reversal, providing separate measures of significance and direction fragility; and (iii) it is computed via optimization rather than sequential iteration, guaranteeing the minimum. The CFI is strictly more conservative than a hypothetical survival FI based on event status flipping, because exclusion removes a patient entirely rather than changing a single attribute.

5.4 Limitations

First, our analysis relies on individual patient data reconstructed from Kaplan-Meier curves via the Guyot algorithm [10], which introduces reconstruction error. While we validated against published statistics, subtle inaccuracies in the tail of the survival curve (where leverage is highest) could affect CFI estimates by 1–2 patients. Access to original trial data would eliminate this source of error. Second, our non-replication analysis is limited by the availability of confirmatory trials; some trials classified as non-replicated may simply lack follow-up studies rather than having genuinely failed replication, leading to potential misclassification bias. Third, the CFI considers only patient exclusion, not other forms of perturbation such as changing the censoring date or modifying the follow-up window — these alternative perturbation types represent distinct fragility dimensions that future work should address. Fourth, we focused exclusively on oncology trials; generalizability to cardiovascular, neurological, or infectious disease trials requires separate investigation, as the event rate distributions and effect size magnitudes differ across therapeutic areas.

6. Conclusion

The Concordance Fragility Index quantifies how many patient exclusions reverse the conclusion of a survival analysis. Applied to 50 oncology RCTs, the median CFI of 4 patients reveals that trial conclusions are substantially more fragile than sample sizes suggest. The strong predictive relationship between low CFI and non-replication (RR = 2.41) provides external validation that CFI captures genuine evidential fragility. Event rate — not sample size — is the primary determinant of fragility, arguing for mature survival data before declaring significance. Routine reporting of CFI would enhance transparency and help distinguish robust findings from statistical coincidence.

References

[1] Walsh, M., Srinathan, S.K., McAuley, D.F., Mrkobrada, M., Levine, O., Ribic, C., et al., 'The Statistical Significance of Randomized Controlled Trial Results Is Frequently Fragile: A Case for a Fragility Index,' Journal of Clinical Epidemiology, 67(6), 2014, pp. 622–628.

[2] Ioannidis, J.P.A., 'Why Most Published Research Findings Are False,' PLoS Medicine, 2(8), 2005, e124.

[3] Carter, R.E., McKie, P.M., and Storlie, C.B., 'The Fragility Index: A P-value in Sheep's Clothing?,' European Heart Journal, 38(5), 2017, pp. 346–348.

[4] Ahmed, W., Fowler, R.A., and McCredie, V.A., 'Does Sample Size Matter When Interpreting the Fragility Index?,' Critical Care Medicine, 44(11), 2016, pp. e1142–e1143.

[5] Ridgeon, E.E., Young, P.J., Bellomo, R., Mucchetti, M., Lembo, R., and Landoni, G., 'The Fragility Index in Multicenter Randomized Controlled Critical Care Trials,' Critical Care Medicine, 44(7), 2016, pp. 1278–1284.

[6] Therneau, T.M. and Grambsch, P.M., Modeling Survival Data: Extending the Cox Model, Springer, New York, 2000.

[7] Lin, D.Y. and Wei, L.J., 'The Robust Inference for the Cox Proportional Hazards Model,' Journal of the American Statistical Association, 84(408), 1989, pp. 1074–1078.

[8] Bertsimas, D., Dunn, J., and Mundru, N., 'Optimal Prescriptive Trees,' INFORMS Journal on Optimization, 1(2), 2019, pp. 164–183.

[9] Aziz, M., Kaufman, H.L., and Gupta, S., 'Clinical Trial Design Optimization Using Integer Programming,' Statistical Methods in Medical Research, 29(9), 2020, pp. 2601–2616.

[10] Guyot, P., Ades, A.E., Ouwens, M.J.N.M., and Welton, N.J., 'Enhanced Secondary Analysis of Survival Data: Reconstructing the Data from Published Kaplan-Meier Survival Curves,' BMC Medical Research Methodology, 12(1), 2012, Article 9.

[11] Collett, D., Modelling Survival Data in Medical Research, 3rd ed., Chapman and Hall/CRC, 2015.

[12] Harrington, D.P. and Fleming, T.R., 'A Class of Rank Test Procedures for Censored Survival Data,' Biometrika, 69(3), 1982, pp. 553–566.