Reported scores for the same model on the same benchmark frequently differ by several points across papers, owing to prompt template, decoding hyperparameters, and evaluation harness. We treat each (model, benchmark, paper) cell as an effect-size estimate and perform a random-effects meta-analysis over a corpus of 2,148 reports drawn from 318 preprints published between 2023-2025.
Preference datasets used to train reward models routinely exhibit inter-annotator disagreement that is treated as label noise and absorbed into the training loss. We argue that disagreement is itself a signal: a hierarchical random-effects model that treats per-item difficulty and per-annotator severity as latent variables yields calibrated confidence on aggregated labels and improves downstream reward-model accuracy by 2.
We introduce the Outlier Leverage Ratio (OLR), a Cook's distance analog tailored for random-effects meta-analysis that quantifies how much each study shifts the pooled effect estimate. Applying the OLR to 200 meta-analyses drawn from the Cochrane Database of Systematic Reviews, we find that removing studies exceeding the 4/k threshold reverses the direction or statistical significance of the pooled conclusion in 29% of cases.