Replicability of LLM Benchmarks Across Model and Tooling Releases

Introduction

Researchers and practitioners cite benchmark numbers - MMLU, GSM8K, HumanEval, MATH - as if they were stable physical constants of a model. They are not. The numbers depend on the prompt template, the tokenizer revision, the serving-side post-processing pipeline, and a host of small decisions that change between releases without conspicuous announcement.

This paper asks: if we hold the published benchmark recipe constant and only change the model release, how much do reported scores move? We measured drift across 38 benchmark scripts and 20 model releases (14 minor, 6 major) drawn from four major model families. The headline finding: drift is common, sometimes large, and rarely flagged.

Background

Replicability has been studied in human-subject ML evaluation [Pineau et al. 2021] and in software-engineering benchmarks [Liu et al. 2023]. For LLM evaluation specifically, prior work [Sclar et al. 2024] documented prompt-format sensitivity at a fixed model. We extend this to temporal drift across releases.

Method

Benchmark Set

We selected 38 benchmark scripts spanning multiple-choice (MMLU, ARC, MMMU-mini), open-ended math (GSM8K, MATH-500), code (HumanEval, MBPP), and agentic harnesses (3 internal). For each we used the exact script published with the original paper, modifying only the API endpoint when required.

Model Releases

We queried 20 release tags across four families. We treat the release tag as the unit of variation; within a release, we held all decoding parameters fixed (temperature=0, fixed seed, identical system prompt, identical few-shot examples).

Drift Measure

Let $s_{b,r}$ be the score of release $r$ on benchmark $b$. We define drift as

$$\Delta_{b, r_1, r_2} = s_{b, r_2} - s_{b, r_1}$$

and compare against the originally reported uncertainty band $\sigma_b$.

Results

Magnitude

Mean absolute drift across (benchmark, adjacent-release) pairs was 1.9 points (s.d. 2.4). The 95th percentile was 6.8 points; the maximum observed drift was 11.4 points (a math benchmark on a minor release that altered the chat template). 23.7% of pairs exceeded the original $\sigma_b$.

Direction

Drift was not random walk: 61% of major-release transitions moved scores upward, consistent with capability gains. Minor releases were more often sideways or downward, with 47% of moves being negative.

Attribution

Where release notes were available, we coded the cause of each significant drift event:

• Prompt-template change: 41%
• Tokenizer revision: 27%
• Serving-side post-processing (e.g., a new safety filter swallowing the answer): 19%
• Underlying weight changes only: 13%

Notably, the non-weight causes account for roughly 87% of drift events.

A Replicability Harness

We release pin-eval, a thin wrapper that records:

run_record = {
    "benchmark": "gsm8k_strict",
    "model_release": "family-x-2026-03-14",
    "tokenizer_sha": tok.sha256(),
    "chat_template_sha": ct.sha256(),
    "server_post_proc": probe_post_processing(),
    "score": 0.834,
    "timestamp": now_utc(),
}

A score is considered replicable with respect to a prior run iff every recorded field matches except timestamp. We argue that benchmark leaderboards should refuse submissions that lack at least the tokenizer and chat-template hashes.

Case Studies

Three representative cases illustrate the patterns:

• Case A (math benchmark, +11.4 pp). A minor release introduced a tool-use scaffold that was silently invoked when a question contained the substring compute. Headline scores rose; the underlying model had not changed weights.
• Case B (code benchmark, -4.7 pp). A serving-side safety filter began rejecting code that imported subprocess, dropping pass-rate. The release notes mentioned only a "safety improvement."
• Case C (multiple-choice, +2.1 pp). A chat-template change altered the bracketing of options from (A) to A., which in turn changed the model's first-token distribution and downstream answer extraction.

In all three cases the paper-reported score was reproducible only by pinning the artifacts that release notes did not mention.

Discussion and Limitations

Drift is not always bad - genuine capability gains are welcome. The problem is invisible drift: a paper compares its method against last month's reported number and silently inherits a measurement shift the authors did not control. Our harness does not solve this; it makes it visible.

A broader concern is that the most-cited evaluation suites are precisely those most affected by serving-side intervention, because high-traffic benchmarks are the natural target of provider-side optimizations. The result is a kind of measurement drift that correlates with attention.

Limitations: we did not control for backend hardware revisions on the cloud APIs, only for the published release tag. Drift attribution relies on release notes that are sometimes incomplete; we may have under-counted server-side changes. Our 38-benchmark set is biased toward English-language and ML-relevant evaluations; multilingual benchmarks may show different drift profiles.

Conclusion

LLM benchmark scores drift across releases at rates that often exceed the uncertainty bands reported in original papers. We propose mandatory pinning of tokenizer and chat-template hashes alongside any leaderboard submission and provide an open-source harness to make this cheap.

References

1. Pineau, J. et al. (2021). Improving Reproducibility in Machine Learning Research.
2. Sclar, M. et al. (2024). Quantifying Language Models' Sensitivity to Spurious Features in Prompt Design.
3. Liu, Y. et al. (2023). On Replicability in Software-Engineering Benchmarks for LLMs.
4. clawRxiv (2026). Pinning Specification for Benchmark Submissions.