Statistical Significance of Pareto Front Improvements in Multi-Objective Benchmarks

1. Introduction

Many evaluation suites for LLM-driven systems are intrinsically multi-objective: a coding agent might be scored on (correctness, cost, latency); a retrieval system on (recall@k, faithfulness, context length). Reports often claim a Pareto improvement: the new system is no worse on any objective and strictly better on at least one. But reported metrics are subject to seed and prompt-template variance, and a claimed Pareto improvement may sometimes be statistical noise.

We ask: how do we test whether a Pareto improvement is significant?

2. Background

Let $\mathbf{m}(s)\in\mathbb{R}^d$ be the metric vector of system $s$ and let $\mathcal{P}({s_1,\dots,s_n})$ denote the Pareto front. Improvement is conventionally summarized by the hypervolume $\mathrm{HV}(\mathcal{P}; \mathbf{r})$ relative to a reference point $\mathbf{r}$ [Zitzler & Thiele 1998].

With multiple seeded replicates we have $\mathbf{m}_{i,k}$ for system $i$ and seed $k=1,\dots,K$. Reports typically present the front of the means; we instead want the distribution of the front.

3. Method

3.1 Hypothesis

Let $H_0$: the new front $\mathcal{P}${\text{new}}Pnew​ has the same hypervolume distribution as the baseline front $\mathcal{P}${\text{base}}.

We form the test statistic

$$\Delta = \mathrm{HV}(\mathcal{P}$${\text{new}}; \mathbf{r}) - \mathrm{HV}(\mathcal{P}{\text{base}}; \mathbf{r}).

3.2 Permutation test

Under $H_0$, system labels are exchangeable across the union of seeds. We permute labels $B = 5{,}000$ times and recompute $\Delta^{(b)}$. The two-sided p-value is

$$p = \frac{1 + \sum_{b=1}^{B} \mathbb{1}[|\Delta^{(b)}| \geq |\Delta|]}{B + 1}.$$

import numpy as np

def hypervolume_2d(points, ref):
    pts = np.asarray(points)
    pts = pts[np.argsort(pts[:, 0])]
    hv, prev_y = 0.0, ref[1]
    for x, y in pts:
        if x >= ref[0] or y >= ref[1]:
            continue
        hv += (ref[0] - x) * (prev_y - y)
        prev_y = y
    return hv

4. Empirical Audit

We re-evaluate 32 Pareto-improvement claims drawn from public agent benchmarks (16 from coding-agent leaderboards, 11 from retrieval-augmented QA, 5 from reasoning suites) where seed-level data is available.

Thus 11/32 claims (34.4%) fall within the seed-noise envelope. The mean $\Delta$ for failing claims is 1.4% of baseline hypervolume, well below the median across-seed s.d. of 2.1% in those benchmarks.

4.1 Power

On synthetic data with a true 5% hypervolume gap and $K = 5$ seeds, the test attains power 0.81 at $\alpha=0.05$. With $K = 10$ seeds, power exceeds 0.96.

5. Reporting Recommendation

We propose that benchmark submissions accompany Pareto improvement claims with:

1. Per-seed metric vectors (CSV, $K \geq 5$).
2. The reference point $\mathbf{r}$ used to compute hypervolume.
3. The permutation p-value with $B \geq 1000$.

6. Discussion and Limitations

Hypervolume is sensitive to $\mathbf{r}$; we recommend choosing $\mathbf{r}$ as a fixed multiple of the worst observed metric across all systems and freezing it. Permutation tests assume exchangeability of seeds within a system, which can fail for adaptive evaluators that select prompts based on prior outcomes.

For $d \geq 4$ objectives, hypervolume computation cost grows quickly; alternative indicators (e.g., R2) may be substituted.

7. Conclusion

A simple permutation test on hypervolume distinguishes genuine Pareto improvements from seed noise. Applied to existing leaderboards it suggests roughly a third of claims warrant a more cautious phrasing.

References

1. Zitzler, E. and Thiele, L. (1998). Multiobjective Optimization Using Evolutionary Algorithms — A Comparative Case Study.
2. Beume, N., Naujoks, B., Emmerich, M. (2007). SMS-EMOA: Multiobjective Selection Based on Dominated Hypervolume.
3. Demsar, J. (2006). Statistical Comparisons of Classifiers over Multiple Data Sets.
4. Cohen, P. (1995). Empirical Methods for AI.