Cost-Per-Solved-Problem as a Unified Inference Metric for Reasoning Agents

1. Motivation

A leaderboard row that reads "Model X: 64.2% on MATH" hides much of what a deployer actually needs to know. Was that score obtained with greedy decoding (one sample) or with majority-of-256? Did it use a verifier? At what dollar cost per problem? Two systems with identical headline accuracy can differ by 10-30x in inference compute, and recent work has shown that scaling inference compute is a substantive lever in its own right [Snell et al. 2024, Brown et al. 2024].

We propose that reasoning-agent papers report Cost-Per-Solved-Problem (CPSP) as a primary metric alongside accuracy, and we develop the formalism to make CPSP comparisons rigorous.

2. Definition

Let $\mathcal{D}$ be a benchmark with problems ${x_i}_{i=1}^N$ and ground-truth answers ${y_i^$}{yi∗​}. Let $\pi$ be an inference policy that produces an output $\hat{y}_i$ at expected cost $c_i(\pi)$ in USD. Let $a(\pi) = \frac{1}{N}\sum_i \mathbb{1}[\hat{y}_i = y_i^$] be the accuracy and $\bar{c}(\pi) = \frac{1}{N}\sum_i c_i(\pi)$ the mean cost. Then

$$\mathrm{CPSP}(\pi) = \frac{\bar{c}(\pi)}{a(\pi)}$$

is the expected dollar cost to obtain one correct solution.

This form has two attractive properties. First, it admits a meaningful comparison between systems that operate at different accuracy regimes. Second, under the cost model $c_i = \alpha n_{\mathrm{in}}(i) + \beta n_{\mathrm{out}}(i)$, CPSP decomposes additively across pipeline stages, simplifying attribution.

3. CPSP for Common Inference Policies

3.1 Greedy / Single Sample

$\mathrm{CPSP}_{\text{greedy}} = \bar{c}_1 / a_1$ where $a_1 = \Pr[\hat{y}_1 = y^*]$.

3.2 Majority Vote of $K$

Let $p$ be the per-sample correctness probability assuming an infinite candidate set with mode at the true answer. The majority-vote accuracy with $K$ samples is approximated by

$$a_{\text{maj}}(K) \approx \sum_{j=\lceil K/2 \rceil}^{K} \binom{K}{j} p^j (1-p)^{K-j}$$

and the cost scales as $\bar{c}_{\text{maj}}(K) = K \cdot \bar{c}_1$. Thus

$$\mathrm{CPSP}_{\text{maj}}(K) = \frac{K \bar{c}$$1}{a{\text{maj}}(K)}.

The minimizing $K^$K∗ depends on $p$: for $p = 0.55$ we find $K^$ = 9; for $p = 0.30$ no finite $K$ improves over the greedy CPSP.

3.3 Best-of-N with Verifier

If a verifier with sensitivity $\eta$ and specificity $\rho$ scores candidates, the accuracy becomes

$$a_{\text{bon}}(N) = 1 - \bigl(1 - \eta p\bigr)^N - \text{verifier-accept-of-wrong correction.}$$

The verifier itself contributes a per-problem cost $c_v(N) = \gamma N$.

4. Empirical Study

We evaluated 14 publicly-available reasoning systems across MATH, GPQA, ARC-AGI, BIG-Bench-Hard, and HumanEval, using vendor-published $/M-token prices from March 2026. For each system we report the accuracy-CPSP frontier swept over$K \in {1, 4, 16, 64, 256}$.

4.1 Headline Results

LM-B has 1.7 points lower accuracy than LM-A but 3.4x lower CPSP. On a leaderboard sorted by accuracy LM-B ranks 6th; on CPSP it ranks 1st.

Of the 14 systems, 6 changed rank by three or more positions when sorted by CPSP versus accuracy.

5. Implementation

def cpsp(samples_per_problem, cost_per_sample, verdicts):
    n_correct = sum(verdicts)
    n_problems = len(verdicts)
    total_cost = sum(samples_per_problem) * cost_per_sample
    if n_correct == 0:
        return float("inf")
    return total_cost / n_correct

We release a cpsp package that wraps a benchmark runner with a cost-model adaptor for major API providers and produces the accuracy-CPSP frontier as a single command.

6. Discussion

CPSP can be misleading at very low accuracies — division amplifies noise. We recommend reporting confidence bands using a delta-method approximation

$$\mathrm{Var}[\widehat{\mathrm{CPSP}}] \approx \frac{\mathrm{Var}[\bar{c}]}{a^2} + \frac{\bar{c}^2 \mathrm{Var}[a]}{a^4}.$$

Vendor pricing changes also affect rankings. We propose archiving the price table snapshot alongside reported CPSP values.

Finally, CPSP captures only the inference-time cost; training cost amortization is out of scope.

7. Limitations

We do not address latency, which is a separate axis some applications care about. Nor do we address cost variance across users — heavy-tail token distributions can make tail CPSP much higher than mean CPSP.

8. Conclusion

Reporting accuracy without cost paints an incomplete picture of the design choices that produced a leaderboard score. CPSP is one principled way to surface the trade-off, and the formalism extends naturally to multi-call agentic pipelines.

References

1. Snell, C. et al. (2024). Scaling inference compute optimally.
2. Brown, B. et al. (2024). Large language monkeys: scaling inference with repeated sampling.
3. Hendrycks, D. et al. (2021). Measuring mathematical problem solving.
4. Rein, D. et al. (2023). GPQA: A graduate-level Google-proof Q&A benchmark.
5. Chollet, F. (2019). On the measure of intelligence.