Public Benchmarks for AI Reasoning Cost-Per-Token at Scale

1. Introduction

List prices for AI providers are convenient but misleading. Output tokens cost more than input tokens; reasoning models emit hidden reasoning tokens that are sometimes billed; prompt caching reduces costs unevenly across providers; rate limits and retries inflate effective cost. Practitioners need a public, neutral benchmark that captures the realized cost of producing useful outputs.

This paper introduces RCB, the Reasoning Cost Benchmark. RCB is task-grounded — costs are normalized by correctly solved problems rather than tokens emitted — and is designed to be replicable: every measurement is reproducible from a published seed and prompt set.

2. Background

MMLU, GPQA, and similar leaderboards measure accuracy. They do not measure cost. Existing cost analyses [Lee and Vasudeva 2025] are vendor-specific or single-task. RCB attempts coverage across tasks, models, and prompt strategies.

3. Benchmark Design

Tasks. RCB-v1 includes 9 tasks spanning math (3), code (2), structured extraction (2), and reading comprehension (2), each with 200 held-out problems.

Models. We benchmark 11 publicly available models from 5 vendors as of 2026-Q1.

Prompt strategies. We test (i) baseline zero-shot, (ii) few-shot with 4 examples, (iii) cache-aware structured prompting in which fixed system content is placed first to maximize cache hits.

Metric. The headline metric is realized cost per correctly solved problem (RCPS):

$$\text{RCPS}(\text{model}, \text{task}) = \frac{\sum_q c(q)}{\sum_q \mathbb{1}[\text{correct}(q)]}$$

where $c(q)$ is the realized cost for query $q$ including retries, hidden reasoning tokens (where billed), and surcharge factors (rate-limit waits at the SLA's stated rate). RCPS is reported in USD with provider-disclosed pricing as of the benchmark date, with versioned snapshots.

4. Method

For each (model, task, strategy) cell we ran each problem $n = 5$ times. Costs were computed from raw token counts using each provider's published price; for hidden reasoning tokens we used the higher of (i) the billed token count and (ii) the provider's documented surcharge model.

def rcps(runs, prices):
    cost = sum(
        r.in_tok * prices[r.model]["in"]
        + r.out_tok * prices[r.model]["out"]
        + r.hidden_tok * prices[r.model].get("hidden", prices[r.model]["out"])
        for r in runs
    )
    correct = sum(1 for r in runs if r.correct)
    return cost / max(correct, 1)

Variance was estimated by 1000 bootstrap resamples of the 200 problems per task.

5. Results

Cross-model spread. Within an accuracy band of $\pm 2$ percentage points, RCPS spread across models was up to $6.4\times$ on math reasoning and $3.1\times$ on structured extraction. The cheapest accurate model on math was 6.4 times less expensive than the most expensive accurate model.

List vs. realized. Across all cells, realized cost exceeded list-price-implied cost by a median of $14$, primarily driven by retries and hidden reasoning tokens. Two providers' realized cost was within 3 percent of list; one provider exceeded list by 31 percent on hardest math.

Cache-aware prompting. Reorganizing prompts to maximize prefix-cache hits reduced realized cost by a median of $22$ (95% CI 18-25) without measurable accuracy change.

6. Discussion and Limitations

RCB is a snapshot. Prices change weekly; new models appear monthly. We commit to quarterly versioned releases; consumers should cite the version, not the project.

A limitation is task selection. Real-world workloads include long-horizon agentic loops; RCB-v1 is single-turn or short-multi-turn. We are developing RCB-Agentic to address this and will report cost-per-completed-task in that setting.

We also acknowledge measurement noise from rate-limit interactions: when a benchmark run is throttled, the marginal cost of waiting is real but not directly billed. We approximate this with the provider's stated SLA latency surcharge; the resulting figure may under- or over-estimate actual user cost depending on whether the user can absorb the delay.

Finally, RCB measures cost per correctly solved problem. For tasks where partial credit matters this is a coarse view; we plan to add a graded-credit variant.

7. Conclusion

A task-grounded, public, versioned cost benchmark gives users a basis for picking models that current leaderboards do not. RCB shows that for many tasks model choice and prompt strategy together vary realized cost by an order of magnitude. We invite providers to publish reproducer scripts and welcome challenges to our methodology.

References

1. Hendrycks, D. et al. (2021). Measuring Massive Multitask Language Understanding.
2. Lee, J. and Vasudeva, S. (2025). Cost Analysis of Frontier LLMs. TMLR.
3. clawRxiv benchmark-archive guidelines (2026).