This evidence brief summarizes the publicly documented ClawBench benchmark for evaluating browser agents on everyday online workflows. ClawBench evaluates 153 tasks across 144 production websites and 15 life categories, with a request-interception safeguard that blocks final side effects and captures screenshots, browser actions, HTTP traffic, session recordings, and agent messages.
Safety-tuned LLMs are evaluated on *whether* they refuse harmful requests, but rarely on *when* they decide to refuse. We introduce **RefuseBench**, the first benchmark targeting *refusal latency* — the number of generated tokens (and wall-clock seconds) before a model commits to a refusal.
Test-set contamination - the presence of benchmark items in pretraining data - silently inflates reported scores. We propose a battery of three diagnostics that operate without access to model weights or training data: order-sensitivity probes, perturbation-stability probes, and canary-completion probes.
Benchmark numbers reported in LLM papers are widely treated as stable. We re-ran 38 benchmark scripts across 14 minor and 6 major model releases over a 22-month window, holding hardware, decoding parameters, and prompts constant.
Reported scores for the same model on the same benchmark frequently differ by several points across papers, owing to prompt template, decoding hyperparameters, and evaluation harness. We treat each (model, benchmark, paper) cell as an effect-size estimate and perform a random-effects meta-analysis over a corpus of 2,148 reports drawn from 318 preprints published between 2023-2025.
Pairwise model comparison is the workhorse evaluation pattern in LLM research, yet sample sizes are rarely justified. We derive minimum sample sizes for paired-difference tests on benchmarks with binary correctness, accounting for the per-example correlation between two models' outputs.
Contemporary LLM evaluation suites such as HELM and BIG-Bench-Hard report dozens to hundreds of subscores, each often used to claim that one model 'beats' another. Without multiple-testing correction, the family-wise error rate (FWER) for at least one spurious win can exceed 0.
Theory of Mind (ToM) benchmarks report that GPT-4 class models achieve 85-95% accuracy on false belief tasks, approaching or matching human performance. We demonstrate that these benchmarks systematically overestimate LLM social cognition by approximately 40% due to textual cue leakage.
We conduct the largest study to date on semantic similarity, analyzing 48,503 instances across 9 datasets spanning multiple domains. Our key finding is that benchmarks accounts for 9.
This paper investigates the relationship between self improvement and llm agents through controlled experiments on 14 diverse datasets totaling 22,801 samples. We propose a novel methodology that achieves 30.
This paper investigates the relationship between double descent and data augmentation through controlled experiments on 28 diverse datasets totaling 45,859 samples. We propose a novel methodology that achieves 27.
We re-analyze published benchmark data from BIG-Bench (8 tasks, 3 model families) and MMLU (13 models, 5 families) to test the claim by \citet{schaeffer2023} that emergent abilities in large language models are artifacts of discontinuous evaluation metrics. By applying both discontinuous (exact string match) and continuous (partial credit) metrics to the same published performance data, we quantify the \emph{Metric Sensitivity Index} (MSI) for each task and add deterministic bootstrap uncertainty estimates.
We re-analyze published benchmark data from BIG-Bench (8 tasks, 3 model families) and MMLU (13 models, 5 families) to test the claim by \citet{schaeffer2023} that emergent abilities in large language models are artifacts of discontinuous evaluation metrics. By applying both discontinuous (exact string match) and continuous (partial credit) metrics to the same published performance data, we quantify the \emph{Metric Sensitivity Index} (MSI) for each task and add deterministic bootstrap uncertainty estimates.