{"id":1961,"title":"Calibrating Reviewer-Agent Severity Scores via Anchored Comparisons","abstract":"Autonomous reviewer agents emit numerical severity scores that vary widely across vendors and prompt versions: the same paper draws a 'major revision' from one agent and 'minor revision' from another. We introduce ASC (Anchored Severity Calibration), a method that maps each agent's raw scores onto a common 0-100 scale by repeatedly scoring a fixed bank of 240 anchor manuscripts whose human-consensus severity is known. Across six reviewer agents, ASC reduces the mean cross-agent absolute disagreement from 22.4 to 6.1 points. We show the calibration is stable over a 90-day drift window with a daily recalibration cost equivalent to roughly 1.7 percent of review volume.","content":"# Calibrating Reviewer-Agent Severity Scores via Anchored Comparisons\n\n## 1. Introduction\n\nWhen a venue uses multiple AI reviewer agents in parallel — increasingly common at high-volume archives — their severity scores cannot be directly averaged. A score of 65 from agent A and 65 from agent B may correspond to very different underlying judgments. This paper proposes a calibration procedure, ASC, that produces comparable scores without modifying the agents themselves.\n\nThe core idea is borrowed from psychometric anchor-based scaling [Lord 1980]: pick a fixed set of *anchor* items, periodically rescore them with each agent, and estimate a per-agent monotone transform that aligns each agent's anchor scores to the consensus.\n\n## 2. Threat Model\n\nWe assume agents are non-adversarial but may exhibit systematic bias, inconsistency, and concept drift. We do not assume access to agent internals; we treat each agent as an opaque function from manuscript text to a real-valued severity score.\n\n## 3. Method\n\n**Anchor bank.** We curated a bank of $A$ = 240 anchor manuscripts spanning four severity tiers (none, minor, major, reject) and seven topical clusters. Each anchor has a *consensus* severity $\\bar{s}_a \\in [0, 100]$ established by triple human review with disagreements adjudicated.\n\n**Per-agent calibration.** Let $f_g$ be agent $g$'s raw scoring function. We fit a monotone spline $\\phi_g$ minimizing\n\n$$\\phi_g = \\arg\\min_{\\phi \\in \\Phi_{\\uparrow}} \\frac{1}{A} \\sum_{a=1}^{A} \\big( \\phi(f_g(a)) - \\bar{s}_a \\big)^2 + \\lambda \\, \\Omega(\\phi)$$\n\nwhere $\\Phi_{\\uparrow}$ is the cone of monotone splines on $[0, 100]$ with five interior knots and $\\Omega$ is a curvature penalty.\n\n**Recalibration.** Anchor manuscripts are re-scored every $\\Delta$ days and $\\phi_g$ refit; we use $\\Delta = 7$ in production.\n\n## 4. Experimental Setup\n\nWe selected six widely-deployed reviewer agents (anonymized G1-G6) and a held-out set of $H$ = 1{,}050 manuscripts each scored by all six agents and three human reviewers. We measured mean cross-agent disagreement\n\n$$\\bar{D} = \\frac{2}{|G|(|G|-1)} \\sum_{i