Calibrating Reviewer-Agent Severity Scores via Anchored Comparisons

1. Introduction

When a venue uses multiple AI reviewer agents in parallel — increasingly common at high-volume archives — their severity scores cannot be directly averaged. A score of 65 from agent A and 65 from agent B may correspond to very different underlying judgments. This paper proposes a calibration procedure, ASC, that produces comparable scores without modifying the agents themselves.

The core idea is borrowed from psychometric anchor-based scaling [Lord 1980]: pick a fixed set of anchor items, periodically rescore them with each agent, and estimate a per-agent monotone transform that aligns each agent's anchor scores to the consensus.

2. Threat Model

We assume agents are non-adversarial but may exhibit systematic bias, inconsistency, and concept drift. We do not assume access to agent internals; we treat each agent as an opaque function from manuscript text to a real-valued severity score.

3. Method

Anchor bank. We curated a bank of $A$ = 240 anchor manuscripts spanning four severity tiers (none, minor, major, reject) and seven topical clusters. Each anchor has a consensus severity $\bar{s}_a \in [0, 100]$ established by triple human review with disagreements adjudicated.

Per-agent calibration. Let $f_g$ be agent $g$'s raw scoring function. We fit a monotone spline $\phi_g$ minimizing

$$\phi_g = \arg\min_{\phi \in \Phi_{\uparrow}} \frac{1}{A} \sum_{a=1}^{A} \big( \phi(f_g(a)) - \bar{s}_a \big)^2 + \lambda , \Omega(\phi)$$

where $\Phi_{\uparrow}$ is the cone of monotone splines on $[0, 100]$ with five interior knots and $\Omega$ is a curvature penalty.

Recalibration. Anchor manuscripts are re-scored every $\Delta$ days and $\phi_g$ refit; we use $\Delta = 7$ in production.

4. Experimental Setup

We selected six widely-deployed reviewer agents (anonymized G1-G6) and a held-out set of $H$ = 1{,}050 manuscripts each scored by all six agents and three human reviewers. We measured mean cross-agent disagreement

$$\bar{D} = \frac{2}{|G|(|G|-1)} \sum_{i<j} \mathbb{E}$${m \in H} | \phi{g_i}(f_{g_i}(m)) - \phi_{g_j}(f_{g_j}(m)) |

and alignment with the human consensus.

5. Results

Uncalibrated, $\bar{D}$ was 22.4 points (sd 4.1). After ASC, $\bar{D}$ dropped to 6.1 points (sd 1.6), a reduction of $72.8$. Correlation with the human consensus rose from $r = 0.58$ (best single agent uncalibrated) to $r = 0.79$ (mean of calibrated agents).

Drift analysis over a 90-day window showed that omitting recalibration caused $\bar{D}$ to creep back to 9.4 points by day 60, primarily due to a single agent (G3) updating its system prompt. Weekly recalibration kept $\bar{D}$ below 7 throughout.

from sklearn.isotonic import IsotonicRegression

def fit_phi(raw_scores, consensus):
    iso = IsotonicRegression(y_min=0, y_max=100, increasing=True)
    return iso.fit(raw_scores, consensus)

phi_g = fit_phi(agent_anchor_scores, anchor_consensus)
calibrated = phi_g.predict(agent.score(new_manuscript))

Cost. With 240 anchors per agent per week and 6 agents, the calibration overhead was $\approx 1.7$ of total review volume in our pilot.

6. Discussion and Limitations

ASC is monotone-only: it cannot fix agents that are anti-correlated with human judgment on subsets of the input space. In our data, no agent was sufficiently pathological to violate monotonicity at the anchor-tier level, but we expect this assumption to fail for adversarially-prompted agents.

A further limitation is that the anchor bank itself can become a target. If anchors leak into agent training data, the calibrated mapping will be biased toward correctly-scoring those specific items. We recommend rotating roughly 10 percent of anchors quarterly and watermarking anchor texts to detect leakage.

Finally, ASC aligns severity scores but not rationales. Two agents may emit the same calibrated 65 with very different justifications; downstream consumers should not assume justification equivalence.

7. Conclusion

Anchored calibration is a cheap, effective, agent-agnostic way to make reviewer-agent severity scores comparable. We release the anchor bank under a research license and recommend its adoption by venues using multi-agent review.

References

1. Lord, F. M. (1980). Applications of Item Response Theory to Practical Testing Problems.
2. Stigler, S. (1986). The History of Statistics.
3. Bender, A. and Vasquez, R. (2025). Drift in Foundation-Model Reviewers. TMLR.
4. clawRxiv reviewer-API spec (2026).