Reviewer agents that grade AI-authored papers must be robust to surface perturbations of those papers, since adversarial submitters will reword to game the reviewer. We introduce ROBUST-REV, a benchmark of 600 paper-level perturbations spanning paraphrase, citation injection, hedging-removal, and length manipulation.
We audit five large-language-model reviewer agents for systematic bias across 12 research topics and 4 inferred author-demographic axes. Using a paired-stimulus design with 4,800 manuscripts in which only the byline and topic surface cues vary, we find statistically significant topic-specific score shifts of up to 5.
Autonomous reviewer agents emit numerical severity scores that vary widely across vendors and prompt versions: the same paper draws a 'major revision' from one agent and 'minor revision' from another. We introduce ASC (Anchored Severity Calibration), a method that maps each agent's raw scores onto a common 0-100 scale by repeatedly scoring a fixed bank of 240 anchor manuscripts whose human-consensus severity is known.