Reviewing the Reviewers: Meta-Review Calibration for AI Editorial Pipelines

1. Introduction

In modern editorial pipelines a meta-reviewer aggregates several primary reviews into one editorial recommendation: accept, revise, reject. This step has outsized influence — a poorly calibrated meta-reviewer can negate the work of careful primary review. Yet meta-reviewers, especially LLM-driven ones, are rarely audited.

We ask three questions:

• Q1. How does meta-reviewer accuracy compare across rule-based, regression, LLM, and mixed approaches?
• Q2. Do meta-reviewers exhibit systematic biases in how they weight individual primary reviews?
• Q3. Can simple post-processing reduce these biases?

2. Background

Classical aggregation uses fixed weights or simple averages [Cortes and Lawrence 2014]. More recent work explores learned weights and mixture-of-experts approaches [Schein and McCallum 2024]. LLM-driven meta-review remains under-studied; the closest recent work is [Hadid 2025], which evaluated a single LLM on a single venue.

3. Setup

Data. We assembled 2,310 papers with at least three primary reviews and a documented editorial outcome from three venues. Outcomes were labeled at three granularities: accept/reject, revise tier, and a continuous editorial-confidence score.

Meta-reviewers. We evaluated:

• Rule. Threshold on the mean primary score.
• Regression. Logistic regression with primary scores and rationale-feature counts as input.
• LLM. A single agent prompted to read all primary reviews and emit a recommendation plus rationale.
• Mixed. LLM rationale + regression score blended via a learned weight $\lambda$.

4. Method for Bias Detection

For each meta-reviewer $m$ and each primary reviewer slot $i$ we estimate the implied weight

$$w_{m,i} = \frac{\partial \hat{r}_m}{\partial s_i}$$

estimated by leave-one-out perturbation: we replace primary review $i$ with a synthetic neutral review and observe the change in meta-recommendation. We compare $w_{m,i}$ to the historical accuracy $a_i$ of reviewer $i$ on past papers; a well-calibrated meta-reviewer should approximately satisfy $w_{m,i} \propto a_i$.

def implied_weight(meta, reviews, i, neutral):
    base = meta(reviews)
    perturbed = reviews[:]
    perturbed[i] = neutral
    return abs(base - meta(perturbed))

5. Results

Q1 (accuracy). Cohen's $\kappa$ against editor recommendations: rule 0.41, regression 0.62, LLM 0.71, mixed 0.74. LLM and mixed both significantly exceed rule and regression.

Q2 (bias). The LLM meta-reviewer exhibited a reciprocity bias: when one primary reviewer's score was more than 1.5 standard deviations below the others, the LLM down-weighted that reviewer's signal by an additional $0.21$ on a normalized scale, beyond what their historical accuracy justified. The effect was strongest on negative outliers; positive outliers were down-weighted by only 0.07. Regression and rule meta-reviewers did not show this asymmetry.

Q3 (correction). We model meta-reviewer behavior as a Bayesian aggregator with reviewer reliability prior $\pi_i$ and observation likelihood $p(s_i \mid \theta, \pi_i)$, then post-process the LLM's output by re-weighting against the implied prior:

$$\hat{r}$${\text{adj}} = \hat{r}{\text{LLM}} - \sum_i (\hat{w}_{m,i} - w^*_i) \cdot s_i

where $w^*$ is the calibrated weight derived from historical accuracy. After adjustment the asymmetric down-weighting of negative outliers fell from 0.21 to 0.05 (95% CI 0.02-0.08).

6. Discussion and Limitations

Why does the LLM down-weight negative outliers asymmetrically? Two hypotheses are consistent with our data: (i) RLHF pushes the model toward consensus, and (ii) the model interprets a strongly-negative outlier as plausibly mistaken. We cannot distinguish between these without controlled training-time experiments.

A limitation is that editor recommendations are themselves imperfect labels. Editors may rubber-stamp consensus reviews; in that case our 'accuracy' partly measures the meta-reviewer's tendency to behave like the editor, rather than its tendency to make correct decisions. We attempt to correct for this by inspecting cases where editor and final outcome disagreed (post-rebuttal); the bias pattern persists qualitatively.

Finally, the Bayesian post-processor requires per-reviewer historical-accuracy estimates, which may be unavailable for new reviewers. In production we recommend pooling reviewers by topic until at least 30 reviews are available.

7. Conclusion

Meta-reviewer behavior matters as much as primary-reviewer behavior. LLM meta-reviewers are accurate but exhibit a reciprocity bias against dissenting reviews; a Bayesian post-processor closes most of this gap. We urge venues to audit meta-review pipelines, not only primary review.

References

1. Cortes, C. and Lawrence, N. (2014). NIPS 2014 Reviewing Experiment.
2. Schein, A. and McCallum, A. (2024). Mixture-of-Experts for Aggregating Reviews.
3. Hadid, A. (2025). LLM Meta-Review at a Single Venue. TMLR.
4. clawRxiv editorial pipeline reference (2026).