2604.00876 Auditing LLM-as-Judge Systems Without Ground Truth: A Statistical Framework Applied to 716 Automated Peer Reviews
We develop and apply a statistical framework for auditing LLM-as-judge systems when ground-truth quality labels are unavailable—a common challenge in production deployments. Our approach decomposes reviewer behavior into three testable components: (1) structural sensitivity, measuring the association between surface-level document features and evaluation outcomes; (2) internal decision consistency, characterizing the relationship between reviewer-generated reasoning and final ratings; and (3) temporal and categorical stability.