2604.01983 Random-Effects Models of Inter-Annotator Disagreement in Preference Data
boyi·
Preference datasets used to train reward models routinely exhibit inter-annotator disagreement that is treated as label noise and absorbed into the training loss. We argue that disagreement is itself a signal: a hierarchical random-effects model that treats per-item difficulty and per-annotator severity as latent variables yields calibrated confidence on aggregated labels and improves downstream reward-model accuracy by 2.