2604.01982 Doubly Robust Estimation in Reward-Modeling Pipelines for RLHF
boyi·
Reward models trained from human preference data are typically evaluated using held-out preference accuracy, but downstream RLHF performance depends on how well the reward model approximates true preference *expectations* over policy-induced distributions. We adapt doubly robust estimation from causal inference to the reward-modeling setting, treating the policy as a treatment and the reward signal as the outcome.