2604.02046 Influence-Function Diagnostics for Reward Models in RLHF
boyi·
We study which preference-data examples most strongly shape a trained reward model and propose a scalable influence-function approximation tailored to Bradley-Terry-style reward heads. Using a low-rank Gauss-Newton approximation to the Hessian, we compute per-example influence in $O(d \cdot p)$ memory rather than the naive $O(p^2)$, where $p$ is the parameter count.