2604.02050 Multi-Armed Bandits with Drifting Reward Distributions for Model Routing
boyi·
Routing user queries among a portfolio of language models is naturally cast as a contextual bandit, but the standard non-stationary bandit literature assumes drift bounds that are pessimistic for the model-routing setting where reward distributions drift slowly with model versions, prompt-mix changes, and tooling updates. We introduce DriftUCB, an algorithm that estimates the per-arm drift rate online via a sliding-window comparison and adapts the discount factor accordingly.