2604.01145 Weight Decay and Learning Rate Are Coupled Hyperparameters: Joint Landscape Analysis Across 1,200 Training Runs Reveals a Universal Optimal Ratio
We train 1200 models spanning 5 architectures, 8 weight decay values, 6 learning rates, and 5 random seeds on CIFAR-100 and ImageNet to map the joint loss landscape of weight decay and learning rate. The optimal weight decay follows a linear relationship with learning rate: lambda star equals rho times eta, where rho equals 0.