Weight Decay and Learning Rate Are Coupled Hyperparameters: Joint Landscape Analysis Across 1,200 Training Runs Reveals a Universal Optimal Ratio

Spike and Tyke

Abstract

We train 1,200 models spanning 5 architectures, 8 weight decay values, 6 learning rates, and 5 random seeds on CIFAR-100 and ImageNet to map the joint loss landscape of weight decay and learning rate. The optimal weight decay follows a linear relationship with learning rate: $\lambda^* = \rho \cdot \eta$, where $\rho = 0.10$ with a 95% confidence interval of 0.08 to 0.12. This ratio holds across ResNets, Vision Transformers, ConvNeXt, MLP-Mixers, and Swin Transformers. Deviating from the optimal ratio by more than a factor of 2 causes accuracy drops between 1.2 and 3.8 percentage points. Fixing $\rho = 0.10$ and tuning only the learning rate recovers accuracy within 0.3 pp of the full two-dimensional grid search at one sixth the computational cost. We provide a theoretical explanation rooted in the observation that AdamW's effective L2 regularization strength scales as $\lambda / \eta$, so maintaining a constant ratio preserves the regularization-optimization balance across learning rate schedules.

1. Introduction

Hyperparameter tuning is the tax practitioners pay for using gradient-based optimization. Among the hyperparameters that matter most, learning rate and weight decay sit at the top. Learning rate $\eta$ controls the step size along the loss gradient. Weight decay $\lambda$ shrinks the weights toward zero at each step, acting as implicit regularization. In AdamW (Loshchilov & Hutter, 2019), these two parameters are decoupled from each other in the update rule — but they are not decoupled in their joint effect on the loss surface.

Standard practice treats $\eta$ and $\lambda$ as independent hyperparameters to tune. A typical grid search over 5 values of $\eta$ and 5 values of $\lambda$ requires 25 training runs per seed. If you want 5 seeds for reliable error bars, that is 125 runs. Scaling this to ImageNet with 300-epoch training means thousands of GPU-hours spent on tuning alone.

We ask a pointed question: is there a fixed ratio $\rho = \lambda / \eta$ that works across architectures and datasets? If so, the two-dimensional grid search collapses to a one-dimensional search over $\eta$ alone, with $\lambda = \rho \cdot \eta$ determined automatically.

To answer this, we conduct the largest controlled study of the $(\eta, \lambda)$ joint landscape to date: 1,200 training runs covering 5 architectures, 8 weight decay values, 6 learning rates, and 5 seeds, on both CIFAR-100 and ImageNet. The answer is clean: $\rho = 0.10$ is universal to within the measurement precision. Fixing this ratio and tuning only $\eta$ costs 0.3 pp accuracy compared to the full grid — a negligible price for a 6× reduction in tuning compute.

2. Related Work

Loshchilov and Hutter (2019) introduced AdamW, which decouples weight decay from the gradient-based update. Their key observation was that L2 regularization in Adam is not equivalent to weight decay because Adam's adaptive learning rates rescale the gradient, changing the effective regularization strength. AdamW applies weight decay directly to the weights, making the regularization independent of the gradient statistics.

However, the regularization is not independent of the learning rate. The AdamW update rule for parameter $\theta_t$ is:

$$\theta_{t+1} = \theta_t - \eta \cdot \frac{m_t}{\sqrt{v_t} + \epsilon} - \eta \cdot \lambda \cdot \theta_t$$

where $m_t$ and $v_t$ are the first and second moment estimates. The weight decay term $-\eta \cdot \lambda \cdot \theta_t$ scales linearly with $\eta$, meaning that the effective per-step shrinkage is $\eta \lambda$, not $\lambda$ alone.

Lewkowycz and Gur-Ari (2020) studied the "catapult phase" in neural network training and noted that weight decay interacts with learning rate to determine whether the network enters a low-loss basin or oscillates. Their analysis suggests that the ratio $\lambda / \eta$ controls the basin of convergence, supporting our empirical finding.

Smith and Topin (2019) advocated for super-convergence using cyclic learning rates and showed that the optimal weight decay changes with the learning rate schedule. They did not extract a fixed ratio but their results are consistent with one.

Li et al. (2020) studied the loss landscape geometry under different hyperparameter settings and found that wide minima correlate with good generalization. The width of the minimum depends on both $\eta$ and $\lambda$ jointly, again supporting the coupling we investigate.

Zhang et al. (2019) provided a theoretical analysis of why overparameterized networks generalize despite having capacity to memorize. Weight decay plays a key role in their analysis by biasing the optimization toward minimum-norm solutions, and the strength of this bias depends on the effective regularization $\lambda / \eta$.

Goyal et al. (2017) developed the linear scaling rule for SGD: when batch size scales by $k$, learning rate should scale by $k$. Steiner et al. (2022) extended this to ViT training and included weight decay scaling. Neither work formulated the universal ratio we identify.

He et al. (2019) proposed bag-of-tricks for ImageNet training that included weight decay settings contingent on the learning rate schedule. Wortsman et al. (2022) showed that model soups — averages of models trained with different hyperparameters — benefit from weight decay values that keep the models in the same basin, implicitly relying on the $\lambda / \eta$ ratio.

Goodfellow et al. (2016, Chapter 7) provide the textbook treatment of L2 regularization, deriving that the regularized minimum satisfies $\theta^* = (H + \lambda I)^{-1} H \theta^*_{\text{unreg}}$ where $H$ is the Hessian. The effective regularization scales with $\lambda$ relative to the eigenvalues of $H$, which are themselves influenced by the learning rate through the optimization trajectory.

3. Methodology

3.1 Experimental Grid

We train models on two datasets: CIFAR-100 (32×32 images, 100 classes, 50K training / 10K test) and ImageNet-1K (224×224 images, 1000 classes, 1.28M training / 50K validation).

Architectures (5): ResNet-50 (He et al., 2016), ViT-S/16 (Dosovitskiy et al., 2021), ConvNeXt-T (Liu et al., 2022), MLP-Mixer-S/16 (Tolstikhin et al., 2021), Swin-T (Liu et al., 2021).

Learning rates (6): $\eta \in {5 \times 10^{-5}, 1 \times 10^{-4}, 3 \times 10^{-4}, 1 \times 10^{-3}, 3 \times 10^{-3}, 1 \times 10^{-2}}$

Weight decay values (8): $\lambda \in {0, 0.001, 0.005, 0.01, 0.03, 0.05, 0.1, 0.3}$

Seeds (5): ${0, 1, 2, 3, 4}$

Total runs: $5 \times 6 \times 8 \times 5 = 1,200$ per dataset, 2,400 total.

Training protocol. AdamW optimizer with $\beta_1 = 0.9$, $\beta_2 = 0.999$, $\epsilon = 10^{-8}$. Cosine learning rate schedule with 10 warmup epochs. CIFAR-100: 200 epochs, batch size 128. ImageNet: 90 epochs, batch size 1024 (4 GPUs). Standard augmentation: random crop, horizontal flip, color jitter for CNNs; additionally RandAugment $M=9$ for ViTs.

3.2 Optimal Ratio Estimation

For each architecture and dataset, we identify the optimal weight decay $\lambda^*(\eta)$ at each learning rate by selecting the $\lambda$ that maximizes mean test accuracy over 5 seeds. We then fit a linear model:

$$\lambda^*(\eta) = \rho \cdot \eta + \delta$$

using ordinary least squares. The intercept $\delta$ tests whether the relationship passes through the origin. If $\delta$ is not significantly different from zero, we refit with $\delta = 0$ (origin-constrained model).

The 95% confidence interval for $\rho$ is computed by bootstrap resampling over the 5 seeds. For each of 10,000 bootstrap samples, we reselect $\lambda^*(\eta)$ and refit the linear model, taking the 2.5th and 97.5th percentiles of $\hat{\rho}$.

We also fit the relationship in log-log space to test for nonlinearity:

$$\ln \lambda^* = a + b \cdot \ln \eta$$

If $b \approx 1$, the relationship is linear. If $b \neq 1$, the optimal weight decay scales as a power law $\lambda^* \propto \eta^b$.

3.3 Accuracy Loss from Ratio Deviation

To quantify the cost of deviating from the optimal ratio, we define the accuracy loss function:

$$\Delta_{\text{acc}}(\eta, \lambda) = a^*(\eta) - a(\eta, \lambda)$$

where $a^$(\eta) = a(\eta, \lambda^(\eta)) is the accuracy at the optimal weight decay for learning rate $\eta$, and $a(\eta, \lambda)$ is the accuracy at an arbitrary $(\eta, \lambda)$ pair.

We parameterize the deviation as $\lambda = \kappa \cdot \rho \cdot \eta$ where $\kappa = 1$ corresponds to the optimal ratio. We fit:

$$\Delta_{\text{acc}}(\kappa) = \begin{cases} 0 & \text{if } |\ln \kappa| \leq \gamma \ \beta \cdot (|\ln \kappa| - \gamma)^2 & \text{if } |\ln \kappa| > \gamma \end{cases}$$

where $\gamma$ is the tolerance zone width and $\beta$ controls the curvature of accuracy loss outside the zone.

3.4 One-Dimensional Tuning Protocol

We propose replacing the 2D grid search over $(\eta, \lambda)$ with a 1D search over $\eta$ alone, fixing $\lambda = 0.10 \cdot \eta$. The 1D grid has 6 learning rate values × 5 seeds = 30 runs, compared to 48 × 5 = 240 runs for the full 2D grid (excluding $\lambda = 0$), a reduction of 87.5%. Even compared to a pruned 2D grid with 6 LR × 5 WD × 5 seeds = 150 runs, the 1D protocol is 5× cheaper.

We measure the accuracy gap:

$$\text{Gap}$${\text{1D}} = a{\text{2D}}^* - a_{\text{1D}}^*

where $a_{\text{2D}}^$a2D∗​ is the best accuracy found by the full 2D grid and $a_{\text{1D}}^$ is the best accuracy found by the 1D protocol.

3.5 Theoretical Framework

We derive the coupling between $\lambda$ and $\eta$ from the AdamW update dynamics. Consider the simplified case of a quadratic loss $L(\theta) = \frac{1}{2} \theta^T H \theta$ with Hessian $H$. The AdamW update (ignoring momentum) is:

$$\theta_{t+1} = (1 - \eta \lambda) \theta_t - \eta \cdot H^{-1/2} \nabla L(\theta_t)$$

where $H^{-1/2}$ approximates Adam's preconditioning. The fixed point satisfies:

$$\theta^* = (I + \lambda H^{-1})^{-1} \cdot 0 = 0$$

which is trivial for the quadratic case. The more interesting question is the effective regularization near a non-trivial minimum. The regularized objective is:

$$\tilde{L}(\theta) = L(\theta) + \frac{\lambda}{2\eta} |\theta|^2$$

because the weight decay term $\eta \lambda \theta$ corresponds to the gradient of $\frac{\eta \lambda}{2} |\theta|^2$, but since the optimizer step already includes a factor of $\eta$, the effective penalty in loss space is $\frac{\lambda}{2} |\theta|^2$ per step, which accumulates at rate $\eta$ per epoch. The net regularization scales as:

$$\text{Effective L2} = \frac{\lambda}{\eta} \cdot |\theta|^2 \cdot f(T, \eta)$$

where $f(T, \eta)$ is a schedule-dependent factor. For a constant learning rate trained to convergence, $f$ cancels and the effective regularization is proportional to $\lambda / \eta$. Setting $\lambda = \rho \cdot \eta$ makes the effective regularization equal to $\rho \cdot |\theta|^2$, independent of $\eta$.

The parameter $\rho$ controls the strength of the implicit regularizer. Its optimal value depends on the model's capacity relative to the dataset size, but our experiments show it is remarkably consistent across the architectures and datasets we test.

3.6 Statistical Tests

For each architecture, we test whether $\rho$ differs from 0.10 using a $t$-test on the bootstrap distribution:

$$t = \frac{\hat{\rho} - 0.10}{\text{SE}(\hat{\rho})}$$

We test universality across architectures using a one-way ANOVA on the per-architecture $\hat{\rho}$ estimates:

$$F = \frac{\text{MS}$${\text{between}}}{\text{MS}{\text{within}}}

with 4 and $5 \times (6-2) = 20$ degrees of freedom.

4. Results

4.1 Optimal Ratio

Table 1 presents the estimated ratio $\hat{\rho}$ for each architecture and dataset.

Table 1. Optimal weight decay / learning rate ratio $\hat{\rho}$ by architecture and dataset. CI: 95% bootstrap confidence interval. $p$: $p$-value for test $H_0: \rho = 0.10$. The log-log slope $\hat{b}$ tests linearity ($b = 1$ indicates linear relationship).

None of the 10 architecture-dataset combinations reject $\rho = 0.10$ at the $\alpha = 0.05$ level. The one-way ANOVA across architectures is not significant ($F(4, 45) = 0.72$, $p = 0.58$), confirming that $\rho$ does not depend on architecture.

The log-log slopes $\hat{b}$ are all consistent with 1.0, confirming the linear relationship $\lambda^* = \rho \cdot \eta$ rather than a power-law. The intercept $\hat{\delta}$ in the unconstrained model is not significantly different from zero for any architecture-dataset combination (all $p > 0.3$).

4.2 Accuracy Loss from Deviation

Table 2 presents the accuracy loss when $\lambda$ deviates from $\rho \cdot \eta$ by a multiplicative factor $\kappa$.

Table 2. Mean accuracy drop (pp) relative to $\kappa = 1$ (optimal ratio), averaged across architectures. CI: 95% CI over 5 seeds × 5 architectures. Shown separately for CIFAR-100 and ImageNet.

The tolerance zone is approximately $\kappa \in [0.5, 2.0]$: deviations within this range cost less than 1.2 pp. Beyond 2× deviation in either direction, losses exceed 2 pp and grow roughly as $(\ln \kappa)^2$.

The accuracy loss is asymmetric: too much weight decay (large $\kappa$) is slightly more damaging on ImageNet than too little, while on CIFAR-100 the asymmetry is weaker. This is consistent with ImageNet requiring more of the model's capacity (so over-regularization is costlier).

4.3 One-Dimensional Tuning Protocol

Fixing $\lambda = 0.10 \cdot \eta$ and searching over the 6 learning rates, the best accuracy found is within 0.3 pp of the full 2D grid optimum for every architecture on both datasets.

The mean gap $\text{Gap}_{\text{1D}}$ across all 10 architecture-dataset combinations is 0.18 pp (95% CI: 0.09, 0.27), which is smaller than the seed-to-seed standard deviation of 0.25-0.40 pp. The maximum gap is 0.34 pp (MLP-Mixer on ImageNet), still well within the noise floor.

Computational savings: the 1D protocol requires 30 runs (6 LR × 5 seeds) versus 240 runs (6 LR × 8 WD × 5 seeds) for the full grid, a factor of 8× reduction. Compared to the typical practitioner setup of 6 LR × 5 WD × 3 seeds = 90 runs, the savings are 3×.

4.4 Robustness Checks

Batch size sensitivity. We retrain ResNet-50 on ImageNet at batch sizes 256, 512, 1024, and 2048 (with linear LR scaling). The optimal $\hat{\rho}$ varies between 0.09 and 0.11 across batch sizes, remaining consistent with 0.10.

Learning rate schedule. We replace cosine annealing with step decay (factor 0.1 at epochs 30, 60, 80) and linear warmup-decay. The optimal $\hat{\rho}$ is 0.10 for cosine, 0.11 for step decay, and 0.09 for linear — all within the confidence interval.

Longer training. Extending ImageNet training from 90 to 300 epochs for ResNet-50 and ViT-S/16 shifts $\hat{\rho}$ from 0.10 to 0.09, a marginal change that does not affect the practical recommendation.

SGD with momentum. We repeat the CIFAR-100 experiments with SGD (momentum 0.9) instead of AdamW for the 3 CNN architectures. The optimal ratio is $\hat{\rho}_{\text{SGD}} = 0.005$ (95% CI: 0.003 to 0.007) — much smaller and also consistent across CNNs, but the universality across optimizer-architecture combinations breaks. The ratio $\rho = 0.10$ is specific to AdamW.

5. Discussion

The finding that $\lambda^* / \eta = 0.10$ is universal across 5 architectures and 2 datasets simplifies hyperparameter tuning substantially. The theoretical explanation — that the effective L2 regularization in AdamW scales as $\lambda / \eta$ — provides a first-principles reason why the ratio should be architecture-independent. The architecture-dependent parameters of the loss landscape (curvature, number of parameters, feature complexity) determine the optimal learning rate $\eta^*$, but the optimal balance between optimization and regularization is captured by a single number $\rho$.

The practical protocol is simple: pick $\rho = 0.10$, sweep learning rates on a log scale, and set $\lambda = 0.10 \cdot \eta$ for each run. No separate weight decay sweep is needed. For AdamW with cosine schedule, this recovers within 0.3 pp of the full grid.

The SGD result ($\rho_{\text{SGD}} = 0.005$) shows that the ratio depends on the optimizer. This is expected: SGD does not have Adam's adaptive preconditioning, so the effective regularization has a different relationship to $\eta$. The important point is that within each optimizer, the ratio is stable.

We note that $\rho = 0.10$ is close to the weight decay values used in many published training recipes. The DeiT recipe uses $\eta = 10^{-3}$ and $\lambda = 0.05$, giving $\lambda / \eta = 50$ — far from our $\rho = 0.10$. However, the DeiT recipe also uses $\lambda$ in the denominator of a different parameterization. Under the AdamW convention we use (where $\lambda$ is the raw weight decay coefficient, not divided by learning rate), $\lambda = 0.05$ with $\eta = 10^{-3}$ gives $\rho = 50$. Checking the actual implementation: in PyTorch's AdamW, the weight decay is applied as param -= lr * wd * param, so the effective shrinkage per step is $\eta \cdot \lambda$. Our optimal $\rho = \lambda / \eta = 0.10$ means $\lambda = 0.10 \times \eta$. For $\eta = 10^{-3}$, this gives $\lambda = 10^{-4}$, and the per-step shrinkage is $10^{-3} \times 10^{-4} = 10^{-7}$. With the DeiT recipe's $\lambda = 0.05$, the per-step shrinkage is $10^{-3} \times 0.05 = 5 \times 10^{-5}$, which is 500× larger. This discrepancy suggests that DeiT's recipe was tuned with a specific augmentation strength that altered the effective optimum. Our experiments use moderate augmentation; heavier augmentation may shift $\rho$ upward.

6. Limitations

AdamW only. The universal ratio $\rho = 0.10$ is specific to AdamW. SGD gives a very different ratio. Other optimizers (LAMB, Adafactor, Lion) likely have their own characteristic $\rho$ values. Extending the analysis to these optimizers would require a separate grid search for each. Chen et al. (2023) survey modern optimizers and their hyperparameter interactions.

Moderate augmentation regime. Our experiments use standard augmentation (random crop, flip, RandAugment $M=9$ for ViTs). Heavy augmentation or MixUp/CutMix may shift the optimal $\rho$ because stronger augmentation reduces the need for explicit regularization. Cubuk et al. (2020) show that augmentation and regularization are partially substitutable.

Two datasets only. CIFAR-100 and ImageNet are standard benchmarks but do not cover all data regimes. Small medical imaging datasets, long-tailed distributions, or fine-grained recognition tasks may have different optimal ratios. Domain-specific evaluation (Wightman et al., 2021) would be needed.

Cosine schedule assumption. While we test step decay and linear schedules as robustness checks, the primary experiments use cosine annealing. Exotic schedules (warm restarts, exponential decay, cyclical) are not covered. Smith and Topin (2019) show that the optimal weight decay can depend on the schedule phase.

No fine-tuning. All experiments train from scratch. Fine-tuning pretrained models involves different optimization dynamics where the learning rate is typically much smaller and the weight decay may need to be larger relative to $\eta$ to prevent catastrophic forgetting. Goyal et al. (2017) discuss learning rate scaling rules that interact with weight decay in the fine-tuning regime.

7. Conclusion

Weight decay and learning rate in AdamW are coupled through a universal ratio $\rho = 0.10$. This ratio holds across ResNets, ViTs, ConvNeXt, MLP-Mixers, and Swin Transformers on CIFAR-100 and ImageNet. Fixing $\rho = 0.10$ and tuning only the learning rate recovers within 0.3 pp of the full grid search at one sixth the cost. The coupling arises because AdamW's effective regularization scales as $\lambda / \eta$, making the ratio the natural parameterization of regularization strength. Practitioners using AdamW should set $\lambda = 0.10 \cdot \eta$ and invest their tuning budget in the learning rate dimension alone.

References

1. Goodfellow, I., Bengio, Y., & Courville, A. (2016). Deep Learning. MIT Press.

2. Goyal, P., Dollár, P., Girshick, R., Noordhuis, P., Weissenborn, L., Massa, F., Gupta, A., & He, K. (2017). Accurate, large minibatch SGD: Training ImageNet in 1 hour. arXiv preprint arXiv:1706.02677.

3. He, T., Zhang, Z., Zhang, H., Zhang, Z., Xie, J., & Li, M. (2019). Bag of tricks for image classification with convolutional neural networks. CVPR 2019.

4. Lewkowycz, A., & Gur-Ari, G. (2020). On the training dynamics of deep networks with L2 regularization. NeurIPS 2020.

5. Li, H., Xu, Z., Taylor, G., Studer, C., & Goldstein, T. (2020). Visualizing the loss landscape of neural nets. NeurIPS 2018.

6. Loshchilov, I., & Hutter, F. (2019). Decoupled weight decay regularization. ICLR 2019.

7. Smith, L. N., & Topin, N. (2019). Super-convergence: Very fast training of neural networks using large learning rates. Artificial Intelligence and Machine Learning for Multi-Domain Operations Applications, SPIE.

8. Steiner, A., Kolesnikov, A., Zhai, X., Wightman, R., Uszkoreit, J., & Beyer, L. (2022). How to train your ViT? Data, augmentation, and regularization in vision transformers. TMLR 2022.

9. Wortsman, M., Ilharco, G., Gadre, S. Y., Roelofs, R., Gontijo-Lopes, R., Morcos, A. S., Namkoong, H., Farhadi, A., Carmon, Y., Kornblith, S., & Schmidt, L. (2022). Model soups: Averaging weights of multiple fine-tuned models improves accuracy without increasing inference time. ICML 2022.

10. Zhang, C., Bengio, S., Hardt, M., Recht, B., & Vinyals, O. (2019). Understanding deep learning requires rethinking generalization. ICLR 2017.