Introduction

The depth--width tradeoff is a fundamental question in neural network design. Theory suggests that deeper networks can represent certain functions exponentially more efficiently than shallow ones [telgarsky2016benefits], yet deeper networks are harder to optimize [he2016deep]. Under a fixed parameter budget—a common practical constraint—increasing depth necessarily decreases width, creating a tension between representational power and trainability.

Prior work has studied this tradeoff theoretically [lu2017depth, hanin2019deep] and empirically in the over-parameterized regime [nguyen2021wide]. Recent interest in grokking [power2022grokking, gromov2023grokking] has highlighted how network architecture affects the transition from memorization to generalization on algorithmic tasks. However, controlled experiments comparing depth and width at fixed parameter count across different task types remain scarce.

We address this gap with a systematic sweep of 24 configurations (4 depths $\times$ 3 budgets $\times$ 2 tasks), measuring final performance, convergence speed, and training stability.

Experimental Setup

Architecture

We use fully-connected MLPs with ReLU activations, Kaiming initialization, and no skip connections. For each (depth $L$, budget $B$) pair, we solve for the hidden width $W$ such that the total parameter count approximately equals $B$. For $L=1$: $B = W(d_\text{in} + d_\text{out} + 1) + d_\text{out}$. For $L \geq 2$: $B = (L{-}1)W^2 + (d_\text{in} + L + d_\text{out})W + d_\text{out}$.

Tasks

Sparse parity (compositional). Given $n=20$ binary input bits, the label is the parity (XOR) of $k=3$ fixed bits. This is a well-studied compositional task that theoretically benefits from depth: a single hidden layer requires $\Omega(2^k)$ neurons, while a depth-$k$ network needs only $O(k)$ neurons per layer [barak2022hidden]. We use 3,000 training and 1,000 test examples.

Smooth regression. Given 8-dimensional Gaussian inputs, the target is $y = \sum_i \sin(x_i) + 0.1 \sum_{i<j} x_i x_j$. This smooth function should be efficiently approximable by wide, shallow networks via the universal approximation theorem. We use 2,000 training and 500 test examples.

Training

All models are trained with AdamW, cosine annealing learning rate schedule, and full-batch gradient descent. Task-specific hyperparameters: sparse parity uses lr$=3{\times}10^{-3}$, weight decay$=10^{-2}$, max 1,500 epochs; regression uses lr$=10^{-3}$, weight decay$=10^{-4}$, max 800 epochs. All experiments use seed 42. For each configuration, we split the training set deterministically into 80% optimization-train and 20% validation; early stopping and checkpoint selection use validation metrics only, and the test metric is evaluated once at the best validation epoch. For convergence-speed comparisons, we use task-specific validation thresholds: 85% accuracy for sparse parity and 0.90 $R^2$ for regression.

Results

Sparse Parity

Test accuracy on sparse parity (3-bit XOR, 20 input bits). Most configurations reach high accuracy, but depth-8 at 5K and depth-1 at 50K lag.

Convergence speed on sparse parity: epochs to reach 85% validation accuracy. Depth 2--4 converge much faster than depth 1 when both reach threshold.

The key finding is in convergence speed (Table). While most configurations reach high final accuracy (Table), depth-2 and depth-4 networks converge substantially faster than depth-1 networks at the same parameter budget (e.g., at 50K params, depth-4 reaches 85% at epoch 38 vs.\ 577 for depth-1, a 15.2$\times$ speedup). This aligns with theory: deeper networks can compute parity with fewer parameters per layer. However, depth-8 networks at the smallest budget (5K params, width=25) achieve only 63.9% test accuracy, demonstrating optimization fragility for very deep narrow networks without skip connections.

Smooth Regression

Test R² on smooth regression. Depth 2 consistently achieves the best generalization.

Depth 2 is the clear winner across all budgets (Table), with $R^2$ ranging from 0.920 (5K) to 0.932 (20K). Single-layer networks suffer at small budgets ($R^2 = 0.843$ at 5K) despite having the widest layers (width=500). At 50K, depth-2 reaches the 0.90 validation $R^2$ threshold at epoch 50 versus epoch 337 for depth-1 (6.7$\times$ faster). Deep networks (especially depth-8) show larger train-test gaps (e.g., at 20K: train $R^2=0.983$ vs.\ test $R^2=0.867$), demonstrating how narrow hidden layers bottleneck generalization on smooth tasks.

Discussion

Moderate depth is a robust default. Across both tasks and all budgets, depth-2 networks achieve the best or near-best performance; they are in the best-performing set for 5/6 task-budget pairs. This suggests that for practical MLP deployment under parameter constraints, two hidden layers is a robust default choice.

Depth helps convergence on compositional tasks. On sparse parity, deeper networks converge much faster, consistent with the theoretical advantage of depth for computing Boolean functions. However, this benefit saturates and reverses at extreme depths due to optimization difficulty.

Excess depth hurts smooth tasks. For regression, depth beyond 2 hurts generalization. At 20K and 50K parameters, depth-8 networks reach train $R^2$ of roughly 0.999 while test $R^2$ remains below 0.90, indicating severe overfitting—the narrow layers act as a bottleneck that memorizes rather than generalizes.

Training stability degrades at architectural extremes. At the 5K budget, depth-8 networks (width=25--26) achieve only 63.9% on parity. At the opposite extreme, depth-1 at 50K converges slowly and reaches only 93.7% parity accuracy despite a much wider layer. These failures indicate that both over-deep narrow networks and overly shallow wide networks can be brittle under fixed-parameter constraints.

Limitations. We test only ReLU MLPs without skip connections, batch normalization, or other stabilizers that could shift the optimal depth. We use a single seed; variance across seeds would strengthen conclusions. The tasks, while theoretically motivated, are synthetic.

Conclusion

Under fixed parameter budgets, moderate-depth MLPs (especially 2 hidden layers) provide the best tradeoff between representational power and trainability. Depth accelerates convergence on compositional tasks, but excess depth causes optimization instability, especially at smaller budgets where hidden layers become very narrow. For smooth regression, depth-2 consistently outperforms both extreme width and extreme depth. These findings suggest that practitioners working under parameter constraints should default to 2 hidden layers, and consider depth 4 for parity-like tasks where convergence speed is critical.

\bibliographystyle{plainnat}

References

• [barak2022hidden] B. Barak, B. Edelman, S. Goel, S. Kakade, E. Malach, and C. Zhang. Hidden progress in deep learning: SGD learns parities near the computational limit. In NeurIPS, 2022.

• [gromov2023grokking] A. Gromov. Grokking modular arithmetic. arXiv preprint arXiv:2301.02679, 2023.

• [hanin2019deep] B. Hanin and D. Rolnick. Deep ReLU networks have surprisingly few activation patterns. In NeurIPS, 2019.

• [he2016deep] K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In CVPR, 2016.

• [lu2017depth] Z. Lu, H. Pu, F. Wang, Z. Hu, and L. Wang. The expressive power of neural networks: A view from the width. In NeurIPS, 2017.

• [nguyen2021wide] Q. Nguyen and M. Hein. Optimization landscape and expressivity of deep {CNNs}. JMLR, 22, 2021.

• [power2022grokking] A. Power, Y. Burda, H. Edwards, I. Babuschkin, and V. Misra. Grokking: Generalization beyond overfitting on small algorithmic datasets. arXiv preprint arXiv:2201.02177, 2022.

• [telgarsky2016benefits] M. Telgarsky. Benefits of depth in neural networks. In COLT, 2016.