Introduction

The ReLU activation function $\sigma(x) = \max(0, x)$ naturally induces sparsity: neurons whose pre-activation is negative produce exactly zero output. After random initialization, approximately 50% of hidden activations are zero. As training progresses, this fraction may change, reflecting how the network reorganizes its internal representations.

A neuron is called dead if it outputs zero for every input in a dataset. More broadly, we study the zero fraction: the proportion of all activation values (across all neurons and samples) that are exactly zero. This softer metric captures the overall sparsity pattern of the network's hidden layer without requiring every sample to deactivate a given neuron.

Separately, the phenomenon of grokking — delayed generalization that occurs long after memorization [power2022grokking] — has attracted significant attention. Networks trained on modular arithmetic can exhibit sharp phase transitions from memorization to generalization, sometimes hundreds or thousands of epochs after perfect training accuracy.

We investigate three questions:

• Do ReLU networks self-sparsify during training?
• Does activation sparsity correlate with generalization?
• Do grokking transitions coincide with sparsity transitions?

Methods

Models and Tasks

We train two-layer ReLU MLPs with hidden widths $h \in {32, 64, 128, 256}$ on two tasks:

Modular Addition. Given one-hot encoded pairs $(a, b)$ with $a, b \in \mathbb{Z}_{97}$, predict $(a + b) \bmod 97$. We use 30% of all $97^2 = 9,409$ examples for training, following the grokking setup of [power2022grokking]. Input dimension is $2 \times 97 = 194$; output dimension is 97 (classification). We use lr $= 0.01$ and weight decay $= 1.0$.

Nonlinear Regression. $y = \sin(\mathbf{x}^\top \mathbf{w}_1) + 0.5\cos(\mathbf{x}^\top \mathbf{w}_2) + \epsilon$, where $\mathbf{x} \in \mathbb{R}^{10}$, $\mathbf{w}_1, \mathbf{w}_2$ are fixed random projections, and $\epsilon ~ \mathcal{N}(0, 0.05^2)$. 2000 training and 500 test samples. We use lr $= 0.01$ and weight decay $= 0.1$.

Training and Tracking

All models are trained with AdamW for 3000 epochs using full-batch gradient descent. Random seed is fixed at 42 for reproducibility.

Every 50 epochs, we pass a probe batch (up to 512 training samples) through the network and record five metrics:

• Dead neuron fraction: fraction of hidden neurons with $\max_i \sigma(\mathbf{w}^\top \mathbf{x}_i + b) = 0$ across all probe samples.
• Near-dead fraction: fraction of neurons with mean activation below $10^{-3}$.
• Zero fraction: proportion of all activation values that are exactly zero, measuring overall activation sparsity.
• Activation entropy: Shannon entropy of the discretized activation distribution (50 bins).
• Mean activation magnitude: average absolute value across all neurons and samples.

Statistical Analysis

We compute Spearman rank correlations between final sparsity metrics and generalization measures (test accuracy, generalization gap) across all 8 experiments, and separately within each task. For each correlation, we report a bootstrap 95% confidence interval (800 resamples). For modular addition, we attempt to detect grokking (sharp test accuracy increase after training accuracy saturation).

Results

Task-Dependent Sparsification

Contrary to the simple hypothesis that networks universally self-sparsify, we observe task-dependent sparsification direction:

• Regression: All four widths showed increased zero fraction during training ($+0.024$ to $+0.052$), consistent with self-sparsification.
• Modular addition: All four widths showed decreased zero fraction during training ($-0.050$ to $-0.127$), indicating that memorization of the modular arithmetic structure requires denser activation patterns.

No strictly dead neurons emerged in any experiment; the dead neuron fraction remained 0 throughout training for all 8 runs.

Sparsity Predicts Generalization

Spearman correlations between sparsity metrics and generalization.

The strongest pooled finding is a negative correlation between zero activation fraction and generalization gap ($\rho = -0.857$, $p = 0.007$, 95% CI $[-1.000, -0.351]$). Experiments with higher zero fractions (more sparse activations) tend to have smaller generalization gaps and higher test accuracy in this pooled view. The change in zero fraction during training shows a positive but not conventionally significant trend with test accuracy ($\rho = 0.667$, $p = 0.071$, 95% CI $[0.041, 1.000]$): experiments that increase their sparsity during training tend to generalize better in this small sample.

Task-stratified correlations (modular addition only, regression only) have wide confidence intervals spanning both positive and negative values for most metrics, reflecting low statistical power at $n=4$ per task. This indicates that pooled cross-task trends should be interpreted as hypothesis-generating rather than definitive causal evidence.

Grokking and Sparsity

None of the four modular addition experiments exhibited full grokking (defined as test accuracy exceeding 0.8 with a sharp transition from below 0.4) within 3000 epochs. The highest test accuracy reached was 0.725 (width 256). This is consistent with the literature: grokking in two-layer MLPs on mod-97 addition can require substantially more epochs. However, we observe that modular addition models progressively decrease their activation sparsity during the memorization phase, suggesting that if grokking were to occur, it might coincide with a reversal of this trend.

Width Effects

For regression, smaller models (width 32) achieve the highest final zero fraction (0.595) and the best generalization (test $R^2 = 0.933$, gen gap 0.032). Larger models show slightly lower sparsity and comparable performance. For modular addition, the relationship between width and final performance is non-monotonic: widths 32 and 256 achieve the highest test accuracy ($$0.58 and $$0.73 respectively).

Discussion

Our central finding — that activation sparsity correlates strongly with generalization in pooled analysis — supports the view that sparse representations are associated with better generalization. However, the task-dependent direction of sparsification and wide task-stratified intervals add important nuance.

In regression, where the target function has smooth structure, networks benefit from sparse, selective activation patterns. In modular arithmetic, the combinatorial structure of the task appears to require dense activation patterns for memorization, and generalization (grokking) may require a qualitative shift in representation rather than simple sparsification.

The absence of truly dead neurons across all experiments suggests that the "dying ReLU" phenomenon is not inevitable in small MLPs trained with AdamW, even with strong weight decay. The zero fraction provides a richer signal than the dead neuron fraction for tracking representational changes during training.

Limitations. We study only two-layer ReLU MLPs; deeper architectures may show qualitatively different sparsity dynamics. Results are from a single seed. Per-task hyperparameters (different learning rates and weight decay) mean cross-task comparisons confound task structure with optimizer settings. AdamW's weight decay itself promotes sparsity, complicating causal claims.

Conclusion

Activation sparsity, measured as zero fraction, is strongly associated with generalization in pooled ReLU MLP experiments ($\rho = -0.857$ vs.
generalization gap; 95% CI $[-1.000, -0.351]$). The direction of sparsification during training is task-dependent: regression self-sparsifies while modular addition becomes denser. Task-stratified analyses are uncertain at current sample size, so the pooled signal should be treated as preliminary. Monitoring activation sparsity remains a cheap, informative diagnostic for training dynamics.

Reproducibility. All experiments use seed 42, pinned library versions (PyTorch 2.6.0, NumPy 2.2.4, SciPy 1.15.2), and complete on CPU in about 2.5 minutes in our verified environment. The full analysis is executable via the accompanying SKILL.md.

\bibliographystyle{plainnat}

References

• [power2022grokking] A. Power, Y. Burda, H. Edwards, I. Babuschkin, and V. Misra. Grokking: Generalization beyond overfitting on small algorithmic datasets. In ICLR 2022 Workshop on PAIR\textsuperscript{2Struct}, 2022.

• [li2024relu] B. Li, et al. ReLU strikes back: Exploiting activation sparsity in large language models. In ICLR, 2024.

• [gromov2023grokking] A. Gromov. Grokking modular arithmetic. arXiv:2301.02679, 2023.