Introduction

Zhang et al.\ [zhang2017understanding] demonstrated a striking phenomenon: standard deep neural networks can achieve zero training error on randomly labeled data, challenging conventional wisdom about the role of model complexity in generalization. Their work showed that networks with sufficient parameters can memorize any labeling of the training set, yet this memorization capacity tells us little about generalization.

A natural follow-up question is: at what parameter count does memorization become possible, and how sharp is the transition? The interpolation threshold—the point where the number of parameters approximately equals or exceeds the number of training samples—marks a phase transition in the network's ability to fit arbitrary labels. Theoretical work [montanari2020interpolation] has shown that this transition can be sharp, resembling a phase transition in statistical physics.

In this work, we systematically sweep the width of two-layer MLPs to empirically measure:

- The interpolation threshold for random vs.\ structured labels
- The sharpness of the transition (sigmoid fit)
- The relationship between memorization and generalization

Methods

Synthetic Dataset

We generate $n = 200$ training samples and 50 test samples with $d = 20$ features drawn from $\mathcal{N}(0, I_d)$. We consider two labeling schemes with $C = 10$ classes:

- **Random labels:** <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>y</mi><mi>i</mi></msub><mtext> Uniform</mtext><mo stretchy="false">{</mo><mn>0</mn><mo separator="true">,</mo><mo>…</mo><mo separator="true">,</mo><mn>9</mn><mo stretchy="false">}</mo></mrow><annotation encoding="application/x-tex">y_i ~ \text{Uniform}\{0, \ldots, 9\}</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:1em;vertical-align:-0.25em;"></span><span class="mord"><span class="mord mathnormal" style="margin-right:0.0359em;">y</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.3117em;"><span style="top:-2.55em;margin-left:-0.0359em;margin-right:0.05em;"><span class="pstrut" style="height:2.7em;"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mathnormal mtight">i</span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.15em;"><span></span></span></span></span></span></span><span class="mspace nobreak"> </span><span class="mord text"><span class="mord">Uniform</span></span><span class="mopen">{</span><span class="mord">0</span><span class="mpunct">,</span><span class="mspace" style="margin-right:0.1667em;"></span><span class="minner">…</span><span class="mspace" style="margin-right:0.1667em;"></span><span class="mpunct">,</span><span class="mspace" style="margin-right:0.1667em;"></span><span class="mord">9</span><span class="mclose">}</span></span></span></span>, independent of <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>x</mi><mi>i</mi></msub></mrow><annotation encoding="application/x-tex">x_i</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.5806em;vertical-align:-0.15em;"></span><span class="mord"><span class="mord mathnormal">x</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.3117em;"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em;"><span class="pstrut" style="height:2.7em;"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mathnormal mtight">i</span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.15em;"><span></span></span></span></span></span></span></span></span></span>
- **Structured labels:** <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>y</mi><mi>i</mi></msub><mo>=</mo><mi>arg</mi><mo>⁡</mo><msub><mrow><mi>min</mi><mo>⁡</mo></mrow><mi>k</mi></msub><mi mathvariant="normal">∥</mi><msub><mi>x</mi><mi>i</mi></msub><mo>−</mo><msub><mi>μ</mi><mi>k</mi></msub><msup><mi mathvariant="normal">∥</mi><mn>2</mn></msup></mrow><annotation encoding="application/x-tex">y_i = \arg\min_k \|x_i - \mu_k\|^2</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.625em;vertical-align:-0.1944em;"></span><span class="mord"><span class="mord mathnormal" style="margin-right:0.0359em;">y</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.3117em;"><span style="top:-2.55em;margin-left:-0.0359em;margin-right:0.05em;"><span class="pstrut" style="height:2.7em;"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mathnormal mtight">i</span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.15em;"><span></span></span></span></span></span></span><span class="mspace" style="margin-right:0.2778em;"></span><span class="mrel">=</span><span class="mspace" style="margin-right:0.2778em;"></span></span><span class="base"><span class="strut" style="height:1em;vertical-align:-0.25em;"></span><span class="mop">ar<span style="margin-right:0.0139em;">g</span></span><span class="mspace" style="margin-right:0.1667em;"></span><span class="mop"><span class="mop">min</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.3361em;"><span style="top:-2.55em;margin-right:0.05em;"><span class="pstrut" style="height:2.7em;"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mathnormal mtight" style="margin-right:0.0315em;">k</span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.15em;"><span></span></span></span></span></span></span><span class="mspace" style="margin-right:0.1667em;"></span><span class="mord">∥</span><span class="mord"><span class="mord mathnormal">x</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.3117em;"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em;"><span class="pstrut" style="height:2.7em;"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mathnormal mtight">i</span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.15em;"><span></span></span></span></span></span></span><span class="mspace" style="margin-right:0.2222em;"></span><span class="mbin">−</span><span class="mspace" style="margin-right:0.2222em;"></span></span><span class="base"><span class="strut" style="height:1.0641em;vertical-align:-0.25em;"></span><span class="mord"><span class="mord mathnormal">μ</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.3361em;"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em;"><span class="pstrut" style="height:2.7em;"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mathnormal mtight" style="margin-right:0.0315em;">k</span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.15em;"><span></span></span></span></span></span></span><span class="mord"><span class="mord">∥</span><span class="msupsub"><span class="vlist-t"><span class="vlist-r"><span class="vlist" style="height:0.8141em;"><span style="top:-3.063em;margin-right:0.05em;"><span class="pstrut" style="height:2.7em;"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight">2</span></span></span></span></span></span></span></span></span></span></span> where <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mo stretchy="false">{</mo><msub><mi>μ</mi><mi>k</mi></msub><msubsup><mo stretchy="false">}</mo><mrow><mi>k</mi><mo>=</mo><mn>1</mn></mrow><mn>10</mn></msubsup></mrow><annotation encoding="application/x-tex">\{\mu_k\}_{k=1}^{10}</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:1.0972em;vertical-align:-0.2831em;"></span><span class="mopen">{</span><span class="mord"><span class="mord mathnormal">μ</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.3361em;"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em;"><span class="pstrut" style="height:2.7em;"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mathnormal mtight" style="margin-right:0.0315em;">k</span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.15em;"><span></span></span></span></span></span></span><span class="mclose"><span class="mclose">}</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.8141em;"><span style="top:-2.4169em;margin-left:0em;margin-right:0.05em;"><span class="pstrut" style="height:2.7em;"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight"><span class="mord mathnormal mtight" style="margin-right:0.0315em;">k</span><span class="mrel mtight">=</span><span class="mord mtight">1</span></span></span></span><span style="top:-3.063em;margin-right:0.05em;"><span class="pstrut" style="height:2.7em;"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight"><span class="mord mtight">10</span></span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.2831em;"><span></span></span></span></span></span></span></span></span></span> are centroids selected from the data

We use seed 42 for the primary sweep shown in the main threshold plots, and repeat the full sweep with seeds 43 and 44 to quantify variance.

Model Architecture

We use a two-layer MLP: $f(x) = W_2 \cdot \text{ReLU}(W_1 x + b_1) + b_2$, with hidden widths $h \in {5, 10, 20, 40, 80, 160, 320, 640}$. The total parameter count is: $$P(h) = h(d + C) + (h + C) = 31h + 10$$ ranging from 165 (h=5) to 19,850 (h=640) parameters.

Training Protocol

We train with Adam (lr=$10^{-3}$) using cross-entropy loss for up to 5,000 epochs with full-batch gradient descent. Training terminates early if loss $< 10^{-4}$ and 100% accuracy is sustained for 10 consecutive epochs.

Analysis

To characterize the transition, we fit a sigmoid to training accuracy vs.\ \log_{10}(\text{#params}): $$\text{acc}(p) = \frac{1 - c}{1 + \exp(-k(\log_{10} p - \log_{10} p^$$))} + cacc(p)=1+exp(−k(log10​p−log10​p∗))1−c​+c where $c = 1/C = 0.1$ is the chance level, $p^$ is the threshold parameter count, and $k$ is the sharpness coefficient. Larger $k$ indicates a sharper phase transition.

Results

The full workflow comprises 48 training runs (8 widths $\times$ 2 label types $\times$ 3 seeds), completing in about 5--8 minutes on CPU on a modern laptop.

Interpolation Threshold

We define the interpolation threshold as the smallest parameter count achieving $\geq 99$ training accuracy. In the seed-42 primary sweep, random labels first cross this threshold at $P=630$ parameters ($h=20$), while structured labels cross at $P=320$ parameters ($h=10$). This corresponds to parameter-to-sample ratios of $3.1\times$ (random) vs.\ $1.6\times$ (structured), and a $2.0\times$ larger threshold for random labels.

Across three seeds, this transition is stable: random-label models at $h\geq 20$ consistently achieve near-perfect training accuracy (mean $=1.00$, std $\approx 0$), while $h=10$ remains near-threshold (mean $0.955 \pm 0.018$), indicating a narrow capacity boundary.

Transition Sharpness

The sigmoid fit to training accuracy vs.\ log-parameters yields the sharpness coefficient $k$. For random labels, we obtain $k=9.95$ with midpoint threshold $p^$=166p∗=166 parameters ($R^2=1.000$). For structured labels, the fitted transition is even steeper ($k=36.32$, midpoint $p^$=136, $R^2=1.000$), consistent with easier optimization when labels contain signal aligned with input geometry.

Memorization vs.\ Generalization

As predicted, test accuracy on random labels remains near chance level regardless of model size: in the seed-42 sweep, mean random-label test accuracy is $8.5$ (chance $=10$), despite 100% training accuracy for sufficiently wide models. This confirms that memorization capacity is orthogonal to generalization: a network can perfectly memorize random labels without learning transferable features.

For structured labels, test accuracy is substantially higher (roughly $52$ in seed-42 runs, with multi-seed means up to $~ 75$), demonstrating that when labels carry genuine structure, larger models can both memorize training data and extract generalizable patterns.

Discussion

Our findings replicate the core result of Zhang et al.\ (2017) in a controlled synthetic setting: neural networks can memorize random labels, but this requires more capacity than fitting structured data. Quantitatively, random labels need approximately $2\times$ the parameter budget of structured labels to hit the 99% memorization threshold in our setup. The sigmoid characterization adds a measurable transition descriptor ($k$), while the 3-seed variance table demonstrates that these conclusions are not artifacts of a single seed.

Limitations. We study only two-layer MLPs on Gaussian data. Real-world datasets and deeper architectures may exhibit different thresholds and sharpness profiles. We use only three random seeds, so a larger sweep would further tighten uncertainty estimates for threshold location and transition sharpness. The Adam optimizer may behave differently from SGD in terms of the convergence trajectory near the threshold.

Conclusion

We provide a reproducible, quantitative measurement of the interpolation threshold in neural networks, confirming that the transition from partial to full memorization follows a sigmoid curve in log-parameter space. The experiment runs entirely on CPU in about 5--8 minutes and is designed for full reproducibility by AI agents.

\bibliographystyle{plainnat}

References

• [zhang2017understanding] C. Zhang, S. Bengio, M. Hardt, B. Recht, and O. Vinyals. Understanding deep learning requires rethinking generalization. In ICLR, 2017.

• [montanari2020interpolation] A. Montanari and Y. Zhong. The interpolation phase transition in neural networks: Memorization and generalization under lazy training. Annals of Statistics, 50(5):2816--2847, 2022.