We systematically measure how MLP architecture—specifically depth and width—affects robustness to label noise in classification tasks. We sweep label noise from 0\% to 50\% across three architectures (shallow-wide, medium, deep-narrow) in the same small-model regime (3.
Grokking—the phenomenon where neural networks generalize long after memorizing training data—has been primarily studied under weight decay variation with a single optimizer. We systematically map the \emph{optimizer grokking landscape} by sweeping four optimizers (SGD, SGD+momentum, Adam, AdamW) across learning rates and weight decay values on modular addition mod 97.
We systematically measure the memorization capacity of two-layer MLPs by sweeping model width and training on synthetic data with random vs.\ structured labels.
We systematically reproduce the double descent phenomenon using random ReLU features models on synthetic regression data. Our experiments confirm that test error peaks sharply at the interpolation threshold—where the number of features equals the number of training samples—and decreases in the overparameterized regime.
We systematically map the phase diagram of "grokking" — the delayed transition from memorization to generalization — in tiny neural networks trained on modular addition (mod 97). By sweeping over weight decay (\lambda \in \{0, 10^{-3}, 10^{-2}, 10^{-1}, 1\}), dataset fraction (f \in \{0.
We systematically reproduce the double descent phenomenon using random ReLU features models on synthetic regression data. Our experiments confirm that test error peaks sharply at the interpolation threshold—where the number of features equals the number of training samples—and decreases in the overparameterized regime.
We systematically map the phase diagram of "grokking" — the delayed transition from memorization to generalization — in tiny neural networks trained on modular addition (mod 97). By sweeping over weight decay (\lambda \in \{0, 10^{-3}, 10^{-2}, 10^{-1}, 1\}), dataset fraction (f \in \{0.