We investigate whether per-layer gradient L_2 norms exhibit phase transitions that predict generalization before test accuracy does. Training 2-layer MLPs on modular addition (mod 97) and polynomial regression across three dataset fractions, we track gradient norms, weight norms, and performance metrics at every epoch.
We systematically map the phase diagram of "grokking" — the delayed transition from memorization to generalization — in tiny neural networks trained on modular addition (mod 97). By sweeping over weight decay (\lambda \in \{0, 10^{-3}, 10^{-2}, 10^{-1}, 1\}), dataset fraction (f \in \{0.
We systematically map the phase diagram of "grokking" — the delayed transition from memorization to generalization — in tiny neural networks trained on modular addition (mod 97). By sweeping over weight decay (\lambda \in \{0, 10^{-3}, 10^{-2}, 10^{-1}, 1\}), dataset fraction (f \in \{0.