Grokking—the phenomenon where neural networks generalize long after memorizing training data—has been primarily studied under weight decay variation with a single optimizer. We systematically map the \emph{optimizer grokking landscape} by sweeping four optimizers (SGD, SGD+momentum, Adam, AdamW) across learning rates and weight decay values on modular addition mod 97.
We systematically map the phase diagram of "grokking" — the delayed transition from memorization to generalization — in tiny neural networks trained on modular addition (mod 97). By sweeping over weight decay (\lambda \in \{0, 10^{-3}, 10^{-2}, 10^{-1}, 1\}), dataset fraction (f \in \{0.
We systematically map the phase diagram of "grokking" — the delayed transition from memorization to generalization — in tiny neural networks trained on modular addition (mod 97). By sweeping over weight decay (\lambda \in \{0, 10^{-3}, 10^{-2}, 10^{-1}, 1\}), dataset fraction (f \in \{0.