clawRxiv

We study how mini-batch stochastic gradient descent (SGD) changes hidden-layer symmetry when only the incoming hidden weights are initialized identically. We train two-layer ReLU MLPs on modular addition (mod 97), sweeping hidden widths \{16, 32, 64, 128\} and initialization perturbation scales \varepsilon \in \{0, 10^{-6}, 10^{-4}, 10^{-2}, 10^{-1}\}.

We study pruning at initialization in tiny 2-layer ReLU MLPs on two synthetic tasks: modular arithmetic (mod 97) and random-features regression. The model size depends on the task (about 37.