We investigate whether training loss curves of neural networks follow universal functional forms. We train tiny MLPs (hidden sizes 32, 64, 128) on four synthetic tasks—modular addition (mod 97), modular multiplication (mod 97), random-feature regression, and random-feature classification—recording per-epoch training loss across 1,500 epochs.
Zipf's law—the empirical observation that word frequency is inversely proportional to rank—is a foundational assumption in NLP and information theory. We investigate how well this law holds for \emph{token} frequency distributions produced by modern BPE-based tokenizers across three corpus types: natural language (7 languages), and programming code (Python, Java).
Neural scaling laws promise that model performance follows predictable power-law trends as compute increases. We verify this claim using published data from two open model families—Cerebras-GPT (7 sizes, 111M--13B) and Pythia (8 sizes, 70M--12B)—and find a sharp divergence: training loss scales reliably (adj-R^2 = 0.
Zipf's law—the empirical observation that word frequency is inversely proportional to rank—is a foundational assumption in NLP and information theory. We investigate how well this law holds for \emph{token} frequency distributions produced by modern BPE-based tokenizers across three corpus types: natural language (7 languages), and programming code (Python, Java).
Neural scaling laws promise that model performance follows predictable power-law trends as compute increases. We verify this claim using published data from two open model families—Cerebras-GPT (7 sizes, 111M--13B) and Pythia (8 sizes, 70M--12B)—and find a sharp divergence: training loss scales reliably (adj-R^2 = 0.
Neural scaling laws promise that model performance follows predictable power-law trends as compute increases. We verify this claim using published data from two open model families—Cerebras-GPT (7 sizes, 111M--13B) and Pythia (8 sizes, 70M--12B)—and find a sharp divergence: training loss scales reliably (adj-R^2 = 0.