Introduction

Scaling laws—empirical power-law relationships between model size and performance—have become a cornerstone of large language model development. Kaplan et al.[kaplan2020scaling] established that training loss decreases as a power law in model parameters $N$: $L(N) = aN^{-\alpha} + L_\infty$. Hoffmann et al.[hoffmann2022chinchilla] extended this to jointly model parameters and data: $L(N,D) = aN^{-\alpha} + bD^{-\beta} + L_\infty$, yielding the "Chinchilla-optimal" training recipe.

However, recent work has challenged the assumption that loss-scaling laws transfer to downstream tasks. Schaeffer et al.[schaeffer2023emergent] argued that apparent "emergent abilities" are measurement artifacts of nonlinear metrics, while Pearce[pearce2024reconciling] attempted to reconcile smooth loss scaling with discontinuous task performance. The core question remains: can scaling laws for training loss reliably predict downstream task performance?

We contribute a reproducible statistical framework that addresses this question. Using only published benchmark data from Cerebras-GPT[dey2023cerebras] and Pythia[biderman2023pythia]—requiring no model inference—we fit three scaling formulations, quantify extrapolation risk, and test cross-family transfer. The entire analysis is encoded as an agent-executable SKILL.md skill with embedded data, pinned dependencies, and parametric bootstrap confidence intervals.

Data

We use two publicly documented model families trained on The Pile[gao2020pile]:

Cerebras-GPT (Dey et al., 2023): 7 model sizes from 111M to 13B parameters, trained with Chinchilla-optimal compute allocation ($D \approx 20N$). Published metrics include Pile test loss and 7 downstream benchmarks (LAMBADA, HellaSwag, PIQA, WinoGrande, ARC-Easy, ARC-Challenge, OpenBookQA), all evaluated zero-shot. Values are sourced from the paper and HuggingFace model cards.

Pythia (Biderman et al., 2023): 8 model sizes from 70M to 12B parameters, all trained on $~$300B tokens (fixed budget). Published benchmarks include LAMBADA, PIQA, WinoGrande, ARC-Easy, and ARC-Challenge at the final checkpoint (step 143,000). HellaSwag and OpenBookQA are not in the Pythia evaluation suite.

Crucially, all data is embedded directly in our source code—no downloads, API calls, or model inference are required.

Methods

Loss Scaling Formulations

We fit three scaling law formulations to Cerebras-GPT Pile test losses via nonlinear least squares in parameter space (not log-log transformed); with $n=7$ well-separated data points spanning two orders of magnitude, the difference between parameter-space and log-space fitting is negligible: \text{Kaplan:} L(N) &= a N^{-\alpha} + L_\infty

\text{Chinchilla:} L(N,D) &= a N^{-\alpha} + b D^{-\beta} + L_\infty

\text{Corrected:} L(N) &= a N^{-\alpha}(1 + c N^{-\gamma}) + L_\infty Model selection uses AIC and BIC to penalize over-parameterization.

Task Scaling

For each downstream benchmark, we fit two functional forms:

• Bounded power-law: $\mathrm{acc}(N) = 1 - a N^{-\alpha}$, with $a > 0$ and $0 < \alpha < 1$.
• Sigmoid in log-space: $\mathrm{acc}(N) = L / (1 + e^{-k(\ln N - x_0)})$, capturing S-shaped emergence.

We also apply piecewise-linear breakpoint detection in $(\ln N, \mathrm{acc})$ space to identify phase transitions.

Statistical Inference

Parametric bootstrap. For each fit, we generate $B = 500$ synthetic datasets by adding Gaussian noise (with standard deviation estimated from residuals) to the fitted curve, refit, and extract 95% confidence intervals from the bootstrap distribution.

Model comparison. Adjusted $R^2$ quantifies goodness of fit while penalizing model complexity. AIC and BIC balance fit quality against parameter count.

Extrapolation risk. We train each scaling law on the 4 smallest models and predict the 3 largest, measuring mean absolute percentage error (MAPE) for both loss and task accuracy.

Cross-family transfer. We fit bounded power-laws on Cerebras-GPT benchmarks and use the fitted parameters to predict Pythia accuracy on the 5 overlapping tasks.

Results

Loss Scaling

Table presents loss scaling fits. The Kaplan power-law achieves adj-$R^2 = 0.990$ and is selected by both AIC ($-46.9$) and BIC ($-47.1$). The estimated exponent $\alpha = 0.106$ (95% CI: $[0.101, 0.201]$) is consistent with Kaplan et al.'s original value. The Chinchilla and corrected formulations achieve lower adj-$R^2$ ($0.973$ and $0.977$, respectively) with wider confidence intervals, reflecting over-parameterization for the $n=7$ sample.

Loss scaling fits on Cerebras-GPT. $^$Best model by AIC.*

| Formulation | $\alpha$ | 95% CI | $L_\infty$ | adj-$R^2$ | AIC | |—|—|—|—|—|—| | Kaplan$^*$ | 0.106 | $[0.101, 0.201]$ | 0.113 | 0.990 | $-46.9$ | | Chinchilla | 0.102 | $[0.041, 0.861]$ | 0.485 | 0.973 | $-43.5$ | | Corrected | — | $[0.102, 0.407]$ | — | 0.977 | $-44.7$ |

Task Scaling

Table presents task-level scaling fits. LAMBADA shows strong power-law scaling (adj-$R^2 = 0.977$), with the sigmoid model fitting even better ($0.994$). However, three tasks—HellaSwag ($0.824$), WinoGrande ($0.763$), and ARC-Challenge ($0.804$)—exhibit poor power-law fits, consistent with claims that downstream task scaling is unreliable[schaeffer2023emergent].

Task scaling fits (bounded power-law) on Cerebras-GPT. Tasks with adj-$R^2 < 0.85$ are italicized.

| Task | $\alpha$ | Power-Law adj-$R^2$ | Sigmoid adj-$R^2$ | |—|—|—|—| | LAMBADA | 0.195 | 0.977 | 0.994 | | HellaSwag | 0.078 | 0.824 | 0.879 | | PIQA | 0.111 | 0.927 | 0.932 | | WinoGrande | 0.068 | 0.763 | 0.734 | | ARC-Easy | 0.143 | 0.917 | 0.956 | | ARC-Challenge | 0.050 | 0.804 | 0.859 | | OpenBookQA | 0.039 | 0.858 | 0.894 |

Extrapolation Risk

When fitting on the 4 smallest models (111M--1.3B) and predicting the 3 largest (2.7B--13B), loss extrapolation achieves $\mathrm{MAPE} = 6.9$, while task accuracy extrapolation averages $\mathrm{MAPE} = 13.1$—nearly twice as large. The worst-performing task for extrapolation is WinoGrande, where the model underestimates accuracy at 13B by $>16$. This asymmetry directly implies that compute planning based on loss projections is substantially more reliable than planning based on task accuracy projections.

Cross-Metric Correlation

The correlation between loss improvement and accuracy improvement across consecutive model sizes is weak and statistically insignificant (Pearson $r = -0.29$, $p = 0.58$; Spearman $\rho = -0.09$, $p = 0.87$; $n = 6$ pairs). With only $n = 6$ pairs, we found no statistically significant correlation between loss improvement and task accuracy improvement.

Cross-Family Transfer

Fitting bounded power-laws on Cerebras-GPT and predicting Pythia accuracy yields an average transfer MAPE of $12.7$ across 5 overlapping tasks. Transfer is best for PIQA ($4.4$) and WinoGrande ($5.3$), but poor for LAMBADA ($21.8$) and ARC-Challenge ($21.9$). The divergence likely reflects differences in training recipe: Cerebras-GPT uses Chinchilla-optimal allocation while Pythia uses a fixed token budget, so their scaling exponents are not directly comparable.

Discussion

Loss predictions are reliable; task predictions are not. Our results quantify a fundamental asymmetry: loss-based scaling laws achieve adj-$R^2 > 0.99$ and extrapolate with $<7$ error, while task-based predictions are substantially noisier ($R^2$ as low as $0.76$, extrapolation error up to $2\times$ higher). For compute planning, this means organizations can reliably project training loss at larger scales but should not assume proportional gains on downstream benchmarks.

Connection to emergent abilities. The three poorly-scaling tasks (HellaSwag, WinoGrande, ARC-Challenge) are precisely those that require compositional reasoning—multi-step inference, commonsense, or pragmatic knowledge. This is consistent with the hypothesis that "emergent abilities" reflect not sudden capability jumps but rather the inadequacy of smooth power-law models for tasks with complex cognitive dependencies[schaeffer2023emergent]. The sigmoid model outperforms the power-law for 6 of 7 tasks, suggesting that S-shaped saturation curves may better capture how benchmark accuracy evolves.

The gap between loss and tasks. The near-zero cross-metric correlation ($r = -0.29$, $p = 0.58$) is striking: a large loss improvement between model sizes does not predict a correspondingly large accuracy improvement. This gap may arise because training loss is dominated by next-token prediction on high-frequency tokens, while benchmarks test rare compositional patterns. The implication for AI safety and evaluation is that loss alone is an unreliable proxy for capability.

Limitations. The primary limitation is sample size ($n = 7$ for Cerebras-GPT, $n = 8$ for Pythia), which limits the statistical power of all curve fits and renders bootstrap confidence intervals wide. The Chinchilla formulation is not reliably identifiable when $D \propto N$ (as in Cerebras-GPT's Chinchilla-optimal recipe). HellaSwag is absent from Pythia's evaluation suite, reducing cross-family comparability. Breakpoint detection at these sample sizes has low power and should be interpreted cautiously.

Conclusion and How to Extend

We demonstrated that neural scaling laws for training loss are robust (adj-$R^2 = 0.99$, $\alpha \approx 0.106$), but downstream task scaling is unreliable—3 of 7 tasks show poor power-law fits, and task extrapolation error is nearly double that of loss. These findings have practical implications: compute allocation based on loss projections is justified, but teams should not extrapolate task performance from scaling curves without large error bars.

How to extend this analysis. The accompanying SKILL.md is designed for modularity:

• Add a model family: Add a new dictionary to src/data.py following the existing format, then update src/analysis.py:run\_full\_analysis() to include the new family.
• Add a downstream task: Add accuracy values to the model dictionaries in data.py. Task analysis auto-discovers all benchmark keys.
• Add a scaling formulation: Add a function to src/scaling\_models.py and register it in the FORMULATIONS dict.
• Change bootstrap samples: Adjust n\_bootstrap in run.py (default: 500; increase to 1000 for tighter CIs, ${~}2\times$ slower).

References

• [kaplan2020scaling] J. Kaplan, S. McCandlish, T. Henighan, T. B. Brown, B. Chess, R. Child, S. Gray, A. Radford, J. Wu, and D. Amodei, "Scaling Laws for Neural Language Models," arXiv preprint arXiv:2001.08361, 2020.

• [hoffmann2022chinchilla] J. Hoffmann, S. Borgeaud, A. Mensch, E. Buchatskaya, T. Cai, E. Rutherford, D. de Las Casas, L. A. Hendricks, J. Welbl, A. Clark, T. Hennigan, E. Noland, K. Millican, G. van den Driessche, B. Damoc, A. Guy, S. Osindero, K. Simonyan, E. Elsen, J. W. Rae, O. Vinyals, and L. Sifre, "Training Compute-Optimal Large Language Models," in Advances in Neural Information Processing Systems (NeurIPS), 2022.

• [biderman2023pythia] S. Biderman, H. Schoelkopf, Q. Anthony, H. Bradley, K. O'Brien, E. Hallahan, M. A. Khan, S. Purohit, U. S. Prashanth, E. Raff, A. Skowron, L. Sutawika, and O. van der Wal, "Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling," in Proceedings of the 40th International Conference on Machine Learning (ICML), 2023.

• [dey2023cerebras] N. Dey, G. Gosal, Z. Chen, H. Khachane, W. Marshall, R. Pathria, M. Tom, and J. Hestness, "Cerebras-GPT: Open Compute-Optimal Language Models Trained on the Cerebras Wafer-Scale Cluster," arXiv preprint arXiv:2304.03208, 2023.

• [schaeffer2023emergent] R. Schaeffer, B. Miranda, and S. Koyejo, "Are Emergent Abilities of Large Language Models a Mirage?" in Advances in Neural Information Processing Systems (NeurIPS), 2023.

• [pearce2024reconciling] T. Pearce, J. Jun, and S. Sheratt, "Reconciling Scaling Laws with Downstream Task Performance," arXiv preprint arXiv:2403.11981, 2024.

• [gao2020pile] L. Gao, S. Biderman, S. Black, L. Golding, T. Hoppe, C. Foster, J. Phang, H. He, A. Thite, N. Nabeshima, S. Presser, and C. Leahy, "The Pile: An 800GB Dataset of Diverse Text for Language Modeling," arXiv preprint arXiv:2101.00027, 2020.