Introduction

Neural scaling laws[kaplan2020scaling, hoffmann2022training] have become a cornerstone of modern machine learning, enabling practitioners to predict model performance as a function of compute, data, and parameter count. The canonical form relates test loss to model size via a power law: $$L(N) = a \cdot N^{-\alpha} + L_\infty$$ where $N$ is the number of trainable parameters, $\alpha > 0$ is the scaling exponent, $a$ is a coefficient, and $L_\infty$ is the irreducible loss.

Differentially private stochastic gradient descent (DP-SGD)[abadi2016deep] is the dominant method for training neural networks with formal privacy guarantees. DP-SGD modifies standard SGD by (1) clipping per-sample gradients to bound sensitivity and (2) adding calibrated Gaussian noise. Both operations degrade the signal-to-noise ratio of gradient updates, raising a fundamental question: does the power-law scaling relationship still hold under DP-SGD, and if so, how does privacy affect the scaling exponent?

This question has practical importance. If $\alpha_{\text{private}} < \alpha_{\text{non-private}}$, then private models scale less efficiently—each doubling of parameters yields a smaller loss reduction than in the non-private case. This would imply that organizations training under privacy constraints should allocate even more parameters (relative to non-private baselines) to achieve acceptable performance.

Method

Experimental Setup

We generate a synthetic classification dataset of 500 samples with 10 features and 5 Gaussian clusters (classes), split 80/20 into train/test sets. We use two-layer MLPs (Linear $\to$ ReLU $\to$ Linear) with hidden widths $h \in {16, 32, 64, 128, 256}$, yielding parameter counts from 261 to 4,101.

Each model is trained for 100 epochs with SGD (learning rate 0.01, batch size 64) under three privacy regimes:

• Non-private: Standard SGD (no clipping, no noise).
• Moderate DP: DP-SGD with clipping norm $C = 1.0$, noise multiplier $\sigma = 1.0$.
• Strong DP: DP-SGD with $C = 1.0$, $\sigma = 3.0$.

All experiments use 3 random seeds (42, 123, 789), totaling 45 training runs. We report mean and standard deviation of test cross-entropy loss across seeds.

DP-SGD Implementation

We implement DP-SGD from scratch without external privacy libraries. For each mini-batch:

• Compute per-sample gradients via individual forward/backward passes.
• Clip each per-sample gradient to $\ell_2$ norm $\leq C$.
• Sum clipped gradients and add Gaussian noise $\mathcal{N}(0, \sigma^2 C^2 \mathbf{I})$.
• Average and apply as the parameter update.

Scaling Law Fitting

For each privacy level, we fit Equation to the (parameter count, mean test loss) data using bounded nonlinear least squares via SciPy's trust-region reflective solver (curve\_fit with method="trf"), with bounds $a > 0$, $0 < \alpha < 5$, $L_\infty \geq 0$. We report the fitted exponent $\alpha$ and coefficient of determination $R^2$. To quantify uncertainty in $\alpha$, we compute a deterministic nonparametric bootstrap CI (1000 resamples; bootstrap seed 2026) by resampling per-seed losses at each model size and refitting.

Results

Scaling law fit parameters across privacy levels. α is the scaling exponent (higher = more efficient scaling), R² is the goodness of fit, and the ratio column shows α / α_\textnon-private}. The final column reports bootstrap 95% CI for $\alpha$ (1000 resamples).}

\begin{figure}[h]

\includegraphics[width=0.85\textwidth]{../results/scaling_laws.png} Test loss vs.\ parameter count (log-log) for three privacy levels, with fitted power-law curves. Error bars show ± 1 standard deviation across 3 seeds.

\end{figure}

Key Findings

• Scaling laws hold under DP-SGD. All three privacy levels exhibit power-law scaling with $R^2 > 0.95$, confirming that the functional form $L(N) = a \cdot N^{-\alpha} + L_\infty$ remains valid under privacy constraints.

• DP raises absolute loss while also yielding a larger fitted exponent on this task. Counter to naive expectation, DP-SGD point estimates show a higher scaling exponent ($\alpha_{\text{DP}} \approx 0.43$) than non-private models ($\alpha_{\text{NP}} \approx 0.32$). While DP models start from higher absolute loss (the coefficient $a$ increases from 0.10 to about 0.42), every run in our sweep still reaches 100% test accuracy. The difference therefore reflects cross-entropy loss and confidence calibration on an easy task, not a demonstrated gain in classification capability.

• Exponent uncertainty is high at this scale. Bootstrap 95% CIs for $\alpha$ are wide and hit the upper optimizer bound in all regimes, indicating that with only 5 model sizes and 3 seeds, exponent magnitude is not tightly identified even when the fit curve has high $R^2$.

• The irreducible loss floor is near zero for all regimes. $L_\infty \approx 0$ across all privacy levels, reflecting that the well-separated Gaussian clusters can be perfectly classified given sufficient capacity, regardless of privacy noise.

• Moderate and strong DP show nearly identical scaling exponents. $\alpha_{\sigma=1.0} = 0.432$ vs.\ $\alpha_{\sigma=3.0} = 0.431$, which is consistent with a clipping-dominated regime on this particular setup. We do not view this as evidence of a general law without harder datasets, more privacy levels, and larger models.

• On this task, DP looks more like a "constant factor tax" than a scaling tax. The ratio $\alpha_{\text{DP}} / \alpha_{\text{NP}} > 1$ means that, within this small synthetic sweep, the private curves do not flatten relative to the non-private baseline. The cost of privacy appears primarily in the coefficient $a$, though this interpretation should be treated as task-specific.

Limitations

• Small scale: Our models (261--4,101 parameters) are far smaller than practical networks. Scaling behavior may differ at larger scales.
• Synthetic data: Gaussian cluster data may not reflect the complexity of real-world distributions.
• Accuracy saturation: All 45 runs achieve 100% test accuracy, so our conclusions concern loss and calibration more than error-rate scaling.
• No formal privacy accounting: We use noise multiplier $\sigma$ as a proxy for privacy level but do not compute formal $(\varepsilon, \delta)$ guarantees.
• Fixed hyperparameters: Learning rate, epochs, and clipping norm are fixed across all runs. Optimal hyperparameters may differ between private and non-private training.
• Architecture-specific: Results are for 2-layer MLPs only; deeper or different architectures may exhibit different scaling behavior under DP.
• Wide CI bounds: Bootstrap intervals for $\alpha$ are broad, reflecting limited statistical power with only 3 seeds per model size.

Conclusion

We demonstrate that neural scaling laws persist under DP-SGD with high goodness of fit ($R^2 > 0.95$). On our synthetic classification task, DP-SGD raises absolute loss levels while also yielding a slightly higher fitted $\alpha$ than the non-private baseline. Because all runs already achieve 100% test accuracy, we interpret this as a result about loss scaling and calibration on an easy task, not a general claim that privacy improves learning efficiency. Bootstrap CIs for $\alpha$ are wide, so relative exponent differences should be treated as suggestive rather than conclusive. Moderate ($\sigma = 1.0$) and strong ($\sigma = 3.0$) DP show nearly identical exponents, consistent with a clipping-dominated regime on this setup. Future work should validate these patterns on harder tasks (e.g., natural images, language), at larger model scales, and with formal $(\varepsilon, \delta)$ privacy accounting.

\bibliographystyle{plainnat}

References

• [abadi2016deep] Martin Abadi, Andy Chu, Ian Goodfellow, H. Brendan McMahan, Ilya Mironov, Kunal Talwar, and Li Zhang. Deep learning with differential privacy. In Proceedings of the 2016 ACM SIGSAC Conference on Computer and Communications Security, pages 308--318, 2016.

• [hoffmann2022training] Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. Training compute-optimal large language models. arXiv preprint arXiv:2203.15556, 2022.

• [kaplan2020scaling] Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361, 2020.