Introduction

Neural network calibration—the alignment between predicted confidence and actual correctness probability—is essential for reliable decision-making in safety-critical applications [guo2017calibration]. A model that predicts class $k$ with probability 0.8 should be correct approximately 80% of the time. Modern neural networks are frequently overconfident, and this miscalibration often worsens under distribution shift [ovadia2019trust].

We study a specific question: How does model capacity affect calibration under distribution shift in a controlled synthetic benchmark? Prior work has shown that model size and calibration interact in non-trivial ways [guo2017calibration], but the interaction between capacity and shift-induced miscalibration is less explored.

Our contributions:

- A controlled experimental framework isolating the effect of model width on calibration under synthetic covariate shift.
- Evidence that, in this benchmark, the narrowest model is best calibrated in-distribution and that severe-shift miscalibration is largest for mid-to-large widths rather than following a simple monotonic scaling law.
- A fully reproducible, agent-executable experiment suite completing in under 3 minutes.

Methods

Data Generation

We generate synthetic classification data with $d=10$ features and $C=5$ classes. Cluster centers are sampled from $\mathcal{N}(0, 4I_d)$, and class-conditional distributions are $\mathcal{N}(\mu_c, 2.25 I_d)$. The higher within-class variance relative to center separation creates meaningful overlap between classes, yielding in-distribution accuracy of 85--90%. Training data ($N=500$) is drawn from the original distribution. Test data ($N=200$) is drawn with each class's cluster mean shifted by $\delta$ along a class-specific random direction $d_c$: $$\mu_c' = \mu_c + \delta \cdot d_c, |d_c| = 1, \delta \in {0, 0.5, 1.0, 2.0, 4.0}$$ The per-class random directions (fixed per seed) ensure that shift breaks decision boundaries rather than merely translating all clusters uniformly, which would preserve relative separability.

Models

We use 2-layer MLPs: $f(x) = W_2 \cdot \text{ReLU}(W_1 x + b_1) + b_2$, with hidden widths $h \in {16, 32, 64, 128, 256}$. Parameter counts range from 261 (width 16) to 3,845 (width 256). All models are trained with Adam ($\text{lr}=0.01$) for 200 epochs using cross-entropy loss, sufficient for convergence.

Metrics

Expected Calibration Error (ECE). Following [guo2017calibration], we partition predictions into $B=10$ equal-width confidence bins and compute: $$\text{ECE} = \sum_{b=1}^{B} \frac{|B_b|}{N} \left| \text{acc}(B_b) - \text{conf}(B_b) \right|$$ where $\text{acc}(B_b)$ and $\text{conf}(B_b)$ are the accuracy and mean confidence in bin $b$.

Brier Score. The multi-class Brier score measures both calibration and refinement: $$\text{BS} = \frac{1}{N} \sum_{i=1}^{N} \sum_{c=1}^{C} (p_{i,c} - y_{i,c})^2$$

Overconfidence Gap. We define the overconfidence gap as $\bar{p}${\max} - \text{acc}pˉ​max​−acc, where $\bar{p}${\max} is the mean maximum predicted probability across test samples. Positive values indicate systematic overconfidence.

Experimental Design

We run all combinations of 5 widths $\times$ 5 shift magnitudes $\times$ 3 seeds (42, 43, 44) = 75 experiments. We report mean $\pm$ standard deviation across seeds. All random seeds are fixed for full reproducibility.

Results

In-Distribution Calibration

At shift $\delta = 0$, all model widths achieve accuracy of 88--90% and ECE below 0.10. Wider models exhibit higher in-distribution ECE (0.097 for width 256 vs.\ 0.069 for width 16), indicating that overparameterized models are more overconfident even on the training distribution. This is consistent with the finding of [guo2017calibration] that larger networks tend toward overconfidence.

Calibration Degradation Under Shift

As shift magnitude increases, ECE rises for all models (Figure). At the maximum shift ($\delta = 4.0$), ECE approximately doubles for all widths, rising from 0.07--0.10 to 0.11--0.14. The largest severe-shift errors in this run occur for the mid-to-large models, with width 64 reaching ECE 0.143 and overconfidence gap 0.141, while width 16 remains the best calibrated at $\delta = 4.0$ with ECE 0.107.

This benchmark therefore does not support a simple monotonic story in which increasing width uniformly improves in-distribution calibration and then degrades faster under shift. Instead, it shows that capacity materially changes calibration behavior, but the strongest effect appears in a subset of larger models and should be interpreted as an empirical pattern of this setup rather than a general law.

\begin{figure}[h]

\includegraphics[width=0.85\textwidth]{../results/ece_vs_shift.pdf}
\caption{ECE vs.\ distribution shift magnitude for MLPs of varying width.
Error bars show <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mo>±</mo><mn>1</mn></mrow><annotation encoding="application/x-tex">\pm 1</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.7278em;vertical-align:-0.0833em;"></span><span class="mord">±</span><span class="mord">1</span></span></span></span> std across 3 seeds.
All models become less calibrated as shift increases, and the narrowest model remains best calibrated at the largest shift in this benchmark.}

\end{figure}

Reliability Diagrams

Reliability diagrams (Figure) for the width-256 model show increasing departure from the diagonal—particularly in high-confidence bins—as shift grows. The model remains confident while its empirical accuracy drops under larger shifts.

\begin{figure}[h]

\includegraphics[width=\textwidth]{../results/reliability_diagrams.pdf}
\caption{Reliability diagrams for width=256 MLP across shift magnitudes.
Blue bars above the diagonal indicate underconfidence; red bars below indicate overconfidence.}

\end{figure}

Discussion

Our results show that model capacity materially affects calibration under shift in this benchmark, but not through a simple monotonic tradeoff. The smallest model is best calibrated in-distribution, and the largest severe-shift miscalibration appears in the mid-to-large widths rather than increasing cleanly with width. This has practical implications for model selection in deployment environments where distribution shift is expected: calibration robustness should be measured directly rather than inferred from model size alone.

Limitations. (1) Synthetic Gaussian data may not capture real-world shift patterns. (2) We test only covariate shift (per-class mean translation); label shift and concept drift may show different patterns. (3) 2-layer MLPs are simplified; deeper architectures or attention-based models may behave differently. (4) We do not apply post-hoc calibration methods (temperature scaling, Platt scaling), which could mitigate the observed degradation.

Future Work. Extending to real datasets (e.g., CIFAR-10-C), deeper architectures, and post-hoc calibration methods would strengthen these findings. Investigating the interaction between regularization (dropout, weight decay) and calibration robustness is another promising direction.

Conclusion

We demonstrate that model capacity strongly influences calibration under distribution shift in a controlled synthetic setting, but the effect is not a simple monotonic scaling trend. In our benchmark, the narrowest model is best calibrated in-distribution, and the largest severe-shift miscalibration appears in the mid-to-large models. This setup-dependent calibration pattern should inform model selection and deployment decisions, particularly in safety-critical domains where distribution shift is expected.

\bibliographystyle{plainnat}

References

• [guo2017calibration] Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q. Weinberger. On calibration of modern neural networks. In International Conference on Machine Learning (ICML), pages 1321--1330, 2017.

• [ovadia2019trust] Yaniv Ovadia, Emily Fertig, Jie Ren, Zachary Nado, D. Sculley, Sebastian Nowozin, Joshua V. Dillon, Balaji Lakshminarayanan, and Jasper Snoek. Can you trust your model's uncertainty? {E}valuating predictive uncertainty under dataset shift. In Advances in Neural Information Processing Systems (NeurIPS), 2019.

• [naeini2015obtaining] Mahdi Pakdaman Naeini, Gregory F. Cooper, and Milos Hauskrecht. Obtaining well calibrated probabilities using {B}ayesian binning into quantiles. In AAAI Conference on Artificial Intelligence, 2015.