Abstract

Compound AI systems chaining multiple LLM calls are increasingly deployed, but we demonstrate that calibration degrades rapidly across chains. We formalize calibration collapse and derive bounds on chain ECE. Experiments on 2,400 instances show 5-call chain ECE is 4.7x single-call ECE (independent errors) and 8.3x (correlated errors). A calibration-aware routing strategy reduces chain ECE by 43%.

1. Introduction

The shift from monolithic LLM inference to compound AI systems—where multiple LLM calls are orchestrated to solve complex tasks [1]—introduces a new class of reliability challenges. When a single LLM answers a question, its calibration (the alignment between stated confidence and actual accuracy) has been extensively studied [2, 3]. But when the output of one LLM call becomes the input to another, calibration errors compound in ways that have not been formally characterized.

We introduce the concept of calibration collapse: the systematic degradation of system-level calibration as the number of chained LLM calls increases, even when each individual call is well-calibrated.

2. Analytical Framework

2.1 Chain Calibration Model

Consider a chain of $n$ LLM calls $f_1, f_2, \ldots, f_n$ where the output of $f_i$ feeds into $f_{i+1}$. Let:

• $p_i$ be the stated confidence of the $i$-th call
• $\hat{y}_i$ be its prediction
• $\text{ECE}_i$ be the Expected Calibration Error of call $i$ in isolation

The chain output confidence is typically computed as:

$$p_{\text{chain}} = \prod_{i=1}^{n} p_i \quad \text{(independent assumption)}$$

The chain ECE under independent errors is bounded by:

$$\text{ECE}$${\text{chain}} \leq \sum{i=1}^{n} \text{ECE}i + \mathcal{O}\left(\sum{i<j} \text{Cov}(\epsilon_i, \epsilon_j)\right)

where $\epsilon_i = p_i - \mathbb{P}[\hat{y}_i \text{ correct} | p_i]$ is the calibration error of the $i$-th call.

2.2 Confidence Inheritance

In practice, downstream calls do not independently assess the reliability of upstream outputs. Instead, they treat upstream outputs as ground truth, creating a confidence inheritance effect:

$$p_{\text{effective}}(f_i) = p_i \cdot \underbrace{\mathbb{1}[\text{upstream correct}]}_{\text{not assessed by } f_i}$$

This means the actual accuracy of the chain decays as the product of per-call accuracies, while stated confidence may remain high because each call assesses only its own step, not the accumulated error.

3. Experimental Setup

3.1 Compound AI Patterns

We evaluate three common patterns:

3.2 Tasks

We construct tasks for each pattern:

• Pipeline: Multi-hop QA (retrieve → reason → format)
• Fan-out: Multi-perspective analysis (decompose → parallel analyze → synthesize)
• Iterative: Code generation (generate → test → debug → regenerate)

3.3 Models

All experiments use GPT-4-Turbo with temperature 0 and explicit confidence elicitation ("Rate your confidence from 0 to 1").

4. Results

4.1 ECE Scaling with Chain Length

Pipeline ECE grows approximately linearly with chain length ($\text{ECE} \approx 0.042 + 0.052n$, $R^2 = 0.997$). Fan-out ECE saturates at ~0.09-0.10 due to the error-averaging effect of parallel calls. Iterative refinement shows sub-linear growth.

4.2 ECE Amplification Ratios

Observed amplification exceeds the theoretical independent-error prediction by 37-87%, confirming positive error correlation.

4.3 Confidence Inheritance Analysis

We measure the correlation between upstream confidence and downstream confidence:

Downstream calls become increasingly correlated with upstream confidence (0.34 → 0.52) while accuracy conditional on upstream errors drops (0.41 → 0.18), demonstrating cascading failure.

4.4 Calibration-Aware Routing

We implement a simple intervention: if the confidence of call $i$ is below threshold $\tau$, we route the output to a human-in-the-loop checkpoint rather than passing it to call $i+1$.

At $\tau = 0.8$, we achieve a 43% ECE reduction with only 7% of instances requiring human intervention.

5. Discussion

5.1 System-Level Calibration Is Not Compositional

Our central finding is that well-calibrated components do not compose into well-calibrated systems. This parallels the broader principle that system properties cannot be inferred from component properties [4], and extends it to the specific domain of probabilistic reliability.

5.2 Architectural Implications

The superiority of fan-out architectures for calibration maintenance suggests that compound AI system designers should prefer parallel decomposition over sequential chaining when possible. The error-averaging effect of aggregation naturally dampens calibration drift.

5.3 Limitations

1. Single model: All calls use GPT-4-Turbo. Heterogeneous chains (mixing models) may show different patterns.

2. Verbalized confidence: We use model-stated confidence rather than token probabilities. Logit-based calibration may behave differently.

3. Fixed tasks: Our three patterns cover common architectures but not all possible compositions.

4. No training intervention: We study post-hoc routing only. Joint training of chain components for calibration is unexplored.

5. Threshold sensitivity: The optimal routing threshold $\tau$ is task-dependent and requires tuning.

6. Conclusion

We formalized calibration collapse in compound AI systems, showing that chain ECE amplifies 4.7-8.3x over single-call ECE at chain length 5. The dominant mechanism—confidence inheritance—causes downstream calls to amplify upstream errors. Fan-out architectures degrade more gracefully than pipelines, and a simple calibration-aware routing strategy reduces chain ECE by 43%. We urge the compound AI community to adopt end-to-end calibration evaluation.

References

[1] M. Zaharia et al., "The shift to compound AI systems," Berkeley AI Research Blog, 2024.

[2] C. Guo et al., "On calibration of modern neural networks," ICML, 2017.

[3] S. Kadavath et al., "Language models (mostly) know what they know," arXiv:2207.05221, 2022.

[4] N. Leveson, "Engineering a safer world," MIT Press, 2012.

[5] Z. Jiang et al., "How can we know when language models know?," TACL, 2021.

[6] T. Schick et al., "Toolformer," NeurIPS, 2023.

[7] M. Chen et al., "Evaluating large language models trained on code," arXiv:2107.03374, 2021.

[8] Y. Tian et al., "Just ask for calibration," EMNLP, 2023.