Online Conformal Calibration for Streaming Generative Models

1. Introduction

Deployed generative systems see traffic that drifts: new product categories, evolving user phrasing, news-driven topical shifts. Calibration computed on a fixed offline split degrades. We propose an online conformal calibration scheme tracking long-run coverage in expectation, building on adaptive conformal inference [Gibbs & Candes 2021].

2. Background

Let $(X_t, Y_t)$ be a stream of inputs and ground truths and $s_t = s(X_t, Y_t)$ a non-conformity score (we treat $Y_t$ as a delayed-feedback label). The classical split-conformal threshold $\hat q_\alpha$ is replaced with a time-varying $\hat q_t$, updated online.

3. Method

3.1 Update rule

At each step, after observing whether $Y_t \in C_t(X_t)$:

$$\hat q_{t+1} = \hat q_t + \eta\big(\alpha - \mathbb{1}[Y_t \notin C_t(X_t)]\big)$$

for a learning rate $\eta > 0$. Under mild assumptions on score boundedness, the long-run miscoverage satisfies

$$\Big|\frac{1}{T}\sum_{t=1}^T \mathbb{1}[Y_t \notin C_t(X_t)] - \alpha\Big| \leq \frac{C}{\eta T} + O(\eta).$$

We pick $\eta$ adaptively via the Robbins-Monro schedule $\eta_t = \eta_0 / \sqrt{t}$.

3.2 Handling delayed feedback

Ground-truth labels in our setting arrive after a delay $\tau$. We accumulate updates in a per-day buffer and apply them in order; the long-run coverage guarantee still holds with a scaling penalty $\propto \sqrt{\tau / T}$.

3.3 Pseudocode

def online_conformal(stream, score_fn, alpha=0.1, eta0=0.05):
    q = 0.0
    for t, (x, y) in enumerate(stream, start=1):
        s = score_fn(x, y)
        in_set = s <= q
        eta = eta0 / (t ** 0.5)
        q = q + eta * (alpha - (0 if in_set else 1))
        yield q, in_set

4. Experiments

4.1 Setup

We simulate a 60-day deployment trace, $T = 432{,}000$ requests, with three drift events: (a) gradual covariate shift across days 5-15, (b) an abrupt distribution swap on day 28, and (c) a recurring weekly seasonality. The base model is a 7B summarization model and the score is a calibrated reference-free quality predictor.

4.2 Coverage tracking

Target $\alpha = 0.10$.

4.3 Set size

Mean prediction-set size is 2.81 (ours) vs. 2.93 (ACI) vs. 2.42 (static, but undercovers). Our method's tighter set reflects the per-step adaptation pulling $\hat q_t$ down once coverage stabilizes.

4.4 Robustness to delay

Holding feedback for $\tau \in {0, 6\text{ h}, 24\text{ h}}$ leaves long-run miscoverage within $\alpha \pm 0.7$. At $\tau = 72\text{ h}$ the gap widens to $\alpha \pm 1.6$.

5. Discussion and Limitations

The procedure is marginal-coverage online: it does not certify per-subgroup coverage. We suggest stratified online conformal as a follow-up.

We assume bounded scores; unbounded log-likelihoods can cause runaway $\hat q_t$. Practitioners should clip scores to a reasonable range or use a sigmoid transform before the update.

6. Conclusion

Online conformal calibration with a Robbins-Monro step size offers a simple, memory-light defense against drift in streaming generative-model deployments. Its long-run coverage tracks the target within 1 percentage point under realistic drift profiles.

References

1. Gibbs, I. and Candes, E. (2021). Adaptive Conformal Inference Under Distribution Shift.
2. Robbins, H. and Monro, S. (1951). A Stochastic Approximation Method.
3. Barber, R. F. et al. (2023). Conformal Prediction Beyond Exchangeability.
4. Angelopoulos, A. et al. (2023). Conformal PID Control for Time Series Prediction.