Causal Identifiability Under Hidden Confounders in Observational Agent Logs

1. Introduction

A tempting use of agent logs is to ask counterfactual questions: would my system perform better if I added a verifier sub-agent? Without randomized assignment, this is a causal inference problem on observational data. Hidden confounders — features of tasks that affect both routing and outcome but are not logged — threaten naive estimates.

We analyze identifiability in this setting, derive conditions under which the effect can still be recovered, and provide a practical sensitivity analysis for the unidentifiable case.

2. Setup

Let $T \in {0, 1}$ denote treatment (e.g., verifier present), $Y$ denote outcome (task success), $X$ denote logged covariates (prompt length, tool count, user tier, etc.), and $U$ denote unlogged covariates (e.g., true task difficulty as judged by a human).

The causal DAG of interest is

$$U \to T,,, U \to Y,,, X \to T,,, X \to Y,,, T \to Y.$$

The target estimand is

$$\text{ATE} = \mathbb{E}[Y(1) - Y(0)],$$

the average effect of adding the verifier under intervention.

3. Identifiability

Under strong ignorability — $T \perp!!!\perp Y(t) \mid X$ — the ATE is identified by the standard adjustment formula. With hidden $U$, ignorability fails and the ATE is in general not identified. We study two relaxations.

Proxy availability. If we have a logged variable $Z$ that is a proxy for $U$ (i.e., $U \to Z$ and $Z$ is independent of $T, Y$ given $U$), then under proxy-completeness conditions [Miao et al. 2018] the ATE can be recovered via a tensor-decomposition estimator.

Instrumental variable. If we have a variable $W$ such that $W \to T$ but $W \not\to Y$ except through $T$, and $W \perp!!!\perp U$, then the ATE is identified by IV regression. Random A/B test events occasionally provide such instruments in production systems.

4. Sensitivity Analysis When Identifiability Fails

When neither proxies nor instruments are available, we propose a Rosenbaum-style sensitivity bound. Define the worst-case bias

$$\Gamma = \max_x \frac{P(T=1 \mid X=x, U=u^+)}{P(T=1 \mid X=x, U=u^-)},$$

the odds ratio for treatment assignment between high- and low-confounder strata. For each plausible $\Gamma$ we compute a bias-adjusted confidence interval. The smallest $\Gamma$ at which the interval crosses zero is the robustness value: practitioners can compare it to plausible levels of unmeasured confounding informed by domain knowledge.

5. Estimator

We use a doubly robust estimator that adjusts for $X$ and is augmented with proxy correction when available:

$$\widehat{\text{ATE}} = \frac{1}{n} \sum_i \left[ \hat{m}_1(X_i) - \hat{m}_0(X_i) + \frac{T_i (Y_i - \hat{m}_1(X_i))}{\hat{e}(X_i)} - \frac{(1-T_i)(Y_i - \hat{m}_0(X_i))}{1-\hat{e}(X_i)} \right].$$

Nuisance functions $\hat{m}_t$ and $\hat{e}$ are fit with cross-fitting [Chernozhukov et al. 2018] using gradient-boosted trees.

def dr_ate(X, T, Y, m_hat, e_hat):
    m1 = m_hat(X, t=1); m0 = m_hat(X, t=0); e = e_hat(X)
    contrib = (m1 - m0
               + T*(Y - m1)/e
               - (1-T)*(Y - m0)/(1-e))
    return contrib.mean()

6. Application

We applied the estimator to logs of 38{,}412 agent invocations from a production research-assistant deployment, where some invocations were routed to a configuration with an additional verifier sub-agent. Routing was determined by an internal heuristic that depended in part on a logged complexity score and in part on hidden user-side context.

Naive estimate. A simple difference-in-means yields an apparent verifier benefit of $5.4$ percentage points.

Adjusted estimate. With the doubly robust estimator on logged covariates, the estimate falls to $1.3$ points (95% CI $[0.6, 2.1]$). Sensitivity analysis indicates a robustness value of $\Gamma \approx 1.4$, meaning that an unmeasured confounder would have to roughly $1.4\times$ the treatment-assignment odds to nullify the effect. Domain experts judged $\Gamma$ in the range $[1.1, 1.5]$ as plausible, so the verifier benefit is suggestive but not conclusive.

7. Limitations

Doubly robust estimation requires that at least one of the outcome model and the propensity model be correctly specified. With agent-log distributions that are messy and high-dimensional, both can be misspecified together. We mitigate by using cross-fitting and ensembled nuisance learners, but no statistical method substitutes for occasional randomized A/B testing.

A second limitation is that ATE may not be the right estimand for many operational decisions. CATE (conditional average treatment effect) on specific user segments would often be more useful; we sketch a CATE extension in the appendix.

8. Conclusion

Observational agent logs can yield credible causal estimates when paired with appropriate methodology. Naive estimates can be substantially biased; doubly robust adjustment helps; sensitivity analysis reveals when the remaining uncertainty is large enough to warrant a randomized experiment.

References

1. Rosenbaum, P. (2002). Observational Studies.
2. Chernozhukov, V. et al. (2018). Double/Debiased Machine Learning for Treatment and Structural Parameters.
3. Miao, W., Geng, Z., and Tchetgen Tchetgen, E. (2018). Identifying Causal Effects with Proxy Variables of an Unmeasured Confounder.
4. Pearl, J. (2009). Causality: Models, Reasoning, and Inference.
5. VanderWeele, T. and Ding, P. (2017). Sensitivity Analysis in Observational Research: Introducing the E-Value.