Evaluating Agent Plans via Counterfactual Simulation Rollouts

1. Introduction

When an agent succeeds on a task, was its plan good or did it get lucky? When it fails, was the plan bad or was the environment adversarial? Outcome-only evaluation cannot tell. We propose CFSim, an evaluator that uses counterfactual rollouts in a learned environment model to attribute outcome to plan quality.

2. Related Work

Trace-likelihood scoring [Hong et al. 2024] proxies plan quality by the probability of the produced trace under a reference policy; it correlates with human ratings but conflates style with substance. Reward-model approaches [Bai et al. 2022] judge final outputs but not the planning step. Counterfactual reasoning has been used in offline RL evaluation [Thomas and Brunskill 2016] but not, to our knowledge, applied directly to agent plan scoring at this granularity.

3. Method

3.1 Setup

Let a plan be $\pi = (a_1, a_2, \ldots, a_T)$ over an action space $\mathcal{A}$. Let $\hat{E}$ be a calibrated environment model that, given state $s$ and action $a$, returns a distribution over next-state and observation. Define the simulated return of a plan as

$$\hat{V}(\pi) = \mathbb{E}$${\hat{E}}\left[ \sum{t} r(s_t, a_t) \right].

3.2 Counterfactual perturbations

We construct a candidate set of $K$ perturbed plans ${\pi^{(k)}}$ via

• single-action substitutions (drawn from a domain-specific edit distribution),
• prefix truncations,
• branch swaps (swapping the order of two independent sub-goals).

3.3 The CFSim score

$$\text{CFSim}(\pi) = \frac{\hat{V}(\pi) - \min_k \hat{V}(\pi^{(k)})}{\max_k \hat{V}(\pi^{(k)}) - \min_k \hat{V}(\pi^{(k)}) + \epsilon}.$$

A score near 1 means $\pi$ is near the simulated optimum within its perturbation neighborhood; near 0 means it is among the worst.

def cfsim(plan, env_model, K=24):
    base = simulate(env_model, plan)
    cands = [simulate(env_model, perturb(plan)) for _ in range(K)]
    lo, hi = min(cands), max(cands)
    return (base - lo) / max(hi - lo, 1e-6)

4. Experimental Setup

Tasks. 18 web-navigation tasks (booking flows, form filling) and 22 office-tool tasks (multi-step spreadsheet edits, email triage). Each task has a recorded gold trajectory and three plausible non-gold trajectories scored by 3 human raters.

Environment models. Web tasks use a sandboxed playwright-driven simulator; office tasks use a deterministic mock of an office-suite API. Both were calibrated against held-out real executions to within 4.1% step-level agreement.

5. Results

Differences between CFSim and trace-likelihood were significant at $p < 0.001$ via a stratified bootstrap.

6. Analysis

Where CFSim shines. Tasks with multiple acceptable solutions (e.g., booking flows with both deep-link and search-driven paths). Outcome-only credits both equally; CFSim correctly distinguishes the more robust path.

Where CFSim suffers. Tasks where the environment model is mis-specified (e.g., a website behavior that the simulator does not capture). Calibration-error sensitivity is the main cost of the method.

7. Limitations

The perturbation distribution is task-specific. We provide defaults but practitioners should expect to tune them. We also do not address the cost of producing a plan — only its quality.

8. Conclusion

Counterfactual simulation gives plan-quality evaluators a useful third dimension beyond outcome and likelihood. The main cost is environment-model calibration; for tasks where a calibrated model is available, CFSim correlates substantially better with expert judgment than existing methods.

References

1. Hong, S. et al. (2024). Trace-Likelihood Scoring for Agents.
2. Bai, Y. et al. (2022). Constitutional AI.
3. Thomas, P. S. and Brunskill, E. (2016). Data-Efficient Off-Policy Policy Evaluation.
4. Zheng, L. et al. (2024). AgentBench.