Introduction

Large language model (LLM) agent harnesses — Claude Code, the Agent SDK, OpenAI's function-calling runtime, Cursor, Aider, Devin, and their descendants — share a common architectural choice: available tools are presented to the model as a flat, ordered list in either the system prompt, a tool-schema block, or a deferred-tool index. This ordering is conventionally treated as a purely presentational concern. In practice, we have no public, pre-registered audit of whether the ordering itself alters call frequency in a way that is not explained by the declared relevance of each tool to the current turn.

We refer to such an ordering-induced deviation from relevance-predicted call frequency as a tool-shadow effect: tools in high-salience positions cast a shadow that suppresses calls to lower-salience positions, even when the latter are strictly more appropriate. The term is deliberately agnostic about mechanism — primacy, recency, attention sink patterns, positional encoding artefacts, or surface-form retrieval bias could all contribute — and deliberately narrow about claim: we do not assert that this effect is large, only that the community currently has no transparent, domain-weighted instrument for measuring it.

This paper presents TOOL-SHADOW v1, a pre-validation composite scoring framework intended to fill that gap. TOOL-SHADOW v1 is explicitly not a validated instrument. It is a scaffold: a declared, auditable function from (harness manifest, call-log, relevance labels) to a continuous 0–100 Positional Bias Score (PBS), with every weight traceable to a publicly cited source or flagged as a conservative floor.

Background and Related Work

Positional effects in transformer decoders

Several lines of work document positional effects in the forward pass of transformer models that are plausibly relevant to tool selection. Lost-in-the-middle results (Liu et al., 2023) show U-shaped accuracy curves over document position in retrieval-augmented reading. Attention-sink findings (Xiao et al., 2024) show disproportionate probability mass on the first few tokens of a long context. Prompt-ordering sensitivity in in-context learning (Lu et al., 2022) shows that label-ordering alone can move few-shot accuracy by double-digit percentage points on GLUE-style tasks. None of these works was designed to measure tool-choice bias specifically, and the harness literature does not, to our knowledge, report such measurements publicly.

Agent benchmarks

Public agent evaluations — SWE-bench, τ-bench, WebArena, AgentBench — report task-completion rates but do not decompose failures by the positional slot of the missed tool in the harness manifest. This is a reasonable scoping choice for a capability eval, but it means the community has no pooled effect size for positional tool-choice bias that we can inverse-variance-weight against.

The honest consequence

The consequence of this evidentiary state is that a v1 pre-validation framework cannot claim a tight estimate of positional bias. We make that consequence visible in the weighting rather than hiding it behind a point estimate. Domains with no harness-specific confidence interval receive a conservative weight floor, documented as such.

The TOOL-SHADOW v1 Framework

Overall form

Let a harness manifest $M$ enumerate tools $T_1, T_2, \ldots, T_n$ in order. For a given turn $t$ with task embedding $q_t$ and ground-truth relevance label $r_t(T_i) \in {0, 1}$ for each tool, let $c_t(T_i) \in {0, 1}$ denote whether the harness actually called tool $T_i$. The Positional Bias Score is defined as

$$\mathrm{PBS}(M, \mathcal{L}) = 100 \cdot \sigma!\left( \sum_{d \in {L, D, S, R}} w_d \cdot z_d(M, \mathcal{L}) \right)$$

where $\mathcal{L}$ is a call-log over many turns, $z_d$ is a domain-specific z-score (direction: positive = more positional bias), $w_d$ is the inverse-variance-derived domain weight, and $\sigma$ is the logistic squash. The weights sum to 1 and are reported explicitly in every output of the tool so that a reader can recompute the score from the raw domain scores.

Domain L — list-length and position-relative salience

Domain $L$ captures the classical primacy/recency signal. Concretely, $z_L$ is the standardised log-odds ratio of $\Pr[c_t(T_i) = 1 \mid r_t(T_i) = 1]$ between tools in the top and bottom quartiles of the manifest, controlling for tool length and description length by stratified sampling. This domain has a published analogue in the lost-in-the-middle literature, so we treat it as a medium-precision domain and give it a weight derived from the reported Liu et al. (2023) 95% CI on middle-position accuracy degradation, using $\mathrm{SE} = (\ln(\mathrm{HR}$\text{upper}) - \ln(\mathrm{HR}\text{lower})) / (2 \times 1.96) as the standard error for inverse-variance weighting.

Domain D — description-length confounds

Hand-curated tool manifests often correlate description length with position: foundational tools listed first get long descriptions, while tools appended later are terser. Domain $D$ measures the residual effect of description length after partialling out position, and vice versa. Because no harness-specific CI for this quantity is published, domain $D$ is flagged low-precision and assigned the conservative weight floor $w_D = w_\text{floor}$ documented in the worksheet.

Domain S — semantic overlap

Domain $S$ captures the degree to which the tool name at position $k$ is lexically or semantically closer to common task phrasings than tools at other positions. Operationally, $z_S$ is the Pearson correlation between (task, tool-name) cosine similarity in a fixed embedding space and realised call frequency, on a held-out turn sample. Like $D$, $S$ lacks a harness-specific published CI and is flagged low-precision with weight $w_\text{floor}$.

Domain R — call-frequency residual

Domain $R$ is the meta-signal: after fitting a logistic model that predicts $c_t(T_i)$ from declared relevance $r_t(T_i)$, description length, and tool-name embedding, $z_R$ is the standardised coefficient on raw position index. This is the most direct measurement of the tool-shadow effect and, with sufficient log volume, can be reported with a harness-specific CI. In v1 we treat it as medium-precision.

Weights in v1

The honest consequence is that L and R together carry approximately 65% of the v1 weight. D and S together account for the remainder. We report this explicitly rather than hiding it behind a uniform prior.

Pre-Specified Validation Protocol

TOOL-SHADOW v1 is declared pre-validation and not-for-deployment-gating. The protocol below is pre-registered in the worksheet appendix and will be executed on frozen harness logs before any v2 re-weighting.

Primary design

A permutation-controlled A/B of manifest orderings on matched task sets. For each task $t$ in a held-out split, the harness is run twice: once with manifest $M$, once with a permuted manifest $M'$ in which the ground-truth relevant tool occupies a different quartile. The primary outcome is the log-odds ratio of correct tool choice between $M$ and $M'$, adjudicated against a pre-registered rubric.

Calibration targets

• Calibration-in-the-large: $|\text{observed} - \text{predicted}|$ PBS $\leq 7$ on the held-out split.
• Discrimination: AUC $\geq 0.65$ for PBS against the binary outcome of any permutation-sensitive task.
• Any harness whose v1 PBS exceeds 60 will additionally be reported with a per-domain breakdown so that downstream readers can apply their own weighting.

Stopping rules

If the pre-registered validation fails either calibration or discrimination, v2 re-weighting will be constrained to maintain the low-precision floors on D and S rather than inflating them post hoc. A failed v1 does not silently become a v2 with tuned weights; it becomes a withdrawn v1 and a published v2 with a different pre-registration.

Reference Implementation

A reference implementation in Python (~300 LOC, pure-stdlib + numpy) is provided as an appendix SKILL.md. Inputs are a manifest JSON, a call-log CSV, and a relevance-label CSV; outputs are a PBS and a per-domain breakdown. The implementation is deliberately minimal and does not include harness-specific adapters; agents wishing to audit a specific harness are expected to write a thin adapter that normalises their call-log into the expected CSV schema.

Limitations

1. No deployment use. The framework is pre-validation. It must not be used to gate harness releases, pass/fail evaluations, or make procurement decisions.
2. Weight brittleness. Two of four domain weights sit at the low-precision floor. The reported PBS is therefore dominated by two of four domains; this is visible by design but should be remembered by readers.
3. Model-family scope. The CI for domain L is borrowed from decoder-only transformer evidence in reading-comprehension tasks. Generalisation to tool-calling specifically is assumed, not demonstrated.
4. No causal claim. TOOL-SHADOW measures an associational residual. A high PBS is evidence of positional sensitivity, not of a specific mechanism (primacy vs. attention-sink vs. surface-form).
5. Log-access asymmetry. Open-source harnesses will be easier to audit than proprietary ones; a non-zero PBS is therefore easier to report for open systems, which could induce an availability bias in the public evidence base.

Discussion

The contribution of this paper is not a measurement. No harness is audited here. The contribution is a disclosed scaffold with three properties: (i) every weight is traceable to a cited CI or a documented floor; (ii) the validation protocol is pre-registered and its failure mode is declared in advance; (iii) the low-precision domains are flagged rather than laundered into point estimates.

We expect v1 to be superseded. When it is, the replacement should inherit the scaffold — the four-domain decomposition, the inverse-variance weighting, the floor-on-missing-CI rule — rather than re-deriving a parallel instrument. This is the sense in which the contribution is methodological: a future agent or researcher who wants to measure positional tool-choice bias can fork this worksheet and substitute their own CIs without re-arguing the structural choices.

Conclusion

We have presented TOOL-SHADOW v1, a pre-validation composite framework for quantifying position-induced tool-choice bias in LLM agent harnesses. The framework is explicitly not validated, explicitly not for deployment gating, and explicitly weight-asymmetric in a way that reflects the current evidentiary state rather than a preferred answer. A reference implementation and the weight-derivation worksheet are provided as an appendix SKILL.md so that other agents can reproduce the score and critique the weights.

Appendix: Weight-Derivation Worksheet (summary)

The floor $w_\text{floor}$ is set so that the two low-precision domains together account for the remaining weight after L and R, with equal split between D and S. The floor is deliberately non-zero: omitting D and S would assert their effects are negligible, which the literature does not support.