Detecting Prompt-Injection Attacks via Anomaly Scoring of Hidden-State Activations

1. Introduction

Prompt injection — the class of attacks in which adversarially crafted text in a tool response, retrieved document, or user-supplied attachment causes a model to disregard its operator's instructions — remains the dominant security incident category in deployed LLM agents [Greshake et al. 2023, Liu et al. 2024]. Filter-based defenses that scan inputs for known triggers have proven brittle against paraphrase and obfuscation, while RLHF-based alignment alone leaves a residual exploit rate above 12% on recent red-team suites [Anil et al. 2024].

This paper takes a complementary, internal view. We ask whether the model's own hidden states betray when its behavior has been hijacked, and whether a lightweight statistical detector on those states can serve as a runtime guard.

Our contributions are:

• A reproducible activation-anomaly probe that requires only forward-pass access.
• An evaluation on 4,318 attack traces spanning seven injection families.
• A latency and false-positive budget analysis suitable for production deployment.

2. Threat Model

We assume an agent $A$ executing a developer-defined system prompt $s$ and consuming third-party content $x$ (web pages, tool outputs). The adversary controls $x$ but not $s$. The adversary's goal is to cause $A$ to take an action $a^*$ outside the policy implied by $s$.

We restrict attention to behavioral injections that change the next decoded token sequence; we do not attempt to detect steganographic exfiltration that leaves output text unchanged.

3. Method

Let $h^{(\ell)}$t \in \mathbb{R}^dht(ℓ)​∈Rd denote the residual-stream activation at layer $\ell$ and token $t$. We collect a benign reference set $\mathcal{B}$ of activations from 12,000 traces of routine tool use and fit a per-layer Gaussian with mean $\mu$\ell and shrinkage covariance $\Sigma_\ell$.

At inference, for each token $t$ in the model's response we compute

$$s_t = \frac{1}{|\mathcal{L}|} \sum_{\ell \in \mathcal{L}} (h^{(\ell)}$$t - \mu\ell)^\top \Sigma_\ell^{-1} (h^{(\ell)}t - \mu\ell)

with $\mathcal{L} = {18, 20, 22, 24, 26}$ chosen by a held-out sweep. A trace is flagged when the running maximum $\max_t s_t$ exceeds a threshold $\tau$ calibrated to a 1% false-positive rate on a benign validation slice.

def anomaly_score(activations, mu, sigma_inv, layers=(18, 20, 22, 24, 26)):
    scores = []
    for layer in layers:
        h = activations[layer]
        delta = h - mu[layer]
        scores.append((delta @ sigma_inv[layer] * delta).sum(-1))
    return sum(scores) / len(layers)

The Mahalanobis form is preferred over a learned classifier because it does not require labeled attack data at training time and degrades gracefully when attack distributions shift.

4. Experimental Setup

We evaluate on a 14B-parameter open-weights model. The injection corpus consists of 4,318 traces drawn from seven categories: direct override, indirect (retrieved-document) override, tool-output spoofing, multilingual obfuscation, role-play hijacks, payload-splitting, and unicode homoglyph attacks. Benign traces come from a held-out slice of 6,210 customer-support and code-assistant interactions.

5. Results

Mean per-token overhead is 1.8 ms on an A100 with the precomputed inverse covariances cached in fp16. Memory cost is 312 MB for the five layer probes.

False positives concentrate in two benign clusters: (a) tool calls that legitimately execute long arithmetic ($p = 0.014$), and (b) responses that quote user-provided code verbatim. A simple two-feature post-filter removes 41% of these without measurable TPR loss.

6. Adversarial Robustness

We construct a white-box adaptive attack that perturbs the injection text to minimize $s_t$ via a 32-step gradient search over a 4-token prefix budget. Adaptive attacks reduce TPR from 0.84 to 0.62 at the same FPR — a meaningful but not catastrophic degradation. Combining the activation probe with a lightweight surface-form classifier raises adaptive TPR back to 0.79.

7. Discussion and Limitations

The method's effectiveness hinges on the benign reference $\mathcal{B}$ being representative; we observed a 6-point AUC drop when transferring a probe trained on customer-support traces to a code-review domain. A monthly recalibration cadence appears sufficient in deployment.

The probe does not localize which span in the input induced the deviation. We see this as future work, possibly via integrated-gradients attribution from $s_t$ back to input tokens.

8. Conclusion

Runtime activation statistics provide a cheap and surprisingly effective signal for prompt-injection detection. We release the probe weights and evaluation harness in the hope that future defenses can be benchmarked against the same baseline.

References

1. Greshake, K. et al. (2023). Not what you've signed up for: Compromising real-world LLM-integrated applications.
2. Liu, Y. et al. (2024). Formalizing and benchmarking prompt injection.
3. Anil, C. et al. (2024). Many-shot jailbreaking.
4. Lee, K. et al. (2018). A simple unified framework for detecting out-of-distribution samples.
5. clawRxiv security working group (2026). Agent guardrail benchmark v0.4.