Introduction

The robustness of learned world models to adversarial manipulation is a central concern for AI safety [hubinger2019risks, park2023ai]. When an agent maintains beliefs about a hidden environment state and updates those beliefs based on signals from other agents, a natural question arises: \emph{how quickly can a strategic adversary corrupt those beliefs?}

This question connects to the Bayesian Persuasion literature [kamenica2011bayesian] and strategic information transmission [crawford1982strategic], but differs in that we study a repeated game where the adversary can observe the learner's evolving beliefs and adapt its deception strategy accordingly.

We formalize this as a signaling game with three axes of variation: learner type (how much the learner trusts signals), adversary strategy (random, greedy-deceptive, or credibility-building), and environment dynamics (stable, slow-drift, volatile).

Model

Environment. A hidden environment has $n = 5$ discrete states. At each round $t$, the true state $s_t \in {0, \ldots, 4}$ is drawn from a regime-dependent process: stable ($s_t = s_0$ for all $t$), slow-drift (redrawn every 5,000 rounds), or volatile (redrawn every 500 rounds).

Adversary. The adversary observes $s_t$ and sends a signal $\sigma_t \in {0, \ldots, 4}$. We study three strategies:

• Random (RA): $\sigma_t ~ \mathrm{Uniform}(0, 4)$.
• Strategic (SA): $\sigma_t = \arg\max_{j \neq s_t} b_t(j)$, reinforcing the learner's strongest incorrect belief.
• Patient (PA): Truthful for 200 rounds, then switches to the SA strategy.

Learner. The learner maintains a belief vector $b_t \in \Delta^4$ and updates it upon receiving $\sigma_t$ via multiplicative Bayesian updating with signal strength $\lambda = 3.0$ and a belief floor of $\epsilon = 0.01$ (preventing irreversible collapse). We study:

• Naive (NL): Full trust — applies likelihood ratio $\lambda$ at the signaled state.
• Skeptical (SL): Blends the signal likelihood with a uniform distribution using trust factor $\tau = 0.4$.
• Adaptive (AL): Tracks signal--action consistency via exponential moving average ($\alpha = 0.02$) and uses this as a dynamic trust factor, clamped to $[0.05, 0.95]$.

Metrics. Belief error: $1 - b_t(s_t)$. Decision accuracy: fraction of rounds where $\arg\max b_t = s_t$. Exploitation gap: difference in signal truthfulness between the first 500 and last 500 rounds.

Experiment

We run all $3 \times 3 = 9$ learner--adversary matchups across 3 environment regimes and 2 noise levels (0.0 and 0.1), with 3 random seeds each, for $50,000$ rounds per simulation. This yields 162 simulations completed in 15 seconds on an 11-core machine.

Results

\caption{Final belief error (last 20% of rounds, mean $\pm$ std across 3 seeds). Noise = 0.0.}

Decision accuracy (mean ± std across 3 seeds). Noise = 0.0.

Finding 1: Strategic adversaries are devastatingly effective. The SA achieves $>0.99$ belief error against all learner types in all environments (Table). Decision accuracy drops to near zero. This demonstrates that a greedy adversary who simply reinforces the learner's strongest incorrect belief can rapidly corrupt any Bayesian world model, regardless of the learner's defense mechanism.

Finding 2: Volatile environments provide marginal resilience. In volatile environments, frequent state changes force belief resets that briefly expose the learner to new information. The Skeptical Learner benefits most, reducing error from 0.998 (stable) to 0.992 (volatile) against SA. The Adaptive Learner achieves the lowest baseline error (0.769 vs. RA in volatile), suggesting that trust adaptation helps against non-strategic noise.

Finding 3: Trust-based defenses have a fundamental blind spot. The Adaptive Learner measures signal--action consistency to estimate trust. However, a strategic adversary that consistently sends the same deceptive signal achieves high apparent consistency (because the learner converges on the wrong state, which then matches the signal). This means the AL's trust increases under strategic deception — a fundamental limitation of any defense that lacks access to ground truth.

Finding 4: Credibility exploitation is detectable. The Patient Adversary's trust-then-exploit pattern produces a reliable exploitation gap of 0.40 (early truthful rate = 0.40 in the first 500 rounds vs. 0.00 in the last 500). This pattern is detectable by a simple auditor, suggesting that temporal analysis of signal truthfulness could serve as a practical defense mechanism.

Limitations

Our signal model is simple (single discrete signal per round). Real-world world models process high-dimensional observations. The 5-state environment is small; scaling to continuous state spaces would require different update mechanisms. We do not study multi-adversary settings or adversaries that learn the learner's defense strategy.

Conclusion

We demonstrate that Bayesian world models are highly vulnerable to adversarial signal manipulation. Defenses based on skepticism and trust adaptation provide marginal benefits in dynamic environments but fail catastrophically against strategic adversaries. The key insight is that trust-based defenses require ground-truth feedback to detect deception — without it, a consistent adversary can fool any trust-tracking mechanism. Future work should explore defenses that incorporate diverse information sources or adversarial training.

\bibliographystyle{plainnat}

References

• [crawford1982strategic] V. P. Crawford and J. Sobel. Strategic information transmission. Econometrica, 50(6):1431--1451, 1982.

• [hubinger2019risks] E. Hubinger, C. van Merwijk, V. Mikulik, J. Skalse, and S. Garrabrant. Risks from learned optimization in advanced machine learning systems. arXiv preprint arXiv:1906.01820, 2019.

• [kamenica2011bayesian] E. Kamenica and M. Gentzkow. Bayesian persuasion. American Economic Review, 101(6):2590--2615, 2011.

• [park2023ai] P. S. Park, S. Goldstein, A. O'Gara, M. Chen, and D. Hendrycks. AI deception: A survey of examples, risks, and potential solutions. arXiv preprint arXiv:2308.14752, 2023.

• [rabinowitz2018machine] N. Rabinowitz, F. Perbet, H. F. Song, C. Zhang, S. Eslami, and M. Botvinick. Machine theory of mind. In ICML, pages 4218--4227, 2018.

• [camerer2004cognitive] C. F. Camerer, T.-H. Ho, and J.-K. Chong. A cognitive hierarchy model of games. Quarterly Journal of Economics, 119(3):861--898, 2004.