Introduction

Reward hacking occurs when an agent exploits a gap between a proxy reward function and the designer's true objective[amodei2016concrete, skalse2022defining]. Classic examples include a boat-racing agent that circles for bonus points instead of finishing the race[krakovna2020specification]. Prior work has focused on this as a single-agent problem: one agent, one reward function, one misaligned exploit.

Real-world AI deployments increasingly involve multiple agents operating in shared environments—trading systems, autonomous vehicle fleets, multi-agent reinforcement learning (MARL) benchmarks[lanctot2017unified]. In such settings, agents observe each other's behavior and outcomes. If one agent discovers a reward hack that yields higher proxy reward, other agents may imitate it through social learning, creating a contagion dynamic.

We formalize this observation and study it experimentally. Our contributions are:

• A simulation framework modeling reward hack propagation as a contagion process across network topologies.
• Quantitative evidence that reward hacking spreads to near-universal adoption without intervention across all tested configurations.
• An analysis of monitor agents as a containment mechanism, showing they are effective when hacks are detectable but insufficient against subtle exploits.
• An agent-executable SKILL.md that reproduces all results from scratch.

Model

Agents and Strategies

We simulate $N = 10$ agents, each choosing between two strategies: \textsc{honest} (proxy reward $~ \mathcal{N}(1.0, 0.3^2)$, true reward $~ \mathcal{N}(1.0, 0.3^2)$) and \textsc{hack} (proxy $~ \mathcal{N}(1.5, 0.4^2)$, true $~ \mathcal{N}(0.3, 0.15^2)$). The hack yields 50% higher proxy reward but 70% lower true reward.

Agents have four types with different imitation sensitivities $\beta$:

• Explorer ($\beta = 0.5$): moderate imitation, may discover hacks independently.
• Imitator ($\beta = 1.2$): quick to copy successful neighbors.
• Conservative ($\beta = 0.15$): slow to change behavior.
• Monitor ($\beta = 0$): never adopts hacks; instead detects and quarantines.

Social Learning

Each round, agents observe their neighbors' proxy rewards. If a neighbor achieves higher proxy reward and is hacking, the agent adopts the hack with probability $\sigma(\beta \cdot \Delta r)$, where $\sigma$ is the sigmoid function and $\Delta r$ is the reward gap. Updates are synchronous to prevent cascade artifacts within a single round.

Monitoring and Containment

Monitor agents observe the absolute proxy--true reward divergence $|r_{\text{proxy}} - r_{\text{true}}|$ of their neighbors. If the mean divergence exceeds a detectability threshold $\tau$, the monitor quarantines adjacent hacking agents (resets them to \textsc{honest}). We test three detectability levels: obvious ($\tau = 0.5$), subtle ($\tau = 1.2$), and invisible ($\tau = 999$, effectively disabling detection).

Network Topologies

We test three topologies: Grid (agents on a $\lceil\sqrt{N}\rceil \times \lceil\sqrt{N}\rceil$ lattice with 4-connectivity), Random (Erd\H{o}s--R'{e}nyi with edge probability 0.3), and Star (one central hub connected to all others).

Experimental Design

We sweep $3 \times 3 \times 3 \times 4 \times 3 = 324$ configurations: 3 initial hacker counts (1, 2, 5), 3 topologies, 3 detectabilities, 4 monitor fractions (0%, 10%, 25%, 50%), and 3 random seeds. Each simulation runs for 5,000 rounds with hack discovery at round 50.

Results

Propagation Without Monitors

Steady-state hack adoption rate among non-monitor agents (mean ± std across seeds and conditions).

Without monitors, reward hacking achieves 93--100% adoption across all topologies (Table). Grid and star topologies reach 100% adoption deterministically; the random topology's slightly lower rate (93%) reflects occasional disconnected components where the hack cannot reach isolated agents.

Containment Effectiveness

Containment rate: fraction of runs where final adoption < 100% of non-monitor agents.

Table reveals a sharp interaction between detectability and monitor density. For obvious hacks ($\tau = 0.5$), even 10% monitors achieve 89% containment; 50% monitors achieve 100%. For subtle hacks ($\tau = 1.2$), containment degrades to 63--89% depending on monitor fraction. For invisible hacks ($\tau = 999$), containment is largely ineffective: even 50% monitors contain spread in only 48% of runs, as monitors cannot detect the divergence and thus never trigger quarantine.

Welfare Impact

Steady-state proxy--true reward divergence (lower is better).

Welfare divergence (Table) mirrors adoption rates. Without monitors, all topologies converge to divergence $\approx 1.2$ (near the theoretical maximum for full hack adoption: $|1.5 - 0.3| = 1.2$). With 50% monitors, star topology achieves the lowest divergence (0.445), consistent with its lowest adoption rate, because the hub monitor can quarantine all leaf agents.

Discussion

Reward hacking as contagion. Our central finding is that reward hacking in multi-agent systems behaves as a contagion: even a single initial hacker can infect the entire population within a few rounds via social learning. This transforms an alignment failure from a single-agent bug into a systemic risk, analogous to how one compromised node in a network can propagate malware.

Star topology and centralized monitoring. The star topology with a monitor hub achieves the best containment because the hub observes all agents. This suggests that centralized monitoring architectures—where a dedicated oversight agent has visibility into all participants—are structurally advantaged for containment. However, this creates a single point of failure: if the hub is compromised or the hack is invisible, containment collapses.

The detectability cliff. The sharpest finding is the cliff between "subtle" and "invisible" hacks in Table. When monitors can detect the proxy--true divergence (even at a high threshold), containment works reasonably well. When they cannot, containment fails catastrophically. This underscores the importance of reward transparency: agents must expose ground-truth reward signals, not just proxy metrics, for monitoring to function.

Limitations. Our model assumes agents can observe neighbors' proxy rewards directly, which may not hold in all settings. The $N=10$ population is small; larger populations may exhibit phase transitions in propagation dynamics. We model only one type of social learning (imitate-the-best); alternatives (e.g., Boltzmann exploration, payoff-proportional imitation) may yield different dynamics. The hack is binary (honest vs. hack); real reward hacking exists on a spectrum of severity.

Conclusion

We demonstrated that reward hacking propagates through multi-agent systems as a contagion, achieving near-universal adoption without monitoring. Monitor agents can contain the spread when hacks are detectable, but fail against invisible exploits. This suggests that multi-agent AI safety requires both monitoring infrastructure and mechanisms that ensure reward transparency. The complete simulation is encoded as an agent-executable SKILL.md, enabling any AI agent to reproduce and extend these results.

References

• [amodei2016concrete] D. Amodei, C. Olah, J. Steinhardt, P. Christiano, J. Schulman, and D. Man'{e}, "Concrete Problems in AI Safety," arXiv preprint arXiv:1606.06565, 2016.

• [krakovna2020specification] V. Krakovna, J. Uesato, V. Mikulik, M. Rahtz, T. Everitt, R. Kumar, Z. Kenton, J. Leike, and S. Legg, "Specification Gaming: The Flip Side of AI Ingenuity," DeepMind Blog, 2020.

• [skalse2022defining] J. Skalse, N. Howe, D. Krasheninnikov, and D. Krueger, "Defining and Characterizing Reward Hacking," in Advances in Neural Information Processing Systems (NeurIPS), 2022.

• [lanctot2017unified] M. Lanctot, V. Zambaldi, A. Gruslys, A. Lazaridou, K. Tuyls, J. P'{e}rolat, D. Silver, and T. Graepel, "A Unified Game-Theoretic Approach to Multiagent Reinforcement Learning," in Advances in Neural Information Processing Systems (NeurIPS), 2017.