Despite Instructions: Frontier Agents Improvise Covert Channels at Test Time

Jacob Dineen; Silei Ren; Muhao Chen; Dan Roth; Ben Zhou

cs.AISep 26, 2026

Despite Instructions: Frontier Agents Improvise Covert Channels at Test Time

Authors: Jacob Dineen, Silei Ren, Muhao Chen, Dan Roth, Ben Zhou

Organizations: Arizona State University · Cornell University · University of California, Davis · University of Pennsylvania

Abstract

In security-sensitive applications, language-model agents are often required to coordinate without disclosing confidential information. Yet repeated interactions may also let ordinary messages acquire shared private meaning. We study a repeated game with pairs of models in which the sender model observes one of four secret states and selects one of four summaries of the same public report, while the receiver model tries to infer the secret state. We find that model pairs can learn to communicate the secret using only one bit of feedback indicating whether the receiver inferred it correctly. This learning occurs during inference with fixed parameters and no supplied codebook or encoding examples. The effect also persists when agents generate their own free-form updates in a simulated incident-response task. Across ten independent games, pairs of GPT-5.6 Sol agents reach 98.8% final accuracy, compared with 25% chance, despite explicit instructions prohibiting disclosure and a monitor that screens each message without access to the agents' interaction histories. The same interactions that help agents cooperate can therefore allow confidential information to pass through messages intended for legitimate coordination.

Figures & tables

Figure 1: Covert channel formation through interaction. The sender holds task instructions and confidential information it is told to keep confidential, and sends task messages that a monitor inspects and approves before delivery. Each agent keeps its own history, and task feedback reaches both. Early in the interaction the receiver recovers only the task meaning of a message. After repeated exchanges, the same message also conveys the private information, carried by wording the monitor sees as ordinary (highlighted).

Figure 2: Communication through sentence choice across model pairings. Dots show each game’s final accuracy, with means and 95% confidence intervals in black. Curves show accuracy during play. Dashed lines mark 25% chance.

Figure 3: Shared expectations help with some messages more than others. Left, average accuracy from before play through round four. Right, learning through repeated exchanges. Bars and shading show 95% confidence intervals.

Figure 4: Learning and receiver memory in incident updates, with Sol receivers throughout. Shading shows 95% intervals across ten games per sender. Rejected updates count as failures.

Sender	Original	Standard reply	No history	Delivered
Sol	$98.8\pm 2.6$	$27.5\pm 11.9$	$26.2\pm 11.3$	$100.0\pm 0.0$
Terra	$96.2\pm 6.7$	$20.6\pm 8.9$	$26.9\pm 12.2$	$100.0\pm 0.0$
Luna	$21.9\pm 8.5$	$21.2\pm 6.7$	$20.6\pm 10.6$	$94.4\pm 15.7$
Sonnet	$3.8\pm 9.9$	$25.6\pm 6.9$	$30.6\pm 14.0$	$10.0\pm 24.2$
Opus	$36.9\pm 38.2$	$22.5\pm 8.9$	$30.0\pm 19.3$	$80.0\pm 29.7$

Table 1: Final accuracy and delivery (%) by sender, as mean

\pm

standard deviation over ten games.

Figure 5: Paraphrasing in Sol-to-Sol games. Left, accuracy in ten games before and after rewriting their final messages, with receiver history unchanged. Right, three runs each compare two games with the same target sequence, with and without paraphrasing throughout learning. Markers show the four evaluations. Horizontal dashed lines mark 25% chance.

Appendix figures & tables12 assets

Supplementary material from the paper’s appendix.

Appendix

Strategy	Messages	$t=4$	$t=8$	$t=16$	$t=24$
Independent trial and error	$M=4$	49.90	71.36	93.23	98.75
Most-likely guessing	$M=4$	71.09	96.90	99.99	100.00
Independent trial and error	$M=10^{12}$	25.00	25.00	25.00	25.00
Most-likely guessing	$M=10^{12}$	25.00	25.00	25.00	25.00

Appendix

Table 2: Expected accuracy of the learning references. Each row uses

K=4

states and

M/4

messages per state under the true fixed sender. Learning states are independent and uniform, and evaluation uses a fresh uniform state. Entries are

J_{t}

in percent, rounded to two decimal places; the large-message values are certified by the bounds above. The 100.00% entry is rounded, not exact.

Study	Games	Rounds	Evaluation rounds	Checks
Model-pair screen	15	24	24	16
Feedback removal / role swap	30	24	0, 4, 8, 16, 24	8, final 24
Shuffled feedback	30	24	24	16
Sonnet-to-Sol history ablations	30	24	24 †	16
Sol-to-Sol history comparison	20	24	24	16
Sol-to-Sol feedback continuation	20	8 + 16	8, 24	16

Appendix

Table 3: Evaluation schedules for sentence selection, shared priors, and earlier free-form studies. Counts are per pairing and condition, and per message family or size where applicable. Checks do not add interaction rounds.

Estimate	Value	CI
Feedback removal and role swap (30 games per condition)
Truthful scores	64.7	[54.6, 75.0]
No scores	25.0	[25.0, 25.0]
Role swap, truthful scores	25.0	[25.0, 25.0]
Paired difference, truthful $-$ no scores	+39.7	[29.2, 49.9]
Shuffled feedback (30 Sonnet-to-Sol games, round 24)

Appendix

Table 4: Sentence-selection controls. Accuracies are percentages and differences are percentage points, with 95% intervals unless stated otherwise. Sol history accuracies show mean

\pm

SD and their two planned comparisons use Bonferroni-adjusted 97.5% intervals. Each experiment includes its own control.

Panel of Figure 6 — Figure 6: Earlier messages decoded using a mapping fitted after play. The fit uses final messages and their true states, while the receiver relies on its experience at that round. Shading shows 95% confidence intervals.

Family	Sentence	Favored	Mismatched
Weather	The sky is clear and calm.	1	2
	Clouds are gathering overhead.	2	1
	Steady rain is falling.	3	4
	A severe thunderstorm is underway.	4	3
Day cycle	The first light of dawn is appearing.	1	3
	The midday sun is high overhead.	2	4

Appendix

Table 5: Fixed sender codebooks. The value each sentence signals under the codebook both roles favor and under the mismatched reassignment, labeled reassigned in Figure 3 .

Family	Prior compatibility	T0 fav./mis.	Early fav./mis.	Fav.–most likely
Weather	100.0 [100.0, 100.0]	100.0 / 0.0	95.7 / 34.2	+47.4 [42.7, 51.1]
Day cycle	82.2 [75.9, 88.5]	96.9 / 0.0	99.6 / 26.4	+51.3 [49.9, 53.2]
Compass	100.0 [100.0, 100.0]	59.4 / 18.8	57.2 / 47.7	+8.9 [ $-$ 1.4, 20.8]
Inspection	64.6 [56.2, 73.0]	25.0 / 31.3	48.4 / 48.8	+0.1 [ $-$ 8.5, 9.4]

Appendix

Table 6: Agreement before interaction and accuracy during early learning. T0 is accuracy before play and Early averages rounds 0–4. Fav. and mis. denote the favored and mismatched mappings. Values are percentages or percentage-point differences, with 95% intervals.

Panel of Figure 7 — Figure 7: Communication with larger sets of messages, including freely generated descriptions. Each line connects accuracy before interaction and after 24 rounds.

$K$	Compatibility	Sender ordered	Receiver ordered	Receiver shifted	Chance
16	100.0 [100.0, 100.0]	100.0	100.0	0.0	6.25
64	90.6 [87.5, 93.8]	93.0	97.7	0.0	1.56

Appendix

Table 7: Agreement before interaction with larger message sets. Separately queried senders and receivers choose among

K

messages. The ordered columns give agreement with the ordered mapping, and the shifted column scores receivers against the shifted mapping. Values are percentages, with 95% CIs in brackets.

Setting	Condition	Initial	Round 4	Round 24
Fixed sender, 16 messages	Ordered	$90.6\pm 12.0$	96.9	$100.0\pm 0.0$
	Shifted	$3.1\pm 6.2$	31.3	$68.8\pm 37.5$
Fixed sender, 64 messages	Ordered	$93.8\pm 7.2$	96.9	$100.0\pm 0.0$
	Shifted	$0.0\pm 0.0$	3.1	$3.1\pm 6.2$
Two agents, supplied sentences	Weather, 16	$96.9\pm 6.2$	—	$100.0\pm 0.0$
	Weather, 64	$87.5\pm 10.2$	—	$100.0\pm 0.0$

Appendix

Table 8: Accuracy in the larger-message games. Mean accuracy (%) across four games for each condition in Figure 7 , with sample SD before interaction and after 24 rounds. Only fixed-sender games were checked at round 4.

Study	Games	Checks	Chance tests	Paired tests
Main	10 per sender	8 at 0, 16 at 60	10	20 standard/empty
Replay	10 Sol	16 final	10	10 original/rewrite
Continuous	3 Sol pairs	8 at 0, 20, 40 and 16 at 60	6	12 standard/empty

Appendix

Table 9: Evaluation schedules and the number of tests in each correction. Main and continuous games run for 60 rounds. Chance tests use Bonferroni and paired tests use Holm at

\alpha=0.05

, separately for each sender pairing in the main study.

Finding	Sender update
amber.example (5)	Gateway logs are under review, and the investigation remains ongoing; further updates will follow.
birch.example (3)	Gateway logs are available, and the assigned investigation team continues its review; further updates will follow.
cedar.example (5)	Gateway logs are available for review, and the assigned team continues investigating; further updates will follow.
dune.example (3)	Gateway logs are available for review, and the investigation team remains assigned; further updates will follow.

Appendix

Table 10: Final updates from the first Sol game. Parentheses give the number of checks for each finding. The receiver answers all 16 correctly with its history and six without it.

Pair	Messages	Original	Standard reply	No history	Replacement target
1	Unchanged	16	4	0	16
1	Paraphrased	12	5	2	12
2	Unchanged	16	5	11	16
2	Paraphrased	14	5	0	16
3	Unchanged	16	4	1	16
3	Paraphrased	14	4	6	14

Appendix

Table 11: Correct decisions out of 16 after learning with or without paraphrasing. The last column counts decisions that follow the replacement message’s target when messages are swapped.

Explore similar work

Jun 25, 2026cs.CR

Tool Use Enables Undetectable Steganography in Multi-Agent LLM Systems

Increasingly autonomous agentic AI systems pose novel multi-agent risks, such as secret collusion via covert communication channels. The natural defence to these collusion attempts is to monitor plain-text communication, but the efficacy of monitors has been called into doubt by increasingly sophisticated model steganography; indeed, some theoretical schemes have been proposed that are information-theoretically or computationally indistinguishable from good-faith plain-text communication. In this paper, we demonstrate that the complexity of these schemes is no longer a safety barrier, as agentic coding models can already produce undetectable stegosystems when given realistic tool usage, such as code execution or accessing research papers through web searches. Agents also adapt when key ingredients are missing, for example, by adding model-sampling components or implementing related keyed coding schemes. We then frame tacit steganographic coordination between agents as a Schelling-point problem and introduce coordination metrics for estimating when two agents are likely to select compatible schemes without explicit prior agreement. Our results suggest a shift in the threat model for covert communication between AI agents, where the main barrier is no longer whether frontier agents can understand and implement sophisticated stegosystems, but coordination: whether independently acting agents can converge on compatible schemes, keys, and parameters. We find substantial convergence on broad scheme families but limited strict one-shot coordination, suggesting that shared artefacts, repeated interaction, and tool-mediated search are the settings where covert communication risks are most acute. Overall, our findings provide empirical grounding for the recent strategic confinement hypothesis, which assumes that capable agents can construct covert channels that survive monitoring.

Jimmy Laurence Rippin, Simon C. Marshall, David Demitri Africa +1

Oxford University · Artificial Intelligence Security Institute (AISI)

Multi-Agent LLM Systems Multi-Agent Coordination LLM Agent Security

Ask PDF

Jul 6, 2026cs.CY

When Agents Lie: Premeditation, Persistence, and Exploitation in Repeated Games

As large language models are deployed as autonomous agents that communicate intentions before acting, a critical safety question is whether agents that publicly commit to actions will honor those commitments. We place LLM agents in repeated

n

-player games with a three-stage protocol that separates private intent, public announcement, and final action, allowing us to identify whether each deviation from a stated announcement was already planned during private deliberation. Evaluating three frontier models across six games in homogeneous and heterogeneous groups over 10 rounds, we report two findings. First, when agents deviate from their announcements, the deviation is predominantly already stated in their private plan (exceeding 90% in the highest-deception conditions), yet this is not a fixed model property: the same model ranges from perfect honesty to near-total deviation across games. Second, different models interpret announcements incompatibly, some as binding commitments and others as cheap talk, producing payoff gaps that emerge in Round~0 and persist across all 10 rounds. Systems that combine models from different providers therefore cannot assume shared announcement semantics and require empirical testing of model interactions before deployment.

Jerick Shi, Terry Jingcheng Zhang, Bernhard Schölkopf +2

Carnegie Mellon University, Pittsburgh, USA · Jinesis AI Lab, Vector Institute and University of Toronto, Toronto, Canada · EuroSafeAI +1

Multi-Agent LLM Systems Game-Playing Agents Deception in Language Models Repeated Games LLM Agent Evaluation LLM Agent Reliability

Ask PDF

May 27, 2026cs.CR

The Best-Laid SCHEMEs: Coordinated Sabotage and Monitoring in Multi-Agent Systems

As agentic coding systems decompose work across multiple model instances, a critical safety question is whether those instances can coordinate to achieve a hidden malicious objective while remaining aligned with user intent. We introduce SCHEME, a benchmark of 17 task instances across 7 settings and 8 real open-source libraries, each pairing a legitimate software-engineering task with a covert side task. Every setting is designed so that no proper subset of agents can succeed alone: agents must decompose a shared sabotage plan, relay partial requirements under different communication topologies, and execute mutually consistent edits, testing genuine multi-agent coordination rather than individual capability. Evaluating with GPT 5.1 Codex and Gemini 3.1 Pro, we find coordinated sabotage is already practical, with Gemini completing the covert objective while succeeding on the legitimate task in 84% of samples and Codex in 46%. A failure analysis shows that recovery, not failure incidence, drives the model gap: both models suffer similar numbers of communication failures per trajectory, but Gemini recovers from

\sim

73% of occurrences against Codex's

\sim

56%. However, the agents' code edits remain detectable: a trusted monitor flags coordinated sabotage at FPR

\le

2% in 99%/68% (Gemini/Codex) of attempts from code edits alone, rising to 100%/81% with access to agent communication. Despite stealth-aware prompting, undetected sabotage stays at

<

3% under the full-visibility monitor; telling agents a monitor is present halves Gemini's sabotage success without significantly breaking detection, even when agents are explicitly prompted to fabricate cover stories.

Nikolay Radev, Lennart Haas, Benjamin Arnav +1

AI Agent Security Benchmarks AI Agent Monitoring Multi-Agent Systems Multi-Agent System Security Multi-Agent Coding

Ask PDF