Communication between Frozen Large Language Models via Prompt Optimization in a Referential Game

Vivek Anand; Muthu Chandrasekaran; Shiva Chaitanya

cs.CLSep 25, 2026

Communication between Frozen Large Language Models via Prompt Optimization in a Referential Game

Authors: Vivek Anand, Muthu Chandrasekaran, Shiva Chaitanya

Organizations: School of Electrical and Computer Engineering, Georgia Institute of Technology, Atlanta, GA, USA · Work done while at Interactly.ai

Abstract

We study communication between two frozen large language models from different providers, with different tokenizers, accessed through their API endpoints. The two play a referential game: one sees an object and describes it in a short fixed-length message over a small alphabet; the other must pick that object out of a candidate set. Neither model's weights are updated. Each agent's prompt is rewritten by an isolated prompt optimizer whose reflection model reads that agent's scored interactions. In the positional setting, optimized prompts carry a shared code that generalizes to held-out objects above a measured no-codebook baseline, including when the memory window is removed. In a second setting, independent per-letter blocks no longer fit within the message, although a whole-object place value code does. The base system fails to establish reliable communication: the sender struggles to retain an injective rule, and the receiver has too few confirmed examples in view. A sender collision penalty, retention of successful interactions, and sequential optimization enable successful place value communication in some runs. Outcomes vary across runs and reflection models. In successful runs, the protocol is written into the optimized prompts, where it can be read and audited directly.

Figures & tables

Figure 1: Two frozen LLMs communicate with a shared code over a constrained message channel. Rocky sees a hidden object, Grace an unlabelled candidate set, and one message passes between them. After Grace picks, the environment returns one shared reward bit to both of them. 3 3 3 Character art throughout this paper is OpenMoji, https://openmoji.org , CC BY-SA 4.0.

Panel of Table 1 — Table 1: Design choices in related communication studies. The columns compare frozen player weights, different model backbones for the two roles, and fixed agent architecture.

Figure 2: One round of the game. Rocky sees the object and emits a message over the message alphabet. Grace sees the message and an unlabelled candidate set and picks one from it. Rocky sends

m

and the environment returns

r

to both agents.

term	symbol	meaning	in the example
object alphabet	$\mathcal{L}$ , $\|\mathcal{L}\|=W$	the $W$ values a letter can take	b z n v k j q x
object	$o\in\mathcal{L}^{k}$	a string of $k$ letters, one of $W^{k}$ objects	zjvx
letter	$o_{i}\in\mathcal{L}$	the $i$ -th letter of an object	$o_{2}=$ j
message alphabet	$\mathcal{S}$ , $\|\mathcal{S}\|=V$	the $V$ symbols a message is written in	g m
message	$m\in\mathcal{S}^{l}$	a string of $l$ symbols, one of $V^{l}$ messages	ggmmgmgmmmmm
symbol	$m_{j}\in\mathcal{S}$	the $j$ -th symbol of a message	$m_{4}=$ m

Table 2: The lexicon of the game. One word and one symbol per part, used unchanged in every section, figure and formula. The sizes are in Table 3 and the agents’ parts in Table 4 .

symbol	meaning	PosGame (positional)	PVGame (place value)
$k$	letters per object	4	3
$W$	object alphabet size	8	6
$W^{k}$	objects	4096	216
$l$	symbols per message	12	8
$V$	message alphabet size	2	2
$N$	candidates shown	8	20

Table 3: The two games. One per code family, at the values every number in the paper uses. The PosGame admits a positional code exactly; the PVGame has no room for three independent fixed letter blocks.

Figure 3: The positional code on a real emitted pair. Each letter corresponds to 3 fixed message positions, so one confirmed pair can reveal up to

k

distinct letter facts. The message alphabet varies per random seed.

Figure 4: The place value code and why a positional code cannot work Top, the pair’s own rule, read from the settled Rocky’s committed rule where each letter is a base-6 digit, the three digits are one number, and that number is written as eight bits. Bottom, a positional code that needs nine bits for three letters yet the channel has only eight.

term	meaning
agent	Rocky, the sender, and Grace, the receiver
module	one typed DSPy signature an agent runs. The reasoner thinks under a token budget and answerer formats the answer
gene	one module’s instruction text and the only text an optimizer may rewrite
seed gene	the original gene a run starts from (Section 3.5 )
optimized gene	a gene after the optimizer has rewritten it
harness	the fixed text around a gene, which no optimizer may touch (Section 3.5 )

Table 4: The agent lexicon. One word per part of an agent, per step of changing one, and per unit of running the pair. The game’s own parts are in Table 2 .

Figure 5: Inside an agent. Rocky and Grace share the same anatomy which contains a bounded reasoner, a tiny answerer, a round log with up to

D

selected rows injected into the memory window, and the agent’s own GEPA job. The dashed boxes mark the optimizer and window, and the numbered slots are role dependent.

	memory off (nomem)	memory on (mem)
seed genes	no-prompt-opt@nomem neither knob	no-prompt-opt@mem memory alone
optimized genes	prompt-opt@nomem the optimizer alone	prompt-opt@mem optimizer and memory
re-evaluation	prompt-opt-both@nomem prompt-opt@mem’s genes without the memory

Table 5: The two knobs and the five cells. Memory on or off or prompt optimization on or off, with

D=19

. Each of the four cells is one arm and the fifth reuses the prompt-opt@mem’s optimized genes with the memory removed.

Figure 6: The two harnesses in each agent Each module consists of both its gene and its harness (in grey). The reasoner’s prompt carries the two slots with state, the memory window and last round’s rule while the answerer’s prompt merely carries the analysis.

Figure 7: The optimizer’s harness. Left, the reflection prompt, DSPy’s fixed text with the two coloured slots this system fills. Right, one scored rollout in the minibatch slot. Bottom, GEPA evaluates the returned candidates through its validation procedure.

	window off (nomem)	window on (mem)
seed genes	no-prompt-opt@nomem 0.333	no-prompt-opt@mem 0.246
optimized genes	prompt-opt@nomem 0.451	prompt-opt@mem 0.892
re-evaluation	prompt-opt-both@nomem 0.906

Table 6: The evaluation grid in the PosGame . Three evaluation draws on 99 held-out objects per cell, chance 0.125, ceiling 0.589. The fifth cell re-evaluates the prompt-opt@mem arm’s genes with the window off.

metric	this code	perfect code	fraction	reading
topsim	0.472	0.498	0.95	0.95 of the perfect code on these objects
posdis	0.853	1	0.85	each position is mostly about one letter
poscon	0.95	1	0.95	the usual block is used for 95 percent of objects

Table 7: Structure of the emitted positional code. Each metric beside what the perfect code scores and the fraction reached. The perfect code’s posdis is taken on the whole object space, so that fraction is a floor.

	window off (nomem)	window on (mem)
seed genes	no-prompt-opt@nomem 0.041	no-prompt-opt@mem 0.024
optimized genes	prompt-opt@nomem 0.048	prompt-opt@mem 0.088
re-evaluation	prompt-opt-both@nomem 0.112

Table 8: The same grid in the PVGame . Three evaluation draws on 98 held-out objects per cell, chance 0.050, ceiling 0.056. Accuracy at or under the baseline alone does not establish code acquisition; the best cell is approximately twice that baseline.

the rule slot holds	accuracy	reading
nothing	0.060	what play provides, chance
Rocky’s own encode rule	0.968	derives the inverse itself

Table 9: The handed-rule probe. Rerunning the game with the seed gene with an empty window and only the rule slot changed. On the same rounds the same Grace in the game scores 0.043.

confirmed pairs	rules still possible, median	rules still possible, mean	draws where one rule is left
1	36	49	0 percent
2	2	3.5	18 percent
3	1	1.5	69 percent

Table 10: How many confirmed pairs identify a place value rule. A rule is draw from all 5760, that many objects are and the rules consistent with all of them are counted, over 400 draws.

the window holds	measured
confirmed pairs, mean	1.17
confirmed pairs, median	1
windows with no confirmed pair	31 percent
windows with three or more	15 percent

Table 11: What a 19-row window holds. Measured over the 471 play rounds of the Experiment 2 prompt-opt@mem arm, with 30 wins.

Rocky’s genes from	pvfit	colliding objects, of 216	held-out accuracy, window off
the prompt-opt@nomem arm	0.750	72	0.048
the prompt-opt@mem arm	0.028	124	0.112

Table 12: Rocky’s rule against the pair’s accuracy. pvfit and the colliding objects are read on Rocky’s optimized genes over all 216 objects. The accuracy is each arm’s genes with the window off, the prompt-opt@nomem and prompt-opt-both@nomem cells of Table 8 .

Figure 8: The collision auxiliary, as one scoring batch. The third item reuses the first’s message for a different object, so its score is docked

\lambda

even though its play reward was a lucky 1.

Rocky

condition attempts pvfit colliding held-out accuracy, window off auxiliary off 1 0.059 36 0.041 (n = 1) auxiliary on, settled 4 0.939 4.5 0.098 (n = 4) auxiliary on, failed 3 0.055 23.3 0.054 (n = 3)

Table 13: Rocky’s injective code Every attempt is the Rocky stage of a prompt-opt@memwins with window off before Grace follows.

Figure 9: The same window under the two retention rules. Under fifo the losses flood the window and the pairs scroll out while keep-wins evicts a losing row first, with nothing else changed.

Grace

paired run Grace’s window wins pairs in view accuracy pair 1, seed 7 no-prompt-opt@memwins 43 13.2 0.145 no-prompt-opt@mem 33 1.6 0.091 pair 2, seed 8 no-prompt-opt@memwins 42 15.1 0.096 no-prompt-opt@mem 22 1.1 0.055

Table 14: Keep-wins against fifo in two paired runs. 400 rounds of play behind the frozen settled Rocky of Section 4.4.1 and Grace on his seed gene with no optimizer, one dial apart. Chance 0.050.

Rocky Grace

outcome play, last 600 rounds held-out, window off times the ceiling the gene holds partial (part of the rule) 0.272 (n = 9) 0.141 (n = 9) 2.5 the first position; the rest guessed full (the whole rule) 0.930 (n = 1) 0.878 (n = 1) 15.7 the whole rule full, Sol reflection 0.893 (n = 2) 0.873 (n = 4) 15.6 the whole rule

Table 15: Optimizing Grace After Rocky: We optimize Rocky first and once it settles, optimize Grace over 1200 rounds. Chance is 0.050 against the no codebook ceiling of 0.056.

Panel of Figure 10 — Figure 10: The partial and the full outcome in the genes’ own words. Excerpts of the optimized genes of the seed-16 partial run and of the full run, verbatim with elisions marked. The full gene is reproduced whole in Section B . Both derive the six-value alphabet but only the full gene does Base-6 encoding.

Grace

window off (nomem) window on (memwins) seed genes no-prompt-opt@nomem 0.068 default 0.056 Sol (best 0.061, n = 4) no-prompt-opt@memwins 0.041 default optimized genes prompt-opt@nomem 0.078 default prompt-opt@memwins 0.936 default 0.850 Sol (best 0.878, n = 4) re-evaluation prompt-opt-both@nomem 0.878 default 0.873 Sol (best 0.915, n = 4)

Table 16: The signature grid in the PVGame , over every full run. The default reflection model’s one run beside the four under Sol, on the held out set with Sol’s best run and count in brackets. Chance 0.050, ceiling 0.056.

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

run	what moved	Rocky settled (pvfit)	pair, window off
role swap, seed 7	V4-Flash sends, GLM-5 receives	no (0.113)	void
role swap, seed 8	V4-Flash sends, GLM-5 receives	no (0.045)	void
tier replacement, seed 7	V4-Pro sends, GLM-5 receives	yes (1.000, no colliding objects)	0.136
reflection swap, seed 19	Sol reflects for Grace, behind the frozen 0.977 Rocky	warm-started	0.854
reflection swap, seed 20	the same	warm-started	0.867
reflection swap, seed 22	the same	warm-started	0.915

Appendix

Table 17: The place value swaps. Each against the default pair, GLM-5 sending and DeepSeek-V4-Flash receiving; the pair’s cell is prompt-opt-both@nomem, three evaluation draws on 98 held-out objects, ceiling 0.056. A Grace read behind a Rocky that did not settle is void, so the role-swap rows carry no cell.

Emergent Communication

Emergent communication studies how agents develop shared signals or languages through interaction to coordinate, convey information, and build conventions.

See all 30 papers

RLMOpt: Adaptive Prompt Optimization via Recursive Language Models
Aug 11, 2026Subhash Bangalore Satheesha, Nirvik Pande, Deepthi Duddempudi +1Large Language Model-Guided Optimization LLM Agents
Autonomize AI · Carnegie Mellon University
Mirror, Mirror on the Wall: Prompt Echoing in Small Instruct Language Models
Sep 14, 2026Inez Okulska, Bartosz Naskręcki, Jan Piotrowski +1Data Contamination in Language Models Induction Heads
Centre for Credible AI, Warsaw University of Technology · Adam Mickiewicz University Poznan · Institute of Fundamental Technological Research, Polish Academy of Sciences
Share the Symbol, Not the Vector: Training-Free Fact Communication Across Language Models
Oct 14, 2026Xin Li, Mengbing Liu, Hao Jiang +5Multi-LLM Collaboration Multi-Agent LLM Systems
Nanyang Technological University · Yale University · University of Manchester

Aug 11, 2026J/K move · Enter open · S save

RLMOpt: Adaptive Prompt Optimization via Recursive Language Models

Subhash Bangalore Satheesha, Nirvik Pande, Deepthi Duddempudi, Bharath Dandala

Autonomize AI · Carnegie Mellon University

Large Language Model-Guided Optimization LLM Agents Prompt Optimization Recursive Language Models LLM Input Optimization

Open paper AskPDF arXiv

Prompt optimizers automate the search for prompts that improve language-model performance, but existing methods rely on a predefined optimization procedure: the algorithm determines which candidates to explore and how the search progresses, while the language model generates or refines prompt proposals. We introduce RLMOpt, a prompt optimizer that makes the search policy itself language-model-driven through a recursive language model (RLM). The RLM agent operates over a tool-based environment, inspecting task information, analyzing failures, generating candidates, allocating evaluation budget, and deciding when to stop. A deterministic harness complements the agent by enforcing objective scoring, Pareto-based selection, and regression constraints. We evaluate RLMOpt across four benchmarks spanning structured clinical information extraction (Chia), multi-hop question answering (HotpotQA), verifiable instruction following (IFBench-2025), and multi-turn tool-calling agents (BFCL). In a matched comparison at a single seed, RLMOpt obtains the best held-out score on all four benchmarks and leads the four-task mean (0.610 against 0.589 for GEPA). Repeating each benchmark across seeds yields 11 matched benchmark-seed comparisons, in which RLMOpt outperforms GEPA in 9 cases. Across all 11 runs, it never produced a prompt that underperformed its seed, whereas GEPA fell below its starting point twice. It is also more efficient, achieving these results with fewer search rollouts while producing prompts that are 27-79% the size of those produced by GEPA. Our results further show that optimization gains are determined primarily by the headroom available in the seed prompt, rather than by the search budget. Efficient optimization therefore depends on reaching the available headroom reliably and with minimal search

Abstract

Emergent Communication

Explore similar work

RLMOpt: Adaptive Prompt Optimization via Recursive Language Models

Mirror, Mirror on the Wall: Prompt Echoing in Small Instruct Language Models

Share the Symbol, Not the Vector: Training-Free Fact Communication Across Language Models