MEMO: Memory-Augmented Model Context Optimization for Robust Multi-Turn Multi-Agent LLM Games

Yunfei Xie; Kevin Wang; Bobby Cheng; Jianzhu Yao; Zhizhou Sha; Alexander Duffy; Yihan Xi; Hongyuan Mei

cs.AIMar 9, 2026

MEMO: Memory-Augmented Model Context Optimization for Robust Multi-Turn Multi-Agent LLM Games

Authors: Yunfei Xie, Kevin Wang, Bobby Cheng, Jianzhu Yao, Zhizhou Sha, Alexander Duffy, Yihan Xi, Hongyuan Mei,

Organizations: Rice University · The University of Texas at Austin · A*STAR · Princeton University ·

Abstract

Multi-turn, multi-agent LLM game evaluations often exhibit substantial run-to-run variance. In long-horizon interactions, small early deviations compound across turns and are amplified by multi-agent coupling. This biases win rate estimates and makes rankings unreliable across repeated tournaments. Prompt choice worsens this further by producing different effective policies. We address both instability and underperformance with MEMO (Memory-augmented MOdel context optimization), a self-play framework that optimizes inference-time context by coupling retention and exploration. Retention maintains a persistent memory bank that stores structured insights from self-play trajectories and injects them as priors during later play. Exploration runs tournament-style prompt evolution with uncertainty-aware selection via TrueSkill, and uses prioritized replay to revisit rare and decisive states. Across five text-based games, MEMO raises mean win rate from 25.1% to 49.5% for GPT-4o-mini and from 20.9% to 44.3% for Qwen-2.5-7B-Instruct, using $2,000$ self-play games per task. Run-to-run variance also drops, giving more stable rankings across prompt variations. These results suggest that multi-agent LLM game performance and robustness have substantial room for improvement through context optimization. MEMO achieves the largest gains in negotiation and imperfect-information games, while RL remains more effective in perfect-information settings. All code is open-source and available here: https://github.com/openverse-ai/MEMO

Figures & tables

Panel of Figure 1 — Figure 1: Left Run-to-run performance and stability comparison. Using GPT-4o-mini with MEMO achieves the highest mean win rate (49.5%) with the lowest RSE (6.4%). Right Learning efficiency comparison against the self-play RL baseline method Unstablebaseline . Using Qwen2.5-7B-Instruct, MEMO reaches 60% win rate on Kuhn Poker with only 2,000 games, 19 $\times$ fewer than the 38,000 games required by the RL self-play baseline.

Figure 2: Three paradigms for learning in multi-agent LLM games. (a) Prompt optimization updates the system prompt each round through self-play, but game experience is not effectively retained across rounds, so strategic insights are lost across rounds. (b) Reinforcement learning (RL) updates model weights through self-play but relies on outcome rewards, requiring large sample budgets. (c) MEMO reflects on completed trajectories and accumulates reusable insights in a persistent memory bank across generations, enabling improvement without weight updates or external reward.

Figure 3: The MEMO Framework . At each optimization generation, new candidate contexts are proposed through two strategies: random proposals and memory-augmented updates. These candidates are then evaluated via self-play, and the best-performing candidates are used to update the pool for the next generation. To encourage exploration and mitigate redundant early moves, a prioritized replay module is introduced, enabling efficient search for robust prompts and priors within a single game.

Optimizer	SimpNeg	Kuhn	SimpTak	Avg.
TextGrad	842	986	938	922
MIPRO	145,864	162,084	754,534	354,161
GEPA	110,325	119,365	111,907	113,865
MEMO	87,364	94,160	89,152	90,575

Table 1: Output token cost per prompt optimization method across three games.

Type	Optimizer	Negotiation		Imperfect Info		Perfect Info	Mean Win Rate	Mean RSE
Type	Optimizer	SimpleNegotiation	TwoDollar	KuhnPoker	Briscola	SimpleTak	Mean Win Rate	Mean RSE
GPT-4o-mini
Static	baseline	31.3%	32.2%	39.1%	0.3%	21.4%	25.1%	44.9%
	CoT	27.8%	25.7%	46.5%	30.4%	24.8%	31.1%	28.7%
	ToT	26.3%	27.0%	51.7%	45.1%	23.8%	34.8%	36.5%
Prompt	TextGrad	42.0%	44.6%	55.6 %	7.1%	23.6%	34.6%	18.4%

Table 2: Benchmark results for different approaches using GPT-4o-mini and Qwen2.5-7B-Instruct across multiple tasks. Each win rate is the mean across three evaluation models. Type denotes the optimization paradigm: Static prompting, Prompt optimization, Reinforcement learning (RL), and our method. For full model-wise results, see Appendix H.

Modules			Win Rate			Summary
Tournament	Mem	Replay	TwoDollar	KuhnPoker	Briscola	Mean	$\boldsymbol{\Delta_{\text{base}}}$
			32.2%	39.1%	0.3%	23.8%	–
$\checkmark$			24.7%	54.7%	2.0%	27.1%	+3.3
	$\checkmark$		34.2%	42.0%	26.3%	34.2%	+10.4
$\checkmark$		$\checkmark$	32.0%	54.2%	38.7%	41.6%	+17.8
$\checkmark$	$\checkmark$		48.7%	57.2%	38.4%	48.1%	+24.3

Table 3: GPT-4o-mini ablation experiments comparing combinations of Tournament-based context optimization, Memory bank, and Replay modules. Rows shaded indicate configurations that include the Memory bank. The first row shows the baseline without any optimization.

Training Game	Negotiation		Imperfect Info		Perfect Info	Mean Win Rate
Training Game	SimpleNegotiation	TwoDollar	KuhnPoker	Briscola	Simpletak	Mean Win Rate
GPT-4o-mini
SimpleNegotiation	46.9% (+15.6%)	37.8% (+5.6%)	48.9% (+9.8%)	0.0% (-0.3%)	37.7% (+16.3%)	34.3% (+9.4%)
TwoDollar	31.1% (-0.2%)	48.7% (+16.5%)	53.3% (+14.2%)	1.1% (+0.8%)	47.8% (+26.4%)	36.4% (+11.5%)
KuhnPoker	31.1% (-0.2%)	34.4% (+2.2%)	57.2% (+18.1%)	22.2% (+21.9%)	30.0% (+8.6%)	35.0% (+10.1%)
Briscola	38.9% (+7.6%)	27.8% (-4.4%)	57.8% (+18.7%)	38.4% (+38.1%)	14.3% (-7.1%)	35.4% (+10.6%)

Table 4: Generalization across task. Columns denote the source game where MEMO learns context through self-play, while rows indicate target games where the learned context is evaluated zero-shot . Each entry reports win rates averaged over 50 independent matches.

Figure 4: Transferred GPT-4o-mini context benefits weaker models uniformly but yields mixed results for stronger ones. Per-game win rates with and without the learned context for Grok-4-Fast-Non-Reasoning (left) and Gemini-2.5-Flash-Lite (right).

Appendix figures & tables26 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 5: Ranking sensitivity in KuhnPoker . With environment and evaluator pools fixed, five nearly equivalent prompt variants still flip pairwise outcomes and reshuffle rankings. The heatmap shows Kendall’s $\tau_{b}$ for every pair of prompts: blue indicates similar rankings ( $\tau_{b}\approx 1$ ), white indicates unstable rankings ( $\tau_{b}\approx 0$ ), and orange indicates rank reversals ( $\tau_{b}<0$ ).

Panel of Figure 6 — Figure 6: KuhnPoker Prompt Variant 1

Panel of Figure 7 — Figure 7: KuhnPoker Prompt Variant 2

Panel of Figure 8 — Figure 8: KuhnPoker Prompt Variant 3

Panel of Figure 9 — Figure 9: KuhnPoker Prompt Variant 4

Panel of Figure 10 — Figure 10: KuhnPoker Prompt Variant 5

Figure 11: Ablation studies of experience initialization and replay hyperparameters. Each subplot varies a single parameter while holding the others fixed. The first three panels show TwoDollar replay ablations over buffer size $B$ , priority exponent $\alpha$ , and replay gate $\beta$ . The rightmost panel shows the effect of the experience initialization fraction $\pi$ on TwoDollar. Vertical dotted lines indicate the hyperparameter values used in all other experiments.

Table 16

Panel of Figure 12 — Figure 12: Trajectory Reflection Prompt Template. The model receives a strategically decisive state with its outcome statistics and extracts actionable insights that summarize lessons learned.

Panel of Figure 13 — Figure 13: Memory Operation Prompt Template. The model compares new insights against the existing memory bank and applies add, edit, or remove operations to maintain a coherent and non-redundant library of strategic insights.

Panel of Figure 14 — Figure 14: KuhnPoker Game Starting Prompt

Panel of Figure 15 — Figure 15: Briscola Game Starting Prompt

Panel of Figure 16 — Figure 16: SimpleNegotiation Game Starting Prompt

Panel of Figure 17 — Figure 17: TwoDollar Game Starting Prompt

Panel of Figure 18 — Figure 18: SimpleTak Game Starting Prompt

Panel of Figure 19 — Figure 19: Text-based loss template for Textgrad

Panel of Figure 20 — Figure 20: MIPRO scoring standard

Panel of Figure 21 — Figure 21: GEPA scoring standard

Optimizer	SimpleNegotiation	KuhnPoker	SimpleTak	Avg. Tokens
Textgrad	842	986	938	922
MIPRO	145,864	162,084	754,534	354,161
GEPA	110,325	119,365	111,907	113,865
MEMO (Ours)	87,364	94,160	89,152	90,575

Appendix

Table 11: Output token cost for each prompt optimization method (exact counts).

Optimizer	Negotiation		Imperfect Info		Perfect Info	Mean Win Rate	Mean RSE
Optimizer	SimpleNegotiation	TwoDollar	KuhnPoker	Briscola	SimpleTak	Mean Win Rate	Mean RSE
GPT-4o-mini
baseline	31.3%	32.2%	39.1%	0.3%	21.4%	25.1%	44.9%
MEMO (Ours)	54.9%	52.4%	55.6%	42.7%	41.8%	49.5%	6.4%
Qwen2.5-7B-Instruct
baseline	24.0%	17.1%	45.3%	2.8%	15.1%	20.9%	30.1%

Appendix

Table 14: Benchmark results for baseline and MEMO across multiple tasks. Each win rate is the mean across three evaluation models (Sec. 4.2 ). For per-opponent breakdown, refer to Appendix 13 .

Panel of Figure 22 — Figure 22: Kuhn Poker strategic insights. These insights encode betting principles that balance aggression with hand strength, helping agents avoid predictable play patterns while maximizing expected value.

Panel of Figure 23 — Figure 23: Briscola strategic insights. These insights capture the timing and resource allocation principles for trump cards, enabling agents to maximize point capture rather than using high-value cards indiscriminately.

Panel of Figure 24 — Figure 24: Simple Negotiation insights. These insights reveal that players have asymmetric resource valuations, a concept not explicitly stated in the game description, and encourage proactive information gathering before committing to offers.

Panel of Figure 25 — Figure 25: Two Dollars insight. This insight captures a negotiation tactic that exploits the finite round structure, encouraging agents to use time pressure as a persuasion mechanism.

Figure 26: In this example (Fig. 26 ), the agent plays $\times$ , while the opponent ( $\circ$ ) is one move away from victory. The prompt on the right is learned via self-play in Kuhn Poker and distills transferable behaviors, including opponent modeling, general strategic principles, and strict output-format constraints. Conditioned on this prompt, the agent handles the scenario more reliably; in this short-horizon case, the adapted prompt enables the agent to identify the correct blocking move and prevent the opponent’s immediate win.

Panel of Figure 27 — Figure 27: Simple Negotiation Prompt Transfer to Simple Tak. Although the updated prompt was not trained on Simple Tak, it encourages the model to explicitly reason from the opponent’s perspective during its thought process, resulting in more consistent and reliable performance compared to the basic starting prompt.

Multi-Agent LLM Systems

Multi-agent large language model systems coordinate multiple AI agents to solve tasks through communication, specialization, shared tools, and memory.

See all 1,039 papers

From Player to Master: Enhancing Test-Time Learning of LLM Agents via Reinforcement Learning over Memory
Jun 7, 2026Yishuo Cai, Xingyu Guo, Xuancheng Huang +8Continual Learning for LLM Agents LLM Agent Memory
Peking University, Beijing, China · Central South University, Changsha, China · Zhipu AI, Beijing, China +3
MemTrain: Self-Supervised Context Memory Training
Jun 2, 2026Ziheng Li, Xingrun Xing, Haoqing Wang +2LLM Agent Memory Persistent Memory for Language Models
State Key Laboratory of General Artificial Intelligence, School of Intelligence Science and Technology, Peking University · Samsung Research, Beijing, China
Towards Autonomous Memory Agents
Feb 25, 2026Xinle Wu, Rui Zhang, Mustafa Anis Hussain +1Memory-Augmented Language Model Agents Memory Management for LLM Agents
National University of Singapore, Singapore

Jun 7, 2026J/K move · Enter open · S save

From Player to Master: Enhancing Test-Time Learning of LLM Agents via Reinforcement Learning over Memory

Yishuo Cai, Xingyu Guo, Xuancheng Huang, Jinhua Du+7

Peking University, Beijing, China · Central South University, Changsha, China · Zhipu AI, Beijing, China+3

Continual Learning for LLM Agents LLM Agent Memory Test-Time RL Test-Time Training RL for LLM Agents

Open paper AskPDF arXiv

Large language model (LLM) agents are increasingly deployed in long-running settings where improving through experience at test time becomes important. A common approach is to update an explicit memory after each interaction to guide future decisions. However, most existing methods rely on hand-designed prompting rules, making it difficult to align memory updates with downstream objectives over multi-step horizons consistently. We propose MemoPilot, a plug-in memory copilot that explicitly trains the memory update process to improve a frozen LLM's performance across sequential interactions. We formulate memory updating as a multi-turn decision problem and optimize it end-to-end with multi-turn GRPO. Our training recipe introduces (i) a turn-wise reward signal and (ii) a context-independent, turn-level advantage estimation across rollouts, enabling finer-grained credit assignment and more stable training in multi-turn settings. We evaluate MemoPilot on two testbeds: multi-round Rock-Paper-Scissors (RPS) and Limit Texas Hold'em (LHE). Across both environments, MemoPilot substantially improves test-time learning of a frozen player over strong baselines, ranking first in Elo ratings on both games (1762 on LHE and 1590 on RPS) and outperforming all baseline memory methods and proprietary models, including DeepSeek-V3.2.

Abstract

Multi-Agent LLM Systems

Explore similar work

From Player to Master: Enhancing Test-Time Learning of LLM Agents via Reinforcement Learning over Memory

MemTrain: Self-Supervised Context Memory Training

Towards Autonomous Memory Agents