Evolving Memory, Learning to Meta-Control: Test-Time Adaptation for Experience-augmented LLM Agents

Zeyuan Zhao; Yuanyuan Shi; Peng Yu; Lingyue Fu; Shao Zhang; Ying Wen

cs.LGNov 1, 2026

Evolving Memory, Learning to Meta-Control: Test-Time Adaptation for Experience-augmented LLM Agents

Authors: Zeyuan Zhao, Yuanyuan Shi, Peng Yu, Lingyue Fu, Shao Zhang, Ying Wen

Organizations: Shanghai Jiao Tong University · Shanghai Artificial Intelligence Laboratory

Abstract

Self-evolving agents accumulate experience from task streams, but existing frameworks typically rely on fixed pipelines to retrieve and utilize this experience, neglecting when and how to deploy it. We introduce Online Meta-Control of Experience (OMCE), a framework that updates experience memory and learns a meta-control policy from task feedback during test-time adaptation. OMCE employs a lightweight contextual-bandit meta-controller that uses compact retrieval signals to select among four cognitive actions: DIRECT, EPISODE, GUIDE, and DISTILL. This routing mechanism operationalizes three coupled decisions: whether to invoke experience, which memory format to utilize, and when to consolidate new guidance. OMCE evolves online, leveraging task feedback to update the memory bank and refine the routing policy via contextual bandit updates. Experiments on four online adaptation benchmarks across four backbones show that OMCE improves first-attempt success by up to 20.5 percentage points over the strongest compared baseline, while reducing per-task token use by a factor of 5.7 to 62.5 relative to ACE with GPT-5.2. Controlled ablations further support the benefits of learned routing and continued routing policy adaptation.

Figures & tables

Figure 1: From Fixed Pipelines to Learned Meta-Control. Up: Existing methods typically follow a fixed routine for using and consolidating experience, even when the retrieved experience does not help the current task. Down: OMCE learns from task feedback when to use experience, whether to draw on past cases or general guidance, and when to distill new guidance.

Figure 2: Architecture of OMCE . The lightweight Meta-Controller selects a cognitive action specifying whether and which experience to inject into the task prompt. The agent solves the task with the augmented context containing the query and any injected experience. Memory evolution records outcomes in episodic memory, distills guidance before execution when DISTILL is selected, and refines injected guidance afterward using task feedback.

Benchmark	Model	OSR $\uparrow$ / CSR $\uparrow$
		Base LLM	Reflexion	MemP	ReasoningBank	ACE	OMCE -TS	OMCE -UCB
Multi- Challenge	GPT-5.2	52.8% / 70.1%	- / 98.2%	50.2% / -	53.9% / 66.3%	48.7% / 71.4%	67.4% / 97.1%	66.3% / 95.6%
	Deepseek-V3.2	43.1% / 55.7%	- / 91.6%	46.9% / -	48.4% / 56.0%	50.2% / 70.0%	70.7% / 95.6%	67.4% / 96.7%
	Deepseek-V4-pro	49.5% / 63.0%	- / 90.1%	48.0% / -	55.3% / 66.3%	50.2% / 74.7%	69.6% / 93.8%	64.8% / 91.2%
	Kimi-K2.5	49.8% / 64.8%	- / 89.0%	50.9% / -	56.7% / 72.8%	49.8% / 70.7%	72.2% / 95.6%	73.3% / 96.7%
BBEH	GPT-5.2	33.3% / 49.7%	- / 83.9%	31.9% / -	31.3% / 37.7%	52.4% / 94.4%	53.3% / 78.7%	46.8% / 71.0%

Table 1: Main results under the online adaptation setting. Each entry is reported as OSR / CSR (%), where CSR is measured within

P=4

attempts. For Reflexion, “-” denotes that no first-attempt evaluation applies, as explained in the metric definitions above.

Benchmark	OSR $\uparrow$ / CSR $\uparrow$
Benchmark	Fixed- $a_{\textsc{EPISODE}}$	Fixed- $a_{\textsc{GUIDE}\&\textsc{DISTILL}}$	OMCE -HEU	OMCE -TS	OMCE -UCB
MultiChallenge	56.0% / 91.2%	63.7% / 94.5%	64.5% / 95.2%	67.4% / 97.1%	66.3% / 95.6%
BBEH	50.3% / 76.8%	43.1% / 73.8%	45.6% / 71.7%	53.3% / 78.7%	46.8% / 71.0%
BigCodeBench	49.0% / 59.0%	49.6% / 58.0%	47.4% / 57.4%	49.6% / 58.9%	51.3% / 59.2%
SWE-bench Verified	64.2% / 64.8%	64.4% / 64.8%	64.0% / 64.8%	64.8% / 64.8%	64.8% / 64.8%

Table 2: Routing ablation of OMCE on GPT-5.2, OSR / CSR (%). Fixed-

a_{\textsc{EPISODE}}

and Fixed-

a_{\textsc{GUIDE}\&\textsc{DISTILL}}

always commit to one form of experience; OMCE -HEU selects actions with hand-crafted rules. Appendix D reports the same comparison for the remaining backbones.

			Online memory evolution		Shared memory snapshot
Benchmark	Variant	OMCE	Always-Both-and-Update	Frozen Controller	Always-Both	Random
MultiChallenge	OMCE -TS	67.4% / 97.1%	63.7% / 95.2%	61.9% / 94.1%	63.0% / 83.5%	62.6% / 83.9%
BBEH	OMCE -TS	53.3% / 78.7%	51.1% / 77.6%	49.8% / 75.6%	49.7% / 71.8%	48.2% / 67.4%
BigCodeBench	OMCE -UCB	51.3% / 59.2%	48.3% / 53.5%	49.5% / 65.4%	47.5% / 55.6%	46.5% / 57.0%
SWE-bench Verified	OMCE -TS	64.8% / 64.8%	55.0% / 55.2%	64.2% / 64.6%	48.8% / 49.2%	53.0% / 53.4%

Table 3: Controlled analysis on GPT-5.2, OSR / CSR (%). For each benchmark, OMCE uses the variant with the higher GPT-5.2 OSR in Table 1 (TS on ties). Frozen Controller stops controller updates mid-stream while memory keeps evolving; Always-Both and Random read one shared memory snapshot without updating it.

Figure 3: Performance–efficiency trade-off on four benchmarks using GPT-5.2. The x-axis represents the one-shot success rate, and the y-axis reports the token cost in log scale (average total prompt and output tokens consumed per task).

Figure 4: Self-correction detection in MultiChallenge. The meta-controller routes the query to GUIDE (

a_{\textsc{GUIDE}}

) rather than to episodic cases that enforce the user’s outdated constraint.

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Benchmark	Model	OSR $\uparrow$ / CSR $\uparrow$
		Fixed- $a_{\textsc{EPISODE}}$	Fixed- $a_{\textsc{GUIDE}\&\textsc{DISTILL}}$	OMCE -HEU	OMCE -TS	OMCE -UCB
Multi- Challenge	GPT-5.2	56.0% / 91.2%	63.7% / 94.5%	64.5% / 95.2%	67.4% / 97.1%	66.3% / 95.6%
	Deepseek-V3.2	68.9% / 97.4%	70.3% / 97.4%	69.2% / 97.4%	70.7% / 95.6%	67.4% / 96.7%
	Deepseek-V4-pro	65.6% / 91.9%	67.0% / 95.2%	66.3% / 93.8%	69.6% / 93.8%	64.8% / 91.2%
	Kimi-K2.5	73.3% / 96.7%	72.2% / 96.7%	72.2% / 96.7%	72.2% / 95.6%	73.3% / 96.7%
BBEH	GPT-5.2	50.3% / 76.8%	43.1% / 73.8%	45.6% / 71.7%	53.3% / 78.7%	46.8% / 71.0%

Appendix

Table 4: Full routing ablation of OMCE on four backbones, OSR / CSR (%). Fixed-

a_{\textsc{EPISODE}}

and Fixed-

a_{\textsc{GUIDE}\&\textsc{DISTILL}}

always commit to one form of experience; OMCE -HEU selects actions with hand-crafted rules. The GPT-5.2 block is reproduced from Table 2 for reference.

Quantity	Value
Retrieval count $M$	3
Meta-state dimension $d$	9
Forgetting factor $\gamma$	0.98
Ridge regularization $\lambda$	0.5
LinUCB coefficient $\alpha$	3.0
TS posterior scale $v^{2}$	0.25

Appendix

Table 5: Shared controller and retrieval configuration used by OMCE -TS and OMCE -UCB.

Panel of Figure 5 — Figure 5: Performance–efficiency trade-off on four benchmarks using GPT-5.2, Deepseek-V3.2, Deepseek-V4-pro, and Kimi-K2.5 as backbone LLMs. The x-axis shows one-shot success rate, and the y-axis shows token cost (log scale), defined as the average total number of prompt and output tokens consumed per task.

Pattern & Action	Task Context	Baseline / Wrong Memory	OMCE Behavior & Outcome
Memory Harmful $a_{\textsc{DIRECT}}$	ID: django-16560 Objective: Add violation_error_code to BaseConstraint . Challenge: Prone to wrong-layer patch trap.	$a_{\textsc{GUIDE}}$ : Misled by irrelevant guidance entries, applying patches to the wrong abstraction layer. ✗Failure	Reasoning: Accurately identifies low relevance (confidence 1.0) and avoids hallucinated guidance entries. ✓Success
Type Selection $a_{\textsc{GUIDE}}$	ID: django-13401 Objective: Ensure abstract model fields do not compare as equal across different models. Challenge: Requires adjusting eq , hash , and lt to consider field.model instead of just creation_counter .	$a_{\textsc{EPISODE}}$ : Fails to generalize from specific past case-specific insights, leading to incomplete or incorrect equality logic. ✗Failure	Reasoning: Recognizes that high-level guidance entries provides better architectural instruction for core equality semantics than case-specific insights. ✓Success
DISTILL $a_{\textsc{DISTILL}}$	ID: django-13809 Objective: Add --skip-checks option to the runserver command. Challenge: Requires correctly wiring the new option through the command’s argument parser and execution flow.	$a_{\textsc{DIRECT}}$ : Fails to identify the correct modification layer and misses necessary execution flow updates. ✗Failure	Reasoning: Finds highly similar case-specific insights but no directly applicable guidance entries; distills new guidance entries to identify the correct modification layer. ✓Success

Appendix

Table 6: Qualitative case studies demonstrating the Meta-Controller’s dynamic action selection on SWE-bench Verified.

LLM Agents

Large language model agents use language models to plan, use tools, and act across multi-step tasks and environments.

See all 730 papers

Memory as a Controlled Process: Learned Adaptive Memory Management for LLM Agents
Jul 15, 2026Eric Hanchen Jiang, Zhi Zhang, Yuchen Wu +11Persistent Memory for Language Models Memory Management for LLM Agents
University of California Los Angeles · University of Washington · Northwestern University
ASCENT: Online Test-Time Training of Long-Horizon Agents via Self-Distillation of Verified Experience
Oct 4, 2026Haodong Lu, Dong GongContinual Learning for LLM Agents Test-Time Training
University of New South Wales (UNSW Sydney)
Hippo: Training-Free Agent Self-Evolution via Hippocampal Replay
Oct 17, 2026Yutong Song, Jiang Wu, Yuxuan Ren +2Memory Consolidation for LLM Agents LLM Agent Memory
University of California, Irvine · Independent Researcher · Rice University +1

Jul 15, 2026J/K move · Enter open · S save

Memory as a Controlled Process: Learned Adaptive Memory Management for LLM Agents

Eric Hanchen Jiang, Zhi Zhang, Yuchen Wu, Levina Li+10

University of California Los Angeles · University of Washington · Northwestern University

Persistent Memory for Language Models Memory Management for LLM Agents Agent Memory Management Memory-Augmented Language Model Agents

Open paper AskPDF arXiv

Large Language Model (LLM) agents increasingly rely on external memory systems to accumulate experience across tasks. Yet nearly all existing approaches, from graph-structured memories to reflective insight stores, access memory through fixed, hand-designed heuristics. We argue that this static view of memory is a core bottleneck for agentic learning because optimal memory behavior is fundamentally context-dependent. The early stages of the tasks, benefit from minimal retrieval because memory is sparse; recurring goal types benefit from plan reuse rather than generic nearest-neighbor lookup; stuck agents benefit from re-retrieval with alternative queries; and across long task streams, the memory store itself must be consolidated and pruned to remain useful. We present Memory as a Controlled Process (MemCon), a framework that models memory operations as a Markov Decision Process and learns an online policy that adaptively decides when, what, and how much to retrieve, when to inject a distilled plan, and when to consolidate or forget. MemCon is backend-agnostic: it wraps any existing memory implementation, learns from task-by-task binary feedback with no pretraining and no additional LLM calls, and uses a lightweight tabular contextual bandit with UCB exploration that converges within tens of tasks. Across 6 benchmarks, 3 agent frameworks, and 3 LLM backbones, MemCon consistently outperforms multiple memory baselines by up to 15.2 points in task success while reducing token consumption by 5--20%.

Abstract

LLM Agents

Explore similar work

Memory as a Controlled Process: Learned Adaptive Memory Management for LLM Agents

ASCENT: Online Test-Time Training of Long-Horizon Agents via Self-Distillation of Verified Experience

Hippo: Training-Free Agent Self-Evolution via Hippocampal Replay