Self-evolving agents accumulate experience from task streams, but existing frameworks typically rely on fixed pipelines to retrieve and utilize this experience, neglecting when and how to deploy it. We introduce Online Meta-Control of Experience (OMCE), a framework that updates experience memory and learns a meta-control policy from task feedback during test-time adaptation. OMCE employs a lightweight contextual-bandit meta-controller that uses compact retrieval signals to select among four cognitive actions: DIRECT, EPISODE, GUIDE, and DISTILL. This routing mechanism operationalizes three coupled decisions: whether to invoke experience, which memory format to utilize, and when to consolidate new guidance. OMCE evolves online, leveraging task feedback to update the memory bank and refine the routing policy via contextual bandit updates. Experiments on four online adaptation benchmarks across four backbones show that OMCE improves first-attempt success by up to 20.5 percentage points over the strongest compared baseline, while reducing per-task token use by a factor of 5.7 to 62.5 relative to ACE with GPT-5.2. Controlled ablations further support the benefits of learned routing and continued routing policy adaptation.
Figures & tables
Figure 1: From Fixed Pipelines to Learned Meta-Control. Up: Existing methods typically follow a fixed routine for using and consolidating experience, even when the retrieved experience does not help the current task. Down: OMCE learns from task feedback when to use experience, whether to draw on past cases or general guidance, and when to distill new guidance.
Figure 2: Architecture of OMCE . The lightweight Meta-Controller selects a cognitive action specifying whether and which experience to inject into the task prompt. The agent solves the task with the augmented context containing the query and any injected experience. Memory evolution records outcomes in episodic memory, distills guidance before execution when DISTILL is selected, and refines injected guidance afterward using task feedback.
Benchmark
Model
OSR ↑ / CSR ↑
Base LLM
Reflexion
MemP
ReasoningBank
ACE
OMCE -TS
OMCE -UCB
Multi- Challenge
GPT-5.2
52.8% / 70.1%
- / 98.2%
50.2% / -
53.9% / 66.3%
48.7% / 71.4%
67.4% / 97.1%
66.3% / 95.6%
Deepseek-V3.2
43.1% / 55.7%
- / 91.6%
46.9% / -
48.4% / 56.0%
50.2% / 70.0%
70.7% / 95.6%
67.4% / 96.7%
Deepseek-V4-pro
49.5% / 63.0%
- / 90.1%
48.0% / -
55.3% / 66.3%
50.2% / 74.7%
69.6% / 93.8%
64.8% / 91.2%
Kimi-K2.5
49.8% / 64.8%
- / 89.0%
50.9% / -
56.7% / 72.8%
49.8% / 70.7%
72.2% / 95.6%
73.3% / 96.7%
BBEH
GPT-5.2
33.3% / 49.7%
- / 83.9%
31.9% / -
31.3% / 37.7%
52.4% / 94.4%
53.3% / 78.7%
46.8% / 71.0%
Table 1: Main results under the online adaptation setting. Each entry is reported as OSR / CSR (%), where CSR is measured within P=4 attempts. For Reflexion, “-” denotes that no first-attempt evaluation applies, as explained in the metric definitions above.
Benchmark
OSR ↑ / CSR ↑
Fixed- a\textscEPISODE
Fixed- a\textscGUIDE&\textscDISTILL
OMCE -HEU
OMCE -TS
OMCE -UCB
MultiChallenge
56.0% / 91.2%
63.7% / 94.5%
64.5% / 95.2%
67.4% / 97.1%
66.3% / 95.6%
BBEH
50.3% / 76.8%
43.1% / 73.8%
45.6% / 71.7%
53.3% / 78.7%
46.8% / 71.0%
BigCodeBench
49.0% / 59.0%
49.6% / 58.0%
47.4% / 57.4%
49.6% / 58.9%
51.3% / 59.2%
SWE-bench Verified
64.2% / 64.8%
64.4% / 64.8%
64.0% / 64.8%
64.8% / 64.8%
64.8% / 64.8%
Table 2: Routing ablation of OMCE on GPT-5.2, OSR / CSR (%). Fixed- a\textscEPISODE and Fixed- a\textscGUIDE&\textscDISTILL always commit to one form of experience; OMCE -HEU selects actions with hand-crafted rules. Appendix D reports the same comparison for the remaining backbones.
Online memory evolution
Shared memory snapshot
Benchmark
Variant
OMCE
Always-Both-and-Update
Frozen Controller
Always-Both
Random
MultiChallenge
OMCE -TS
67.4% / 97.1%
63.7% / 95.2%
61.9% / 94.1%
63.0% / 83.5%
62.6% / 83.9%
BBEH
OMCE -TS
53.3% / 78.7%
51.1% / 77.6%
49.8% / 75.6%
49.7% / 71.8%
48.2% / 67.4%
BigCodeBench
OMCE -UCB
51.3% / 59.2%
48.3% / 53.5%
49.5% / 65.4%
47.5% / 55.6%
46.5% / 57.0%
SWE-bench Verified
OMCE -TS
64.8% / 64.8%
55.0% / 55.2%
64.2% / 64.6%
48.8% / 49.2%
53.0% / 53.4%
Table 3: Controlled analysis on GPT-5.2, OSR / CSR (%). For each benchmark, OMCE uses the variant with the higher GPT-5.2 OSR in Table 1 (TS on ties). Frozen Controller stops controller updates mid-stream while memory keeps evolving; Always-Both and Random read one shared memory snapshot without updating it.
Figure 3: Performance–efficiency trade-off on four benchmarks using GPT-5.2. The x-axis represents the one-shot success rate, and the y-axis reports the token cost in log scale (average total prompt and output tokens consumed per task).
Figure 4: Self-correction detection in MultiChallenge. The meta-controller routes the query to GUIDE ( a\textscGUIDE ) rather than to episodic cases that enforce the user’s outdated constraint.
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
Benchmark
Model
OSR ↑ / CSR ↑
Fixed- a\textscEPISODE
Fixed- a\textscGUIDE&\textscDISTILL
OMCE -HEU
OMCE -TS
OMCE -UCB
Multi- Challenge
GPT-5.2
56.0% / 91.2%
63.7% / 94.5%
64.5% / 95.2%
67.4% / 97.1%
66.3% / 95.6%
Deepseek-V3.2
68.9% / 97.4%
70.3% / 97.4%
69.2% / 97.4%
70.7% / 95.6%
67.4% / 96.7%
Deepseek-V4-pro
65.6% / 91.9%
67.0% / 95.2%
66.3% / 93.8%
69.6% / 93.8%
64.8% / 91.2%
Kimi-K2.5
73.3% / 96.7%
72.2% / 96.7%
72.2% / 96.7%
72.2% / 95.6%
73.3% / 96.7%
BBEH
GPT-5.2
50.3% / 76.8%
43.1% / 73.8%
45.6% / 71.7%
53.3% / 78.7%
46.8% / 71.0%
Appendix
Table 4: Full routing ablation of OMCE on four backbones, OSR / CSR (%). Fixed- a\textscEPISODE and Fixed- a\textscGUIDE&\textscDISTILL always commit to one form of experience; OMCE -HEU selects actions with hand-crafted rules. The GPT-5.2 block is reproduced from Table 2 for reference.
Quantity
Value
Retrieval count M
3
Meta-state dimension d
9
Forgetting factor γ
0.98
Ridge regularization λ
0.5
LinUCB coefficient α
3.0
TS posterior scale v2
0.25
Appendix
Table 5: Shared controller and retrieval configuration used by OMCE -TS and OMCE -UCB.
Figure 5: Performance–efficiency trade-off on four benchmarks using GPT-5.2, Deepseek-V3.2, Deepseek-V4-pro, and Kimi-K2.5 as backbone LLMs. The x-axis shows one-shot success rate, and the y-axis shows token cost (log scale), defined as the average total number of prompt and output tokens consumed per task.
Pattern & Action
Task Context
Baseline / Wrong Memory
OMCE Behavior & Outcome
Memory Harmful a\textscDIRECT
ID: django-16560 Objective: Add violation_error_code to BaseConstraint . Challenge: Prone to wrong-layer patch trap.
a\textscGUIDE : Misled by irrelevant guidance entries, applying patches to the wrong abstraction layer. ✗Failure
ID: django-13401 Objective: Ensure abstract model fields do not compare as equal across different models. Challenge: Requires adjusting eq , hash , and lt to consider field.model instead of just creation_counter .
a\textscEPISODE : Fails to generalize from specific past case-specific insights, leading to incomplete or incorrect equality logic. ✗Failure
Reasoning: Recognizes that high-level guidance entries provides better architectural instruction for core equality semantics than case-specific insights. ✓Success
DISTILL a\textscDISTILL
ID: django-13809 Objective: Add --skip-checks option to the runserver command. Challenge: Requires correctly wiring the new option through the command’s argument parser and execution flow.
a\textscDIRECT : Fails to identify the correct modification layer and misses necessary execution flow updates. ✗Failure
Reasoning: Finds highly similar case-specific insights but no directly applicable guidance entries; distills new guidance entries to identify the correct modification layer. ✓Success
Appendix
Table 6: Qualitative case studies demonstrating the Meta-Controller’s dynamic action selection on SWE-bench Verified.
Large Language Model (LLM) agents increasingly rely on external memory systems to accumulate experience across tasks. Yet nearly all existing approaches, from graph-structured memories to reflective insight stores, access memory through fixed, hand-designed heuristics. We argue that this static view of memory is a core bottleneck for agentic learning because optimal memory behavior is fundamentally context-dependent. The early stages of the tasks, benefit from minimal retrieval because memory is sparse; recurring goal types benefit from plan reuse rather than generic nearest-neighbor lookup; stuck agents benefit from re-retrieval with alternative queries; and across long task streams, the memory store itself must be consolidated and pruned to remain useful. We present Memory as a Controlled Process (MemCon), a framework that models memory operations as a Markov Decision Process and learns an online policy that adaptively decides when, what, and how much to retrieve, when to inject a distilled plan, and when to consolidate or forget. MemCon is backend-agnostic: it wraps any existing memory implementation, learns from task-by-task binary feedback with no pretraining and no additional LLM calls, and uses a lightweight tabular contextual bandit with UCB exploration that converges within tens of tasks. Across 6 benchmarks, 3 agent frameworks, and 3 LLM backbones, MemCon consistently outperforms multiple memory baselines by up to 15.2 points in task success while reducing token consumption by 5--20%.
Eric Hanchen Jiang, Zhi Zhang, Yuchen Wu +11
University of California Los Angeles · University of Washington · Northwestern University
A large language model (LLM) agent solves long-horizon tasks through many reasoning-action turns, with one verification signal at termination. Deployed agents face streams of related tasks, making their trajectories a natural resource for improvement. In-context adaptation agents store reflections, memories, or skills as text, so reuse depends on retrieving the right experience and on a frozen policy executing it. We study Online Agentic Test-Time Training (OaTTT), which trains the LLM's weights on its own execution trajectories during deployment. The agent executes each task once, in one pass over the stream, and the executed trajectory with its verification result is the only learning signal for weight updates that persist across tasks. Directly imitating or reinforcing the generated tokens of this single attempt destabilizes the policy. We introduce ASCENT (Agentic Self-distillation for Cross-task EvolutioN at Test-time), which instead self-distills verified experience. A stable version of the LLM, its frozen initial copy, receives the verified trajectory as privileged information and predicts next-token distributions along it with this hindsight. Distilling them into persistent LoRA fast weights updates the agent for later tasks, without an external reference solution or stronger teacher. By further removing invalid-action turns, ASCENT distills enhanced privileged experience for more efficient execution. We characterize its population target and the limits of sparse outcome selection. Across ALFWorld, WebShop, and AppWorld at varied model scales, ASCENT improves task success and interaction efficiency as experience accumulates, outperforms online adaptation methods, and transfers to held-out scenes, showing that an agent can consolidate verified experience into its weights without a separate training phase or memory retrieval. Project page: https://artificer-ai-lab.github.io/ASCENT
Motivated by recursive self-improvement (RSI), we study how LLM agents can learn through external memory without updating model parameters. Extracting useful experience from a single attempt is difficult: flawed reasoning can produce success, while poor execution can obscure a sound strategy. Preserving its value over time is also difficult: our accumulation baseline without consolidation initially improves but eventually performs below a no-memory baseline. Our key idea is to compare attempts at the same task to identify reusable decisions and refine when they apply. We introduce Hippo, a memory framework inspired by selective hippocampal encoding and replay that learns from these comparisons when shortcomings are diagnosed. It compares the resulting experience with existing memory to merge overlaps and resolve conflicts through explicit applicability conditions. This allows prior guidance to be revised without storing every lesson as a separate entry. Experiments on SWE-bench Verified, WebArena, and Mind2Web show performance gains in all three settings and substantially fewer retained experiences than the accumulation baseline. The gains extend across the evaluated base models of varying capabilities, indicating that memory-based adaptation remains useful even for highly capable models.
Yutong Song, Jiang Wu, Yuxuan Ren +2
University of California, Irvine · Independent Researcher · Rice University +1