MetaExp: Learning to Self-Improve through Experience via Verifiable Cross-Task Environment Scaling

Renhao Li; Xinjie Shen; Yang Su; Jianhong Tu; Derek F. Wong; Dayiheng Liu; Min Yang

cs.AIOct 15, 2026

MetaExp: Learning to Self-Improve through Experience via Verifiable Cross-Task Environment Scaling

Authors: Renhao Li, Xinjie Shen, Yang Su, Jianhong Tu, Derek F. Wong, Dayiheng Liu, Min Yang

Organizations: University of Macau · Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences · Georgia Institute of Technology · Alibaba Token Hub, Alibaba Group

Abstract

Language-model agents can accumulate interaction feedback, but reliably turning that feedback into reusable experience remains challenging. Existing methods either optimize external experience scaffolds or train experience-related behavior on specific task distributions, leaving open whether experience learning itself can become a transferable model capability. Training such a capability calls for supervision beyond isolated tasks, which mainly support in-episode problem solving rather than revealing what knowledge remains useful across episodes. In this work, we introduce MetaExp, a scalable framework that trains this capability through verifiable task families with controlled cross-task structure. Each family preserves a reusable factor while varying task-specific realizations, enabling transfer without direct solution reuse. We propose two complementary designs: Shared-Tool families keep black-box tools fixed across tasks to elicit operational experience, while Shared-Workflow families preserve latent procedures across changing realizations to elicit procedural experience. We scale environments under both designs, collect and filter sequential interaction trajectories, and fine-tune Qwen3.5-122B-A10B on 4,956 trajectories from 1,000 task families. With parameters frozen during downstream evolution, MetaExp improves by +8.5 points on SpreadsheetBench, peaks at +2.3 on APEX-Agents, and improves by +14.8 on AutomationBench. The gains persist under domain-excluded training, supporting experience learning as a transferable capability for continued non-parametric improvement after deployment.

Figures & tables

Figure 1: Overview of MetaExp . Heterogeneous seeds are expanded into verifiable task families with complementary Shared-Tool and Shared-Workflow structure. Family-centered task sequences are used to collect execution and feedback-driven reflection trajectories, which train the model to acquire, update, and reuse experience. At evaluation, model parameters are frozen and experience is accumulated from an empty bank on unseen task distributions.

Task success rate (%, $\uparrow$ )
Model	Protocol	$k=0$	$k=5$	$k=10$	$k=20$	$k=40$
GPT-4.1	EvoSkill	44.2	–	–	–	$37.2\pm 2.6$ $(-7.0)$
GPT-5.4	EvoSkill	62.5	–	–	–	$68.1\pm 0.5$ $(+5.6)$
GPT-4.1	SkillGrad	44.2	–	–	–	$54.2\pm 3.5$ $(+10.0)$
GPT-5.4	SkillGrad	62.5	–	–	–	$71.1\pm 1.7$ $(+8.6)$
GLM-5.2	SeqEvo	$88.3\pm 1.2$	$88.9\pm 0.5$ $(+0.6)$	$89.5\pm 0.7$ $(+1.2)$	$89.6\pm 0.4$ $(+1.3)$	$89.4\pm 0.4$ $(+1.1)$

Table 1: Performance and efficiency on SpreadsheetBench. Values are mean

\pm

population standard deviation, with

\Delta_{k}=S(k)-S(0)

shown in parentheses. Green/red denotes improvement/degradation relative to

k=0

. EvoSkill and SkillGrad results are taken from Wang et al. (2026a) and follow their originally reported evolution protocols; all other results use our cold-start SeqEvo protocol. “–” denotes an unreported checkpoint.

Figure 2: Performance on APEX-Agents and AutomationBench. Left: aggregate success rate over adaptation steps, with shaded population standard deviations over three runs. Right: domain-wise success rate and changes

\Delta_{k}

on each adaptation step for MetaExp -Qwen3.5-122B-A10B and Qwen3.5-122B-A10B; green/red denotes improvement/degradation.

Model	$k=0$	$k=5$	$k=10$	$k=20$	$k=40$
Task success rate (%, $\uparrow$ )
Qwen3.5-122B-A10B	$68.9\pm 1.8$	$67.3\pm 1.8$ $(-1.6)$	$67.7\pm 0.5$ $(-1.2)$	$67.0\pm 0.9$ $(-1.9)$	$66.7\pm 1.3$ $(-2.2)$
- MetaExp	$65.0\pm 2.2$	$69.5\pm 3.3$ $(+4.5)$	$72.9\pm 3.7$ $(+7.9)$	$73.1\pm 2.8$ $(+8.1)$	$73.5\pm 3.4$ $(+8.5)$
- Strict-OOD MetaExp	$62.7\pm 1.4$	$66.8\pm 5.0$ $(+4.1)$	$66.2\pm 4.0$ $(+3.5)$	$72.5\pm 3.0$ $(+9.8)$	$72.0\pm 0.8$ $(+9.3)$

Table 2: SpreadsheetBench task success under domain-excluded training. Values are mean

\pm

population standard deviation over three runs, with

\Delta_{k}=S(k)-S(0)

shown in parentheses.

Model	Score (%)	Pass (%)
GLM-5.2	65.1	74.7
Baseline	33.3	47.1
MetaExp	41.4 $(+8.1)$	62.1 $(+15.0)$

Table 3: Model performance on SkillsBench. Values are averaged across three runs.

Task success rate (%, $\uparrow$ )
Benchmark	Shared-Tool Only		Shared-Workflow Only		Full MetaExp
Benchmark	Initial	Final	Initial	Final	Initial	Final
SpreadsheetBench	$62.7\pm 1.0$	$72.4\pm 7.3$ $(+9.7)$	$63.6\pm 1.5$	$69.7\pm 2.7$ $(+6.1)$	$65.0\pm 2.2$	$73.5\pm 3.4$ $(+8.5)$
APEX-Agents	$13.2\pm 1.3$	$12.7\pm 1.3$ $(-0.5)$	$12.2\pm 0.6$	$12.6\pm 0.7$ $(+0.4)$	$12.2\pm 1.7$	$14.0\pm 1.4$ $(+1.8)$
AutomationBench	$7.2\pm 0.5$	$14.8\pm 3.8$ $(+7.6)$	$8.5\pm 0.5$	$15.6\pm 3.6$ $(+7.1)$	$14.5\pm 0.9$	$29.3\pm 3.3$ $(+14.8)$
Execution turns ( $\downarrow$ )

Table 4: Ablation of cross-family task designs on models. Values are mean

\pm

population standard deviation over three runs, with

\Delta=S_{\mathrm{final}}-S_{\mathrm{initial}}

shown in parentheses.

Model	Condition	Skills in bank	Any invocation (%)	Skill as first tool (%)	Calls/task
Baseline-Qwen3.5	$k=5$	$3.0\pm 0.0$	$9.3\pm 2.0$	$2.4\pm 1.3$	$0.094\pm 0.021$
	$k=10$	$5.3\pm 0.5$	$11.3\pm 2.5$	$4.2\pm 1.5$	$0.113\pm 0.025$
	$k=20$	$10.0\pm 0.0$	$18.6\pm 4.7$	$7.8\pm 0.1$	$0.188\pm 0.049$
	$k=40$	$10.0\pm 0.0$	$17.0\pm 8.2$	$5.0\pm 2.6$	$0.172\pm 0.084$
MetaExp -Qwen3.5	$k=5$	$2.0\pm 0.8$	$76.4\pm 16.4$	$63.1\pm 23.2$	$0.765\pm 0.162$
	$k=10$	$3.3\pm 2.6$	$79.5\pm 16.9$	$67.0\pm 23.0$	$0.800\pm 0.164$

Table 5: SpreadsheetBench experience-access behavior. Values are means

\pm

population standard deviations over three seeds.

Model	Conditions	Invoked skill	Did not invoke
Baseline-Qwen3.5	All $k$ pooled	$-4.2$	$-1.2$
MetaExp -Qwen3.5	All $k$ pooled	$+9.5$	$-1.9$
MetaExp -Qwen3.5	$k=20$	$+10.8\pm 4.7$	$-0.3\pm 4.8$
MetaExp -Qwen3.5	$k=40$	$+10.2\pm 6.4$	$-4.7\pm 6.7$

Table 6: Paired success-rate change relative to the same task under an empty-bank condition (%).

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Metric	Baseline-Qwen3.5	MetaExp -Qwen3.5
Skills	30	14
Lexical tokens	29,679	31,155
Tokens per skill	989.3	2,225.4
Failure terms / 1K tokens	1.42	5.42
Verification terms / 1K tokens	3.13	4.56
Opaque IDs / 1K tokens	2.26	0.00

Appendix

Table 7: Structure of the final SpreadsheetBench skill banks at

k=40

, pooled over three seeds.

Case	Baseline-Qwen3.5	MetaExp -Qwen3.5
Cross-sheet lookup	A local VLOOKUP/INDEX–MATCH recipe with several formula variants and a recommendation to write static values.	A symptom–cause–repair–verification taxonomy covering truncated ranges, wrong-sheet references, hidden whitespace, cached formula values, and cell-level validation.
Boundary-sensitive editing	A generic checklist: preserve values, use absolute references, handle missing keys, and verify the output.	Conditional diagnostic rules for inflated worksheet bounds and locale-dependent formula separators, tied to observable failure signals and corrective checks.

Appendix

Table 8: Case study on how different models encode experiences on SpreadsheetBench.

Panel of Figure 3 — Figure 3: Domain distribution of corpus seeds under Shared-Tool and Shared-Workflow construction.

Panel of Figure 4 — Figure 4: Primary-intent distribution of the task-example seeds used in MetaExp .

Panel of Figure 5 — Figure 5: Capability distribution of tool examples and their coverage across seed records.

LLM Agent Self-Improvement

Large language model agent self-improvement studies how agents use experience and feedback to improve their behavior, tools, and workflows.

See all 277 papers

Oct 14, 2026

XferAgent: Cross-Agent Experience Transfer for Self-Improving LLM Agents

Large language model (LLM) agents can reuse experience from prior task executions to improve performance on subsequent tasks. Although existing methods enable cross-agent experience sharing, differences in agents' execution histories and task requirements make shared experiences difficult to reuse effectively. We propose XferAgent, a framework for cross-agent experience transfer that organizes reusable experience in local and shared experience graphs. XferAgent reconciles compatible contributions from heterogeneous agents while avoiding unreliable merges. For each new task, it selects shared experiences and integrates them with the target agent's local experience to support task execution. Experiments across diverse task domains show improvements over representative memory-based methods and effective experience transfer between heterogeneous agent frameworks. Cumulative task success also increases over successive task passes as agents update and reuse their experiences.

Weihao Li, Jun Bai, Shijie Chen +3

McCormick School of Engineering, Northwestern University · School of Computer Science, McGill University · Mila – Quebec AI Institute +1

LLM Agent Self-Improvement Memory-Augmented Language Model Agents Graph-Based Agent Memory Transfer Learning Continual Learning for LLM Agents

Ask PDF

May 19, 2026

Training Language Agents to Learn from Experience

Language agents can adapt from experience in interactive environments, but current reflection-based methods can only self-correct within a single task instance. Whether such experience can be distilled into reusable lessons that improve performance on future unseen tasks remains unclear. We address this problem by introducing the In-context Training (ICT) task, a framework for evaluating cross-task self-improvement in language agents. In ICT, a reflector model observes trajectories collected by an actor model and generates system prompts intended to improve the actor's performance on future unseen tasks. We then propose an RL-based training pipeline for learning such reflections directly from experience, without human-provided examples. Across ALFWorld and MiniHack, our trained reflectors outperform an untrained baseline on most held-out task families, showing that the ability to learn from experience can itself be learned. In some cases, we observe generalisation beyond the benchmark on which the reflector was trained, to substantially different environments. Finally, we introduce MetaGym, a generic Python library for constructing meta-environments, enabling future research on self-improving language agents.

Yuval Shalev, Zifeng Ding, Mateja Jamnik

University of Cambridge

Continual Learning for LLM Agents Self-Improving Agents RL for LLM Agents Self-Reflection in Language Model Agents

Ask PDF

Mar 25, 2026

Experiential Reflective Learning for Self-Improving LLM Agents

Recent advances in large language models (LLMs) have enabled the development of autonomous agents capable of complex reasoning and multi-step problem solving. However, these agents struggle to adapt to specialized environments and do not leverage past interactions, approaching each new task from scratch regardless of their accumulated experience. We introduce Experiential Reflective Learning (ERL), a simple self-improvement framework that enables rapid environment adaptation through experiential learning. ERL reflects on task trajectories and outcomes to generate heuristics, capturing actionable lessons that transfer across tasks. At test time, relevant heuristics are retrieved based on the current task and injected into the agent's context to guide execution. On the Gaia2 benchmark, ERL improves success rate by 7.8% over a ReAct baseline, with large gains in task completion reliability, and outperforms prior experiential learning methods. Through systematic ablations, we find that selective retrieval is essential and that heuristics provide more transferable abstractions than few-shot trajectory prompting. These results demonstrate that reflecting on single-attempt experiences to extract transferable heuristics enables effective agent self-improvement.

Marc-Antoine Allard, Arnaud Teinturier, Victor Xing +1

Illuin Technology

Self-Improving Agents LLM Agent Training Memory-Augmented Language Model Agents

Ask PDF

Abstract

LLM Agent Self-Improvement

Explore similar work

XferAgent: Cross-Agent Experience Transfer for Self-Improving LLM Agents

Training Language Agents to Learn from Experience

Experiential Reflective Learning for Self-Improving LLM Agents