The Right Lesson at the Right Step: Deriving Control Updates for Self-Evolving Agents

Yunhe Su; ZiYi Dong; Tong Yu; Weijian Deng; Hao Li; Bowen Jiang; Pengxu Wei

cs.CLSep 28, 2026

The Right Lesson at the Right Step: Deriving Control Updates for Self-Evolving Agents

Authors: Yunhe Su, ZiYi Dong, Tong Yu, Weijian Deng, Hao Li, Bowen Jiang, Pengxu Wei

Organizations: Sun Yat-sen University · eunomia-bpf community · Tsinghua Shenzhen International Graduate School, Tsinghua University · Nankai University · Department of Computer and Information Science, University of Pennsylvania, Philadelphia, PA, United States · Sun Yat-sen University, Peng Cheng Laboratory

Abstract

Self-evolving agents improve future behavior by reusing past experience, typically as global prompts, memories, or reflections. Yet these mechanisms rarely control where experience takes effect. In long tool-use workflows, the same lesson may correct one decision but distract another, making experience reuse a problem of localized control rather than memory alone. We introduce EvoCUE (Evolution through Control Updates from Evidence), a framework for learning reusable control-program updates from completed agent executions. EvoCUE represents the agent as an explicit state-machine controller, whose nodes perform model or tool calls and whose edges define where control passes next. This makes the workflow editable at precise locations, so each learned update can specify what to add, where it acts, and when it applies. From completed trajectories, EvoCUE uses residual goals and observed execution traces to propose localized instruction or skill edits. Each candidate is evaluated at the point where it would act by resuming the parent and edited controllers from the same checkpoint and comparing their final outcomes. Accepted edits are compiled with applicability rules, confirmed on held-out tasks, and inherited by later executions. We evaluate EvoCUE on long tool-use environments where learned conventions must reach the right execution step. From a minimal AppWorld controller without benchmark-specific onboarding instructions, EvoCUE learns the missing task-completion convention and substantially improves success on Test-Normal and Test-Challenge. On PAST-Bench office workflows, EvoCUE transfers organizational requirements from prior episodes to later tasks, improving task-execution quality. These results show that self-evolving agents should place experience inside the control flow, rather than only store it as text.

Figures & tables

Figure 1: Motivation of EvoCUE . (a) A trip-booking assistant completes the requested operations but pays for the flight without required approval. The missing approval is a residual goal that localizes the failure to the payment step. (b) The lesson helps only when available at the payment decision. The bars show the same effect on AppWorld for a learned instruction about reporting a finished task. (c) EvoCUE restores the past task to before the edit can act, continues it with and without the edit, and keeps the edit only if it improves the outcome.

			Test-Normal ( $n=168$ )		Test-Challenge ( $n=417$ )
Method	Train tasks	Tokens (M)	Initial	Learned	Initial	Learned
GEPA ∗ ( Agrawal et al., 2026 )	90	20.1	13.1	13.7	6.0	5.0
AFlow ∗ ( Zhang et al., 2025b )	90	112.0	8.9	13.7	5.8	4.8
AWM ( Wang et al., 2025b )	90	14.2	11.9	11.9	7.2	6.0
MaAS ( Zhang et al., 2025a )	90	25.0	7.7	8.3	4.3	3.1
EvoCUE (ours)	32	15.9	13.7	79.2	6.7	70.0

Table 1: Frozen AppWorld task success (%). EvoCUE gains 65.5 and 63.3 points. All methods start from the same minimal agent prompt, without official onboarding instructions. Tokens: known learning tokens (Appendix D ). ∗ Final program equals the initial one; differences come from reruns.

	Pilot families		Unseen families
Method	Initial	Learned	Initial	Learned
GEPA ( Agrawal et al., 2026 )	0.599	0.628	0.279	0.300 ∗
AFlow ( Zhang et al., 2025b )	0.594	0.582	0.308	0.292
AWM ( Wang et al., 2025b )	0.590	0.564	0.303	0.441 †
MaAS ( Zhang et al., 2025a )	0.557	0.620	0.316	0.327 †
EvoCUE (ours)	0.608	0.772	0.305	0.332 ∗

Table 2: PAST-Bench task-execution scores in

[0,1]

. Pilot families: 16 later tasks after learning from 16 past episodes. Unseen families: 8 later tasks from four families never used during development. ∗ No update was accepted, so the program equals the initial one. † First of two learning passes.

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Setting	AppWorld	PAST pilot
Experience used for this instance	32-task pool	16 episodes
Maximum learning rounds	3	2
Maximum candidates per round	2	2
Paired tasks per candidate	8	4
Whole-program confirmation tasks	8	4
Prior training hypotheses	8	0

Appendix

Table 3: Settings of the reported learning instances. Historical hypotheses are prior model-generated training proposals, not instructions added to the initial controller.

Domain ( $n$ )	Metric	GEPA	AFlow	AWM	MaAS	EvoCUE
$\tau^{2}$ -bench (16)	solved	10 $\to$ 11	14 $\to$ 14	12 $\to$ 12	11 $\to$ 12	13 $\to$ 11 ∗
EvoClawBench (16)	score	1.000 $\to$ 0.992	1.000 $\to$ 0.499	0.999 $\to$ 0.938	0.688 $\to$ 0.938	0.998 $\to$ 1.000 ∗
MBPP (341)	passed	324 $\to$ 325	323 $\to$ 322	319 $\to$ 318	313 $\to$ 319	323 $\to$ 322 ∗
HumanEval (131)	passed	127 $\to$ 127	127 $\to$ 127	128 $\to$ 127	124 $\to$ 126	no update †
MATH (486)	solved	411 $\to$ 426	413 $\to$ 408	407 $\to$ 408	387 $\to$ 391	no update †
HotpotQA (800)	F1	0.785 $\to$ 0.785	0.783 $\to$ 0.795	0.781 $\to$ 0.798	0.781 $\to$ 0.781	0.786 $\to$ 0.782 ∗

Appendix

Table 4: Six further domains: initial

\to

learned score of each method on the test tasks, with the model and tools fixed.

\tau^{2}

-bench (airline and retail) and EvoClawBench are 16-task pilots. ∗ EvoCUE accepted no update, so its learned controller equals

C_{0}

and the difference is repeated execution. † No update accepted; the unchanged controller was not re-evaluated on the test tasks.

Evaluation	Program	Calls	Known tokens (M)	Missing
AppWorld Normal	Initial	2,869	31.56	21
	Learned	3,214	34.98	49
AppWorld Challenge	Initial	8,166	117.76	78
	Learned	8,683	125.28	86
PAST pilot	Initial	208	1.17	0
	Learned	146	0.56	0

Appendix

Table 5: Recorded inference resources for the initial and learned programs. Tokens are known totals in millions; missing denotes calls without reported token usage. AppWorld rows are the frozen test evaluations, and PAST rows are the 16-task frozen pilot evaluation.

Explore similar work

HarnessEvolve: Learning from Reference Trajectories for Reliable Agent Self-Evolution
Sep 1, 2026Wen Jiang, Mingmin Chu, Yimeng Tian +6Self-Evolving Agents Self-Evolution
ICT AI Competence Center, Huawei Technologies, Shanghai, China
EXG: Self-Evolving Agents with Experience Graphs
May 18, 2026Yuxin Jin, Siyuan Zhang, Hanchen Wang +3Self-Evolving Agents Reasoning Benchmark
University of Technology Sydney Sydney, Australia · The University of New South Wales Sydney, Australia
Learning What to Remember and What to Internalize in LLM Self-Evolution via Adaptive Memory-Parameter Coordination
Aug 2, 2026Tianyun Ji, Zhenya Huang, Jiayu Liu +3Self-Evolving Agents Self-Evolution
University of Science and Technology of China Hefei, Anhui, China · City University of Hong Kong Hong Kong, China · Hefei Normal University Hefei, Anhui, China +1

Sep 1, 2026, cs.LGJ/K move · Enter open · S save

HarnessEvolve: Learning from Reference Trajectories for Reliable Agent Self-Evolution

Wen Jiang, Mingmin Chu, Yimeng Tian, Qianxin Zhang+5

ICT AI Competence Center, Huawei Technologies, Shanghai, China

Self-Evolving Agents Self-Evolution Evolutionary Optimization Methods Gating Shortcut Learning

Open paper AskPDF arXiv

Self-evolving agents advance toward autonomy by optimizing their harness---prompts, skills, tools, and execution logic---based on environmental feedback. This paradigm, however, is hampered by three challenges: \textit{credit assignment failure}, where terminal success/failure feedback makes it ambiguous which step caused the error; \textit{shortcut learning}, where agents memorize task-specific patterns rather than acquire generalizable capabilities; and \textit{catastrophic forgetting}, where unguarded updates degrade previously acquired competence. In this paper, we introduce HarnessEvolve, a self-evolving framework that learns from reference trajectories to achieve reliable agent self-evolution. HarnessEvolve decouples the execution agent from the evolutionary pipeline, assigning execution, evaluation, optimization, and gating to independent agent modules, enabling generalizable and stable harness improvements. Specifically, HarnessEvolve overcomes credit assignment failure by generating reference trajectories (execution paths produced when given the ground-truth answers) and aligning failed executions against them to extract error signals, which are clustered to reveal systematic failure patterns. To prevent shortcut learning and catastrophic forgetting, candidate harness updates must pass two gates: a quality gate that filters data leakage and prompt bloat, and a performance gate that accepts each update if it improves on the current batch without degrading recent batches, with epoch-end validation on a held-out set selecting the best-performing accepted agent snapshot. We conduct extensive experiments on several benchmarks spanning open-domain and enterprise scenarios, using different models and agent frameworks. Results demonstrate that HarnessEvolve consistently outperforms state-of-the-art baselines across all benchmarks and settings, confirming reliability across task domains.

Abstract

Explore similar work

HarnessEvolve: Learning from Reference Trajectories for Reliable Agent Self-Evolution

EXG: Self-Evolving Agents with Experience Graphs

Learning What to Remember and What to Internalize in LLM Self-Evolution via Adaptive Memory-Parameter Coordination