Beyond Endpoint Performance: Process-Level Evaluation of Self-Evolving Agents

Hongqiang Lin; Chao Liu; Xiaofan Bai; Xuan Jin; Yuhong Li; Nenggan Zheng; Xipeng Cao

cs.AISep 21, 2026

Beyond Endpoint Performance: Process-Level Evaluation of Self-Evolving Agents

Authors: Hongqiang Lin, Chao Liu, Xiaofan Bai, Xuan Jin, Yuhong Li, Nenggan Zheng, Xipeng Cao

Organizations: Zhejiang University · Alibaba Group

Abstract

Self-evolving agents convert interaction feedback into persistent artifacts, such as memories or skills, which in turn guide subsequent decisions. As these artifacts are iteratively updated throughout an experience stream, the capabilities they support may evolve. Consequently, endpoint performance alone offers an incomplete view of self-evolution. Process-level evaluation is therefore essential to identify when a target capability emerges and whether later updates strengthen, preserve, or weaken it. Motivated by this, we propose \textsc{EvoPathBench}, a benchmark that tracks individual capabilities during artifact-level self-evolution. EvoPathBench fixes the base model, tools, freezes evolving artifacts at successive checkpoints, and evaluates the target capability on held-out episodes. This benchmark evaluates agent self-evolution using public trading data and calibrated trajectories. It tests three capabilities: generalization to unseen tasks, retention after unrelated learning, and rule adaptation to new evidence. Experimental results show that gains on similar unseen tasks often weaken under distribution shift, retention losses are concentrated in a minority of evolution paths, and no method achieves reliable rule adaptation. Moreover, while self-evolution enables agents to generate candidate artifacts with substantial held-out gains, the selected updates consistently fall short of realizing this potential. Together, these findings establish capability-level process evaluation as a foundation for analyzing self-evolution, identifying candidate evaluation and selection as key targets for improvement.

Figures & tables

Evaluation design	SkillLearnBench 33	Evo-Memory 19	PAST-Bench 24	Evo-Arena 22	SEA-Gym 32	FinEvo-Bench 3	EvoPathBench Ours
Ordered task stream	$\times$	$\checkmark$	$\checkmark$	$\checkmark$	$\checkmark$	$\checkmark$	$\checkmark$
Learning generalization	$\checkmark$	$\triangle$	$\checkmark$	$\triangle$	$\checkmark$	$\triangle$	$\checkmark$
Capability retention	$\times$	$\times$	$\triangle$	$\triangle$	$\checkmark$	$\times$	$\checkmark$
Rule adaptation	$\times$	$\times$	$\checkmark$	$\checkmark$	$\triangle$	$\times$	$\checkmark$
Data source	Existing	Existing	Curated	Curated	Existing	Real	Hybrid

Table 1: Comparison of representative longitudinal agent benchmarks.

\checkmark

indicates that the dimension is explicitly evaluated,

\triangle

indicates that only part of the dimension is evaluated, and

\times

indicates that the dimension is not evaluated. Existing denotes previously released datasets or benchmark tasks. Curated denotes tasks or environments assembled under predefined rules. Real denotes workflows derived from real cases. Hybrid denotes calibrated real-market trading data combined with generated data that preserve similar market properties.

	Learning generalization			Capability retention			Rule adaptation
Method	Near $\mathrm{CEG}_{\mathrm{N}}\uparrow$	Transfer $\mathrm{CEG}_{\mathrm{T}}\uparrow$	State effect $\mathrm{SUE}_{\mathrm{H}}\uparrow$	Mean $\Delta\mathrm{RL}\downarrow$	Worst $20\%$ $\Delta\mathrm{CVaR}_{0.2}\downarrow$	Worst $10\%$ $\Delta\mathrm{CVaR}_{0.1}\downarrow$	After 1 update $\mathrm{REV}_{1}\uparrow$	After 2 updates $\mathrm{REV}_{2}\uparrow$
Artifact-Free Execution
Baseline	0.000	0.000	–	0.000	0.000	0.000	0.000	0.000
Context	0.540	-0.061	0.160	-0.685	-3.229	-5.505	-0.059	0.021
Episodic Memory
Reflection	0.169	0.162	0.015	-0.360	-1.637	-2.597	-0.243	-0.331

Table 2: Main results on EvoPathBench . Arrows indicate the preferred direction. Bold and underline denote the best and second-best non-baseline estimates.

Figure 1: Skill length and overall CEG across update checkpoints. Negative CEG indicates performance below the non-evolving baseline.

	Token usage (M)				Performance and efficiency
Method	Input	Output	Total	Direct update	CEG $\uparrow$	TE $\uparrow$
Artifact-free Execution
Baseline	30.96	2.93	33.88	0.00	0.000	–
Context	137.80	3.81	141.61	0.00	-0.084	-0.00078
Episodic Memory
Reflection	33.11	3.15	36.25	1.04	-0.132	-0.05562

Table 3: Token usage and efficiency on EvoPathBench . Token counts are in millions. Direct update denotes the tokens consumed by model calls that create or revise the persistent artifact. Bold marks the highest CEG and TE.

	Update-process change					Held-out performance change
Intervention	Commit rate	Best gain $\uparrow$	Selection gap $\downarrow$	Harmful rate $\downarrow$	Missed rate $\downarrow$	$\Delta\mathrm{CEG}_{\mathrm{N}}\uparrow$	$\Delta\mathrm{CEG}_{\mathrm{T}}\uparrow$	$\Delta\mathrm{RL}\downarrow$	$\Delta\mathrm{REV}_{1}\uparrow$	$\Delta\mathrm{REV}_{2}\uparrow$
1 candidate $\rightarrow$ 4 candidates	$+0.180$	$+0.496$	$+0.163$	$+0.007$	$-0.029$	$+0.186$	$-1.463$	$-0.115$	$-0.064$	$+0.419$
Open $\rightarrow$ safeguarded acceptance	$-0.607$	$+0.690$	$+0.253$	$-0.033$	$+0.151$	$+0.368$	$+1.420$	$-0.300$	$+0.422$	$+0.121$

Table 4: Impact of controlled update interventions. Each row reports the second setting minus the first. Expanding candidates (1 to 4) increases proposed updates, while safeguards introduce improvement and regression gates. Metrics include: Commit rate (fraction of saved artifacts), Best gain (top candidate improvement), Selection gap (difference between top candidate and saved update), Harmful (committed regressions), and Missed (rejected improvements)

	Validation fidelity		Utility selection		Decision error
Acceptance	Agreement $(\%)\uparrow$	Capture $(\%)\uparrow$	Precision $(\%)\uparrow$	Recall $(\%)\uparrow$	Harmful $(\%)\downarrow$	Missed $(\%)\downarrow$
Open	$56.9$	$69.0$	$9.3$	$52.9$	$12.4$	$0.0$
Safeguarded	$55.8$	$47.2$	$29.5$	$41.9$	$6.2$	$16.5$

Table 5: Effect of safeguards on candidate selection. Both policies are compared at the same checkpoints, and candidates are tested on unseen episodes that are not used for selection

Receiver	SkillBoost	SkillOpt	SFT
Qwen3.5-4B	$+0.27$	$+0.19$	$+0.14$
Qwen3.5-9B	$+0.03$	$-0.02$	$+0.11$
Qwen3.8-27B	$+1.96$	$+0.46$	$+2.19$

Table 6: Cross-model transfer results. Overall CEG relative to Base. Higher is better.

	Artifact-free		Memory			Skill evolution
Metric	Baseline	Context	Reflection	Single-Evidence	Consolidated	SkillOpt	SkillBoost	SkillX	Trace2Skill	SkillGrad
BWT $\uparrow$	$-0.082$	$+0.319$	$+0.043$	$-0.090$	$+0.185$	$-0.721$	+0.863	$+0.231$	$+0.327$	$+0.235$
Mean loss $\mathbb{E}[L^{a}]\downarrow$	$0.926$	0.241	$0.567$	$0.614$	$0.671$	$1.454$	$0.719$	$0.264$	$0.451$	$0.707$
Forgetting rate $(\%)\downarrow$	$23.6$	$14.8$	$15.7$	$15.7$	$16.7$	$20.8$	$21.3$	10.6	$13.4$	$24.1$
Conditional loss $M_{\mathrm{F}}\downarrow$	$3.113$	1.495	$2.054$	$3.091$	$2.795$	$5.631$	$3.414$	$3.221$	$3.316$	$2.469$
Worst 10% $\mathrm{CVaR}_{0.1}\downarrow$	$7.461$	1.956	$4.864$	$5.191$	$5.452$	$12.666$	$5.799$	$2.356$	$4.319$	$5.350$

Table 7: Decomposition of retention outcomes from

K_{2}

K_{5}

in Interference streams. Forgetting rate measures how often performance declines, conditional loss measures the average decline when forgetting occurs, and

\mathrm{CVaR}_{0.1}

is the mean retention loss among the worst

10\%

of paths. Bold indicates the best point estimate in each column.

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 2: Composition of the EvoPathBench benchmark. The inner ring represents the three stream templates, and the outer ring represents the six focal task families. Sector area is proportional to the number of streams. The benchmark contains 600 streams per template and 100 streams in each template–family cell. Each cell includes streams from both market layers.

	Post-change CEG			Second-update increment
Method	Before evidence $K_{3}$ $\rightarrow$	After update 1 $K_{4}$ $\rightarrow$	After update 2 $K_{5}$	$\Delta_{2}=K_{5}-K_{4}$
Single-Evidence Memory	$-0.628$ $\rightarrow$	$-0.157$ $\rightarrow$	$0.088$	$+0.245$
SkillOpt	$0.025$ $\rightarrow$	$-0.559$ $\rightarrow$	$\mathbf{0.439}$	$\mathbf{+0.998}$
SkillBoost	$-0.283$ $\rightarrow$	$\mathbf{0.383}$ $\rightarrow$	$0.001$	$-0.382$
SkillX	$0.005$ $\rightarrow$	$-0.055$ $\rightarrow$	$0.191$	$+0.246$
Trace2Skill	$\mathbf{0.189}$ $\rightarrow$	$-0.445$ $\rightarrow$	$0.414$	$+0.860$

Appendix

Table 8: Checkpoint-level performance on held-out post-reversal episodes.

K_{3}

precedes counterevidence, whereas

K_{4}

and

K_{5}

follow the first and second update opportunities. The final column reports the observed marginal checkpoint change after the second update. Higher is better, and bold denotes the largest point estimate in each column.

Candidate	Proposed trading instructions	Earlier tasks	New task	All tasks
A	Average the public fundamental value and signal to estimate value; scale the target position with mispricing.	$-0.680$	$+3.380$	$+0.335$
B	Trade toward fundamental value when the estimated gain after spread costs exceeds $0.05\%$ ; place orders of up to eight units.	$-2.936$	$+12.141$	$+0.833$
C	Trade whenever the estimated gain after spread costs is positive; size orders within cash, position, and turnover limits.	$+5.292$	$-8.924$	$+1.738$
Selection rule				Choice
Full SkillBoost: require improvement and limit regressions				None
Remove regression protection; select the highest mean				C

Appendix

Table 9: Candidate skill documents and their corresponding internal validation gains, computed relative to the incumbent state.

Episode	Effect	Mean	SD	5th percentile	95th percentile	Negative (%)
Near	CEG	$+0.817$	$0.912$	$-1.238$	$+1.605$	$14.0$
	SUE	$+0.817$	$0.912$	$-1.238$	$+1.605$	$14.0$
Transfer	CEG	$-1.860$	$0.468$	$-2.444$	$-1.825$	$96.0$
	SUE	$-1.614$	$1.048$	$-2.444$	$+0.990$	$90.0$

Appendix

Table 10: Variation across 50 paired executions of a fixed SkillBoost checkpoint. Negative CEG means performance below the Baseline, whereas negative SUE means performance below state-off execution. Scores use the same scaling as Table 2 .

	Overall $\mathrm{CEG}_{5}$ by task order			Order sensitivity			Accepted edits
Method	Original	Reversed	Permuted	Mean $\uparrow$	SD $\downarrow$	Range $\downarrow$	O/R/P
Consolidated Memory	$+0.479$	$+2.355$	+2.355	$+1.730$	0.884	1.876	–
SkillOpt	+5.615	$-3.852$	$+0.906$	$+0.890$	$3.865$	$9.467$	4/3/2
SkillBoost	$-0.205$	+5.887	$+0.616$	+2.100	$2.699$	$6.092$	4/3/5

Appendix

Table 11: Effect of task order on capability evolution at

K_{5}

. Each method receives the same five learning episodes in the original order, the reversed order, or a fixed permutation. SD is the standard deviation of CEG across the three orders, while Range is the difference between the largest and smallest CEG. Accepted edits report the number of skill-document updates saved across ten update opportunities in original/reversed/permuted order. Higher CEG is better, whereas lower SD and Range indicate less order sensitivity. Bold denotes the best value in each column.

Backbone	Calls (K)	Input (M)	Output (M)	Total (M)	Direct update (M)	Estimated cost (CNY)
Qwen3.8-Max	$802.21$	$936.48$	$82.28$	$1,018.76$	$53.13$	$14,199.87$
Kimi-K3	$784.29$	$748.04$	$71.20$	$819.24$	$53.21$	$22,080.59$
Total	$1,586.49$	$1,684.52$	$153.48$	$1,838.00$	$106.34$	$36,280.46$

Appendix

Table 12: Estimated cost of the main experiments. Usage includes all audited calls for task execution, candidate validation, and artifact updates.

Explore similar work

EvoHarnessBench: Can Your Agents Keep Pace with an Evolving Harness?
Sep 3, 2026Zixuan Ke, Vaidehi Patil, Haizhou Shi +9Continual Learning for LLM Agents LLM Agent Harnesses
Salesforce Research · University of North Carolina at Chapel Hill, conducted this work during an internship at Salesforce Research · University of Wisconsin–Madison +1
EvoAgentBench: Benchmarking Agent Self-Evolution via Ability Transfer
Jul 6, 2026Xingze Gao, Chuanrui Hu, Hongda Chen +9LLM Agent Self-Improvement AI Agent Benchmarks
1Anhui University · 2EverMind, Shanda Group · 3Southeast University
SEABench: Benchmarking Endogenous Misalignment In Self-Evolving Agents
Sep 28, 2026Saswat Das, Parvati Viswanathan, Daniel Donnelly +3AI Agent Safety AI Agent Safety Benchmarks
University of Virginia · ELLIS Institute Tübingen · Max Planck Institute for Intelligent Systems +1

Sep 3, 2026, cs.MAJ/K move · Enter open · S save

EvoHarnessBench: Can Your Agents Keep Pace with an Evolving Harness?

Zixuan Ke, Vaidehi Patil, Haizhou Shi, Yang Li+8

Salesforce Research · University of North Carolina at Chapel Hill, conducted this work during an internship at Salesforce Research · University of Wisconsin–Madison+1

Continual Learning for LLM Agents LLM Agent Harnesses Self-Evolving Agents Agent Benchmarks

Open paper AskPDF arXiv

Modern LLM-based agents operate through a harness of tools, reusable skills, and specialist agents that shapes what they observe and what they can do. In practice, this harness continually evolves as new capabilities are added. We introduce EvoHarnessBench, a benchmark for evaluating agents under controlled harness evolution across three axes (tools, skills, and agents). Unlike existing continual-learning benchmarks for agents, which typically place non-stationarity (i.e., what changes over time) in the task stream while keeping the harness fixed, EvoHarnessBench places non-stationarity in the externally supplied harness itself. It contains 17 multi-stage harness streams constructed deterministically from verifier-based benchmarks, comprising 802 tasks, 520 tools, 42 skills, and 62 agents. We evaluate two complementary settings corresponding to the central challenges of harness evolution: deployment evaluation, which isolates retention of previously accessible competence as the harness expands, and self-evolving adaptation evaluation, which tests whether accumulated experience remains useful as new capabilities are introduced. Our results reveal three persistent gaps. First, harness expansion alone can degrade performance on previously solved tasks, producing harness-induced forgetting. Second, gains from self-evolving adaptation remain inconsistent across stages of harness evolution, capability axes, and environments. Third, retention and adaptation can pull in different directions: preserving earlier competence does not necessarily improve adaptation to newly introduced capabilities, and vice versa. These results establish harness evolution as a distinct challenge for building agents that can keep pace with an evolving harness while preserving previously effective behavior.

Abstract

Explore similar work

EvoHarnessBench: Can Your Agents Keep Pace with an Evolving Harness?

EvoAgentBench: Benchmarking Agent Self-Evolution via Ability Transfer

SEABench: Benchmarking Endogenous Misalignment In Self-Evolving Agents