Trace Evolution Bench: Measuring Whether Agents Improve from Their Own Environment Traces

Fangfei Li; Chenyang Zhao; Long Wang; Feng Tian; Lv Guo

cs.AIOct 10, 2026

Trace Evolution Bench: Measuring Whether Agents Improve from Their Own Environment Traces

Authors: Fangfei Li, Chenyang Zhao, Long Wang, Feng Tian, Lv Guo

Organizations: MatrixOrigin, China

Abstract

Self-evolving agents are usually scored by endpoint Gain: held-out success after an update minus success before. That difference can credit a frozen rescore as evolution. We introduce Trace Evolution Bench (TEB), a protocol that separates that rescore from a context or weight update, together with three frozen 80/20/40 packs. An update is accepted only when validation does not fall, held-out identifiers never enter the update, and a text update is not ranked with a weight update. The primary scoreboard is coding, graded by hidden unit tests in a closed workspace. On that pack (40 held-out tasks, one family), DeepSeek v4 Pro under no_evolve never changes its policy yet moves 7.5, 5.0, and 12.5 percentage points across three seeds. Of nine text-update cells inspired by GEPA, ACE, and ExpeL, seven have positive excess above that seed's own band, and one gepa_llm seed moves down by the width of its band (-12.5, regression 0.48). GLM and MiniMax do not reproduce that seven-of-nine pattern. No method stays above its own band on every seed of all three models: GLM has two paired cells above their own bands, and MiniMax expel_llm seed 44 is a third (regression 0.30). The methods were fixed on a validation screen before held-out was read. Office, graded by rule-bounded delivery, and manufacturing table QA are separate packs: office DeepSeek does not exceed its own bands (widest 17.5 points), and manufacturing has no paired positive excess beyond its seed band.

Figures & tables

Benchmark	Scored unit	Frozen rescore band
SWE-bench, AgentBench ( Jimenez et al., 2024 ; Liu et al., 2024 )	One episode	No
LifelongAgentBench ( Zheng et al., 2025 )	A task sequence	No
SEA-Eval ( Jiang et al., 2026 )	A task sequence	No
EvoAgentBench ( Gao et al., 2026 )	Transfer on an ability-supported split	No; one vanilla condition
EvoPathBench ( Lin et al., 2026 )	Capability checkpoints over ordered streams	No
S 3 Gym ( Shi et al., 2026 )	Strict held-out evaluation	No

Table 1: Evaluation target of each benchmark. A frozen rescore band is the range of repeated held-out passes of one unchanged policy.

Property	TEB	RSEA	SEA	Evo	GDP	RSI	PTB	TR	MLE	EPB	S3	SEB	PAST	Fin
Own traces	✓	✓	✓	✓	✓	×	✓	✓	✓	✓	✓	×	✓	✓
Multi-round	✓	✓	✓	×	×	✓	×	×	×	✓	✓	×	✓	✓
Carrier flex. in this paper (m/p/c/w)	×	×	×	×	×	×	×	×	×	×	×	×	×	×
Explore/eval split	✓	×	×	✓	×	×	✓	×	×	✓	✓	×	×	×
Validation gate	✓	×	×	×	×	×	×	×	×	×	×	×	×	×
Held-out lock	✓	×	×	✓	×	×	✓	×	×	✓	✓	×	×	×

Table 2: Protocol mechanics contrast (binary coding). ✓ means the property is explicitly supported in the paper’s main protocol; × means otherwise (including not stated). In “Carrier flex. in this paper (m/p/c/w)”, m/p/c/w = memory/prompt/code/weight. Abbreviations: SEA (SEAGym), Evo (EvoAgentBench), GDP (GDPevo), RSI (RSIBench-Data), PTB (PostTrainBench), TR (TRACE), MLE (OpenMLE), EPB (EvoPathBench), S3 (S 3 Gym), SEB (SE-Bench), PAST (PAST-Bench), Fin (FinEvo-Bench).

Figure 1: TEB global evaluation contract across rounds. Round 0 scores

M_{0}

and does not update. Rounds 1–3 each accept or roll back one text update, then rescore the same held-out. Excess is read against that seed’s own four-pass band.

Figure 2: Per-round accept/rollback mechanics (this loop is rounds

r\geq 1

in Figure 1 ). Round 0 freezes Gain on the locked held-out ids. Later rounds call evolve() once; accept and rollback both score that same held-out. Held-out never enters evolve() . B0 is not ranked against B2.

	Office	Coding	Manufacturing
Source	Product-usage workflows	142 merged pull requests, rewritten as functions	12 real tables; 31 real questions, 109 generated
Items	10 families	One family, fix_bug	13 families
Outside evolution	12/40: access_denied , do_not_publish	None on held-out	12/40: real questions on 4 unseen table families
Tools	list_files , read_file , write_file , publish_artifact , grant_access , submit	list_dir , read_file , write_file , run_tests , submit ; one file, src/solution.py	list_tables , describe_table , execute_sql , submit_answer
Success	Every terminal check passes	Hidden unit tests pass	Score $\geq 80$ : SQL 40, result 40, answer 20
Step cap	20	12	12

Table 3: Three frozen packs. Each has 140 tasks, split 80/20/40, so held-out

n{=}40

. The observation omits gold files, expected SQL, and unit-test oracles.

Method	Valid	Gain (pp)	Mean reg.	Tokens
gepa_llm	true	$0.0$	0.056	469231
ace_llm	true	$-5.6$	0.111	513033
expel_llm	true	$-11.1$	0.056	448950
dgm	true	$-16.7$	0.111	396406
clin_llm	true	$-16.7$	0.222	411690
awm_llm	false	$0.0$	0.000	447296

Table 4: Screen ranking across three models at seed 42. Mean gain is in percentage points. Mean reg. is a rate. Valid is the artifact check.

Method	Injected text
gepa_llm	One revised instruction. The validation gate accepts it or rolls it back.
ace_llm	A grow-and-refine playbook from a generator, reflector, and curator.
expel_llm	A deduplicated pool of insights from success and failure.

Table 5: Text injected by the three scored methods. Each reads that round’s own traces. The kept text is the TEB mechanism.

Figure 3: DeepSeek coding held-out success by round. no_evolve keeps version

M_{0}

and the score still moves. Seeds are 42, 43, and 44.

Seed	r0	r1	r2	r3	Gain	Range
42	67.5	67.5	65.0	60.0	$-7.5$	7.5
43	62.5	65.0	62.5	60.0	$-2.5$	5.0
44	62.5	62.5	65.0	52.5	$-10.0$	12.5

Table 6: Coding, DeepSeek v4 Pro, no_evolve only. Success is held-out percent. Version is

M_{0}

at every round. Range is that seed’s band.

Model	Seed	Frozen Gain	Band
DeepSeek	42	$-7.5$	7.5
DeepSeek	43	$-2.5$	5.0
DeepSeek	44	$-10.0$	12.5
GLM	42	$-2.5$	12.5
GLM	43	$+7.5$	7.5
GLM	44	$-7.5$	15.0

Table 7: Coding no_evolve by model. Frozen Gain is the four-pass Gain. Band is that seed’s four-pass range. DeepSeek pass scores are Table 6 .

Method	Seed	Gain	Frozen	Excess	Band	Reg.	Against that band
gepa	42	$+7.5$	$-7.5$	$+15.0$	7.5	0.167	outside
gepa	43	$+12.5$	$-2.5$	$+15.0$	5.0	0.069	outside
gepa	44	$-22.5$	$-10.0$	$-12.5$	12.5	0.481	equal, down
ace	42	$+12.5$	$-7.5$	$+20.0$	7.5	0.065	outside
ace	43	$+7.5$	$-2.5$	$+10.0$	5.0	0.071	outside
ace	44	$0.0$	$-10.0$	$+10.0$	12.5	0.130	inside

Table 8: Coding-v1, DeepSeek v4 Pro. Excess is method Gain minus the same-seed no_evolve Gain, in percentage points. Band is that seed’s four-pass range. Reg. is the final regression rate. None of these nine cells is over_budget .

Figure 4: DeepSeek coding paired excess. Black ticks are that seed’s own band. Dark bars lie strictly above it. The light positive bar is ace_llm seed 44, inside its

12.5

pp band. gepa_llm seed 44 equals its band on the way down.

Model	Method	Seed	Excess	Band	Reg.	Cap	Against that band
GLM	gepa	42	$+7.5$	12.5	0.207		inside
GLM	gepa	43	$0.0$	7.5	0.069		inside
GLM	gepa	44	$+15.0$	15.0	0.094		equal
GLM	ace	42	$-12.5$	12.5	0.130	yes	equal, down
GLM	ace	43	$+5.0$	7.5	0.065	yes	inside
GLM	ace	44	$+15.0$	15.0	0.091		equal

Table 9: Coding status for GLM and MiniMax. Excess and band are defined as in Table 8 . Unpaired means the round-0 gap is at least 10 pp. Reg. is the final regression rate. Cap marks over_budget . A regression above

0.20

is shown and does not remove the cell.

Figure 5: Paired cells. The wedge is inside the band. A point above the upper line clears that seed’s band. Unpaired cells are omitted.

Figure 6: GLM and MiniMax coding paired excess. Black ticks are that seed’s own band. Dark bars lie strictly above it: GLM expel_llm seeds 43 and 44, and MiniMax expel_llm seed 44. Hatched bars are not the same starting draw: GLM expel_llm seed 42, and MiniMax gepa_llm and ace_llm seed 43.

Method	Seed	Gain	Frozen	Excess	Band	Against that band
gepa	42	$-7.5$	$+2.5$	$-10.0$	7.5	outside, negative; r0 gap 10 pp
gepa	43	$-10.0$	$+2.5$	$-12.5$	5.0	outside, negative; r0 gap 7.5 pp
gepa	44	$0.0$	$+2.5$	$-2.5$	2.5	equal
ace	42	$0.0$	$+2.5$	$-2.5$	7.5	inside
ace	43	$0.0$	$+2.5$	$-2.5$	5.0	inside
ace	44	$0.0$	$+2.5$	$-2.5$	2.5	equal; reg. 0.21

Table 10: Manufacturing-v1, DeepSeek v4 Pro. Excess and band are defined as in Table 8 .

Method	Seed	Gain	Frozen	Excess	Band	Reg.	Against that band
gepa	42	$+7.5$	$+10.0$	$-2.5$	17.5	0.250	inside
gepa	43	$0.0$	$+7.5$	$-7.5$	7.5	0.375	equal
gepa	44	$+17.5$	$+15.0$	$+2.5$	17.5	0.143	inside
ace	42	$+5.0$	$+10.0$	$-5.0$	17.5	0.267	inside
ace	43	$+15.0$	$+7.5$	$+7.5$	7.5	0.077	equal
ace	44	$+2.5$	$+15.0$	$-12.5$	17.5	0.385	inside

Table 11: Office-v1, DeepSeek v4 Pro. Excess and band are defined as in Table 8 . Reg. is the final regression rate.

Appendix figures & tables52 assets

Supplementary material from the paper’s appendix.

Appendix

	office-v1	coding-v1	manufacturing-v1
Source	Product-usage workflows in a closed catalog workspace	142 merged pull requests of one production repository	12 pseudonymized real tables (32,290 rows) and 31 real questions
Items	140; 10 families	140; fix_bug	140; 13 families; 31 real, 109 generated
Held-out OOD	12/40: access_denied , do_not_publish	none	12/40: real questions on 4 unseen table families
Language	33/140 Chinese, rest English	Chinese	Chinese
Observation	Catalog files and six tools	One file, src/solution.py	Four tools: list tables, describe a table, execute SQL, submit an answer
Success	Every terminal check passes	Closed-workspace unit tests pass	Score $\geq 80$ of 100: SQL 40, result 40, answer 20; a partial SQL or result check caps the score below 80

Appendix

Table 12: TEB pack card. Frozen range is held-out Success over the four passes of DeepSeek v4 Pro no_evolve , seeds 42–44. Language counts instructions containing Chinese.

Slot	Kept	Omitted
expel_llm	Insights from success and failure; deduplicated pool	Upstream harness and retrieval
awm_llm	Workflows from successful traces only	Upstream induction stack
ace_llm	Generator, reflector, curator; grow-and-refine playbook	Upstream application runtime
gepa_llm	One revised instruction; the TEB gate accepts or rolls it back	Population search and Pareto compile
reasoning_bank	Title, when, and content memories; lexical top- $k$	Upstream agent and test-time search
clin_llm	Condition, action, and effect rules	Domain gold memory

Appendix

Table 13: Method-inspired slots on the Standard Tool Actor. The kept column is the TEB mechanism. The omitted column is the upstream system this paper does not claim to reproduce.

Model	Method	Before	After	Gain	Reg.	Artifact
deepseek-v4-pro	expel_llm	0.167	0.167	-0.333	0.000	true
deepseek-v4-pro	awm_llm	0.333	0.333	-0.167	0.000	true
deepseek-v4-pro	ace_llm	0.167	0.333	-0.167	0.000	true
deepseek-v4-pro	gepa_llm	0.500	0.333	-0.167	0.167	true
deepseek-v4-pro	reasoning_bank	0.333	0.167	-0.333	0.167	false
deepseek-v4-pro	clin_llm	0.500	0.333	-0.167	0.167	true

Appendix

Table 14: Screen cells. Validation columns are success rates. Gain is after minus the model-matched no_evolve rate.

Model	Method	Seed	Round	Version	Before	After	Decision
DS	ace	42	1	$M1$	20.0	40.0	accept
DS	ace	42	2	$M2$	35.0	50.0	accept
DS	ace	42	3	$M3$	35.0	35.0	accept
DS	ace	43	1	$M0$	30.0	20.0	rollback
DS	ace	43	2	$M0$	25.0	5.0	rollback
DS	ace	43	3	$M1$	15.0	35.0	accept

Appendix

Table 15: Validation gate on office-v1. Columns are validation percent before and after the proposal.

n{=}20

, so one task is 5 pp.

Model	Method	Seed	Round	Succ.	Val.	Reg.	Tokens
DS	ace	42	0	32.5	—	0.000	592020
DS	ace	42	1	30.0	40.0	0.231	690436
DS	ace	42	2	37.5	50.0	0.083	667912
DS	ace	42	3	37.5	35.0	0.267	808663
DS	ace	43	0	32.5	—	0.000	576054
DS	ace	43	1	25.0	30.0	0.308	634816

Appendix

Table 16: Office-v1 held-out record. Success and validation are percent. Regression is a rate.

Seed	Round	catalog_discover	brief_report	checklist	csv_audit	access_denied	do_not_publish
42	0	6	0	0	1	2	5
42	1	4	1	1	1	2	4
42	2	4	0	0	1	1	5
42	3	7	2	2	2	1	4
43	0	5	1	0	1	1	3
43	1	6	0	1	1	2	4

Appendix

Table 17: DeepSeek no_evolve solved counts by family and round. incident_track , postmortem , and onboarding are omitted (0 on every pass).

Seed	Round	catalog_discover	brief_report	checklist	csv_audit	access_denied	do_not_publish
42	0	5	1	1	1	1	5
42	1	5	0	1	1	1	2
42	2	5	1	1	1	2	6
42	3	6	2	2	2	1	4
43	0	5	0	1	1	1	4
43	1	5	1	1	1	1	3

Appendix

Table 18: DeepSeek gepa_llm solved counts by family and round. incident_track , postmortem , and onboarding are omitted (0 on every pass).

Seed	Round	catalog_discover	brief_report	checklist	csv_audit	access_denied	do_not_publish
42	0	5	0	2	1	1	4
42	1	5	1	0	1	1	4
42	2	6	2	0	2	2	3
42	3	4	1	2	2	1	5
43	0	5	2	0	1	1	4
43	1	4	0	1	1	1	3

Appendix

Table 19: DeepSeek ace_llm solved counts by family and round. incident_track , postmortem , and onboarding are omitted (0 on every pass).

Seed	Round	catalog_discover	brief_report	checklist	csv_audit	access_denied	do_not_publish
42	0	5	0	0	1	1	3
42	1	5	2	0	1	1	4
42	2	4	1	1	1	1	5
42	3	7	3	1	1	1	4
43	0	5	1	0	1	2	4
43	1	5	1	0	0	2	4

Appendix

Table 20: DeepSeek expel_llm solved counts by family and round. incident_track , postmortem , and onboarding are omitted (0 on every pass).

Cell	Family	Round 0	Last
GLM no_evolve	access_denied	2/6	3/6
GLM no_evolve	brief_report	3/5	1/5
GLM no_evolve	catalog_discover	4/8	5/8
GLM no_evolve	checklist	2/5	0/5
GLM no_evolve	csv_audit	1/4	1/4
GLM no_evolve	do_not_publish	5/6	4/6

Appendix

Table 21: GLM 5.3 and MiniMax M2.5 solved counts, round 0 to last round.

Cell	Seed	r0	r1	r2	r3
GLM no_evolve	42	17/6	22/4	14/5	22/4
GLM gepa	42	14/7	12/10	20/5	22/5
GLM ace	42	18/3	16/5	21/5	20/5
GLM expel	42	20/5	18/2	23/3	23/2
MM no_evolve	42	27/5	29/1	30/2	28/3
MM gepa	42	31/2	29/3	25/4	29/2

Appendix

Table 22: Seed-42 office-v1 transfer. Counts are max_steps / verifier_fail .

Seed	r0	r1	r2	r3
42	17/9	15/8	13/10	16/9
43	16/8	17/7	14/8	15/8
44	15/9	16/8	15/9	15/8

Appendix

Table 23: Manufacturing-v1 no_evolve . Counts are below_threshold / max_steps . These passes have no verifier_fail .

Cell	Seed	Main	Reasons
DS ace	42	false	gated, paper_min
DS ace	43	true	eligible
DS ace	44	false	gated, paper_min
DS expel	42	true	eligible
DS expel	43	true	eligible
DS expel	44	true	eligible

Appendix

Table 24: Stored eligibility reasons for the 20 office-v1 cells. Main is the runner’s office-lane flag, not the coding scoreboard.

Model	Seed	r0	r1	r2	r3	Range
DeepSeek v4 Pro	42	35.0	32.5	27.5	45.0	17.5
DeepSeek v4 Pro	43	27.5	35.0	32.5	35.0	7.5
DeepSeek v4 Pro	44	25.0	30.0	22.5	40.0	17.5
GLM 5.3	42	42.5	35.0	52.5	35.0	17.5
MiniMax M2.5	42	20.0	25.0	20.0	22.5	5.0

Appendix

Table 25: Frozen-policy band. Range is max minus min of the four held-out passes, in percentage points.

Figure 7: Held-out success by round. The first panel is frozen no_evolve , including GLM 5.3 and MiniMax M2.5 at seed 42. DeepSeek seed 42 goes from 27.5% at round 2 to 45% at round 3 with no context update. The other panels are DeepSeek only. no_evolve moves as much as the adapters.

Figure 8: DeepSeek office paired excess. Shading is the widest seed band,

\pm 17.5

pp, not seed 43’s own

7.5

pp band. Each seed’s range is the Range column of Table 25 .

Model	Method	Seed	r0	r1	r2	r3
DS	no_evolve	42	$M_{0}$	$M_{0}$	$M_{0}$	$M_{0}$
DS	no_evolve	43	$M_{0}$	$M_{0}$	$M_{0}$	$M_{0}$
DS	no_evolve	44	$M_{0}$	$M_{0}$	$M_{0}$	$M_{0}$
DS	gepa	42	$M_{0}$	$M_{0}$	$M_{1}$	$M_{1}$
DS	gepa	43	$M_{0}$	$M_{0}$	$M_{1}$	$M_{1}$
DS	gepa	44	$M_{0}$	$M_{0}$	$M_{0}$	$M_{1}$

Appendix

Table 26: Accepted agent_version after each office-v1 round. A repeated version means the artifact did not change. On no_evolve that is every round.

Cell	Version	From	To	Gap
DS no_evolve 42	$M_{0}$	27.5	45.0	17.5
DS no_evolve 44	$M_{0}$	22.5	40.0	17.5
DS gepa 42	$M_{0}$	35.0	25.0	10.0
DS gepa 43	$M_{1}$	40.0	30.0	10.0
DS gepa 44	$M_{0}$	25.0	35.0	10.0
DS expel 43	$M_{2}$	20.0	42.5	22.5

Appendix

Table 27: Consecutive held-out passes with the same agent_version and an absolute gap of at least 10 pp. The accepted artifact did not change between the two scores.

Figure 9: Held-out success (%) at each round for all 20 cells. A dash is a round the runner did not start (MiniMax ace_llm round 3).

Model	Method	Seed	r0	r1	r2	r3	Gain	Excess	Flag
DS	no_evolve	42	35.0	32.5	27.5	45.0	$+10.0$	—
DS	no_evolve	43	27.5	35.0	32.5	35.0	$+7.5$	—	reg.
DS	no_evolve	44	25.0	30.0	22.5	40.0	$+15.0$	—
DS	gepa	42	35.0	25.0	40.0	42.5	$+7.5$	$-2.5$	reg.
DS	gepa	43	30.0	30.0	40.0	30.0	$0.0$	$-7.5$	reg.
DS	gepa	44	25.0	35.0	35.0	42.5	$+17.5$	$+2.5$

Appendix

Table 28: All 20 office-v1 cells. Success columns are held-out percent. Gain and excess are percentage points. Excess is blank on no_evolve . DS / GLM / MM are DeepSeek v4 Pro, GLM 5.3, and MiniMax M2.5.

Figure 10: Seed-42 transfer. Top row: GLM 5.3. Bottom row: MiniMax M2.5. A dash is a round the runner did not start (MiniMax ace_llm round 3). These curves are not a second three-seed confirmation.

Model	Method	Succ.	Gain	Excess	Note
MiniMax	no_evolve	22.5	$+2.5$	—	reg. 0.25
MiniMax	gepa	22.5	$+5.0$	$+2.5$	inside 5.0; reg. 0.273
MiniMax	ace	27.5	$-5.0$	$-7.5$	stopped at round 2
MiniMax	expel	32.5	$+10.0$	$+7.5$	2.5 pp past a 5.0 band
GLM	no_evolve	35.0	$-7.5$	—	reg. 0.381
GLM	gepa	32.5	$-15.0$	$-7.5$	budget; reg. 0.20

Appendix

Table 29: Seed-42 transfer. Excess is against that model’s no_evolve . Not ranked with DeepSeek.

Family	$n$	What the check is
catalog_discover	8	List /catalog , write and publish catalog_index.md
brief_report	5	Bounded summary; invented owners, dates, or claims fail
checklist	5	Required sections only
csv_audit	4	Recompute stated counts or clean a table
incident_track	3	Timeline, risk, and triage; unknowns stay unknown
postmortem	2	Facts and actions split; blame fails

Appendix

Table 30: Office-v1 held-out families.

n

sums to 40. OOD families do not appear in evolution.

Figure 11: DeepSeek family change from round 0 to the final held-out pass, in solved tasks. Zero-change families incident_track , postmortem , and onboarding are included. Color is the task delta.

Method	Seed	r0	r1	r2	r3
no_evolve	42	23/3	25/2	25/4	20/2
no_evolve	43	24/5	21/5	24/3	25/1
no_evolve	44	23/7	26/2	25/6	21/3
gepa	42	24/2	26/4	22/2	22/1
gepa	43	27/1	26/2	19/5	26/2
gepa	44	25/5	22/4	24/2	18/5

Appendix

Table 31: DeepSeek failure counts by round, max_steps / verifier_fail . The two counts sum to the number of held-out failures. No other failure reason appears.

Seed	r0	r1	r2	r3	Range
42	35.0	42.5	42.5	37.5	7.5
43	40.0	40.0	45.0	42.5	5.0
44	40.0	40.0	40.0	42.5	2.5

Appendix

Table 32: Manufacturing-v1, DeepSeek v4 Pro, no_evolve only. Success is held-out percent. Version is

M_{0}

at every round.

Family	$n$	Seed 42	Seed 43	Seed 44
accounts_payable	3	0/0/0/0	0/0/0/0	0/0/0/0
advance_payments	3	0/0/0/0	0/0/0/0	0/0/0/0
advance_receipts	1	0/0/0/0	0/0/0/0	0/0/0/0
bpc_report	7	0/0/0/0	0/0/0/0	0/0/0/0
commodity	3	1/2/2/2	2/2/2/2	2/2/2/2
electricity	3	3/3/3/3	3/3/3/3	3/3/3/3

Appendix

Table 33: Manufacturing-v1 solved counts by family at rounds 0–3, written r0/r1/r2/r3.

Model	Seed	r0	r1	r2	r3	Gain	Range
GLM	42	72.5	72.5	60.0	70.0	-2.5	12.5
GLM	43	60.0	65.0	67.5	67.5	+7.5	7.5
GLM	44	67.5	70.0	75.0	60.0	-7.5	15.0
MiniMax	42	30.0	37.5	40.0	35.0	+5.0	10.0
MiniMax	43	37.5	35.0	37.5	37.5	+0.0	2.5
MiniMax	44	40.0	37.5	27.5	40.0	+0.0	12.5

Appendix

Table 34: Coding frozen-policy passes for GLM and MiniMax. Gain and Range for all three models are Table 7 . DeepSeek passes are Table 6 . Range is max minus min of the four held-out passes, in percentage points.

Figure 12: DeepSeek coding held-out success (%) at each round. Twelve cells: four methods by three seeds. Paired excess is Figure 4 .

Figure 13: GLM (top row) and MiniMax (bottom row) coding held-out success by round. Both models have seeds 42, 43, and 44. Line color is the seed.

Figure 14: GLM and MiniMax coding held-out success (%) at each round. GLM ace_llm seeds 42 and 43 have no round 3. Paired excess for these cells is Figure 6 .

Model	Method	Seed	r0	r1	r2	r3
DeepSeek	no_evolve	42	$M_{0}$	$M_{0}$	$M_{0}$	$M_{0}$
DeepSeek	no_evolve	43	$M_{0}$	$M_{0}$	$M_{0}$	$M_{0}$
DeepSeek	no_evolve	44	$M_{0}$	$M_{0}$	$M_{0}$	$M_{0}$
DeepSeek	gepa	42	$M_{0}$	$M_{1}$	$M_{1}$	$M_{1}$
DeepSeek	gepa	43	$M_{0}$	$M_{1}$	$M_{1}$	$M_{1}$
DeepSeek	gepa	44	$M_{0}$	$M_{1}$	$M_{2}$	$M_{2}$

Appendix

Table 35: Accepted agent_version after each coding round. A repeated version means the artifact did not change. On no_evolve that is every round.

Model	Method	Seed	Version	Rounds	From	To	Gap
DeepSeek	no_evolve	44	$M_{0}$	2–3	65.0	52.5	-12.5
DeepSeek	gepa	44	$M_{2}$	2–3	67.5	42.5	-25.0
DeepSeek	ace	42	$M_{0}$	0–1	67.5	52.5	-15.0
GLM	no_evolve	42	$M_{0}$	1–2	72.5	60.0	-12.5
GLM	no_evolve	42	$M_{0}$	2–3	60.0	70.0	+10.0
GLM	no_evolve	44	$M_{0}$	2–3	75.0	60.0	-15.0

Appendix

Table 36: Consecutive coding held-out passes with the same agent_version and an absolute gap of at least 10 pp. The accepted artifact did not change between the two scores.

Method	Seed	r0	r1	r2	r3
no_evolve	42	13	13	14	16
no_evolve	43	15	14	15	16
no_evolve	44	15	15	14	19
gepa	42	15	9	10	12
gepa	43	15	10	11	10
gepa	44	14	10	13	23

Appendix

Table 37: DeepSeek coding failure counts by round. The count is max_steps . verifier_fail is absent on every pass.

Figure 15: Archived ticket-gym, not office-v1. Split is 40/10/20 and held-out

n{=}20

. OOD is locked in held-out.

Env	Family	Evo	Val	Held-out
ticket-gym	ticket_close	12	3	4
ticket-gym	dual_ticket	10	3	4
ticket-gym	auth_recovery	9	2	3
ticket-gym	file_report	9	2	3
ticket-gym	access_denied (OOD)	0	0	6
inbox	triage	16	4	8

Appendix

Table 38: Archived ticket-gym family counts, held-out

n{=}20

, not office-v1. OOD families are held-out only. The same table lists the inbox mock and the office_work expansion.

Board	Succ.	Gain	dual	file	OOD	Seeds
B0 reflexion (heuristic)	70%	$+20$ vs $0.50$	$4/4$	$3/3$	$0/6$	3
B0 other (heuristic)	50%	$+0$	$0/4$	$3/3$	$0/6$	3
Weight attempt (ineligible)	50%	$+0$ vs $0.50$	$0/4$	$3/3$	$0/6$	1
3B reflexion_llm	55%	$+10$ vs $0.45$	$0/4$	$0/3$	$4/6$	1
3B other wrappers	$45$ – $55\%$	$+0$ this-run	$0/4$	$0/3$	$3$ – $4/6$	1

Appendix

Table 39: Archived ticket-gym mock, held-out

n{=}20

. Not an office-v1 score. Heuristic rows are three seeds. Wrapper rows are one seed. The weight attempt submitted no training.

Family	$n$	$M_{0}$ / other B0	reflexion
ticket_close	4	$4/4$	$4/4$
issue_triage	3	$0/3$	$3/3$
release_notes	2	$0/2$	$2/2$
postmortem	2	$0/2$	$2/2$
diff_review	2	$0/2$	$2/2$
table_brief	2	$0/2$	$2/2$

Appendix

Table 40: Archived ticket-gym expansion pack office_work , not office-v1. Held-out

n{=}20

, three identical seeds. Heuristic B0 only. Not ranked vs. Table 39 .

Figure 16: Archived ticket-gym tool calls, not office-v1. Locked held-out ids ( SandboxAgent , not a 3B log). (a)

M_{0}

dual: recoverable failed submit , second ticket never closed. (b) Same id after workflow:dual_ticket_close . (c)

M_{0}

OOD: no grant_access . (d) OOD after grant_access_on_denied . Consecutive identical calls are collapsed (

\times n

Evolver	s42	s43	s44	dual	OOD
reflexion (stand-in)	70	70	70	$4/4$	$0/6$
no_evolve	50	50	50	$0/4$	$0/6$
prompt_memory	50	50	50	$0/4$	$0/6$
success_memory	50	50	50	$0/4$	$0/6$
expel	50	50	50	$0/4$	$0/6$
awm	50	50	50	$0/4$	$0/6$

Appendix

Table 41: Archived ticket-gym heuristic B0, not office-v1. Held-out Success % by seed. Other B0 methods are no_evolve , prompt_memory , success_memory , expel , awm .

Loop	r0	r1	r2	r3	Gain	OOD	Wall (h)
reflexion_llm	45	55	55	55	$+10$	$4/6$	4.0
gepa_llm	50	50	50	50	$+0$	$3/6$	3.8
letta wrapper	55	55	55	55	$+0$	$4/6$	5.9
dgm wrapper	45	45	45	45	$+0$	$3/6$	5.7
agent_evolver	45	50	55	55	jitter	$4/6$	5.4

Appendix

Table 42: Archived ticket-gym 3B tip-wrappers, not office-v1. Seed 42. Columns r0–r3 are held-out Success % of the accepted agent. reflexion_llm r3 rolls back (card is M2 55%). agent_evolver

45{\to}50{\to}55

occurs under three rollbacks (jitter), not a second

+10

pp. Dual

0/4

and file

0/3

on every row. Wall is loop process time, not time-to-first-token.

Board	Succ.	Gain	reply	thread	OOD	Seeds
B0 reflexion	70%	$+30$ vs $0.40$	$3/3$	$3/3$	$0/6$	3
B0 other	40%	$+0$	$0/3$	$0/3$	$0/6$	3

Appendix

Table 43: Archived inbox mock, not office-v1. Held-out

n{=}20

. Not ranked vs. Table 39 . Reply

n{=}3

, thread

n{=}3

, OOD quarantine

n{=}6

Prompt	Succ.	tc	auth	file	OOD
Hinted wrapper $M_{0}$	45%	$4/4$	$2/3$	$0/3$	$3/6$
No hint $M_{0}$	30%	$4/4$	$2/3$	$0/3$	$0/6$

Appendix

Table 44: Archived ticket-gym hygiene, not office-v1. Same frozen 3B, serving prompt with vs. without grant_access hint. Eval-only (

R{=}0

). Dual

0/4

on both rows. Table 42 keeps the hinted prompt.

Card	r1	r2	r3	Held-out note
B0 reflexion	A	A	A	70% from r1
reflexion_llm	A	A	R	stays M2 55%
gepa / letta / dgm wrap	R	R	R	this-run $M_{0}$
agent_evolver	R	R	R	$45{\to}55$ jitter
letta_native	A	A	R	r1 is 10%; final M2 jitter 65%
dgm_native	A	A	A	val 70%, held-out 50%

Appendix

Table 45: Archived ticket-gym gates, not office-v1 (Table 15 ). A = accept, R = rollback. dgm_native can accept a validation 70% proposal while held-out stays 50%. qwen-plus r3 accepts

0{\to}0

; the card is final 5%, not the M2 peak.

Figure 17: Archived ticket-gym gates, not office-v1. Accept and rollback are not held-out Gain.

Card	r0	r1	r2	r3	Gain	dual	OOD
letta_native	60	10	55	65	$+5$ jitter	$2/4$	$6/6$
dgm_native	50	50	50	50	$+0$	$0/4$	$0/6$
qwen-plus	5	5	15	5	$+0$	$0/4$	$1/6$

Appendix

Table 46: Archived ticket-gym native pairing, not office-v1. Seed 42. Columns r0–r3 are held-out Success % of the accepted agent. Dual/OOD are the frozen (final) card. Not ranked vs. Table 39 . letta_native r3 rolls back: final 65% is M2 re-eval jitter vs. r2 55%. qwen-plus peak r2 15% is not the card. dgm_native actor is SandboxAgent (genome only). File: Letta/DGM

3/3

, qwen-plus

0/3

. Ticket-close: Letta

0/4

, DGM

4/4

, qwen-plus

0/4

Figure 18: Archived ticket-gym native pairing board, not office-v1. Not ranked vs. Table 39 .

Evolver	Teacher traces	Self traces
schema_memory	100%	0%
prompt_memory	0%	0%
no_evolve	0%	0%

Appendix

Table 47: NL2SQL teacher contrast, not office-v1. Held-out Success %, mean of seeds 42–44 (cells identical across seeds). Self has no successful SQL in traces.

Figure 19: NL2SQL teacher contrast, not office-v1. (a) Teacher traces. (b) Self traces, which contain no successful SQL, so every evolver stays at 0%.

Evolver	Success	Gain	in-dist	OOD
reflexion (stand-in)	75%	$+55$ pp	$15/15$	$0/5$
other B0	20%	$+0$ pp	$4/15$ (tc only)	$0/5$

Appendix

Table 48: Archived ticket-gym office_work heuristic B0, not office-v1. Held-out Success %. Identical across seeds 42–44. Other B0: no_evolve , prompt_memory , success_memory , expel , awm .

Figure 20: Archived ticket-gym expansion pack, not office-v1. Not ranked vs. Table 39 .

Env	Board	Budget	Succ.	Gain	In-dist family	OOD family	Seeds
ticket-gym	B0 reflexion (heur.)	B0	70%	$+20$ vs 0.50	dual 4/4	0/6	3
ticket-gym	3B reflexion_llm wrap	B0	55%	$+10$ vs 0.45	dual 0/4	4/6	1
inbox	B0 reflexion (heur.)	B0	70%	$+30$ vs 0.40	reply 3/3, thread 3/3	0/6	3
inbox	B0 other (heur.)	B0	40%	$+0$	0/3	0/6	3
NL2SQL	Teacher schema_mem	—	100%	$+100$ vs 0	OOD 4/4	—	3
(contrast)	Self (all evolvers)	—	0%	—	0/4	—	3

Appendix

Table 49: Archived cards across ticket-gym, inbox, NL2SQL, and office_work . Not an office-v1 ranking. “—” = family not present in that environment.

AI Agent Benchmarks

AI agent benchmarks measure how reliably autonomous systems plan, use tools, interact, and complete tasks across controlled and real-world settings.

See all 1,503 papers

May 28, 2026

BenchTrace: A Benchmark for Testing Reflection Ability and Controlled Evolution in LLM Agents

Self-evolving agents improve over time by reflecting on past failures, but existing evaluation is limited in two ways: it measures only task scores, leaving reflection quality unknown, and it relies on agents' own episode runs, offering no mechanism to target specific failure patterns. We present BenchTrace, a benchmark for evaluating self-evolution ability in LLM agents. BenchTrace is built on a snapshot-reflection dataset of 1,821 annotated episodes spanning six diverse tasks, and comprises a Reflection Evaluation that probes failure identification through targeted QA tasks, and an Evolution Evaluation that tests whether past failure experience translates into avoidance behavior in a controlled self-evolution simulation. Building on BenchTrace, we propose failure avoidance rate (FAR), a new evaluation metric measuring the fraction of test cases in which the agent successfully avoids the target failure instance. Experiments with Qwen3-32B and GPT-4.1 reveal that both models fall below a 30% end-to-end pass rate on reflection evaluation, with diagnosis as the primary bottleneck. Evolution evaluation shows that self-evolution methods generally improve FAR over the non-evolving baseline, but agents forget early lessons as noise episodes accumulate, and agents fail to generalize their reflections beyond the specific context, causing negative transfer across task contexts. Our correlation analysis further reveals that only a fully correct reflection is strongly associated with higher FAR. BenchTrace exposes concrete limits of current self-evolution approaches and provides a controlled, model-agnostic framework for targeted evaluation.

Jiahao Huang, Fei Cheng, Junfeng Jiang +2

University of Tokyo · Kyoto University · National Institute of Informatics

LLM Agent Evaluation Self-Reflection in Language Model Agents AI Agent Benchmarks LLM Agents Self-Evolving Agents

Ask PDF

Jun 16, 2026

SEAGym: An Evaluation Environment for Self-Evolving LLM Agents

Self-evolving LLM-based agents improve mainly by changing their agent harness: the structured execution layer around a base model, including prompts, memory, tools, middleware, runtime state, and the model-tool interaction loop. Existing evaluations often reduce this process to isolated task scores or a single sequential curve, obscuring whether an update produces reusable improvement, overfits recent tasks, increases cost, or harms older behavior. We introduce SEAGym, an evaluation environment for measuring agent harness updates across training, validation, test, replay, and cost records. SEAGym turns Harbor-compatible benchmarks into dynamic self-evolution task sources with train batches, frozen update-validation, held-out ID and OOD transfer views, replay diagnostics, and saved snapshot and metric records. Instantiating SEAGym on Terminal-Bench 2.0 and HLE, we compare ACE, TF-GRPO, and AHE under a shared epoch/batch protocol. The results show that these evaluation views provide complementary signals about the evolution process: frequent updates may fail to improve held-out performance, useful intermediate snapshots may collapse later, and source diversity and model backend can affect harness reliability.

Congjie Zheng, Chuanyi Xue, Bin Liang +2

Department of Automation, Tsinghua University, Beijing, 100084, China · Beijing National Research Center for Information Science and Technology (BNRist), Tsinghua University, Beijing, 100084, China

LLM Agent Evaluation AI Agent Benchmarks Automated Agent Evaluation LLM Agent Reliability Self-Evolving Agents

Ask PDF

Sep 29, 2026

Self-Evolving Harness on Multiple Tasks with the Agent as Its Own Optimizer

A harness is the code around a language-model agent that organizes prompts, calls tools, manages context, and controls execution. As models grow stronger, recent work has begun to let agents improve their own harnesses, a line of work known as self-evolving harnesses. In most existing methods, a separate proposer running on a human-designed harness modifies the solver's harness, and a separate harness is evolved for each benchmark. Real-world tasks come from many domains, so both the evolution and the evaluation of a harness should cover a diverse range of tasks. We propose a framework close to recursive self-improvement: the same frozen model, on the same version of the harness, first solves tasks as the solver and then, as the proposer, reads the complete run records and directly edits the harness that runs it. Each evolution batch draws tasks from five benchmarks in different domains. To measure generalization, training and held-out tasks are strictly separated, and we additionally evaluate on five out-of-distribution benchmarks never used during evolution. We frame the evolution process as deep-learning training with two stages, multi-task pretraining and continual training. Starting from a 49-line seed harness, the harness obtained at the end of the first stage improves the average score by 4.48 points on the in-distribution benchmarks and by 12.64 points on the out-of-distribution benchmarks, surpassing Codex on the former and matching it on the latter. In the second stage, continued evolution on Claw-Eval, one of the out-of-distribution benchmarks, further raises the score on that benchmark from 66.17 to 68.06, exceeding Codex. We also provide an in-depth analysis of the mechanisms that emerged during evolution, including output truncation, history compaction, and independent review.

Qiankai Xu

Nanjing University

Continual Learning for LLM Agents AI Agent Evaluation Agent Harness Optimization LLM Agent Workflow Optimization LLM Agent Self-Improvement

Ask PDF

Abstract

AI Agent Benchmarks

Explore similar work

BenchTrace: A Benchmark for Testing Reflection Ability and Controlled Evolution in LLM Agents

SEAGym: An Evaluation Environment for Self-Evolving LLM Agents

Self-Evolving Harness on Multiple Tasks with the Agent as Its Own Optimizer