cs.AIOct 10, 2026

Trace Evolution Bench: Measuring Whether Agents Improve from Their Own Environment Traces

Authors: Fangfei Li, Chenyang Zhao, Long Wang, Feng Tian, Lv Guo

Organizations: MatrixOrigin, China

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Self-evolving agents are usually scored by endpoint Gain: held-out success after an update minus success before. That difference can credit a frozen rescore as evolution. We introduce Trace Evolution Bench (TEB), a protocol that separates that rescore from a context or weight update, together with three frozen 80/20/40 packs. An update is accepted only when validation does not fall, held-out identifiers never enter the update, and a text update is not ranked with a weight update. The primary scoreboard is coding, graded by hidden unit tests in a closed workspace. On that pack (40 held-out tasks, one family), DeepSeek v4 Pro under no_evolve never changes its policy yet moves 7.5, 5.0, and 12.5 percentage points across three seeds. Of nine text-update cells inspired by GEPA, ACE, and ExpeL, seven have positive excess above that seed's own band, and one gepa_llm seed moves down by the width of its band (-12.5, regression 0.48). GLM and MiniMax do not reproduce that seven-of-nine pattern. No method stays above its own band on every seed of all three models: GLM has two paired cells above their own bands, and MiniMax expel_llm seed 44 is a third (regression 0.30). The methods were fixed on a validation screen before held-out was read. Office, graded by rule-bounded delivery, and manufacturing table QA are separate packs: office DeepSeek does not exceed its own bands (widest 17.5 points), and manufacturing has no paired positive excess beyond its seed band.

Figures & tables

BenchmarkScored unitFrozen rescore band
SWE-bench, AgentBench ( Jimenez et al., 2024 ; Liu et al., 2024 )One episodeNo
LifelongAgentBench ( Zheng et al., 2025 )A task sequenceNo
SEA-Eval ( Jiang et al., 2026 )A task sequenceNo
EvoAgentBench ( Gao et al., 2026 )Transfer on an ability-supported splitNo; one vanilla condition
EvoPathBench ( Lin et al., 2026 )Capability checkpoints over ordered streamsNo
S 3 Gym ( Shi et al., 2026 )Strict held-out evaluationNo
Table 1: Evaluation target of each benchmark. A frozen rescore band is the range of repeated held-out passes of one unchanged policy.
PropertyTEBRSEASEAEvoGDPRSIPTBTRMLEEPBS3SEBPASTFin
Own traces✓✓✓✓✓×✓✓✓✓✓×✓✓
Multi-round✓✓✓××✓×××✓✓×✓✓
Carrier flex. in this paper (m/p/c/w)××××××××××××××
Explore/eval split✓××✓××✓××✓✓×××
Validation gate✓×××××××××××××
Held-out lock✓××✓××✓××✓✓×××
Table 2: Protocol mechanics contrast (binary coding). ✓ means the property is explicitly supported in the paper’s main protocol; × means otherwise (including not stated). In “Carrier flex. in this paper (m/p/c/w)”, m/p/c/w = memory/prompt/code/weight. Abbreviations: SEA (SEAGym), Evo (EvoAgentBench), GDP (GDPevo), RSI (RSIBench-Data), PTB (PostTrainBench), TR (TRACE), MLE (OpenMLE), EPB (EvoPathBench), S3 (S 3 Gym), SEB (SE-Bench), PAST (PAST-Bench), Fin (FinEvo-Bench).
Figure 1: TEB global evaluation contract across rounds. Round 0 scores M0M_{0} and does not update. Rounds 1–3 each accept or roll back one text update, then rescore the same held-out. Excess is read against that seed’s own four-pass band.
Figure 2: Per-round accept/rollback mechanics (this loop is rounds r≥1r\geq 1 in Figure 1 ). Round 0 freezes Gain on the locked held-out ids. Later rounds call evolve() once; accept and rollback both score that same held-out. Held-out never enters evolve() . B0 is not ranked against B2.
 OfficeCodingManufacturing
SourceProduct-usage workflows142 merged pull requests, rewritten as functions12 real tables; 31 real questions, 109 generated
Items10 familiesOne family, fix_bug13 families
Outside evolution12/40: access_denied , do_not_publishNone on held-out12/40: real questions on 4 unseen table families
Toolslist_files , read_file , write_file , publish_artifact , grant_access , submitlist_dir , read_file , write_file , run_tests , submit ; one file, src/solution.pylist_tables , describe_table , execute_sql , submit_answer
SuccessEvery terminal check passesHidden unit tests passScore ≥80\geq 80 : SQL 40, result 40, answer 20
Step cap201212
Table 3: Three frozen packs. Each has 140 tasks, split 80/20/40, so held-out n=40n{=}40 . The observation omits gold files, expected SQL, and unit-test oracles.
MethodValidGain (pp)Mean reg.Tokens
gepa_llmtrue0.00.00.056469231
ace_llmtrue−5.6-5.60.111513033
expel_llmtrue−11.1-11.10.056448950
dgmtrue−16.7-16.70.111396406
clin_llmtrue−16.7-16.70.222411690
awm_llmfalse0.00.00.000447296
Table 4: Screen ranking across three models at seed 42. Mean gain is in percentage points. Mean reg. is a rate. Valid is the artifact check.
MethodInjected text
gepa_llmOne revised instruction. The validation gate accepts it or rolls it back.
ace_llmA grow-and-refine playbook from a generator, reflector, and curator.
expel_llmA deduplicated pool of insights from success and failure.
Table 5: Text injected by the three scored methods. Each reads that round’s own traces. The kept text is the TEB mechanism.
Figure 3: DeepSeek coding held-out success by round. no_evolve keeps version M0M_{0} and the score still moves. Seeds are 42, 43, and 44.
Seedr0r1r2r3GainRange
4267.567.565.060.0−7.5-7.57.5
4362.565.062.560.0−2.5-2.55.0
4462.562.565.052.5−10.0-10.012.5
Table 6: Coding, DeepSeek v4 Pro, no_evolve only. Success is held-out percent. Version is M0M_{0} at every round. Range is that seed’s band.
ModelSeedFrozen GainBand
DeepSeek42−7.5-7.57.5
DeepSeek43−2.5-2.55.0
DeepSeek44−10.0-10.012.5
GLM42−2.5-2.512.5
GLM43+7.5+7.57.5
GLM44−7.5-7.515.0
Table 7: Coding no_evolve by model. Frozen Gain is the four-pass Gain. Band is that seed’s four-pass range. DeepSeek pass scores are Table 6 .
MethodSeedGainFrozenExcessBandReg.Against that band
gepa42+7.5+7.5−7.5-7.5+15.0+15.07.50.167outside
gepa43+12.5+12.5−2.5-2.5+15.0+15.05.00.069outside
gepa44−22.5-22.5−10.0-10.0−12.5-12.512.50.481equal, down
ace42+12.5+12.5−7.5-7.5+20.0+20.07.50.065outside
ace43+7.5+7.5−2.5-2.5+10.0+10.05.00.071outside
ace440.00.0−10.0-10.0+10.0+10.012.50.130inside
Table 8: Coding-v1, DeepSeek v4 Pro. Excess is method Gain minus the same-seed no_evolve Gain, in percentage points. Band is that seed’s four-pass range. Reg. is the final regression rate. None of these nine cells is over_budget .
Figure 4: DeepSeek coding paired excess. Black ticks are that seed’s own band. Dark bars lie strictly above it. The light positive bar is ace_llm seed 44, inside its 12.512.5 pp band. gepa_llm seed 44 equals its band on the way down.
ModelMethodSeedExcessBandReg.CapAgainst that band
GLMgepa42+7.5+7.512.50.207 inside
GLMgepa430.00.07.50.069 inside
GLMgepa44+15.0+15.015.00.094 equal
GLMace42−12.5-12.512.50.130yesequal, down
GLMace43+5.0+5.07.50.065yesinside
GLMace44+15.0+15.015.00.091 equal
Table 9: Coding status for GLM and MiniMax. Excess and band are defined as in Table 8 . Unpaired means the round-0 gap is at least 10 pp. Reg. is the final regression rate. Cap marks over_budget . A regression above 0.200.20 is shown and does not remove the cell.
Figure 5: Paired cells. The wedge is inside the band. A point above the upper line clears that seed’s band. Unpaired cells are omitted.
Figure 6: GLM and MiniMax coding paired excess. Black ticks are that seed’s own band. Dark bars lie strictly above it: GLM expel_llm seeds 43 and 44, and MiniMax expel_llm seed 44. Hatched bars are not the same starting draw: GLM expel_llm seed 42, and MiniMax gepa_llm and ace_llm seed 43.
MethodSeedGainFrozenExcessBandAgainst that band
gepa42−7.5-7.5+2.5+2.5−10.0-10.07.5outside, negative; r0 gap 10 pp
gepa43−10.0-10.0+2.5+2.5−12.5-12.55.0outside, negative; r0 gap 7.5 pp
gepa440.00.0+2.5+2.5−2.5-2.52.5equal
ace420.00.0+2.5+2.5−2.5-2.57.5inside
ace430.00.0+2.5+2.5−2.5-2.55.0inside
ace440.00.0+2.5+2.5−2.5-2.52.5equal; reg. 0.21
Table 10: Manufacturing-v1, DeepSeek v4 Pro. Excess and band are defined as in Table 8 .
MethodSeedGainFrozenExcessBandReg.Against that band
gepa42+7.5+7.5+10.0+10.0−2.5-2.517.50.250inside
gepa430.00.0+7.5+7.5−7.5-7.57.50.375equal
gepa44+17.5+17.5+15.0+15.0+2.5+2.517.50.143inside
ace42+5.0+5.0+10.0+10.0−5.0-5.017.50.267inside
ace43+15.0+15.0+7.5+7.5+7.5+7.57.50.077equal
ace44+2.5+2.5+15.0+15.0−12.5-12.517.50.385inside
Table 11: Office-v1, DeepSeek v4 Pro. Excess and band are defined as in Table 8 . Reg. is the final regression rate.
Appendix figures & tables52 assets

Supplementary material from the paper’s appendix.

Appendix

 office-v1coding-v1manufacturing-v1
SourceProduct-usage workflows in a closed catalog workspace142 merged pull requests of one production repository12 pseudonymized real tables (32,290 rows) and 31 real questions
Items140; 10 families140; fix_bug140; 13 families; 31 real, 109 generated
Held-out OOD12/40: access_denied , do_not_publishnone12/40: real questions on 4 unseen table families
Language33/140 Chinese, rest EnglishChineseChinese
ObservationCatalog files and six toolsOne file, src/solution.pyFour tools: list tables, describe a table, execute SQL, submit an answer
SuccessEvery terminal check passesClosed-workspace unit tests passScore ≥80\geq 80 of 100: SQL 40, result 40, answer 20; a partial SQL or result check caps the score below 80
Appendix
Table 12: TEB pack card. Frozen range is held-out Success over the four passes of DeepSeek v4 Pro no_evolve , seeds 42–44. Language counts instructions containing Chinese.
SlotKeptOmitted
expel_llmInsights from success and failure; deduplicated poolUpstream harness and retrieval
awm_llmWorkflows from successful traces onlyUpstream induction stack
ace_llmGenerator, reflector, curator; grow-and-refine playbookUpstream application runtime
gepa_llmOne revised instruction; the TEB gate accepts or rolls it backPopulation search and Pareto compile
reasoning_bankTitle, when, and content memories; lexical top- kkUpstream agent and test-time search
clin_llmCondition, action, and effect rulesDomain gold memory
Appendix
Table 13: Method-inspired slots on the Standard Tool Actor. The kept column is the TEB mechanism. The omitted column is the upstream system this paper does not claim to reproduce.
ModelMethodBeforeAfterGainReg.Artifact
deepseek-v4-proexpel_llm0.1670.167-0.3330.000true
deepseek-v4-proawm_llm0.3330.333-0.1670.000true
deepseek-v4-proace_llm0.1670.333-0.1670.000true
deepseek-v4-progepa_llm0.5000.333-0.1670.167true
deepseek-v4-proreasoning_bank0.3330.167-0.3330.167false
deepseek-v4-proclin_llm0.5000.333-0.1670.167true
Appendix
Table 14: Screen cells. Validation columns are success rates. Gain is after minus the model-matched no_evolve rate.
ModelMethodSeedRoundVersionBeforeAfterDecision
DSace421M1M120.040.0accept
DSace422M2M235.050.0accept
DSace423M3M335.035.0accept
DSace431M0M030.020.0rollback
DSace432M0M025.05.0rollback
DSace433M1M115.035.0accept
Appendix
Table 15: Validation gate on office-v1. Columns are validation percent before and after the proposal. n=20n{=}20 , so one task is 5 pp.
ModelMethodSeedRoundSucc.Val.Reg.Tokens
DSace42032.5—0.000592020
DSace42130.040.00.231690436
DSace42237.550.00.083667912
DSace42337.535.00.267808663
DSace43032.5—0.000576054
DSace43125.030.00.308634816
Appendix
Table 16: Office-v1 held-out record. Success and validation are percent. Regression is a rate.
SeedRoundcatalog_discoverbrief_reportchecklistcsv_auditaccess_denieddo_not_publish
420600125
421411124
422400115
423722214
430510113
431601124
Appendix
Table 17: DeepSeek no_evolve solved counts by family and round. incident_track , postmortem , and onboarding are omitted (0 on every pass).
SeedRoundcatalog_discoverbrief_reportchecklistcsv_auditaccess_denieddo_not_publish
420511115
421501112
422511126
423622214
430501114
431511113
Appendix
Table 18: DeepSeek gepa_llm solved counts by family and round. incident_track , postmortem , and onboarding are omitted (0 on every pass).
SeedRoundcatalog_discoverbrief_reportchecklistcsv_auditaccess_denieddo_not_publish
420502114
421510114
422620223
423412215
430520114
431401113
Appendix
Table 19: DeepSeek ace_llm solved counts by family and round. incident_track , postmortem , and onboarding are omitted (0 on every pass).
SeedRoundcatalog_discoverbrief_reportchecklistcsv_auditaccess_denieddo_not_publish
420500113
421520114
422411115
423731114
430510124
431510024
Appendix
Table 20: DeepSeek expel_llm solved counts by family and round. incident_track , postmortem , and onboarding are omitted (0 on every pass).
CellFamilyRound 0Last
GLM no_evolveaccess_denied2/63/6
GLM no_evolvebrief_report3/51/5
GLM no_evolvecatalog_discover4/85/8
GLM no_evolvechecklist2/50/5
GLM no_evolvecsv_audit1/41/4
GLM no_evolvedo_not_publish5/64/6
Appendix
Table 21: GLM 5.3 and MiniMax M2.5 solved counts, round 0 to last round.
CellSeedr0r1r2r3
GLM no_evolve4217/622/414/522/4
GLM gepa4214/712/1020/522/5
GLM ace4218/316/521/520/5
GLM expel4220/518/223/323/2
MM no_evolve4227/529/130/228/3
MM gepa4231/229/325/429/2
Appendix
Table 22: Seed-42 office-v1 transfer. Counts are max_steps / verifier_fail .
Seedr0r1r2r3
4217/915/813/1016/9
4316/817/714/815/8
4415/916/815/915/8
Appendix
Table 23: Manufacturing-v1 no_evolve . Counts are below_threshold / max_steps . These passes have no verifier_fail .
CellSeedMainReasons
DS ace42falsegated, paper_min
DS ace43trueeligible
DS ace44falsegated, paper_min
DS expel42trueeligible
DS expel43trueeligible
DS expel44trueeligible
Appendix
Table 24: Stored eligibility reasons for the 20 office-v1 cells. Main is the runner’s office-lane flag, not the coding scoreboard.
ModelSeedr0r1r2r3Range
DeepSeek v4 Pro4235.032.527.545.017.5
DeepSeek v4 Pro4327.535.032.535.07.5
DeepSeek v4 Pro4425.030.022.540.017.5
GLM 5.34242.535.052.535.017.5
MiniMax M2.54220.025.020.022.55.0
Appendix
Table 25: Frozen-policy band. Range is max minus min of the four held-out passes, in percentage points.
Figure 7: Held-out success by round. The first panel is frozen no_evolve , including GLM 5.3 and MiniMax M2.5 at seed 42. DeepSeek seed 42 goes from 27.5% at round 2 to 45% at round 3 with no context update. The other panels are DeepSeek only. no_evolve moves as much as the adapters.
Figure 8: DeepSeek office paired excess. Shading is the widest seed band, ±17.5\pm 17.5 pp, not seed 43’s own 7.57.5 pp band. Each seed’s range is the Range column of Table 25 .
ModelMethodSeedr0r1r2r3
DSno_evolve42M0M_{0}M0M_{0}M0M_{0}M0M_{0}
DSno_evolve43M0M_{0}M0M_{0}M0M_{0}M0M_{0}
DSno_evolve44M0M_{0}M0M_{0}M0M_{0}M0M_{0}
DSgepa42M0M_{0}M0M_{0}M1M_{1}M1M_{1}
DSgepa43M0M_{0}M0M_{0}M1M_{1}M1M_{1}
DSgepa44M0M_{0}M0M_{0}M0M_{0}M1M_{1}
Appendix
Table 26: Accepted agent_version after each office-v1 round. A repeated version means the artifact did not change. On no_evolve that is every round.
CellVersionFromToGap
DS no_evolve 42M0M_{0}27.545.017.5
DS no_evolve 44M0M_{0}22.540.017.5
DS gepa 42M0M_{0}35.025.010.0
DS gepa 43M1M_{1}40.030.010.0
DS gepa 44M0M_{0}25.035.010.0
DS expel 43M2M_{2}20.042.522.5
Appendix
Table 27: Consecutive held-out passes with the same agent_version and an absolute gap of at least 10 pp. The accepted artifact did not change between the two scores.
Figure 9: Held-out success (%) at each round for all 20 cells. A dash is a round the runner did not start (MiniMax ace_llm round 3).
ModelMethodSeedr0r1r2r3GainExcessFlag
DSno_evolve4235.032.527.545.0+10.0+10.0— 
DSno_evolve4327.535.032.535.0+7.5+7.5—reg.
DSno_evolve4425.030.022.540.0+15.0+15.0— 
DSgepa4235.025.040.042.5+7.5+7.5−2.5-2.5reg.
DSgepa4330.030.040.030.00.00.0−7.5-7.5reg.
DSgepa4425.035.035.042.5+17.5+17.5+2.5+2.5 
Appendix
Table 28: All 20 office-v1 cells. Success columns are held-out percent. Gain and excess are percentage points. Excess is blank on no_evolve . DS / GLM / MM are DeepSeek v4 Pro, GLM 5.3, and MiniMax M2.5.
Figure 10: Seed-42 transfer. Top row: GLM 5.3. Bottom row: MiniMax M2.5. A dash is a round the runner did not start (MiniMax ace_llm round 3). These curves are not a second three-seed confirmation.
ModelMethodSucc.GainExcessNote
MiniMaxno_evolve22.5+2.5+2.5—reg. 0.25
MiniMaxgepa22.5+5.0+5.0+2.5+2.5inside 5.0; reg. 0.273
MiniMaxace27.5−5.0-5.0−7.5-7.5stopped at round 2
MiniMaxexpel32.5+10.0+10.0+7.5+7.52.5 pp past a 5.0 band
GLMno_evolve35.0−7.5-7.5—reg. 0.381
GLMgepa32.5−15.0-15.0−7.5-7.5budget; reg. 0.20
Appendix
Table 29: Seed-42 transfer. Excess is against that model’s no_evolve . Not ranked with DeepSeek.
FamilynnOODWhat the check is
catalog_discover80List /catalog , write and publish catalog_index.md
brief_report50Bounded summary; invented owners, dates, or claims fail
checklist50Required sections only
csv_audit40Recompute stated counts or clean a table
incident_track30Timeline, risk, and triage; unknowns stay unknown
postmortem20Facts and actions split; blame fails
Appendix
Table 30: Office-v1 held-out families. nn sums to 40. OOD families do not appear in evolution.
Figure 11: DeepSeek family change from round 0 to the final held-out pass, in solved tasks. Zero-change families incident_track , postmortem , and onboarding are included. Color is the task delta.
MethodSeedr0r1r2r3
no_evolve4223/325/225/420/2
no_evolve4324/521/524/325/1
no_evolve4423/726/225/621/3
gepa4224/226/422/222/1
gepa4327/126/219/526/2
gepa4425/522/424/218/5
Appendix
Table 31: DeepSeek failure counts by round, max_steps / verifier_fail . The two counts sum to the number of held-out failures. No other failure reason appears.
Seedr0r1r2r3Range
4235.042.542.537.57.5
4340.040.045.042.55.0
4440.040.040.042.52.5
Appendix
Table 32: Manufacturing-v1, DeepSeek v4 Pro, no_evolve only. Success is held-out percent. Version is M0M_{0} at every round.
FamilynnSeed 42Seed 43Seed 44
accounts_payable30/0/0/00/0/0/00/0/0/0
advance_payments30/0/0/00/0/0/00/0/0/0
advance_receipts10/0/0/00/0/0/00/0/0/0
bpc_report70/0/0/00/0/0/00/0/0/0
commodity31/2/2/22/2/2/22/2/2/2
electricity33/3/3/33/3/3/33/3/3/3
Appendix
Table 33: Manufacturing-v1 solved counts by family at rounds 0–3, written r0/r1/r2/r3.
ModelSeedr0r1r2r3GainRange
GLM4272.572.560.070.0-2.512.5
GLM4360.065.067.567.5+7.57.5
GLM4467.570.075.060.0-7.515.0
MiniMax4230.037.540.035.0+5.010.0
MiniMax4337.535.037.537.5+0.02.5
MiniMax4440.037.527.540.0+0.012.5
Appendix
Table 34: Coding frozen-policy passes for GLM and MiniMax. Gain and Range for all three models are Table 7 . DeepSeek passes are Table 6 . Range is max minus min of the four held-out passes, in percentage points.
Figure 12: DeepSeek coding held-out success (%) at each round. Twelve cells: four methods by three seeds. Paired excess is Figure 4 .
Figure 13: GLM (top row) and MiniMax (bottom row) coding held-out success by round. Both models have seeds 42, 43, and 44. Line color is the seed.
Figure 14: GLM and MiniMax coding held-out success (%) at each round. GLM ace_llm seeds 42 and 43 have no round 3. Paired excess for these cells is Figure 6 .
ModelMethodSeedr0r1r2r3
DeepSeekno_evolve42M0M_{0}M0M_{0}M0M_{0}M0M_{0}
DeepSeekno_evolve43M0M_{0}M0M_{0}M0M_{0}M0M_{0}
DeepSeekno_evolve44M0M_{0}M0M_{0}M0M_{0}M0M_{0}
DeepSeekgepa42M0M_{0}M1M_{1}M1M_{1}M1M_{1}
DeepSeekgepa43M0M_{0}M1M_{1}M1M_{1}M1M_{1}
DeepSeekgepa44M0M_{0}M1M_{1}M2M_{2}M2M_{2}
Appendix
Table 35: Accepted agent_version after each coding round. A repeated version means the artifact did not change. On no_evolve that is every round.
ModelMethodSeedVersionRoundsFromToGap
DeepSeekno_evolve44M0M_{0}2–365.052.5-12.5
DeepSeekgepa44M2M_{2}2–367.542.5-25.0
DeepSeekace42M0M_{0}0–167.552.5-15.0
GLMno_evolve42M0M_{0}1–272.560.0-12.5
GLMno_evolve42M0M_{0}2–360.070.0+10.0
GLMno_evolve44M0M_{0}2–375.060.0-15.0
Appendix
Table 36: Consecutive coding held-out passes with the same agent_version and an absolute gap of at least 10 pp. The accepted artifact did not change between the two scores.
MethodSeedr0r1r2r3
no_evolve4213131416
no_evolve4315141516
no_evolve4415151419
gepa421591012
gepa4315101110
gepa4414101323
Appendix
Table 37: DeepSeek coding failure counts by round. The count is max_steps . verifier_fail is absent on every pass.
Figure 15: Archived ticket-gym, not office-v1. Split is 40/10/20 and held-out n=20n{=}20 . OOD is locked in held-out.
EnvFamilyEvoValHeld-out
ticket-gymticket_close1234
ticket-gymdual_ticket1034
ticket-gymauth_recovery923
ticket-gymfile_report923
ticket-gymaccess_denied (OOD)006
inboxtriage1648
Appendix
Table 38: Archived ticket-gym family counts, held-out n=20n{=}20 , not office-v1. OOD families are held-out only. The same table lists the inbox mock and the office_work expansion.
BoardSucc.GaindualfileOODSeeds
B0 reflexion (heuristic)70%+20+20 vs 0.500.504/44/43/33/30/60/63
B0 other (heuristic)50%+0+00/40/43/33/30/60/63
Weight attempt (ineligible)50%+0+0 vs 0.500.500/40/43/33/30/60/61
3B reflexion_llm55%+10+10 vs 0.450.450/40/40/30/34/64/61
3B other wrappers4545 – 55%55\%+0+0 this-run0/40/40/30/333 – 4/64/61
Appendix
Table 39: Archived ticket-gym mock, held-out n=20n{=}20 . Not an office-v1 score. Heuristic rows are three seeds. Wrapper rows are one seed. The weight attempt submitted no training.
FamilynnM0M_{0} / other B0reflexion
ticket_close44/44/44/44/4
issue_triage30/30/33/33/3
release_notes20/20/22/22/2
postmortem20/20/22/22/2
diff_review20/20/22/22/2
table_brief20/20/22/22/2
Appendix
Table 40: Archived ticket-gym expansion pack office_work , not office-v1. Held-out n=20n{=}20 , three identical seeds. Heuristic B0 only. Not ranked vs. Table 39 .
Figure 16: Archived ticket-gym tool calls, not office-v1. Locked held-out ids ( SandboxAgent , not a 3B log). (a) M0M_{0} dual: recoverable failed submit , second ticket never closed. (b) Same id after workflow:dual_ticket_close . (c) M0M_{0} OOD: no grant_access . (d) OOD after grant_access_on_denied . Consecutive identical calls are collapsed ( ×n\times n ).
Evolvers42s43s44dualOOD
reflexion (stand-in)7070704/44/40/60/6
no_evolve5050500/40/40/60/6
prompt_memory5050500/40/40/60/6
success_memory5050500/40/40/60/6
expel5050500/40/40/60/6
awm5050500/40/40/60/6
Appendix
Table 41: Archived ticket-gym heuristic B0, not office-v1. Held-out Success % by seed. Other B0 methods are no_evolve , prompt_memory , success_memory , expel , awm .
Loopr0r1r2r3GainOODWall (h)
reflexion_llm45555555+10+104/64/64.0
gepa_llm50505050+0+03/63/63.8
letta wrapper55555555+0+04/64/65.9
dgm wrapper45454545+0+03/63/65.7
agent_evolver45505555jitter4/64/65.4
Appendix
Table 42: Archived ticket-gym 3B tip-wrappers, not office-v1. Seed 42. Columns r0–r3 are held-out Success % of the accepted agent. reflexion_llm r3 rolls back (card is M2 55%). agent_evolver 45→50→5545{\to}50{\to}55 occurs under three rollbacks (jitter), not a second +10+10 pp. Dual 0/40/4 and file 0/30/3 on every row. Wall is loop process time, not time-to-first-token.
BoardSucc.GainreplythreadOODSeeds
B0 reflexion70%+30+30 vs 0.400.403/33/33/33/30/60/63
B0 other40%+0+00/30/30/30/30/60/63
Appendix
Table 43: Archived inbox mock, not office-v1. Held-out n=20n{=}20 . Not ranked vs. Table 39 . Reply n=3n{=}3 , thread n=3n{=}3 , OOD quarantine n=6n{=}6 .
PromptSucc.tcauthfileOOD
Hinted wrapper M0M_{0}45%4/44/42/32/30/30/33/63/6
No hint M0M_{0}30%4/44/42/32/30/30/30/60/6
Appendix
Table 44: Archived ticket-gym hygiene, not office-v1. Same frozen 3B, serving prompt with vs. without grant_access hint. Eval-only ( R=0R{=}0 ). Dual 0/40/4 on both rows. Table 42 keeps the hinted prompt.
Cardr1r2r3Held-out note
B0 reflexionAAA70% from r1
reflexion_llmAARstays M2 55%
gepa / letta / dgm wrapRRRthis-run M0M_{0}
agent_evolverRRR45→5545{\to}55 jitter
letta_nativeAARr1 is 10%; final M2 jitter 65%
dgm_nativeAAAval 70%, held-out 50%
Appendix
Table 45: Archived ticket-gym gates, not office-v1 (Table 15 ). A = accept, R = rollback. dgm_native can accept a validation 70% proposal while held-out stays 50%. qwen-plus r3 accepts 0→00{\to}0 ; the card is final 5%, not the M2 peak.
Figure 17: Archived ticket-gym gates, not office-v1. Accept and rollback are not held-out Gain.
Cardr0r1r2r3GaindualOOD
letta_native60105565+5+5 jitter2/42/46/66/6
dgm_native50505050+0+00/40/40/60/6
qwen-plus55155+0+00/40/41/61/6
Appendix
Table 46: Archived ticket-gym native pairing, not office-v1. Seed 42. Columns r0–r3 are held-out Success % of the accepted agent. Dual/OOD are the frozen (final) card. Not ranked vs. Table 39 . letta_native r3 rolls back: final 65% is M2 re-eval jitter vs. r2 55%. qwen-plus peak r2 15% is not the card. dgm_native actor is SandboxAgent (genome only). File: Letta/DGM 3/33/3 , qwen-plus 0/30/3 . Ticket-close: Letta 0/40/4 , DGM 4/44/4 , qwen-plus 0/40/4 .
Figure 18: Archived ticket-gym native pairing board, not office-v1. Not ranked vs. Table 39 .
EvolverTeacher tracesSelf traces
schema_memory100%0%
prompt_memory0%0%
no_evolve0%0%
Appendix
Table 47: NL2SQL teacher contrast, not office-v1. Held-out Success %, mean of seeds 42–44 (cells identical across seeds). Self has no successful SQL in traces.
Figure 19: NL2SQL teacher contrast, not office-v1. (a) Teacher traces. (b) Self traces, which contain no successful SQL, so every evolver stays at 0%.
EvolverSuccessGainin-distOOD
reflexion (stand-in)75%+55+55 pp15/1515/150/50/5
other B020%+0+0 pp4/154/15 (tc only)0/50/5
Appendix
Table 48: Archived ticket-gym office_work heuristic B0, not office-v1. Held-out Success %. Identical across seeds 42–44. Other B0: no_evolve , prompt_memory , success_memory , expel , awm .
Figure 20: Archived ticket-gym expansion pack, not office-v1. Not ranked vs. Table 39 .
EnvBoardBudgetSucc.GainIn-dist familyOOD familySeeds
ticket-gymB0 reflexion (heur.)B070%+20+20 vs 0.50dual 4/40/63
3B reflexion_llm wrapB055%+10+10 vs 0.45dual 0/44/61
inboxB0 reflexion (heur.)B070%+30+30 vs 0.40reply 3/3, thread 3/30/63
B0 other (heur.)B040%+0+00/30/63
NL2SQLTeacher schema_mem—100%+100+100 vs 0OOD 4/4—3
(contrast)Self (all evolvers)—0%—0/4—3
Appendix
Table 49: Archived cards across ticket-gym, inbox, NL2SQL, and office_work . Not an office-v1 ranking. “—” = family not present in that environment.
More in this topic

AI Agent Benchmarks

AI agent benchmarks measure how reliably autonomous systems plan, use tools, interact, and complete tasks across controlled and real-world settings.

See all 1,503 papers

Explore similar work

CardsList

  1. BenchTrace: A Benchmark for Testing Reflection Ability and Controlled Evolution in LLM Agents

    May 28, 2026Jiahao Huang, Fei Cheng, Junfeng Jiang +2LLM Agent EvaluationSelf-Reflection in Language Model Agents

    University of Tokyo · Kyoto University · National Institute of Informatics

  2. SEAGym: An Evaluation Environment for Self-Evolving LLM Agents

    Jun 16, 2026Congjie Zheng, Chuanyi Xue, Bin Liang +2LLM Agent EvaluationAI Agent Benchmarks

    Department of Automation, Tsinghua University, Beijing, 100084, China · Beijing National Research Center for Information Science and Technology (BNRist), Tsinghua University, Beijing, 100084, China

  3. Self-Evolving Harness on Multiple Tasks with the Agent as Its Own Optimizer

    Sep 29, 2026Qiankai XuContinual Learning for LLM AgentsAI Agent Evaluation

    Nanjing University