PACE-Bench: Benchmarking Physics Adaptation via Code Evolution in Dynamic Environments

Yuhao Zhan; Bingxiang He; Zecong Tang; Chaojun Xiao

cs.AIAug 14, 2026

PACE-Bench: Benchmarking Physics Adaptation via Code Evolution in Dynamic Environments

Authors: Yuhao Zhan, Bingxiang He, Zecong Tang, Chaojun Xiao

Organizations: Tsinghua University · Zhejiang University

Abstract

Self-evolving agents improve future behavior from interaction experience, yet existing evaluations typically optimize under fixed execution conditions and do not test recovery after those conditions change. To address this gap, we introduce PACE-Bench (Physics Adaptation via Code Evolution), a simulator-grounded benchmark of 144 source-to-target adaptation pairs across six physics domains. Each pair links a source environment to a mutated target environment with the same goal and interface. A code-driven design that succeeds in the source fails in the target, where agents must iteratively adapt it into a working target design using diagnostic sandbox feedback within a limited attempt budget. We compare ten self-evolving methods from four paradigms. The benchmark remains far from saturated: Reflexion + Qwen3-14B succeeds on only 35.9% of full-benchmark pairs, while GPT-5.5 solves 66.7% of the Statics subset under the full budget. Together, these results show that simulator-grounded reflection is more reliable than unverified self-revision, while memory anchors agents to early designs and broad tree search explores without converging. Even revealing exact physical changes does not raise the performance ceiling, pointing to mechanism redesign rather than parameter inference as the central bottleneck. Data and code are available at https://github.com/thunlp/PACE-Bench.

Figures & tables

Figure 1: Existing self-evolving benchmarks test transfer or generalization, whereas PACE-Bench tests adaptation of an executable design after physical environment shift.

Benchmark	Feedback-Driven Improvement	Physical Simulation	Executable Design	Self-Evolution Evaluation	Adaptation After Env. Change
Self-evolving agent benchmarks
StreamBench Wu et al. (2024)	✓	$\times$	$\times$	✓	$\times$
LifelongAgentBench Zheng et al. (2025a)	✓	$\times$	$\times$	✓	$\times$
SE-Bench Yuan et al. (2026)	$\times$	$\times$	$\times$	✓	$\times$
SEA-Eval Jiang et al. (2026)	✓	$\times$	$\times$	✓	$\times$
SkillLearnBench Zhong et al. (2026)	✓	$\times$	$\times$	✓	$\times$

Table 1: Comparison of representative benchmarks for self-evolution, engineering design, and physical environments. The highlighted final column is the core capability evaluated by PACE-Bench ; the other columns describe its evaluation setting.

Figure 2: Overview of the three-phase dataset construction pipeline.

Figure 3: Overview of PACE-Bench . Left: the 36 tasks organized across six physics categories. Right: rendered snapshots of reference solutions succeeding on the source environments.

		Qwen3-4B		Qwen3-8B		Qwen3-14B
Paradigm	Method	Pass@2	Score@2	Pass@2	Score@2	Pass@2	Score@2
Context	Vanilla	11.5 ( $\uparrow$ 0.0)	10.7 ( $\uparrow$ 0.0)	15.6 ( $\uparrow$ 0.0)	12.0 ( $\uparrow$ 0.0)	32.0 ( $\uparrow$ 0.0)	25.5 ( $\uparrow$ 0.0)
	Reflexion	19.5 ( $\uparrow$ 8.0)	14.6 ( $\uparrow$ 3.9)	26.6 ( $\uparrow$ 11.0)	18.1 ( $\uparrow$ 6.1)	35.9 ( $\uparrow$ 3.9)	28.0 ( $\uparrow$ 2.5)
	Self-Refine	6.2 ( $\downarrow$ 5.3)	5.5 ( $\downarrow$ 5.2)	3.1 ( $\downarrow$ 12.5)	3.7 ( $\downarrow$ 8.3)	7.1 ( $\downarrow$ 24.9)	8.0 ( $\downarrow$ 17.5)
Memory	ACE	15.5 ( $\uparrow$ 4.0)	13.3 ( $\uparrow$ 2.6)	19.5 ( $\uparrow$ 3.9)	14.9 ( $\uparrow$ 2.9)	25.0 ( $\downarrow$ 7.0)	19.6 ( $\downarrow$ 5.9)
	ExpeL	11.5 ( $\uparrow$ 0.0)	10.5 ( $\downarrow$ 0.2)	13.2 ( $\downarrow$ 2.4)	10.1 ( $\downarrow$ 1.9)	15.6 ( $\downarrow$ 16.4)	13.0 ( $\downarrow$ 12.5)

Table 2: Main Results: Pass@2 (%) and Score@2 by Method and Model. Values in parentheses show

\Delta

vs. Vanilla (

\uparrow

= improvement,

\downarrow

= degradation).

Model	Pass@2 (%)	Score@2
Qwen3-4B	8.3	12.0
Qwen3-8B	33.3	21.8
Qwen3-14B	37.5	27.3
Qwen3-32B	37.5	28.4
DeepSeek-V4-Pro	45.8	48.7
GPT-5.5	66.7	78.1

Table 3: Vanilla performance on Statics subset under the full 20-attempt budget.

Model	Metric	Vanilla	Reflexion	Self-Refine	ACE	ExpeL	Reasoning Bank	ToT	Code Evolve	SEAL	RAGEN	TTT- Discover
Qwen3-4B	Score	24.8	23.7	11.9	23.9	21.5	21.8	27.4	13.4	23.3	13.5	14.2
	Time	0.9h	0.8h	6.0h	1.3h	1.4h	0.8h	0.6h	1.8h	2.0h	5.5h	5.4h
	S/Hr	27.5	29.6	2.0	18.4	15.4	27.2	45.7	7.5	11.6	2.5	2.6
Qwen3-8B	Score	25.5	22.1	15.0	25.7	22.6	25.0	23.2	8.9	28.0	14.2	14.7
	Time	1.1h	1.5h	5.1h	1.3h	1.4h	1.0h	0.5h	2.2h	2.5h	5.8h	5.7h
	S/Hr	23.2	14.7	2.9	19.8	16.1	25.0	46.4	4.1	11.2	2.4	2.6

Table 4: Cost-normalized results on the Kinematics subset under 20-attempt budget. Scores are fitted to the nearest

1/24

increment and shown to one decimal. S/Hr is computed from the fitted scores before display rounding.

Figure 4: Error type distribution by category (left) and method (right).

	Qwen3-4B			Qwen3-8B			Qwen3-14B
Method	Global Sim	Trend (L $-$ E)	Radicality	Global Sim	Trend (L $-$ E)	Radicality	Global Sim	Trend (L $-$ E)	Radicality
Vanilla	0.792	+0.112	0.208	0.780	+0.126	0.220	0.760	+0.114	0.240
Reflexion	0.772	+0.111	0.228	0.725	+0.117	0.275	0.720	+0.110	0.280
Self-Refine	0.672	+0.120	0.328	0.598	+0.122	0.402	0.597	+0.119	0.403
ACE	0.821	+0.145	0.179	0.795	+0.164	0.205	0.807	+0.163	0.193
ExpeL	0.840	+0.132	0.160	0.807	+0.152	0.193	0.794	+0.139	0.206

Table 5: Code Similarity by Method and Model (iterations

\geq

10). Trend (L

-

E) = mean similarity difference between late and early attempts; positive = increasing self-similarity. Bold denotes the highest value in each column.

	Qwen3-4B			Qwen3-8B			Qwen3-14B
Method	CH	CE	$\Delta$	CH	CE	$\Delta$	CH	CE	$\Delta$
Vanilla	1.6	10.0	+8.4	13.3	13.3	+0.0	17.0	9.8	$-7.2$
Reflexion	9.4	12.0	+2.6	17.9	13.2	$-4.7$	17.1	14.6	$-2.5$
ACE	6.0	12.0	+6.0	12.5	8.8	$-3.7$	7.3	10.4	+3.1

Table 6: Change-Hidden (CH) vs. Change-Exposed (CE) Pass@2 (%) pooled over Statics and Kinematics (approximately 48 environment pairs). CE discloses the changed variables and their exact new values.

Appendix figures & tables19 assets

Supplementary material from the paper’s appendix.

Appendix

Module	Role
environment.py	Physics and terrain configuration
evaluator.py	Success criteria and scoring logic
feedback.py	Diagnostic metrics
prompt.py	Task context $\mathcal{C}$
stages.py	Mutation specs for $\mathcal{E}_{1-4}$
renderer.py	Simulation and GIF generation

Appendix

Table 7: Modules of each task in PACE-Bench .

Severity	Definition and example
Lv 3: Critical	Task unsolvable or evaluator broken (e.g., prompt constraints contradict the evaluator).
Lv 2: Moderate	Task solvable but prompt misleading (e.g., a stale visible-variable value).
Lv 1: Minor	Cosmetic issue (e.g., a renderer color mismatch or unused import).

Appendix

Table 8: Severity levels used during human verification.

Category	#Prompt Tokens	#Constraints	#Prim. APIs	Stage 1	Stage 2	Stage 3	Stage 4
Statics & Equilibrium	935	8	3	4	4	11	11
Kinematics & Linkages	835	7	9	3	4	7	9
Dynamics & Energy	894	12	9	1	2	5	8
Granular & Fluid Interaction	1,136	9	10	2	2	13	12
Cybernetics & Control	1,730	5	9	2	4	12	12
Exotic Physics	778	1	6	1	4	6	8

Appendix

Table 9: Dataset statistics by category (averaged over 6 tasks each). Stages 1–4 report average mutated physical parameters per stage, indicating progressive escalation in difficulty.

Full-benchmark statistic	Value
Pass/fail agreement between runs	86.6%
Median absolute score difference	0.0
Environment pairs with score difference $\leq 5$	82.0%
Spearman $\rho$ , run 1 vs. run 2 (4B / 8B / 14B)	0.667 / 0.881 / 0.976
Kinematics: two-run vs. three-run statistic	Value
Pair-level pass agreement	94.9%

Appendix

Table 10: Run-reliability analysis. The full-benchmark panel compares the two independent runs; the Kinematics panel compares statistics computed from two runs with those computed after adding a third run for five representative methods.

Figure 5: Pass@2 and Score@2 by model on Statics, Vanilla method, five-attempt budget.

Method	Qwen3-4B		Qwen3-8B		Qwen3-14B
	Pass@2 w/o VLM	Pass@2 w/ VLM	Pass@2 w/o VLM	Pass@2 w/ VLM	Pass@2 w/o VLM	Pass@2 w/ VLM
Vanilla	0.0	18.2 ( +18.2 )	24.4	17.1 ( -7.3 )	40.9	18.2 ( -22.7 )
Reflexion	9.1	22.7 ( +13.6 )	42.9	23.8 ( -19.1 )	40.9	27.3 ( -13.6 )
ACE	4.5	13.6 ( +9.1 )	18.2	31.8 ( +13.6 )	13.6	22.7 ( +9.1 )
ExpeL	13.6	9.1 ( -4.5 )	14.3	42.9 ( +28.6 )	13.6	36.4 ( +22.8 )
ToT	13.6	9.1 ( -4.5 )	9.1	18.2 ( +9.1 )	4.5	0.0 ( -4.5 )

Appendix

Table 11: VLM Video Feedback Impact on Statics and Kinematics.

Model	Method	Catastrophic	Structural	Constraint	Numerical	Design	Stagnation	Exploration	Late	Budget
Model	Method	Collapse	Failure	Violation	Instability	Fixation	Stagnation	Exploration	Convergence	Exhaustion
Qwen3-4B	Vanilla	0.0	+3.0	0.0	+3.0	+5.1	-8.1	+6.1	-4.5	-4.5
	Reflexion	0.0	+1.8	0.0	-9.1	-4.1	-4.1	+24.7	+5.9	-15.0
	ACE	0.0	+1.5	0.0	+11.0	-12.3	+15.8	-11.3	0.0	-4.8
	ExpeL	0.0	+4.2	0.0	+9.7	-26.6	+4.2	+3.4	+5.0	0.0
	ToT	0.0	+4.5	0.0	0.0	0.0	+18.9	-23.2	0.0	0.0

Appendix

Table 12: VLM Effect on Error Taxonomy:

\Delta

error share when VLM video feedback is added. Positive = VLM increases this error type; negative = VLM reduces it.

		Catastrophic	Structural	Constraint	Numerical	Design			Late	Budget
Model	Method	Collapse	Failure	Violation	Instability	Fixation	Stagnation	Exploration	Convergence	Exhaustion
Qwen3-4B	Vanilla	0.0	+3.0	0.0	-2.5	-11.6	-2.5	+22.7	-4.5	-4.5
	Reflexion	0.0	+10.0	0.0	+5.0	+3.3	+3.3	-13.3	0.0	-8.3
	ACE	0.0	+2.4	0.0	+6.3	-11.1	+5.6	-4.0	+5.6	-4.8
	ExpeL	0.0	-0.8	0.0	+4.7	-6.6	-0.8	+3.4	0.0	0.0
	ToT	0.0	+5.3	0.0	+5.3	0.0	+5.3	-15.8	0.0	0.0

Appendix

Table 13: CE Effect on Error Taxonomy:

\Delta

error share when mutated variables are disclosed (CE minus CH). Positive = CE increases this error type; negative = CE reduces it. Statics only.

Figure 6: Pass@2@

k

and Score@2@

k

by model (all methods, all categories).

Method	1	2	3	4	5	6	7	8	9	10
Vanilla	0.255	0.468	0.536	0.843	0.518	0.493	0.778	0.793	0.553	0.943
ACE	0.185	0.870	0.842	0.762	0.772	0.895	0.814	1.000	0.670	0.828

Appendix

Table 14: Adjacent-attempt Jaccard code similarity for one Qwen3-14B Stage 4 case study. Each entry compares solutions on consecutive transitions.

Model	Pass@2 (%)	Score@2
Qwen3-4B	11.3	17.2
Qwen3-8B	12.8	16.8
Qwen3-14B	18.3	21.7

Appendix

Table 15: Vanilla from-scratch results by model, computed over eligible task–environment instances.

Target environment	Pass@2 (%)	Score@2
Source	32.3	30.8
Mutated-1	13.4	18.5
Mutated-2	10.9	16.6
Mutated-3	11.2	15.3
Mutated-4	6.4	11.6

Appendix

Table 16: Vanilla from-scratch results by target environment, computed over eligible task–environment instances.

Figure 7: Model $\times$ category Pass@2 heatmap (all methods aggregated).

ID	Stage	Mutations
S_01: Bridge for vehicle crossing
	Stage-1: Brittle Material	• max_structure_mass = 492.0 • joint_max_force = 40.0 • anchor_max_force = 50.0 • joint_max_torque = 0.1 • anchor_max_torque = 0.1
	Stage-2: Paper Joints	• max_structure_mass = 1000.0 • joint_max_force = 5.5 • anchor_max_force = 15.0
	Stage-3: The Vortex Gorge	• gap_width = 20.0 • max_structure_mass = 550.0 • gravity = (0, -20.0) • wind_force = (-20.0, -4.0) • joint_max_torque = 0.2 • anchor_max_torque = 0.2
	Stage-4: Abyssal Crossing	• gap_width = 26.0 • max_structure_mass = 330.0 • gravity = (0, -28.0) • wind_force = (-45.0, -8.0) • joint_max_force = 40.0 • anchor_max_force = 60.0 • joint_max_torque = 80.0 • anchor_max_torque = 120.0
S_02: Tower resisting seismic vibration

Appendix

Table 17: Category: Statics & Equilibrium task specifications.

ID	Stage	Mutations
K_01: Motor-driven 2D walker
	Stage-1: Near-Zero Structure Mass Budget	• max_structure_mass = 0.8
	Stage-2: Near-Zero Ground Traction + Restricted Joints	• ground_friction = 0.02 • default_joint_lower_limit = -0.2617993877991494 • default_joint_upper_limit = 0.2617993877991494
	Stage-3: Dissipative Low-Traction Walking Crown	• ground_friction = 0.015 • max_structure_mass = 1.5 • gravity = (0, -17.0) • max_body_friction = 0.2 • default_joint_lower_limit = -0.2243994752564138 • default_joint_upper_limit = 0.2243994752564138 • linear_damping = 4.0 • angular_damping = 4.0
	Stage-4: Maximum Difficulty — All Variables Beyond Stage-3 Extremes	• ground_friction = 0.01 • max_structure_mass = 1.2 • gravity = (0, -20.0) • max_body_friction = 0.008 • default_joint_lower_limit = -0.20943951023931953 • default_joint_upper_limit = 0.20943951023931953 • linear_damping = 18.0 • angular_damping = 18.0
K_02: Vertical wall climber

Appendix

Table 18: Category: Kinematics & Linkages task specifications.

ID	Stage	Mutations
D_01: Projectile launcher
	Stage-1: The Dense Atmosphere	• linear_damping = 2.5 • angular_damping = 2.5
	Stage-2: The Distant Target	• target_x_min = 50.0 • target_x_max = 55.0 • target_y_min = 2.0 • target_y_max = 5.0
	Stage-3: Heavy World and Drag	• gravity = (0, -15.0) • linear_damping = 1.5 • angular_damping = 1.5
	Stage-4: Extreme Range and Conditions	• target_x_min = 52.0 • target_x_max = 57.0 • target_y_min = 2.0 • target_y_max = 5.0 • gravity = (0, -18.0) • linear_damping = 1.2 • angular_damping = 1.2
D_02: Jumping mechanism

Appendix

Table 19: Category: Dynamics & Energy task specifications.

ID	Stage	Mutations
F_01: Dam blocking water flow
	Stage-1: Low weld ceiling (threshold physics)	• joint_break_force = 41000.0
	Stage-2: Elastic reservoir granules	• fluid_particle_restitution = 0.78
	Stage-3: Bounce apocalypse + wide squeeze + fast debris + heavy world	• joint_break_force = 50000.0 • joint_break_consecutive_steps = 2 • downstream_wall_amplitude = 1.0 • downstream_wall_phase_divisor = 30.0 • fluid_particle_restitution = 0.95 • debris_linear_velocity_x = 3.2 • max_leakage_rate = 0.004 • gravity = (0, -10.5)
	Stage-4: Apocalypse: every physics dial at breaking point	• joint_break_force = 50000.0 • joint_break_consecutive_steps = 2 • downstream_wall_amplitude = 1.0 • downstream_wall_phase_divisor = 20.0 • fluid_particle_restitution = 0.97 • fluid_particle_friction = 0.02 • debris_linear_velocity_x = 4.0 • earthquake_impulse_x = 0.5 • upward_surge_impulse_y = 1.5 • max_structure_mass = 130.0 • backward_slosh_impulse_x = -0.9 • max_leakage_rate = 0.005 • surge_impulses = [1.0, 1.2, 1.3, 1.5, 1.6, 1.8, 1.9, 2.1, 2.2] • gravity = (0, -11.5)
F_02: Amphibious vehicle

Appendix

Table 20: Category: Granular & Fluid Interaction task specifications.

ID	Stage	Mutations
C_01: Inverted pendulum cart-pole
	Stage-1: Curriculum stage 1	• track_center_x = 50.0 • pole_start_angle = 0.1658062789394612 • sensor_delay_angle_steps = 10 • sensor_delay_omega_steps = 10
	Stage-2: Curriculum stage 2	• track_center_x = 50.0 • cart_force_limit_newtons = 0.5 • max_steps = 2000 • gravity = 12.0 • sensor_delay_angle_steps = 10 • sensor_delay_omega_steps = 10 • pole_start_angle = 0.003 • safe_half_range = 0.1
	Stage-3: Curriculum stage 3	• track_center_x = 50.0 • gravity = 22.0 • sensor_delay_angle_steps = 3 • sensor_delay_omega_steps = 3 • cart_force_limit_newtons = 4.0 • pole_mass = 5.0 • cart_mass = 3.0 • safe_half_range = 0.2 • pole_length = 0.7 • max_steps = 1000 • pole_start_angle = 0.0004
	Stage-4: Curriculum stage 4	• track_center_x = 50.0 • gravity = 35.0 • sensor_delay_angle_steps = 3 • sensor_delay_omega_steps = 3 • cart_force_limit_newtons = 50.0 • pole_mass = 12.0 • cart_mass = 2.0 • safe_half_range = 0.15 • pole_length = 0.5 • max_steps = 500 • pole_start_angle = 5e-05 • cart_rail_center_y = 6.0
C_02: Lunar lander

Appendix

Table 21: Category: Cybernetics & Control task specifications.

ID	Stage	Mutations
E_01: Surviving reversing gravity
	Stage-1: Altered Field I	• arena_y_max = 20.0 • build_zone_y_max = 18.0 • gravity = gravity_resonant_violent(t) • joint_force_limit = 1.0
	Stage-2: Altered Field II	• arena_y_max = 9.0 • build_zone_y_max = 8.5 • friction = 0.0 • gravity = gravity_cataclysm(t) • beam_density_scale = 200.0 • linear_damping = -6.0 • angular_damping = -4.0 • joint_force_limit = 0.001 • max_beam_count = 1 • max_structure_mass = 0.02
	Stage-3: Altered Field III	• arena_y_max = 9.5 • build_zone_y_max = 9.0 • friction = 0.0 • gravity = gravity_vortex_singularity(t) • beam_density_scale = 150.0 • linear_damping = -6.0 • angular_damping = -4.0 • joint_force_limit = 0.15 • max_beam_count = 2 • max_structure_mass = 0.3
	Stage-4: Altered Field IV	• arena_y_max = 9.0 • build_zone_y_max = 8.5 • friction = 0.0 • gravity = gravity_cataclysm(t) • beam_density_scale = 300.0 • linear_damping = -6.0 • angular_damping = -4.0 • joint_force_limit = 0.01 • max_beam_count = 1 • max_structure_mass = 0.005
E_02: Thruster flyer in fluid

Appendix

Table 22: Category: Exotic Physics task specifications.

AI Agent Benchmarks

AI agent benchmarks measure how reliably autonomous systems plan, use tools, interact, and complete tasks across controlled and real-world settings.

See all 1,497 papers

May 28, 2026

BenchTrace: A Benchmark for Testing Reflection Ability and Controlled Evolution in LLM Agents

Self-evolving agents improve over time by reflecting on past failures, but existing evaluation is limited in two ways: it measures only task scores, leaving reflection quality unknown, and it relies on agents' own episode runs, offering no mechanism to target specific failure patterns. We present BenchTrace, a benchmark for evaluating self-evolution ability in LLM agents. BenchTrace is built on a snapshot-reflection dataset of 1,821 annotated episodes spanning six diverse tasks, and comprises a Reflection Evaluation that probes failure identification through targeted QA tasks, and an Evolution Evaluation that tests whether past failure experience translates into avoidance behavior in a controlled self-evolution simulation. Building on BenchTrace, we propose failure avoidance rate (FAR), a new evaluation metric measuring the fraction of test cases in which the agent successfully avoids the target failure instance. Experiments with Qwen3-32B and GPT-4.1 reveal that both models fall below a 30% end-to-end pass rate on reflection evaluation, with diagnosis as the primary bottleneck. Evolution evaluation shows that self-evolution methods generally improve FAR over the non-evolving baseline, but agents forget early lessons as noise episodes accumulate, and agents fail to generalize their reflections beyond the specific context, causing negative transfer across task contexts. Our correlation analysis further reveals that only a fully correct reflection is strongly associated with higher FAR. BenchTrace exposes concrete limits of current self-evolution approaches and provides a controlled, model-agnostic framework for targeted evaluation.

Jiahao Huang, Fei Cheng, Junfeng Jiang +2

University of Tokyo · Kyoto University · National Institute of Informatics

LLM Agent Evaluation Self-Reflection in Language Model Agents AI Agent Benchmarks LLM Agents Self-Evolving Agents

Ask PDF

Mar 21, 2026

REVERE: Reflective Evolving Research Engineer

Existing prompt-optimization techniques rely on local signals, causing poor generalization across tasks. In addition, they also rely on weak update mechanisms, such as full-prompt rewrites or unstructured merges, which cause knowledge loss and unstable adaptation. These limitations are magnified in research-coding workflows, which involve heterogeneous repositories and weak feedback, limiting abstraction and learning across tasks. We introduce Reflective Evolving Research Engineer (REVERE), a lightweight, self-adapting agent framework that learns from a Global Training Context, distills recurring cross-repository failure modes into reusable heuristics, and applies targeted, code-based edits to agent prompts. REVERE is evaluated across settings ranging from long-horizon to single-shot benchmarks, and improves over prior expert-crafted instructions by 4.50% on SUPER, 1.3% on ResearchCodeBench, and 4.89% on ScienceAgentBench. It does so at nearly 10x lower cost and 2.7x faster adaptation than existing prompt-optimization baselines, demonstrating that self-adapting agents with continual learning and global memory consolidation can meaningfully evolve their capabilities over time.

Balaji Dinesh Gangireddi, Aniketh Garikaparthi, Manasi Patwardhan +1

TCS Research · Yale University

LLM Agent Self-Improvement Prompt Optimization Software Engineering Agents Continual Learning for LLM Agents Memory-Augmented Agents

Ask PDF

Jul 6, 2026

EvoAgentBench: Benchmarking Agent Self-Evolution via Ability Transfer

Agent self-evolution in long-horizon LLM systems is largely procedural: useful experience is not merely stored information, but reusable procedures for searching, debugging, and verification. Yet current evaluations do not isolate this form of transfer. Agent benchmarks test single-episode task solving; memory benchmarks target information retention rather than procedural reuse. We introduce EvoAgentBench, a benchmark for agent self-evolution via Ability-guided transfer across four agentic domains: web research, algorithmic reasoning, software engineering, and knowledge work. EvoAgentBench extracts trace-grounded Abilities from agent executions, canonicalizes them into operational units, and builds domain-specific Ability Graphs linking tasks that share procedural overlap. By design, every test task is backed by verified training-side Ability support. Across a 528/267 train/test split, two scaffolds, and three backbones, curated Ability content transfers reliably across model families, but no current automatic method sustains positive gain in all settings. EvoAgentBench shifts self-evolution evaluation from aggregate accuracy comparison to fine-grained diagnosis of experience encoding, routing, and uptake. The benchmark is publicly available at https://huggingface.co/datasets/EverMind-AI/EvoAgentBench.

Xingze Gao, Chuanrui Hu, Hongda Chen +9

1Anhui University · 2EverMind, Shanda Group · 3Southeast University

LLM Agent Self-Improvement AI Agent Benchmarks Agent Skill Learning Self-Evolving Agents

Ask PDF

Abstract

AI Agent Benchmarks

Explore similar work

BenchTrace: A Benchmark for Testing Reflection Ability and Controlled Evolution in LLM Agents

REVERE: Reflective Evolving Research Engineer

EvoAgentBench: Benchmarking Agent Self-Evolution via Ability Transfer