PACE-Bench: Benchmarking Physics Adaptation via Code Evolution in Dynamic Environments
Organizations: Tsinghua University · Zhejiang University
Abstract
Self-evolving agents improve future behavior from interaction experience, yet existing evaluations typically optimize under fixed execution conditions and do not test recovery after those conditions change. To address this gap, we introduce PACE-Bench (Physics Adaptation via Code Evolution), a simulator-grounded benchmark of 144 source-to-target adaptation pairs across six physics domains. Each pair links a source environment to a mutated target environment with the same goal and interface. A code-driven design that succeeds in the source fails in the target, where agents must iteratively adapt it into a working target design using diagnostic sandbox feedback within a limited attempt budget. We compare ten self-evolving methods from four paradigms. The benchmark remains far from saturated: Reflexion + Qwen3-14B succeeds on only 35.9% of full-benchmark pairs, while GPT-5.5 solves 66.7% of the Statics subset under the full budget. Together, these results show that simulator-grounded reflection is more reliable than unverified self-revision, while memory anchors agents to early designs and broad tree search explores without converging. Even revealing exact physical changes does not raise the performance ceiling, pointing to mechanism redesign rather than parameter inference as the central bottleneck. Data and code are available at https://github.com/thunlp/PACE-Bench.
Figures & tables
| Benchmark | Feedback-Driven Improvement | Physical Simulation | Executable Design | Self-Evolution Evaluation | Adaptation After Env. Change |
| Self-evolving agent benchmarks | |||||
| StreamBench Wu et al. (2024) | ✓ | ✓ | |||
| LifelongAgentBench Zheng et al. (2025a) | ✓ | ✓ | |||
| SE-Bench Yuan et al. (2026) | ✓ | ||||
| SEA-Eval Jiang et al. (2026) | ✓ | ✓ | |||
| SkillLearnBench Zhong et al. (2026) | ✓ | ✓ | |||
| Qwen3-4B | Qwen3-8B | Qwen3-14B | |||||
| Paradigm | Method | Pass@2 | Score@2 | Pass@2 | Score@2 | Pass@2 | Score@2 |
| Context | Vanilla | 11.5 ( 0.0) | 10.7 ( 0.0) | 15.6 ( 0.0) | 12.0 ( 0.0) | 32.0 ( 0.0) | 25.5 ( 0.0) |
| Reflexion | 19.5 ( 8.0) | 14.6 ( 3.9) | 26.6 ( 11.0) | 18.1 ( 6.1) | 35.9 ( 3.9) | 28.0 ( 2.5) | |
| Self-Refine | 6.2 ( 5.3) | 5.5 ( 5.2) | 3.1 ( 12.5) | 3.7 ( 8.3) | 7.1 ( 24.9) | 8.0 ( 17.5) | |
| Memory | ACE | 15.5 ( 4.0) | 13.3 ( 2.6) | 19.5 ( 3.9) | 14.9 ( 2.9) | 25.0 ( 7.0) | 19.6 ( 5.9) |
| ExpeL | 11.5 ( 0.0) | 10.5 ( 0.2) | 13.2 ( 2.4) | 10.1 ( 1.9) | 15.6 ( 16.4) | 13.0 ( 12.5) | |
| Model | Pass@2 (%) | Score@2 |
| Qwen3-4B | 8.3 | 12.0 |
| Qwen3-8B | 33.3 | 21.8 |
| Qwen3-14B | 37.5 | 27.3 |
| Qwen3-32B | 37.5 | 28.4 |
| DeepSeek-V4-Pro | 45.8 | 48.7 |
| GPT-5.5 | 66.7 | 78.1 |
| Model | Metric | Vanilla | Reflexion | Self-Refine | ACE | ExpeL | Reasoning Bank | ToT | Code Evolve | SEAL | RAGEN | TTT- Discover |
| Qwen3-4B | Score | 24.8 | 23.7 | 11.9 | 23.9 | 21.5 | 21.8 | 27.4 | 13.4 | 23.3 | 13.5 | 14.2 |
| Time | 0.9h | 0.8h | 6.0h | 1.3h | 1.4h | 0.8h | 0.6h | 1.8h | 2.0h | 5.5h | 5.4h | |
| S/Hr | 27.5 | 29.6 | 2.0 | 18.4 | 15.4 | 27.2 | 45.7 | 7.5 | 11.6 | 2.5 | 2.6 | |
| Qwen3-8B | Score | 25.5 | 22.1 | 15.0 | 25.7 | 22.6 | 25.0 | 23.2 | 8.9 | 28.0 | 14.2 | 14.7 |
| Time | 1.1h | 1.5h | 5.1h | 1.3h | 1.4h | 1.0h | 0.5h | 2.2h | 2.5h | 5.8h | 5.7h | |
| S/Hr | 23.2 | 14.7 | 2.9 | 19.8 | 16.1 | 25.0 | 46.4 | 4.1 | 11.2 | 2.4 | 2.6 |
| Qwen3-4B | Qwen3-8B | Qwen3-14B | |||||||
| Method | Global Sim | Trend (L E) | Radicality | Global Sim | Trend (L E) | Radicality | Global Sim | Trend (L E) | Radicality |
| Vanilla | 0.792 | +0.112 | 0.208 | 0.780 | +0.126 | 0.220 | 0.760 | +0.114 | 0.240 |
| Reflexion | 0.772 | +0.111 | 0.228 | 0.725 | +0.117 | 0.275 | 0.720 | +0.110 | 0.280 |
| Self-Refine | 0.672 | +0.120 | 0.328 | 0.598 | +0.122 | 0.402 | 0.597 | +0.119 | 0.403 |
| ACE | 0.821 | +0.145 | 0.179 | 0.795 | +0.164 | 0.205 | 0.807 | +0.163 | 0.193 |
| ExpeL | 0.840 | +0.132 | 0.160 | 0.807 | +0.152 | 0.193 | 0.794 | +0.139 | 0.206 |
| Qwen3-4B | Qwen3-8B | Qwen3-14B | |||||||
| Method | CH | CE | CH | CE | CH | CE | |||
| Vanilla | 1.6 | 10.0 | +8.4 | 13.3 | 13.3 | +0.0 | 17.0 | 9.8 | |
| Reflexion | 9.4 | 12.0 | +2.6 | 17.9 | 13.2 | 17.1 | 14.6 | ||
| ACE | 6.0 | 12.0 | +6.0 | 12.5 | 8.8 | 7.3 | 10.4 | +3.1 | |
Appendix figures & tables19 assets
Supplementary material from the paper’s appendix.
Appendix
| Module | Role |
| environment.py | Physics and terrain configuration |
| evaluator.py | Success criteria and scoring logic |
| feedback.py | Diagnostic metrics |
| prompt.py | Task context |
| stages.py | Mutation specs for |
| renderer.py | Simulation and GIF generation |
| Severity | Definition and example |
| Lv 3: Critical | Task unsolvable or evaluator broken (e.g., prompt constraints contradict the evaluator). |
| Lv 2: Moderate | Task solvable but prompt misleading (e.g., a stale visible-variable value). |
| Lv 1: Minor | Cosmetic issue (e.g., a renderer color mismatch or unused import). |
| Category | #Prompt Tokens | #Constraints | #Prim. APIs | Stage 1 | Stage 2 | Stage 3 | Stage 4 |
| Statics & Equilibrium | 935 | 8 | 3 | 4 | 4 | 11 | 11 |
| Kinematics & Linkages | 835 | 7 | 9 | 3 | 4 | 7 | 9 |
| Dynamics & Energy | 894 | 12 | 9 | 1 | 2 | 5 | 8 |
| Granular & Fluid Interaction | 1,136 | 9 | 10 | 2 | 2 | 13 | 12 |
| Cybernetics & Control | 1,730 | 5 | 9 | 2 | 4 | 12 | 12 |
| Exotic Physics | 778 | 1 | 6 | 1 | 4 | 6 | 8 |
| Full-benchmark statistic | Value |
| Pass/fail agreement between runs | 86.6% |
| Median absolute score difference | 0.0 |
| Environment pairs with score difference | 82.0% |
| Spearman , run 1 vs. run 2 (4B / 8B / 14B) | 0.667 / 0.881 / 0.976 |
| Kinematics: two-run vs. three-run statistic | Value |
| Pair-level pass agreement | 94.9% |
| Method | Qwen3-4B | Qwen3-8B | Qwen3-14B | |||
| Pass@2 w/o VLM | Pass@2 w/ VLM | Pass@2 w/o VLM | Pass@2 w/ VLM | Pass@2 w/o VLM | Pass@2 w/ VLM | |
| Vanilla | 0.0 | 18.2 ( +18.2 ) | 24.4 | 17.1 ( -7.3 ) | 40.9 | 18.2 ( -22.7 ) |
| Reflexion | 9.1 | 22.7 ( +13.6 ) | 42.9 | 23.8 ( -19.1 ) | 40.9 | 27.3 ( -13.6 ) |
| ACE | 4.5 | 13.6 ( +9.1 ) | 18.2 | 31.8 ( +13.6 ) | 13.6 | 22.7 ( +9.1 ) |
| ExpeL | 13.6 | 9.1 ( -4.5 ) | 14.3 | 42.9 ( +28.6 ) | 13.6 | 36.4 ( +22.8 ) |
| ToT | 13.6 | 9.1 ( -4.5 ) | 9.1 | 18.2 ( +9.1 ) | 4.5 | 0.0 ( -4.5 ) |
| Model | Method | Catastrophic | Structural | Constraint | Numerical | Design | Stagnation | Exploration | Late | Budget |
| Collapse | Failure | Violation | Instability | Fixation | Convergence | Exhaustion | ||||
| Qwen3-4B | Vanilla | 0.0 | +3.0 | 0.0 | +3.0 | +5.1 | -8.1 | +6.1 | -4.5 | -4.5 |
| Reflexion | 0.0 | +1.8 | 0.0 | -9.1 | -4.1 | -4.1 | +24.7 | +5.9 | -15.0 | |
| ACE | 0.0 | +1.5 | 0.0 | +11.0 | -12.3 | +15.8 | -11.3 | 0.0 | -4.8 | |
| ExpeL | 0.0 | +4.2 | 0.0 | +9.7 | -26.6 | +4.2 | +3.4 | +5.0 | 0.0 | |
| ToT | 0.0 | +4.5 | 0.0 | 0.0 | 0.0 | +18.9 | -23.2 | 0.0 | 0.0 |
| Catastrophic | Structural | Constraint | Numerical | Design | Late | Budget | ||||
| Model | Method | Collapse | Failure | Violation | Instability | Fixation | Stagnation | Exploration | Convergence | Exhaustion |
| Qwen3-4B | Vanilla | 0.0 | +3.0 | 0.0 | -2.5 | -11.6 | -2.5 | +22.7 | -4.5 | -4.5 |
| Reflexion | 0.0 | +10.0 | 0.0 | +5.0 | +3.3 | +3.3 | -13.3 | 0.0 | -8.3 | |
| ACE | 0.0 | +2.4 | 0.0 | +6.3 | -11.1 | +5.6 | -4.0 | +5.6 | -4.8 | |
| ExpeL | 0.0 | -0.8 | 0.0 | +4.7 | -6.6 | -0.8 | +3.4 | 0.0 | 0.0 | |
| ToT | 0.0 | +5.3 | 0.0 | +5.3 | 0.0 | +5.3 | -15.8 | 0.0 | 0.0 |
| Method | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 |
| Vanilla | 0.255 | 0.468 | 0.536 | 0.843 | 0.518 | 0.493 | 0.778 | 0.793 | 0.553 | 0.943 |
| ACE | 0.185 | 0.870 | 0.842 | 0.762 | 0.772 | 0.895 | 0.814 | 1.000 | 0.670 | 0.828 |
| Model | Pass@2 (%) | Score@2 |
| Qwen3-4B | 11.3 | 17.2 |
| Qwen3-8B | 12.8 | 16.8 |
| Qwen3-14B | 18.3 | 21.7 |
| Target environment | Pass@2 (%) | Score@2 |
| Source | 32.3 | 30.8 |
| Mutated-1 | 13.4 | 18.5 |
| Mutated-2 | 10.9 | 16.6 |
| Mutated-3 | 11.2 | 15.3 |
| Mutated-4 | 6.4 | 11.6 |
| ID | Stage | Mutations |
|---|---|---|
| S_01: Bridge for vehicle crossing | ||
| Stage-1: Brittle Material | • max_structure_mass = 492.0 • joint_max_force = 40.0 • anchor_max_force = 50.0 • joint_max_torque = 0.1 • anchor_max_torque = 0.1 | |
| Stage-2: Paper Joints | • max_structure_mass = 1000.0 • joint_max_force = 5.5 • anchor_max_force = 15.0 | |
| Stage-3: The Vortex Gorge | • gap_width = 20.0 • max_structure_mass = 550.0 • gravity = (0, -20.0) • wind_force = (-20.0, -4.0) • joint_max_torque = 0.2 • anchor_max_torque = 0.2 | |
| Stage-4: Abyssal Crossing | • gap_width = 26.0 • max_structure_mass = 330.0 • gravity = (0, -28.0) • wind_force = (-45.0, -8.0) • joint_max_force = 40.0 • anchor_max_force = 60.0 • joint_max_torque = 80.0 • anchor_max_torque = 120.0 | |
| S_02: Tower resisting seismic vibration | ||
| ID | Stage | Mutations |
|---|---|---|
| K_01: Motor-driven 2D walker | ||
| Stage-1: Near-Zero Structure Mass Budget | • max_structure_mass = 0.8 | |
| Stage-2: Near-Zero Ground Traction + Restricted Joints | • ground_friction = 0.02 • default_joint_lower_limit = -0.2617993877991494 • default_joint_upper_limit = 0.2617993877991494 | |
| Stage-3: Dissipative Low-Traction Walking Crown | • ground_friction = 0.015 • max_structure_mass = 1.5 • gravity = (0, -17.0) • max_body_friction = 0.2 • default_joint_lower_limit = -0.2243994752564138 • default_joint_upper_limit = 0.2243994752564138 • linear_damping = 4.0 • angular_damping = 4.0 | |
| Stage-4: Maximum Difficulty — All Variables Beyond Stage-3 Extremes | • ground_friction = 0.01 • max_structure_mass = 1.2 • gravity = (0, -20.0) • max_body_friction = 0.008 • default_joint_lower_limit = -0.20943951023931953 • default_joint_upper_limit = 0.20943951023931953 • linear_damping = 18.0 • angular_damping = 18.0 | |
| K_02: Vertical wall climber | ||
| ID | Stage | Mutations |
|---|---|---|
| D_01: Projectile launcher | ||
| Stage-1: The Dense Atmosphere | • linear_damping = 2.5 • angular_damping = 2.5 | |
| Stage-2: The Distant Target | • target_x_min = 50.0 • target_x_max = 55.0 • target_y_min = 2.0 • target_y_max = 5.0 | |
| Stage-3: Heavy World and Drag | • gravity = (0, -15.0) • linear_damping = 1.5 • angular_damping = 1.5 | |
| Stage-4: Extreme Range and Conditions | • target_x_min = 52.0 • target_x_max = 57.0 • target_y_min = 2.0 • target_y_max = 5.0 • gravity = (0, -18.0) • linear_damping = 1.2 • angular_damping = 1.2 | |
| D_02: Jumping mechanism | ||
| ID | Stage | Mutations |
|---|---|---|
| F_01: Dam blocking water flow | ||
| Stage-1: Low weld ceiling (threshold physics) | • joint_break_force = 41000.0 | |
| Stage-2: Elastic reservoir granules | • fluid_particle_restitution = 0.78 | |
| Stage-3: Bounce apocalypse + wide squeeze + fast debris + heavy world | • joint_break_force = 50000.0 • joint_break_consecutive_steps = 2 • downstream_wall_amplitude = 1.0 • downstream_wall_phase_divisor = 30.0 • fluid_particle_restitution = 0.95 • debris_linear_velocity_x = 3.2 • max_leakage_rate = 0.004 • gravity = (0, -10.5) | |
| Stage-4: Apocalypse: every physics dial at breaking point | • joint_break_force = 50000.0 • joint_break_consecutive_steps = 2 • downstream_wall_amplitude = 1.0 • downstream_wall_phase_divisor = 20.0 • fluid_particle_restitution = 0.97 • fluid_particle_friction = 0.02 • debris_linear_velocity_x = 4.0 • earthquake_impulse_x = 0.5 • upward_surge_impulse_y = 1.5 • max_structure_mass = 130.0 • backward_slosh_impulse_x = -0.9 • max_leakage_rate = 0.005 • surge_impulses = [1.0, 1.2, 1.3, 1.5, 1.6, 1.8, 1.9, 2.1, 2.2] • gravity = (0, -11.5) | |
| F_02: Amphibious vehicle | ||
| ID | Stage | Mutations |
|---|---|---|
| C_01: Inverted pendulum cart-pole | ||
| Stage-1: Curriculum stage 1 | • track_center_x = 50.0 • pole_start_angle = 0.1658062789394612 • sensor_delay_angle_steps = 10 • sensor_delay_omega_steps = 10 | |
| Stage-2: Curriculum stage 2 | • track_center_x = 50.0 • cart_force_limit_newtons = 0.5 • max_steps = 2000 • gravity = 12.0 • sensor_delay_angle_steps = 10 • sensor_delay_omega_steps = 10 • pole_start_angle = 0.003 • safe_half_range = 0.1 | |
| Stage-3: Curriculum stage 3 | • track_center_x = 50.0 • gravity = 22.0 • sensor_delay_angle_steps = 3 • sensor_delay_omega_steps = 3 • cart_force_limit_newtons = 4.0 • pole_mass = 5.0 • cart_mass = 3.0 • safe_half_range = 0.2 • pole_length = 0.7 • max_steps = 1000 • pole_start_angle = 0.0004 | |
| Stage-4: Curriculum stage 4 | • track_center_x = 50.0 • gravity = 35.0 • sensor_delay_angle_steps = 3 • sensor_delay_omega_steps = 3 • cart_force_limit_newtons = 50.0 • pole_mass = 12.0 • cart_mass = 2.0 • safe_half_range = 0.15 • pole_length = 0.5 • max_steps = 500 • pole_start_angle = 5e-05 • cart_rail_center_y = 6.0 | |
| C_02: Lunar lander | ||
| ID | Stage | Mutations |
|---|---|---|
| E_01: Surviving reversing gravity | ||
| Stage-1: Altered Field I | • arena_y_max = 20.0 • build_zone_y_max = 18.0 • gravity = gravity_resonant_violent(t) • joint_force_limit = 1.0 | |
| Stage-2: Altered Field II | • arena_y_max = 9.0 • build_zone_y_max = 8.5 • friction = 0.0 • gravity = gravity_cataclysm(t) • beam_density_scale = 200.0 • linear_damping = -6.0 • angular_damping = -4.0 • joint_force_limit = 0.001 • max_beam_count = 1 • max_structure_mass = 0.02 | |
| Stage-3: Altered Field III | • arena_y_max = 9.5 • build_zone_y_max = 9.0 • friction = 0.0 • gravity = gravity_vortex_singularity(t) • beam_density_scale = 150.0 • linear_damping = -6.0 • angular_damping = -4.0 • joint_force_limit = 0.15 • max_beam_count = 2 • max_structure_mass = 0.3 | |
| Stage-4: Altered Field IV | • arena_y_max = 9.0 • build_zone_y_max = 8.5 • friction = 0.0 • gravity = gravity_cataclysm(t) • beam_density_scale = 300.0 • linear_damping = -6.0 • angular_damping = -4.0 • joint_force_limit = 0.01 • max_beam_count = 1 • max_structure_mass = 0.005 | |
| E_02: Thruster flyer in fluid | ||
AI Agent Benchmarks
AI agent benchmarks measure how reliably autonomous systems plan, use tools, interact, and complete tasks across controlled and real-world settings.
See all 1,497 papers