Frontier-Eng: Benchmarking Self-Evolving Agents on Real-World Engineering Tasks with Generative Optimization

Yizhe Chi; Deyao Hong; Dapeng Jiang; Tianwei Luo; Kaisen Yang; Boshi Zhang; Zhe Cao; Xiaoyan Fan

cs.AIApr 14, 2026

Frontier-Eng: Benchmarking Self-Evolving Agents on Real-World Engineering Tasks with Generative Optimization

Authors: Yizhe Chi, Deyao Hong, Dapeng Jiang, Tianwei Luo, Kaisen Yang, Boshi Zhang, Zhe Cao, Xiaoyan Fan,

Organizations: Navers Lab, Einsia.AI

Abstract

Current LLM agent benchmarks, which predominantly focus on binary pass/fail tasks such as code generation or search-based question answering, often neglect the value of real-world engineering that is often captured through the iterative optimization of feasible designs. To this end, we introduce Frontier-Eng, a human-verified benchmark for generative optimization -- an iterative propose-execute-evaluate loop in which an agent generates candidate artifacts, receives executable verifier feedback, and revises them under a fixed interaction budget -- spanning $47$ tasks across five broad engineering categories. Unlike previous suites, Frontier-Eng tasks are grounded in industrial-grade simulators and verifiers that provide continuous reward signals and enforce hard feasibility constraints under constrained budgets. We evaluate eight frontier language models using representative search frameworks, finding that while GPT 5.4 achieves the most robust performance, the benchmark remains challenging for all models. Our analysis suggests a dual power-law decay in improvement frequency ( $\sim$ 1/iteration) and magnitude ( $\sim$ 1/improvement count). We further show that although width improves parallelism and diversity, depth remains crucial for hard-won improvements under a fixed budget. Frontier-Eng establishes a new standard for assessing the capacity of AI agents to integrate domain knowledge with executable feedback to solve complex, open-ended engineering problems.

Figures & tables

Figure 1: Overview of Frontier-Eng. Top: We contrast binary-reward agent benchmarks (search/coding with pass–fail style outcomes) against generative optimization , where an agent repeatedly proposes code edits, receives verifier feedback, and improves under a fixed interaction budget. Bottom left: Frontier-Eng covers $47$ tasks across five engineering categories, spanning heterogeneous artifact types, objective structures, and simulator families. Bottom right: The integration pipeline enforces read-only evaluators, isolated execution, feasibility validation, and verifier-parsed scoring, so gains must come from genuine solution improvement rather than reward hacking.

Figure 2: Method and benchmark composition overview of Frontier-Eng. The figure summarizes the unified task interface $\tau=(\mathcal{C},x_{0},\mathcal{E})$ , the iterative propose–evaluate–improve loop under a fixed budget, and the benchmark-level aggregation pipeline for cross-task comparison; it also shows the data composition of the benchmark, which contains 47 tasks organized into five engineering categories.

Figure 3: Case study under openevolve for SustainableDataCenterControl . The figure illustrates how policies evolve from the contributor baseline through repeated propose–evaluate–improve iterations under a fixed verifier and budget. It highlights representative optimization trajectories and final outcomes across model families, emphasizing three practical signals: improvement speed in early iterations, achievable feasible gain over baseline, and stability of late-stage refinements.

Task	Opus	Qwen	Seed	GPT	OSS	GLM	DS	Grok	Gemini
Aerodynamics / CarAerodynamicsSensing	8	$\star$ 2	8	4	6	5	$\star$ 2	8	$\star$ 2
Astrodynamics / MannedLunarLanding	4	8.5	6	2	5	$\star$ 1	3	8.5	7
ComputerSystems / MallocLab	$\star$ 1	8	7	9	4.5	2	4.5	3	6
Cryptographic / AES-128	4	9	8	$\star$ 1	3	7	2	5	6
Cryptographic / SHA-256	4	8	5	$\star$ 1	2	6	9	3	7
Cryptographic / SHA3-256	5	7	3	$\star$ 1	2	4	6	9	8

Table 2: Within-task rank comparison of nine foundation models on the full 47-task set under openevolve (100 iterations, same initial programs, same frozen verifiers), with gpt-5.4 replaced by its retest results under the same protocol. Header abbreviations are unified as: Opus

=

claude-opus-4.6 , Qwen

=

qwen3-coder-next , Seed

=

seed-2.0-pro , GPT

=

gpt-5.4 , OSS

=

gpt-oss-120b , GLM

=

glm-5 , DS

=

deepseek-v3.2 , Grok

=

grok-4.20 , Gemini

=

gemini-3.1-pro-preview . Rank 1 denotes the best model on that task; ties receive average ranks. Bold with a superscript star marks the best (lowest) rank in each row. The Average row reports mean within-task rank over all 47 tasks (lower is better).

Figure 4: Aggregate diagnostics of nine models on all 47 tasks under openevolve , after replacing gpt-5.4 with the retest results. Left: Dolan–Moré performance profile (higher and more left-shifted curves indicate better performance). Ratios are computed on per-task translated scores (see Section 2.4 ):

\alpha=1.0

means a model attains the task-best translated score (including ties), while

\alpha=1.1

means the translated score is at least

1/1.1\approx 90.9\%

of the translated best. Under this definition, gpt-5.4 is best on

24/47

tasks (

51.1\%

), and is within the

1.1

near-best band on

33/47

tasks (

70.2\%

). Right: Win rate over baseline (fraction of tasks for which the model’s best score exceeds the initial score).

Framework	Opus Rank	OSS Rank	Opus Wins	OSS Wins	Ties
abmcts	$\star$ 1.43	1.55	$\star$ 26	20	1
openevolve	$\star$ 1.28	1.64	$\star$ 30	13	4
shinkaevolve	$\star$ 1.28	1.62	$\star$ 29	13	5

Table 3: Average ranks, wins, and ties of the two models under the three search frameworks on all tasks. We use Opus

=

claude-opus-4.6 and OSS

=

gpt-oss-120b . A win is counted when one model’s best score is strictly higher on a task; ties are listed separately. For rank aggregation, each task assigns rank 1 to the better model and rank 2 to the other model, while ties assign rank 1 to both models. Bold with

\star

marks the better model in each row (lower is better for ranks; higher is better for wins).

Model + Framework	gain(%)	early-share	last-improve	best-updates	plateau
Opus + abmcts	61.6	0.89	0.53	4.0	0.65
Opus + openevolve	$\star$ 73.3	0.97	$\star$ 0.70	5.5	0.51
Opus + shinkaevolve	72.9	0.95	0.44	4.0	0.62
OSS + abmcts	50.4	$\star$ 0.99	0.30	3.0	0.72
OSS + openevolve	58.9	$\star$ 0.99	0.53	$\star$ 6.0	0.57
OSS + shinkaevolve	68.3	0.89	0.64	5.0	$\star$ 0.45

Table 4: Trajectory tendency statistics for each model–framework combination, with Opus

=

claude-opus-4.6 and OSS

=

gpt-oss-120b . Metrics are defined as: gain (median baseline-relative improvement), early-share (fraction of total improvement achieved in the first quarter), last-improve (normalized step index of the final improvement; larger is later), best-updates (count of running-best updates), and plateau (longest no-improvement fraction). Bold with

\star

marks the best value in each metric column (higher is better for gain, early-share, last-improve, and best-updates; lower is better for plateau).

Figure 5: Dual

-1

power-law decay in improvement dynamics across 47 tasks (GPT-OSS-120B, OpenEvolve , 500 iterations). (a) Histogram of improvement events by iteration; dashed line is a

\propto t^{-1}

fit (

R^{2}{=}0.84

). (b) Median normalized improvement magnitude by improvement rank

k

within each task’s trajectory; dashed line is a

\propto k^{-1}

fit, showing that improvement size shrinks with ordinal rank just as improvement frequency shrinks with iteration.

Figure 6: Normalized best-of- $n$ score across depth and width configurations (GPT-OSS-120B, OpenEvolve ). Each cell shows the mean normalized score for $n$ chains each run to depth $d$ . Scores are normalized per task by the maximum across all configurations. Red borders mark $n\times d=256$ : scores decrease with $n$ , showing depth dominates width.

Engineering QA benchmarks
Benchmark	Interactive env.	Hard constraints	Exec. verifier	Cross- domain	Best-found under budget
Transportation systems ( Syed and others, 2024 )	$\times$	$\times$	$\times$	$\times$	$\times$
Control engineering ( Kevian and others, 2024 )	$\times$	$\times$	$\times$	$\times$	$\times$
EEE-Bench ( Li et al., 2024 )	$\times$	$\times$	$\times$	$\times$	$\times$
CIRCUIT ( Skelic et al., 2025 )	$\times$	$\times$	$\times$	$\times$	$\times$
Open-world agent benchmarks

Table 5: Selected adjacent benchmarks and the comparison dimensions used to position Frontier-Eng.

\checkmark

\times

, and

\checkmark

denote that a property is fully present, absent, or partially present (e.g., for some tasks or only indirectly emphasized). “Hard constraints” refers to explicit valid/invalid conditions beyond answer accuracy alone. “Best-found under budget” refers to evaluation by the best feasible solution found within a fixed interaction budget.

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Task	Best Combination	Final Score	Runner-up Combination	Final Score
MallocLab	Claude + OpenEvolve	97.00	Claude + ShinkaEvolve	97.00
BatteryFastChargingProfile	Claude + OpenEvolve	120.80	Claude + ABMCTS	113.53
InventoryOptimization_general_meio	Claude + OpenEvolve	0.9929	Claude + ShinkaEvolve	0.9694
SHA3-256	Claude + OpenEvolve	28.90	GPT-OSS + ABMCTS	24.42

Appendix

Table 6: Final-score comparison across models and frameworks for the four focus cases.

Figure 7: Running-best staircase plots for MallocLab .

Figure 8: Running-best staircase plots for BatteryFastChargingProfile .

Figure 9: Running-best staircase plots for InventoryOptimization_general_meio .

Figure 10: Running-best staircase plots for SHA3-256 .

Task	Baseline	Claude	DeepSeek	Gemini	GLM	GPT-5.4	Grok	Qwen	Seed	GPT-OSS
Aerodynamics_CarAerodynamicsSensing	0.9617	0.9624	0.9632	0.9632	0.9628	0.96307	0.9624	0.9632	0.9624	0.9625
Astrodynamics_MannedLunarLanding	4577.437	6027.3126	6079.2455	4674.9462	6839.0331	6660.94	4577.437	4577.437	4733.0435	5589.881
ComputerSystems_MallocLab	28	96	53	48	86	28	57	32	38	53
Cryptographic_AES-128	7.5209	11.8617	12.4591	10.2396	7.9669	39.825	10.8615	5.5501	7.9481	12.1639
Cryptographic_SHA-256	9.8274	16.7955	9.718	9.942	15.1655	26.3405	17.2504	9.8475	15.2838	20.4245
Cryptographic_SHA3-256	16.0932	17.4003	17.0749	16.2255	17.5778	37.4451	16.0594	16.5292	18.3478	19.5513

Appendix

Table 7: Raw per-task scores underlying the Experiment 1 model comparison in Section 3.1 (Table 2 ).

AI Agent Evaluation

AI agent evaluation measures whether tool-using systems complete tasks reliably, safely, and efficiently across interactive settings.

See all 519 papers

EngiAI: A Multi-Agent Framework and Benchmark Suite for LLM-Driven Engineering Design
May 19, 2026Gioele Molinari, Florian Felten, Soheyl Massoudi +1Multi-Agent LLM Systems LLM Agent Evaluation
IDEAL Chair of Artificial Intelligence in Engineering Design ETH Zurich Zurich, Switzerland · IDEAL, ETH Zurich, Zurich, Switzerland · Autom8.build, Brussels, Belgium
Understanding the Challenges in Iterative Generative Optimization with LLMs
Mar 25, 2026Allen Nie, Xavier Daull, Zhiyi Kuang +10Large Language Model-Guided Optimization Self-Improving Agents
Google DeepMind · CNRS · Stanford University +5
PLE: Physical-Engineering's Last Exam
Oct 21, 2026Rujun Han, Yufan Zhuang, Arijit Ray +10AI Agent Benchmarks Benchmark Design
Google Cloud AI Research · Google

May 19, 2026J/K move · Enter open · S save

EngiAI: A Multi-Agent Framework and Benchmark Suite for LLM-Driven Engineering Design

Gioele Molinari, Florian Felten, Soheyl Massoudi, Mark Fuge

IDEAL Chair of Artificial Intelligence in Engineering Design ETH Zurich Zurich, Switzerland · IDEAL, ETH Zurich, Zurich, Switzerland · Autom8.build, Brussels, Belgium

Multi-Agent LLM Systems LLM Agent Evaluation RAG Evaluation

Open paper AskPDF arXiv

Large Language Model (LLM) agents are increasingly applied to engineering design tasks, yet existing evaluation frameworks do not adequately address multi-agent systems that combine simulation, retrieval, and manufacturing preparation. We introduce a benchmark suite with three evaluation dimensions: (1) a workflow benchmark with seven prompt styles targeting distinct cognitive demands-including direct tool use, semantic disambiguation, conditional branching, and working-memory tasks; (2) a Retrieval-Augmented Generation (RAG) benchmark with gated scoring isolating retrieval contributions to parameter selection; and (3) an High Performance Computing (HPC) benchmark evaluating end-to-end ML training orchestration on a SLURM cluster. Alongside the benchmark we present EngiAI, a Multi-Agent System (MAS) reference implementation built on LangGraph that operationalizes the benchmark by coordinating seven specialized agents through a supervisor architecture, unifying topology optimization, document retrieval, HPC job orchestration, and 3D printer control. Across four LLM backends and two EngiBench problems, proprietary models achieve 96-97% average task completion on Beams2D, while open-source 4B-parameter models reach 55-78%, with clear generational improvement. Conditional branching proves most challenging, with task completion dropping to 20-53% for the conditional style on Photonics2D. RAG gating confirms near-perfect retrieval-augmented scores (about 1.0) versus near-zero without retrieval, validating the evaluation design. On HPC orchestration, one model completes all pipeline steps in 100% of runs while another drops to 50%, revealing that multi-step instruction following degrades over long-running workflows.

Abstract

AI Agent Evaluation

Explore similar work

EngiAI: A Multi-Agent Framework and Benchmark Suite for LLM-Driven Engineering Design

Understanding the Challenges in Iterative Generative Optimization with LLMs

PLE: Physical-Engineering's Last Exam