cs.AIOct 21, 2026

PLE: Physical-Engineering's Last Exam

Authors: Rujun Han, Yufan Zhuang, Arijit Ray, Han Yu, Zoey CuiZhu, George Lee, Yifei Ming, Binz Roy, 

Organizations: Google Cloud AI Research · Google

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Frontier language models now score above 90% on popular knowledge benchmarks and are increasingly deployed as agents that carry out multi-hour professional work. Yet no existing benchmark tests whether they can perform the real-world, economically valuable work of a professional physical engineer, such as modelling physics and legal code requirements across a multi-format project corpus, optimising material and installation costs, and issuing 2D/3D engineering design packages that a licensed reviewer would sign off on. Existing engineering benchmarks focus on textbook question answering, isolated tool calls or self-contained toy design problems. As a result, most of these benchmarks are already saturated by leading agents and solving them is far away from delivering real engineering design. We introduce PLE, a benchmark of 85 long-horizon, agentic tasks in professional physical engineering consisting of electrical, civil, structural and mechanical engineering, where solving a task means completing a realistic, billable engineering deliverable. PLE combines two complementary curation paths. Expert-Authored tasks are created end to end by practising engineers from a professional data provider's expert pool. Co-Authored tasks start from problem statements written by practising engineers; an adversarial Creator-Solver agent pipeline constructs each task's realistic project environment, executable grader and reference deliverables before sending them to expert engineers for validation and difficulty intensification. We evaluate a variety of frontier LLMs under seven publicly available agent harnesses. Even the strongest configuration reaches a mean score of only 50.6%, while spending 27M tokens and 63 minutes per task on average, demonstrating the substantial gap between current agents and autonomous physical engineering design.

Figures & tables

Figure 1: Overview of the four professional physical-engineering domains in PLE: (A) Structural, (B) Civil, (C) Electrical, and (D) Mechanical Engineering, illustrated with representative tasks.
Figure 2: Score and cost of every model (harness) pair on the 85 tasks of PLE (mean of three runs; cost per task at list prices, log scale). Colored dots joined by the line form the performance-cost frontier (the highest score for the cost), colored by model family; Claude Fable 5 in Claude Code, Grok 4.7 in Grok Build and Gemini 3.8 Flash in Antigravity are also shown in color off the frontier; gray dots are the remaining pairs. Shading splits the chart at the median cost (6.63)andmedianscore(25.76.63) and median score (25.7%): more score for less cost at upper left, less score for more cost at lower right. The best configuration, Claude Opus 5.5 in Claude Code, scores 50.6% at 17.18 per task.
Table 1: Comparison with representative benchmarks. #Physical Engineering : tasks in professional physical engineering (electrical, civil, structural and mechanical engineering). Real Work Tasks : tasks reflecting authentic professional workflows that get real job work done, rather than textbook problems, competitions, or isolated sub-tasks. Expert Curation : tasks authored or validated by domain professionals. Large Input Corpus : the agent must navigate a realistic multi-format document set (e.g. PDF, spreadsheet, CAD). Large Output Artifacts : the output is a production work-product file rather than an answer string. Objective Function : programmatic scoring of solution quality or correctness. Rubric : expert rubric or expert-judgment grading.
Figure 3: An example PLE 3D task: multi-system MEP coordination of an equipment hall ( MEP coordination, hall A with 8 input files). Input 1: A raw 3D equipment hall model with floor equipment in place but no overhead distribution. Input 2: Supporting engineering documents (floor plans, schedules, schematics and design codes). Output: A complete, collision-free 3D model routing overhead cable trays, power busways, chilled-water piping and fire sprinklers to connect every cabinet while satisfying code clearances and minimising route length.
Figure 4: Overview of the co-authored curation pipeline. (1) Expert Seed Prompt: A practising engineer defines the engineering problem brief and target deliverables via our task nomination interface ( fig. 9 ). (2) Task Creator Agent: An agent researches real engineering sources to construct the task prompt, multi-format project environment, and executable verifier with a reference solution. (3) Task Solver Agent: A sandboxed Solver agent attempts the task across N=3N{=}3 runs without verifier or reference access; if the mean score exceeds the difficulty target, the Creator inspects the solver logs to intensify task difficulty. (4) Expert Review: Tasks meeting the difficulty target undergo final validation and difficulty intensification by licensed engineers before entering PLE.
Table 6
Figure 6: Environment file types by task curation approach. Expert-authored environments are document-centric; co-authored environments add CAD/BIM models, schematics and imagery, web portals and code, and are 12.6×12.6\times larger on disk per task.
Figure 7: Deliverable file type by curation approach (249 deliverables across 85 tasks). Expert-authored tasks mostly issue one formal report or workbook; co-authored tasks issue multi-file packages.
Table 3: Results of Claude Opus 5.5 in each of the seven agent harnesses on all 85 PLE tasks. All numbers are scores in percent (%), the mean task score over three runs (higher is better; best harness per column in bold). Rubric items are judged by Claude Opus 5.5, and co-authored tasks are scored with eq. 1 . Task counts in parentheses.
Figure 8: Per-task scores on all 85 tasks with Claude Opus 5.5 in Claude Code (mean of three runs). The dashed redline indicates the mean score (50.6%).
Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Table 4: Evaluation configuration for all results in section 5 .
Table 5: Mean score in percent of every model (harness) pair (three runs each) on all 85 tasks (best harness per model in bold). Rubric items are judged by Claude Opus 5.5, and co-authored tasks are scored with eq. 1 . Table 3 breaks down the Claude Opus 5.5 row by curation approach and discipline.
Figure 9: Demonstration diagram of the PLE Task Nomination Interface used by practising physical engineers to author the Expert Seed Prompt (Stage 1 of fig. 4 ). Engineers specify the real-world engineering Situation , starting Inputs , quantifiable optimisation Goal , physical and code Constraints , deterministic Verification logic, and Expected Output deliverables (illustrated here with the LV Cable Sizing task).
More in this topic

AI Agent Benchmarks

AI agent benchmarks measure how reliably autonomous systems plan, use tools, interact, and complete tasks across controlled and real-world settings.

See all 1,494 papers

Explore similar work

CardsList
May 19, 2026

EngiAI: A Multi-Agent Framework and Benchmark Suite for LLM-Driven Engineering Design

Large Language Model (LLM) agents are increasingly applied to engineering design tasks, yet existing evaluation frameworks do not adequately address multi-agent systems that combine simulation, retrieval, and manufacturing preparation. We introduce a benchmark suite with three evaluation dimensions: (1) a workflow benchmark with seven prompt styles targeting distinct cognitive demands-including direct tool use, semantic disambiguation, conditional branching, and working-memory tasks; (2) a Retrieval-Augmented Generation (RAG) benchmark with gated scoring isolating retrieval contributions to parameter selection; and (3) an High Performance Computing (HPC) benchmark evaluating end-to-end ML training orchestration on a SLURM cluster. Alongside the benchmark we present EngiAI, a Multi-Agent System (MAS) reference implementation built on LangGraph that operationalizes the benchmark by coordinating seven specialized agents through a supervisor architecture, unifying topology optimization, document retrieval, HPC job orchestration, and 3D printer control. Across four LLM backends and two EngiBench problems, proprietary models achieve 96-97% average task completion on Beams2D, while open-source 4B-parameter models reach 55-78%, with clear generational improvement. Conditional branching proves most challenging, with task completion dropping to 20-53% for the conditional style on Photonics2D. RAG gating confirms near-perfect retrieval-augmented scores (about 1.0) versus near-zero without retrieval, validating the evaluation design. On HPC orchestration, one model completes all pipeline steps in 100% of runs while another drops to 50%, revealing that multi-step instruction following degrades over long-running workflows.

Gioele Molinari, Florian Felten, Soheyl Massoudi +1

IDEAL Chair of Artificial Intelligence in Engineering Design ETH Zurich Zurich, Switzerland · IDEAL, ETH Zurich, Zurich, Switzerland · Autom8.build, Brussels, Belgium

Multi-Agent LLM SystemsLLM Agent EvaluationRAG Evaluation
AskPDF
Apr 14, 2026

Frontier-Eng: Benchmarking Self-Evolving Agents on Real-World Engineering Tasks with Generative Optimization

Current LLM agent benchmarks, which predominantly focus on binary pass/fail tasks such as code generation or search-based question answering, often neglect the value of real-world engineering that is often captured through the iterative optimization of feasible designs. To this end, we introduce Frontier-Eng, a human-verified benchmark for generative optimization -- an iterative propose-execute-evaluate loop in which an agent generates candidate artifacts, receives executable verifier feedback, and revises them under a fixed interaction budget -- spanning 4747 tasks across five broad engineering categories. Unlike previous suites, Frontier-Eng tasks are grounded in industrial-grade simulators and verifiers that provide continuous reward signals and enforce hard feasibility constraints under constrained budgets. We evaluate eight frontier language models using representative search frameworks, finding that while GPT 5.4 achieves the most robust performance, the benchmark remains challenging for all models. Our analysis suggests a dual power-law decay in improvement frequency (∼\sim 1/iteration) and magnitude (∼\sim 1/improvement count). We further show that although width improves parallelism and diversity, depth remains crucial for hard-won improvements under a fixed budget. Frontier-Eng establishes a new standard for assessing the capacity of AI agents to integrate domain knowledge with executable feedback to solve complex, open-ended engineering problems.

Yizhe Chi, Deyao Hong, Dapeng Jiang +18

Navers Lab, Einsia.AI

AI Agent EvaluationAI Agent BenchmarksLarge Language Model-Guided Optimization
AskPDF
Sep 29, 2026

EngiWorld: What Can Frontier Agents Deliver in Professional Engineering Environments?

Autonomous agents have made rapid progress in general-purpose computer use, yet reliable automation of professional industrial engineering remains out of reach. Engineering workflows require reasoning over geometric and physical constraints and keeping dependencies consistent across software and design stages. We present EngiWorld, a benchmark designed around the capabilities that real engineering projects require, from operating a single application to solving engineering problems end to end. It contains 1,301 expert-curated tasks spanning six engineering domains (CAD, CAE, CAM, BIM, EDA, and DCC) and 26 professional software applications, with both GUI and CLI interfaces. The tasks are organized into six types, corresponding to capabilities that real engineering projects require. We further introduce an artifact-centric evaluation methodology built on a unified suite of domain verifiers. The verifiers programmatically check whether final and intermediate artifacts are geometrically valid, physically feasible, and compliant with design rules. Beyond binary success, they also score how well optimized designs meet their specifications. Evaluation of seven frontier models reveals a substantial capability gap: the strongest model scores only 44.7/100, and just 3.6% of runs on tasks that require cross-software coordination succeed. EngiWorld provides a rigorous foundation for measuring progress toward agents that operate professional engineering software end to end.

Hongcheng Gao, Hailong Qu, Yu Lei +19

Tsinghua University · Zhiman Inc. · Chongqing University +8

Software Engineering AgentsAI Agent Benchmarks
AskPDF