Engineering Reasoning and Instruction (ERI) Benchmark: A Large Taxonomy-driven Dataset for Foundation Models and Agents
Authors: MZ Naser, Ahmad Bani Awwad, Zoie McCreery, Radwa Eissa, Ahmad Naser, Gianluca Cusatis, Andrew Metcalf, Kapil Madathil, Jamal Abdalla, Venkatesh Kodur, Mohammad Reza Saeb
Organizations: School of Civil and Environmental Engineering & Earth Sciences, Clemson University, USA · Artificial Intelligence Research Institute for Science and Engineering, Clemson University, USA · Department of Mechanical Engineering, University of Manitoba, Canada · Department of Civil and Environmental Engineering, Northwestern University, USA · Department of Environmental Engineering and Earth Sciences, Clemson University, USA · Departments of Industrial Engineering, Clemson University, USA · Departments of Civil Engineering, American University of Sharjah, UAE · Departments of Civil and Environmental Engineering, Michigan State University, USA · Department of Pharmaceutical Chemistry, Medical University of Gdańsk, Gdańsk, Poland
The Engineering Reasoning and Instruction (ERI) benchmark is a taxonomy-driven instruction dataset designed to train and evaluate engineering-capable large language models (LLMs) and agents. This dataset spans nine engineering fields (namely: civil, mechanical, electrical, chemical, environmental, aerospace, materials, fire, and industrial engineering) and 55 subdomains, and is crossed with seven intent types (i.e., definition, explanation, calculation, comparison, design/synthesis, troubleshooting, and code-related) and three difficulty tiers (undergraduate, graduate, and professional), yielding 57,750 records with field/subdomain/type/difficulty metadata and solution formatting. We examined ERI via seven LLMs and report a statistically significant three-tier performance structure, with frontier models (GPT-5, Claude Sonnet 4, DeepSeek V3.1) achieving mean scores above 4.30 on a five-point scale, while mid-tier and smaller models exhibited progressively higher failure rates and steeper performance degradation on graduate-level questions. To address circularity concerns inherent in LLM benchmarks, we developed a convergent validation protocol that leverages cross-provider independence, multi-judge averaging, and frontier-model agreement analysis to empirically bound hallucination risk to 1.7%. ERI is released with taxonomy specifications, validation scripts, and an evaluation harness to enable reproducible comparisons and regression testing for instruction tuning, routing, retrieval-augmented evaluation, and agentic tool-use workflows in engineering settings.
We introduce CFE-Bench (Classroom Final Exam), a multimodal benchmark for evaluating the reasoning capabilities of large language models across more than 20 STEM domains. CFE-Bench is curated from repeatedly used, authentic university homework and exam problems, paired with reference solutions provided by course instructors. CFE-Bench remains challenging for frontier models: the newly released Gemini-3.1-pro-preview achieves 59.69% overall accuracy, while the second-best model, Gemini-3-flash-preview, reaches 55.46%, leaving substantial room for improvement. Beyond aggregate scores, we conduct a diagnostic analysis by decomposing instructor reference solutions into structured reasoning flows. We find that while frontier models often answer intermediate sub-questions correctly, they struggle to reliably derive and maintain correct intermediate states throughout multi-step solutions. We further observe that model-generated solutions typically contain more reasoning steps than instructor solutions, indicating lower step efficiency and a higher risk of error accumulation. Data and code are available at https://github.com/Analogy-AI/CFE_Bench.
Chongyang Gao, Diji Yang, Shuyan Zhou +4
Northwestern University · UC Santa Cruz · Duke University +3
Most engineering problems involve complex multi-step derivations through checkable intermediate quantities. Grading only the final answer misjudges LLMs, crediting correct answers reached through flawed steps and penalizing wrong answers built on mostly sound ones. Yet, existing benchmarks such as SuperGPQA and EEE-Bench score only final answers, and step-level ones such as TPS-CalcBench rely on LLM judges or human graders, or cover a single domain. We therefore introduce EngTrace, a symbolic benchmark for verifiable process evaluation of engineering reasoning. Its 150 expert-certified templates, spanning five branches, 15 domains, and 42 areas, generate each problem with a gold reasoning trace, so that every intermediate quantity is known exactly. We design and validate an evaluator that checks final answers, coverage of the gold trace's intermediate quantities (milestones), and displayed arithmetic, mostly without an LLM judge. Across 11 LLMs, final answers barely separate the strongest models, while their derivations differ. Wrong answers still state about half of the gold milestones, and domain experts trace most wrong answers to calculation slips, with conceptual errors rarer on easy problems than on harder ones. Overall, as LLMs near the final-answer ceiling, checking derivations, as EngTrace does, reveals differences that final answers miss.
Ayesha Gull, Muhammad Usman Safder, Rania Elbadry +4
Namal University · MBZUAI · The University of Tokyo
Large Language Model (LLM) agents are increasingly applied to engineering design tasks, yet existing evaluation frameworks do not adequately address multi-agent systems that combine simulation, retrieval, and manufacturing preparation. We introduce a benchmark suite with three evaluation dimensions: (1) a workflow benchmark with seven prompt styles targeting distinct cognitive demands-including direct tool use, semantic disambiguation, conditional branching, and working-memory tasks; (2) a Retrieval-Augmented Generation (RAG) benchmark with gated scoring isolating retrieval contributions to parameter selection; and (3) an High Performance Computing (HPC) benchmark evaluating end-to-end ML training orchestration on a SLURM cluster. Alongside the benchmark we present EngiAI, a Multi-Agent System (MAS) reference implementation built on LangGraph that operationalizes the benchmark by coordinating seven specialized agents through a supervisor architecture, unifying topology optimization, document retrieval, HPC job orchestration, and 3D printer control. Across four LLM backends and two EngiBench problems, proprietary models achieve 96-97% average task completion on Beams2D, while open-source 4B-parameter models reach 55-78%, with clear generational improvement. Conditional branching proves most challenging, with task completion dropping to 20-53% for the conditional style on Photonics2D. RAG gating confirms near-perfect retrieval-augmented scores (about 1.0) versus near-zero without retrieval, validating the evaluation design. On HPC orchestration, one model completes all pipeline steps in 100% of runs while another drops to 50%, revealing that multi-step instruction following degrades over long-running workflows.