Most engineering problems involve complex multi-step derivations through checkable intermediate quantities. Grading only the final answer misjudges LLMs, crediting correct answers reached through flawed steps and penalizing wrong answers built on mostly sound ones. Yet, existing benchmarks such as SuperGPQA and EEE-Bench score only final answers, and step-level ones such as TPS-CalcBench rely on LLM judges or human graders, or cover a single domain. We therefore introduce EngTrace, a symbolic benchmark for verifiable process evaluation of engineering reasoning. Its 150 expert-certified templates, spanning five branches, 15 domains, and 42 areas, generate each problem with a gold reasoning trace, so that every intermediate quantity is known exactly. We design and validate an evaluator that checks final answers, coverage of the gold trace's intermediate quantities (milestones), and displayed arithmetic, mostly without an LLM judge. Across 11 LLMs, final answers barely separate the strongest models, while their derivations differ. Wrong answers still state about half of the gold milestones, and domain experts trace most wrong answers to calculation slips, with conceptual errors rarer on easy problems than on harder ones. Overall, as LLMs near the final-answer ceiling, checking derivations, as EngTrace does, reveals differences that final answers miss.
Figures & tables
Figure 1: An example from EngTrace : continuous stirred-tank reactor (CSTR) volume calculation. A symbolic template generates a unique problem instance paired with a gold reasoning trace to enable verifiable process evaluation.
Figure 2: EngTrace ’s Hierarchical Taxonomy spanning 15 domains and 42 areas across five branches: process equilibrium and dynamics (Chemical); field theory and signal transmission (Electrical); structures, soils, and water resources (Civil); production, quality, and stochastic operations (Industrial); and physical mechanics and material properties (Mechanical).
Figure 3: The EngTrace Template Generation Pipeline maps domain-aware constraints to serialized question-solution pairs via physically validated computation, illustrated here using a CSTR volume calculation example.
Figure 4: Final Answer Accuracy by difficulty level for four representative models. The mean over the 58 Easy, 58 Intermediate, and 34 Advanced templates, with 95% intervals, for DeepSeek V4.1 Flash (first on FAC), Claude Sonnet 5 (first on MC), GPT-5.4 mini (closed, fifth on FAC), and gpt-oss-20b (GPT OSS 20B in the figure). Table 20 gives every model’s gap with its tests.
Figure 5: Error categories of the wrong answers read. Each bar gives one model’s readings by category, from three domain experts per wrong answer (57 to 120 readings per model) on responses at the providers’ default settings. Table 24 gives them by level.
Appendix figures & tables20 assets
Supplementary material from the paper’s appendix.
Appendix
Domain
Textbook
Reaction Kinetics
Elements of Chemical Reaction Engineering by H. Scott Fogler
Thermodynamics
Introduction to Chemical Engineering Thermodynamics by J. M. Smith, H. C. Van Ness, and M. M. Abbott
Transport Phenomena
Transport Phenomena by R. Byron Bird, Warren E. Stewart, and Edwin N. Lightfoot
Electromagnetics and Waves
Fundamentals of Applied Electromagnetics by F. T. Ulaby and Umberto Ravaioli
Digital Communications
Digital Communications by John G. Proakis and Masoud Salehi
Signals and Systems
Discrete-Time Signal Processing by Alan V. Oppenheim and Ronald W. Schafer
Appendix
Table 2: Foundational textbooks. The textbooks of each domain, from which the templates take their problems.
Engineering Branch
Authoritative Data Sources
Chemical Engineering
Perry’s Chemical Engineers’ Handbook NIST Chemistry WebBook CRC Handbook of Chemistry and Physics NIST-JANAF Thermochemical Tables NASA Technical Report R-132
Electrical Engineering
Standard Handbook for Electrical Engineers CODATA 2022 Recommended Values refractiveindex.info Database
NAVFAC DM-7.01 and DM-7.02 FHWA HDS-4 and HEC-22 NRCS TR-55 USGS Water-Supply Paper 2339 AISC Shapes Database v16.0 NIST Chemistry WebBook CODATA 2022 Recommended Values NIST SP 811
Industrial Engineering
MIL-STD-105E NIST/SEMATECH e-Handbook of Statistical Methods
Appendix
Table 3: Authoritative data sources. The sources of the constants and material properties the templates sample, by branch.
Domain and area
Easy
Int.
Adv.
Total
Chemical (7 areas)
13
9
8
30
Reaction Kinetics
Conversion and Reactor Sizing
0
0
1
1
Mole Balances
2
1
2
5
Stoichiometry
1
2
1
4
Thermodynamics
Appendix
Table 4: Templates per area and level. 42 areas in 15 domains; each branch has 30 templates; the shaded rows give the branch subtotals and the italic rows name the domain.
Branch
Structure
Difficulty Level
Total
Dom.
Areas
Easy
Int.
Adv.
Chemical
3
7
13
9
8
30
Electrical
3
7
11
12
7
30
Civil
3
11
12
12
6
30
Industrial
3
11
12
12
6
30
Mechanical
3
6
10
13
7
30
Appendix
Table 5: Dataset statistics by branch. Domains, areas, and templates per difficulty level; each branch holds 30 templates.
Variants
Wording
Reasoning Paths
Answer Forms
1
38
58
104
2
17
31
25
3–5
34
25
14
6–15
61
36
7
Appendix
Table 6: Variation within a template. Number of templates whose 15 evaluation-set instances show the given number of distinct question wordings, reasoning paths, and answer forms, with numbers masked; each column sums to 150.
Branch
Wrong Constant
Wrong Unit Conversion
Flipped Sign
Wrong Printed Result
Chemical
van der Waals pressure: R with two digits transposed
annulus flow rate: kPa to Pa by a factor of 100
heat of reaction: Hess’s law reversed
first-order batch reactor: one printed time 4% too high
Electrical
Coulomb’s law: ε0 ten times too large
wave parameters: MHz to Hz by a factor of 105
C/D and D/C conversion: a delay written as an advance
mean and variance: one printed mean 0.2 too high
Civil
Manning discharge: R3/4 for R2/3
cantilever deflection: m to mm by a factor of 100
effective stress: σ′=σ+u
beam reactions: one printed reaction 4% too high
Industrial
safety stock: a two-sided z for a one-sided one
M/M/1 time in system: hours to minutes by a factor of 100
xˉ and R chart limits: the lower limit as center plus A2Rˉ
process capability: one printed Cp 6% too high
Mechanical
torsional shear stress: J with π/4 for π/2
hydrostatic pressure: kPa to Pa by a factor of 100
U-tube manometer: the pipe-fluid column’s sign flipped
natural frequency: one printed ωn 4% too high
Appendix
Table 7: Planted defects. The 20 planted defects, four per branch, each written into a real template of the branch; every expert of the branch reviews all four.
Figure 6: Certification app. (a) The hand check: the expert enters the final answer with its unit before the solution is shown. (b) The review: the result of the hand check, the question beside the gold solution, the source code in a second tab, the template’s other instances, and the verdict form.
Branch
Approve or Reject
AC2 on Scores
κ
AC1
Agr.
Phys.
Math.
Ped.
Chemical
0.492
0.776
77%
0.898
0.952
0.912
Electrical
0.365
0.926
90%
0.941
0.989
0.976
Civil
–
1.000
100%
0.993
0.981
0.974
Industrial
–
1.000
100%
0.962
0.997
0.965
Mechanical
0.627
0.792
80%
0.927
0.973
0.952
Appendix
Table 8: Expert agreement. Agreement among the three experts of each branch in the first round: Fleiss’ κ , Gwet’s AC1, and percent agreement on the decision, and Gwet’s AC2 on the physical, mathematical, and pedagogical scores. –: undefined, since every decision of the branch is an approval.
Check
Setting
Result
Final answer
ϵ fitted on each half
0.915, 0.939
Milestones
0.5%, unit factors
0.705
0.2%, unit factors
0.697
1%, unit factors
0.653
2%, unit factors
0.568
0.5%, no unit factors
0.696
Appendix
Table 9: Scoring settings. Each check’s setting, in bold where it has alternatives, on the expert study: the three-way agreement with the experts on the half of the responses that ϵ was not fitted on (fitted values 0.0015 and 0.0020); the separation between a response’s coverage of its own milestones and of the other instances of its template; the unstated values credited among 88 milestone values multiplied by 1.37; and precision / recall on the incorrect steps inside correct-answer responses.
Figure 7: A worked scoring example. A wrong answer by Qwen3-235B-2507 on an Intermediate chemical-engineering instance. Matching settles 1 of the 7 milestones under a unit factor, the judge rules the other 6, the arithmetic check flags 1 of the 9 steps, and the response scores v(r)=0 and m(r)=3/7 .
Figure 8: What coverage does and does not measure. A: a correct, valid derivation that bypasses two of three gold milestones scores m(r)=1/3 , since Milestone Coverage credits the gold route. B: a misstated rule whose numbers are those of the right rule passes matching, the arithmetic check and the answer check; only a judge can see it, and not every judge does.
Component
Basis
Result
Agreement with the expert labels on 300 responses
Final-answer check
three-way, all 300 responses
0.930
Final-answer check
281 responses not marked partial
0.986 (step matching 0.747)
Milestone matching
P / R / F1
0.927 / 0.924 / 0.926
Milestone matching with the judge †
P / R / F1
0.930 / 0.989 / 0.958
Arithmetic check
P / R, all steps; correct answers
0.872 / 0.317; 0.817 / 0.427
Appendix
Table 10: Validation results. Top: each component against the domain experts’ labels on the expert study (P precision, R recall; correct answers are responses with a correct final answer). Middle: planted defects caught and untouched steps flagged. Bottom: the domain experts’ readings of the evaluated models’ responses. † Measured on the milestone matches and arithmetic flags recorded when the study was scored (milestone F1 0.921 without the judge); the unmarked rows run each check as the evaluation runs it, so the two milestone F1 values, 0.926 and 0.958, are separate measurements and their difference is not the judge’s contribution alone. ‡ 17 of its 240 calls returned no reply after retries, so its counts rest on the 51 conceptual defects and the 111 untouched steps with a reply.
Answer kind
Study
Three-way
Not partial
Readings
Scalar
60
1.000
1.000
139 of 146
Multipart
80
0.800
0.985
50 of 60
Vector
60
0.933
0.949
12 of 16
Array
20
1.000
1.000
8 of 14
Symbolic
40
1.000
1.000
31 of 34
Classification
40
0.975
1.000
4 of 18
Appendix
Table 11: Final-answer check by answer kind. Study: the expert study’s responses of each kind and the check’s agreement with the experts’ verdict, three-way and on the responses the experts did not call partial. Readings: the domain experts’ readings of the evaluated models’ verdicts that agree with the check; the sample is stratified by verdict, so these are counts, not rates for a kind.
Question
Final answer
Gold
Check
Experts
Why they differ
Average time in the system of an M/M/1/K queue (industrial, Advanced)
3.44 minutes
3.45 minutes
correct
incorrect (both)
the rule accepts a value one unit off in its last digit; the unrounded value, L/λeff , is 3.4475 minutes, which rounds to 3.45, not 3.44
Output signal after downsampling by M=3 (electrical, Advanced)
y[n]=cos(πn/8)
y[n]=cos(15πn/8)
partial
correct (both)
the same sequence, since 15π/8=2π−π/8 ; the check reads the stated numbers and finds 1 of its 3 targets
Appendix
Table 12: One discrepancy in each direction. Two final answers of the evaluated models on which the check’s verdict differs from both domain experts’ readings: it credits the first more and the second less than they do.
Model
Organization
Size
Identifier
Weights
Open-Weights LLMs
DeepSeek V4.1 Flash
DeepSeek
552B (8B/16B active)
deepseek/deepseek-v4.1-flash
deepseek-ai/DeepSeek-V4.1-Flash
Gemma 4 26B
Google DeepMind
25.2B (3.8B active)
google/gemma-4-26b-a4b-it
google/gemma-4-26B-A4B-it
GLM-5.3
Z.ai
753B
z-ai/glm-5.3
zai-org/GLM-5.3
GLM-5.3-Flash
Z.ai
320B (18B active)
z-ai/glm-5.3-flash
zai-org/GLM-5.3-Flash
gpt-oss-20b
OpenAI
21B (3.6B active)
openai/gpt-oss-20b
openai/gpt-oss-20b
Appendix
Table 13: Evaluated models. The eleven models of § 5 , grouped by access, and the two flagship anchors run on the 450-instance subset. Size is the developer’s stated parameter count, with the parameters active per token in parentheses for mixture-of-experts models: DeepSeek V4.1 Flash activates 8B in prefill and 16B in decoding, and for GLM-5.3 , whose card states no active count, we give the parameter count of its released weights; closed models do not disclose their size. Identifier is the model’s identifier on OpenRouter, through which we query every model; Weights is the Hugging Face repository of an open-weights model.
Model
Output ceiling
Providers
Completion tokens
Reasoning tokens
Responses with
median / p90
median / p90
reasoning tokens
Open-Weights LLMs
DeepSeek V4.1 Flash
32,768
8
3,060 / 10,296
2,380 / 9,300
100.0%
Gemma 4 26B ‡
32,768
2
5,556 / 11,174
4,754 / 9,905
100.0%
GLM-5.3
32,768
8
3,270 / 17,954
2,473 / 16,298
99.8%
GLM-5.3-Flash
32,768
11
2,500 / 11,328
1,785 / 10,247
99.9%
Appendix
Table 14: Decoding and token use. Per model over its 2,250 responses at matched settings: the output ceiling, the number of providers that served it, the median and 90th-percentile (p90) completion and reasoning tokens as the provider reports them, and the share of responses with any reasoning tokens; the bottom block gives the three models that return no reasoning tokens at their providers’ defaults on their responses there. ‡ Reasoning at medium effort.
Model
Endpoint
Responses
FAC
No readable answer
Matched difference
Templates matched
DeepSeek V4.1 Flash
CoreWeave
782
0.992
0.1%
0.000
141
Parasail
466
0.992
0.0%
−0.002
138
Morph
456
0.991
0.0%
+0.002
133
DeepInfra
411
0.984
0.5%
0.000
78
Makora
87
1.000
0.0%
+0.003
43
InferenceNet §
30
0.800
16.7%
–
0
Appendix
Table 15: Serving endpoints. The endpoints behind the responses of the eight models that more than one endpoint served: the responses each served, their FAC and their share with no readable answer, and the template-matched difference, the endpoint’s mean minus the model’s mean on the same templates, over the templates both cover. The raw FAC per endpoint is not comparable across endpoints, since OpenRouter assigns instances to endpoints unevenly; § marks the 18 endpoints that served or matched fewer than 20 templates, whose difference rests on too few templates to read. ‡ Responses with reasoning at medium effort; a model that one endpoint served is not listed.
Figure 9: Final Answer Accuracy by engineering branch for four representative models. One stacked bar per model, for DeepSeek V4.1 Flash (first on FAC), Claude Sonnet 5 (first on MC), GPT-5.4 mini (closed, fifth on FAC), and gpt-oss-20b (GPT OSS 20B in the figure). Each of the five branches holds 30 of the 150 templates, so a segment is that branch’s share of the model’s FAC: the number inside it is the branch’s FAC, and its height is a fifth of that. The segments add up to the model’s FAC, printed above the bar; color marks the model and hatching the branch.
Figure 10: Final Answer Accuracy by domain for four representative models. The instance mean over each of the 15 domains, in the order of their branches around the rim, for the same four models; the radial axis starts at 0.4. The domain means carry no interval and no test ( Table 21 ). In the figure, gpt-oss-20b is labeled GPT OSS 20B.
Model
FAC change (95% interval)
90% interval
p (Holm)
MC change
DeepSeek V4.1 Flash
−0.013 ( −0.029 to 0.000)
−0.026 to −0.002
1.000
−0.009
Claude Sonnet 5
−0.011 ( −0.025 to 0.000)
−0.022 to −0.003
1.000
−0.005
Kimi K3
+0.002 ( −0.005 to 0.011)
−0.004 to 0.009
1.000
−0.005
GLM-5.3-Flash
−0.011 ( −0.029 to 0.005)
−0.026 to 0.004
1.000
−0.015
GPT-5.4 mini
+0.005 ( −0.012 to 0.024)
−0.009 to 0.021
1.000
+0.023
Muse Glimmer 30B
+0.025 (0.004–0.051)
0.007–0.046
0.527
−0.007
Appendix
Table 22: Change under paraphrase. Paraphrase minus original on the 275 expert-kept pairs over 114 templates, positive when the paraphrase scores higher: the change in FAC with its 95% interval, its 90% interval read against the ±0.05 margin, the Holm-adjusted p of the sign-flip test over templates, and the change in MC.
Vision-Language Models (VLMs) demonstrate strong performance on general multimodal reasoning benchmarks, yet their ability to perform engineering reasoning remains largely unexplored. Unlike general visual question answering, engineering problem solving requires interpreting technical diagrams, selecting governing physical principles, and maintaining physically consistent multi-step reasoning. These capabilities are increasingly important for AI systems used in engineering education, scientific assistance, and technical decision-making, where reasoning failures may produce physically invalid yet superficially plausible solutions. Existing benchmarks primarily evaluate final answers and provide limited assessment of intermediate reasoning processes. We introduce EngVQA, a multimodal benchmark for evaluating engineering reasoning across 5 engineering subjects containing 696 problems. We introduce an 8-stage automatic evaluation framework for assessing VLM-generated solutions. The framework independently evaluates each stage of the solution, enabling fine-grained analysis of reasoning failures. We benchmark multiple state-of-the-art open and closed source VLMs on our evaluation framework and demonstrate substantial limitations in current engineering reasoning capabilities. Human evaluation shows strong agreement with our automated framework, achieving a Pearson correlation of 0.975 and a mean absolute error of 0.67 on a 10-point grading scale. Our results highlight the importance of process-oriented evaluation for reliable assessment of multimodal engineering reasoning systems.
Large language models (LLMs) are increasingly used as graders, verifiers, and process auditors, but most mathematical evaluations still emphasize final-answer accuracy. This can obscure whether a model can verify a non-canonical but valid solution trace. We introduce a controlled linear-equation benchmark for evaluating LLMs in the evaluator role. Each instance asks the model to judge final-answer correctness, step-level trace correctness, and the first incorrect step. Our evaluation of state-of-the-art open LLMs reveals a significant robustness gap: models that accurately evaluate canonical solutions often fail when presented with perturbed but logically equivalent variants. Across GPT-OSS 20B, Qwen3-14B, and Phi-4-Reasoning, base models perform well on canonical traces but degrade substantially on perturbed traces, especially for error localization. On valid perturbed traces, base-model false-rejection rates reach 75.6-85.3%, showing strong sensitivity to canonical solution form. Supervised fine-tuning, distillation, and test-time compute improve robustness in some settings, but gains are model dependent and can trade off against canonical performance. The results show that reliable process-level verification remains challenging, and evaluator robustness should be measured separately from solver accuracy, even in a simple algebraic domain with exact ground truth.
Fateme Mazdarani, Carlos Toxtli
School of Computing Clemson University Clemson, SC, USA
AI-generated answers in high-stakes domains are often fluent but difficult to verify, especially when they contain multi-step reasoning rather than a single final answer. We propose a reasoning-based, reference-free framework for auditing LLM-generated outputs. The method decomposes a generated reasoning trace into segments, labels local premise-target relations using Natural Language Inference (NLI), and organizes these relations into a hypergraph. A deterministic backward AND-OR search then assigns segment-level audit labels that indicate how each segment is grounded within the generated response. We evaluate the framework in two settings: deductive mathematical reasoning with Hard2Verify, and open-ended medical reasoning with UroReason, a new physician-annotated benchmark of LLM reasoning traces from real clinical cases. Across these settings, our NLI-hypergraph audit provides a more reliable reference-free evaluation signal than direct LLM-as-judge baselines. In the clinical setting, state-of-the-art LLM judges often fail to identify problematic reasoning segments, over-accepting fluent but weakly grounded responses. Our results show that QA evaluation should account for how inferential relations compose across a reasoning trace, rather than relying only on final answers or LLMs as verifiers. UroReason will be made available through an API, and our code will be released as open source.
Guneet Singh Kohli, Yuxiang Zhou, Michael Sejr Schlichtkrull +2
Queen Mary University of London · Temple University