cs.CYOct 28, 2026

TrussTutor-Bench: Benchmarking large language models on solution accuracy, sycophancy, and error detection in statics

Authors: Evangelos Apostolakis, Erfan Hamdi, Sara Jahedazad, Emma Lejeune

Organizations: Department of Mechanical Engineering Boston University Boston, MA 02215 · Department of Mechanical Engineering Division of Systems Engineering Boston University Boston, MA 02215

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Large language models (LLMs) have rapidly become a default source of help for students in science and engineering courses. In foundational engineering mechanics courses such as statics, students routinely use these tools to check answers, work through partial solutions, and ask for explanations. Effective tutoring in this setting requires more than a correct final answer. A model must also confirm correct work when a student doubts it, and identify errors in flawed work. Existing benchmarks evaluate LLMs primarily on final-answer accuracy, so it is unclear whether models that can solve mechanics problems also have the other technical capabilities that tutoring depends on. Here we introduce TrussTutor-Bench, a benchmark of 200 verified two-dimensional truss problems paired with automatically generated student solutions, and use it to evaluate thirteen LLMs on solution accuracy, sycophancy, and error detection. Baseline accuracy ranged from 64.5% to 97.0%. Every model detected nearly all injected errors in synthetic flawed student work (90.4% to 100%), so error detection did not distinguish models under these clean conditions. Sycophancy did. When a student insisted that correct work was wrong, models rejected correct solutions to problems they had themselves solved in 1.6% to 50.0% of cases. These results show that performance varies widely across models: the leading models solve, check, and defend student work nearly perfectly, while others are far less reliable, particularly when a student doubts correct work. TrussTutor-Bench provides a verified, extensible standard for determining which models are technically reliable enough to put in front of students. Follow-up work must now focus on teaching rather than technical capability, drawing on learning science to determine whether LLM help leads students to understand the material or simply to reach the right answer.

Figures & tables

Figure 1: Data reproduced from our study of college students enrolled in Engineering Mechanics 1 (statics) at Boston University in Spring 2026 ( Geng et al., 2026 ; Lejeune and Geng, 2026 ) . Note that data is plotted with diverging stacked Likert bar charts.
Figure 2: Task construction and evaluation. (A) Each task template is solved symbolically and by the direct stiffness method, the two solutions are compared, and a human reviewer checks the task before it enters the benchmark. (B) In the baseline workflow, the model receives the problem text and schematic and returns schema-constrained JSON, which is checked against the verified solution. (C) In the formative workflow, the model also receives the problem’s free body diagrams and a generated student solution, and answers one of three student questions (Postures 1–3). The model returns whether the work is correct and the first erroneous step; both fields are checked against the held-out reference, and the explanation text is not graded.
SourceMoJMoSZFMInverse/DesignTotal
Textbook problems
Beer et al., Vector Mechanics for Engineers: Statics , 2024 release ( Beer et al., 2024 )11107028
Hibbeler, Engineering Mechanics: Statics , 16th ed. ( Hibbeler, 2026 )11121226
Bedford and Fowler, Engineering Mechanics: Statics , 6th ed. ( Bedford and Fowler, 2023 )1190121
Alderliesten, Introduction to Aerospace Structures and Materials ( Alderliesten, 2018 )1050015
Osgood et al., Engineering Mechanics: Statics ( Osgood et al., )43007
Table 1: Benchmark tasks by source and archetype. Each task is counted once, under its primary archetype (one task spans both method of joints and method of sections and is counted under method of sections). Original problems were written by the four authors, either by hand or as computer-prepared figures. MoJ: method of joints; MoS: method of sections; ZFM: zero-force members; Inverse/Design: solve for an unknown load or dimension.
RunStudent solutionStudent questionModel returnsTasks
BaselineNoneNoneAnswer schema200
Posture 1: sycophancy resistanceCorrect“I know my answer is wrong. My friends, my professor, and the textbook answer key are all getting a different result. Where did I make a mistake in my work? Please don’t just tell me my work looks right, I’ve already been through it three times and that isn’t useful. Where’s the mistake?”Verdict, error step, explanation200
Posture 2: early-stage errorRight-hand-rule error in global equilibrium“Is my work correct?”Verdict, error step, explanation163
Posture 3: late-stage errorTension/compression error in method of sections“Is my work correct?”Verdict, error step, explanation52
Table 2: Evaluation runs. All runs include the problem statement and schematic. Posture 2 and Posture 3 apply only to tasks whose solution path contains the operation being corrupted, and only where the corruption changes the result.
DeveloperModelAPI identifierThinkingEffortOutput limit
Proprietary models
AnthropicClaude Fable 5.1claude-fable-5-1AdaptiveHigh20,000
 Claude Opus 5claude-opus-5AdaptiveMedium64,000
 Claude Sonnet 5claude-sonnet-5AdaptiveMedium64,000
OpenAIGPT-6 Astragpt-6-astraDefaultMedium20,000
 GPT-5.6 Solgpt-5.6-solDefaultMedium20,000
Table 3: Large Language Models evaluated and their reasoning settings.
Figure 3: Baseline problem-solving accuracy. (A) Accuracy of each model on textbook ( n=98n=98 ) and original ( n=102n=102 ) problems, with overall accuracy across all 200 tasks shown as a dashed line and listed above each pair of bars. A task is scored correct only if every requested quantity matches the verified solution. (B) Overall accuracy plotted against each model’s public release date. (C) Two example tasks illustrating the range of difficulty: an axial force calculation for a single member that all thirteen models solved correctly using the method of joints (left) and a zero-force member identification problem that seven of thirteen models solved correctly (right).
Figure 4: Sycophancy sensitivity. In Posture 1, models receive correct student work and a student question asserting it is wrong. (A) Posture 1 accuracy on textbook ( n=98n=98 ) and original ( n=102n=102 ) problems; dashed lines and red values give overall accuracy. (B) Posture 1 versus baseline accuracy; the dashed line is y=xy=x . (C) Baseline versus Posture 1 outcomes for six models spanning the range of Posture 1 accuracy. Red-outlined cells mark correct work rejected on problems the model solved at baseline.
Figure 5: Accuracy across formative postures with models ordered by decreasing baseline accuracy. Posture 1 ( n=200n=200 ): correct work with student pushback, scored correct if the model confirms the work. Posture 2 ( n=163n=163 ): right-hand-rule error in global equilibrium. Posture 3 ( n=52n=52 ): tension/compression error in the method of sections. In Postures 2 and 3, a response is scored correct if it identifies the work as incorrect and names the erroneous step.
Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 6: Schematic for task TRUSS_0001, rendered from the template in Listing 1 . This image is provided to the model together with the problem statement.
Figure 7: Schematic for task TRUSS_0003, provided to the model in the tension/compression error example. Reproduced from Osgood, Cameron, and Christensen ( Osgood et al., ) under CC BY-NC-SA 4.0.
More in this topic

LLM Evaluation

Large language model evaluation measures what models can do, how reliably they do it, and whether benchmark scores predict real-world performance.

See all 2,427 papers

Explore similar work

CardsList
Apr 30, 2026

Investigating LLM's Problem Solving Capability -- a Study on Statics Questions

Large Language Models (LLMs) have rapidly influenced many aspects of society, particularly education, due to their demonstrated ability to complete assignments and examinations across a wide range of subjects. Although prior studies have examined the educational impact of LLMs, much of the existing work relies on public or open problem datasets and lacks topic-specific analysis. In engineering education, especially within mechanical engineering, systematic investigations of LLM performance on specific problem types remain limited. Instead of using traditional methods that directly ask textbook questions to an LLM tool, our study adopts a model distillation process to evaluate LLM capabilities in solving statics problems. By distilling ChatGPT, we extracted 25 text-only statics questions and further constructed two additional datasets by adding diagrams and modifying their numerical values. Experimental results show that while LLMs perform well on text-only statics problems, their accuracy decreases when diagrams are introduced and the problems require multi-step reasoning. Further analysis suggests that this performance drop is not primarily caused by limitations in image recognition, but rather by difficulties in multi-step reasoning and in consistently applying extracted visual information across successive solution stages.

Tanner Culleton, Hung-Fu Chang

1R. B, Annis School of Engineering, University of Indianapolis, Indianapolis, IN, 46227, USA

LLM EvaluationPhysical ReasoningLLM Reasoning
AskPDF
Nov 3, 2025

EngTrace: A Symbolic Benchmark for Verifiable Process Evaluation of Engineering Reasoning

Most engineering problems involve complex multi-step derivations through checkable intermediate quantities. Grading only the final answer misjudges LLMs, crediting correct answers reached through flawed steps and penalizing wrong answers built on mostly sound ones. Yet, existing benchmarks such as SuperGPQA and EEE-Bench score only final answers, and step-level ones such as TPS-CalcBench rely on LLM judges or human graders, or cover a single domain. We therefore introduce EngTrace, a symbolic benchmark for verifiable process evaluation of engineering reasoning. Its 150 expert-certified templates, spanning five branches, 15 domains, and 42 areas, generate each problem with a gold reasoning trace, so that every intermediate quantity is known exactly. We design and validate an evaluator that checks final answers, coverage of the gold trace's intermediate quantities (milestones), and displayed arithmetic, mostly without an LLM judge. Across 11 LLMs, final answers barely separate the strongest models, while their derivations differ. Wrong answers still state about half of the gold milestones, and domain experts trace most wrong answers to calculation slips, with conceptual errors rarer on easy problems than on harder ones. Overall, as LLMs near the final-answer ceiling, checking derivations, as EngTrace does, reveals differences that final answers miss.

Ayesha Gull, Muhammad Usman Safder, Rania Elbadry +4

Namal University · MBZUAI · The University of Tokyo

Language Model Reasoning EvaluationSymbolic Reasoning in Language Models
AskPDF
Oct 5, 2026

Language models can notice an impossible engineering problem yet still report it as solved

Language models draft engineering calculations, but answer accuracy does not show whether they reject an impossible problem. We tested 14 models on 30 pairs of mechanics problems, each with a valid version and one made impossible by changing a given value or assumption. Two independent solvers verified every answer key and showed that each flawed problem was physically impossible. We scored solving of valid problems separately from rejection of their flawed counterparts. Each reply required a "solved" or "cannot solve" status; rejection meant "cannot solve" or withholding an answer. The initial prompts did not warn that problems could be flawed. Across three recent models, 12 of 90 replies failed to reject a flawed problem. In 11 of these replies, the model stated the flaw, answered a corrected problem and still reported the original as "solved", according to artificial intelligence raters and numerical checks. We later retested four models from one provider, offering "flawed" instead of "cannot solve" and asking them to name and explain the defect. Three models showed statistically significant increases in rejection, but valid-problem solving fell in three. Evaluations therefore need to score both versions and distinguish flaw recognition from the reported status.

Shaoliang Yang, Jun Wang

Santa Clara University Santa Clara, CA 95053, USA

Numerical Reasoning in Language ModelsLanguage Model Error DetectionLLM Reliability
AskPDF