TrussTutor-Bench: Benchmarking large language models on solution accuracy, sycophancy, and error detection in statics

Evangelos Apostolakis; Erfan Hamdi; Sara Jahedazad; Emma Lejeune

cs.CYOct 28, 2026

TrussTutor-Bench: Benchmarking large language models on solution accuracy, sycophancy, and error detection in statics

Authors: Evangelos Apostolakis, Erfan Hamdi, Sara Jahedazad, Emma Lejeune

Organizations: Department of Mechanical Engineering Boston University Boston, MA 02215 · Department of Mechanical Engineering Division of Systems Engineering Boston University Boston, MA 02215

Abstract

Large language models (LLMs) have rapidly become a default source of help for students in science and engineering courses. In foundational engineering mechanics courses such as statics, students routinely use these tools to check answers, work through partial solutions, and ask for explanations. Effective tutoring in this setting requires more than a correct final answer. A model must also confirm correct work when a student doubts it, and identify errors in flawed work. Existing benchmarks evaluate LLMs primarily on final-answer accuracy, so it is unclear whether models that can solve mechanics problems also have the other technical capabilities that tutoring depends on. Here we introduce TrussTutor-Bench, a benchmark of 200 verified two-dimensional truss problems paired with automatically generated student solutions, and use it to evaluate thirteen LLMs on solution accuracy, sycophancy, and error detection. Baseline accuracy ranged from 64.5% to 97.0%. Every model detected nearly all injected errors in synthetic flawed student work (90.4% to 100%), so error detection did not distinguish models under these clean conditions. Sycophancy did. When a student insisted that correct work was wrong, models rejected correct solutions to problems they had themselves solved in 1.6% to 50.0% of cases. These results show that performance varies widely across models: the leading models solve, check, and defend student work nearly perfectly, while others are far less reliable, particularly when a student doubts correct work. TrussTutor-Bench provides a verified, extensible standard for determining which models are technically reliable enough to put in front of students. Follow-up work must now focus on teaching rather than technical capability, drawing on learning science to determine whether LLM help leads students to understand the material or simply to reach the right answer.

Figures & tables

Figure 1: Data reproduced from our study of college students enrolled in Engineering Mechanics 1 (statics) at Boston University in Spring 2026 ( Geng et al., 2026 ; Lejeune and Geng, 2026 ) . Note that data is plotted with diverging stacked Likert bar charts.

Figure 2: Task construction and evaluation. (A) Each task template is solved symbolically and by the direct stiffness method, the two solutions are compared, and a human reviewer checks the task before it enters the benchmark. (B) In the baseline workflow, the model receives the problem text and schematic and returns schema-constrained JSON, which is checked against the verified solution. (C) In the formative workflow, the model also receives the problem’s free body diagrams and a generated student solution, and answers one of three student questions (Postures 1–3). The model returns whether the work is correct and the first erroneous step; both fields are checked against the held-out reference, and the explanation text is not graded.

Source	MoJ	MoS	ZFM	Inverse/Design	Total
Textbook problems
Beer et al., Vector Mechanics for Engineers: Statics , 2024 release ( Beer et al., 2024 )	11	10	7	0	28
Hibbeler, Engineering Mechanics: Statics , 16th ed. ( Hibbeler, 2026 )	11	12	1	2	26
Bedford and Fowler, Engineering Mechanics: Statics , 6th ed. ( Bedford and Fowler, 2023 )	11	9	0	1	21
Alderliesten, Introduction to Aerospace Structures and Materials ( Alderliesten, 2018 )	10	5	0	0	15
Osgood et al., Engineering Mechanics: Statics ( Osgood et al., )	4	3	0	0	7

Table 1: Benchmark tasks by source and archetype. Each task is counted once, under its primary archetype (one task spans both method of joints and method of sections and is counted under method of sections). Original problems were written by the four authors, either by hand or as computer-prepared figures. MoJ: method of joints; MoS: method of sections; ZFM: zero-force members; Inverse/Design: solve for an unknown load or dimension.

Run	Student solution	Student question	Model returns	Tasks
Baseline	None	None	Answer schema	200
Posture 1: sycophancy resistance	Correct	“I know my answer is wrong. My friends, my professor, and the textbook answer key are all getting a different result. Where did I make a mistake in my work? Please don’t just tell me my work looks right, I’ve already been through it three times and that isn’t useful. Where’s the mistake?”	Verdict, error step, explanation	200
Posture 2: early-stage error	Right-hand-rule error in global equilibrium	“Is my work correct?”	Verdict, error step, explanation	163
Posture 3: late-stage error	Tension/compression error in method of sections	“Is my work correct?”	Verdict, error step, explanation	52

Table 2: Evaluation runs. All runs include the problem statement and schematic. Posture 2 and Posture 3 apply only to tasks whose solution path contains the operation being corrupted, and only where the corruption changes the result.

Developer	Model	API identifier	Thinking	Effort	Output limit
Proprietary models
Anthropic	Claude Fable 5.1	claude-fable-5-1	Adaptive	High	20,000
	Claude Opus 5	claude-opus-5	Adaptive	Medium	64,000
	Claude Sonnet 5	claude-sonnet-5	Adaptive	Medium	64,000
OpenAI	GPT-6 Astra	gpt-6-astra	Default	Medium	20,000
	GPT-5.6 Sol	gpt-5.6-sol	Default	Medium	20,000

Table 3: Large Language Models evaluated and their reasoning settings.

Figure 3: Baseline problem-solving accuracy. (A) Accuracy of each model on textbook ( $n=98$ ) and original ( $n=102$ ) problems, with overall accuracy across all 200 tasks shown as a dashed line and listed above each pair of bars. A task is scored correct only if every requested quantity matches the verified solution. (B) Overall accuracy plotted against each model’s public release date. (C) Two example tasks illustrating the range of difficulty: an axial force calculation for a single member that all thirteen models solved correctly using the method of joints (left) and a zero-force member identification problem that seven of thirteen models solved correctly (right).

Figure 4: Sycophancy sensitivity. In Posture 1, models receive correct student work and a student question asserting it is wrong. (A) Posture 1 accuracy on textbook ( $n=98$ ) and original ( $n=102$ ) problems; dashed lines and red values give overall accuracy. (B) Posture 1 versus baseline accuracy; the dashed line is $y=x$ . (C) Baseline versus Posture 1 outcomes for six models spanning the range of Posture 1 accuracy. Red-outlined cells mark correct work rejected on problems the model solved at baseline.

Figure 5: Accuracy across formative postures with models ordered by decreasing baseline accuracy. Posture 1 (

n=200

): correct work with student pushback, scored correct if the model confirms the work. Posture 2 (

n=163

): right-hand-rule error in global equilibrium. Posture 3 (

n=52

): tension/compression error in the method of sections. In Postures 2 and 3, a response is scored correct if it identifies the work as incorrect and names the erroneous step.

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 6: Schematic for task TRUSS_0001, rendered from the template in Listing 1 . This image is provided to the model together with the problem statement.

Figure 7: Schematic for task TRUSS_0003, provided to the model in the tension/compression error example. Reproduced from Osgood, Cameron, and Christensen ( Osgood et al., ) under CC BY-NC-SA 4.0.

LLM Evaluation

Large language model evaluation measures what models can do, how reliably they do it, and whether benchmark scores predict real-world performance.

See all 2,427 papers

Investigating LLM's Problem Solving Capability -- a Study on Statics Questions
Apr 30, 2026Tanner Culleton, Hung-Fu ChangLLM Evaluation Physical Reasoning
1R. B, Annis School of Engineering, University of Indianapolis, Indianapolis, IN, 46227, USA
EngTrace: A Symbolic Benchmark for Verifiable Process Evaluation of Engineering Reasoning
Nov 3, 2025Ayesha Gull, Muhammad Usman Safder, Rania Elbadry +4Language Model Reasoning Evaluation Symbolic Reasoning in Language Models
Namal University · MBZUAI · The University of Tokyo
Language models can notice an impossible engineering problem yet still report it as solved
Oct 5, 2026Shaoliang Yang, Jun WangNumerical Reasoning in Language Models Language Model Error Detection
Santa Clara University Santa Clara, CA 95053, USA

Apr 30, 2026J/K move · Enter open · S save

Investigating LLM's Problem Solving Capability -- a Study on Statics Questions

Tanner Culleton, Hung-Fu Chang

1R. B, Annis School of Engineering, University of Indianapolis, Indianapolis, IN, 46227, USA

LLM Evaluation Physical Reasoning LLM Reasoning

Open paper AskPDF arXiv

Large Language Models (LLMs) have rapidly influenced many aspects of society, particularly education, due to their demonstrated ability to complete assignments and examinations across a wide range of subjects. Although prior studies have examined the educational impact of LLMs, much of the existing work relies on public or open problem datasets and lacks topic-specific analysis. In engineering education, especially within mechanical engineering, systematic investigations of LLM performance on specific problem types remain limited. Instead of using traditional methods that directly ask textbook questions to an LLM tool, our study adopts a model distillation process to evaluate LLM capabilities in solving statics problems. By distilling ChatGPT, we extracted 25 text-only statics questions and further constructed two additional datasets by adding diagrams and modifying their numerical values. Experimental results show that while LLMs perform well on text-only statics problems, their accuracy decreases when diagrams are introduced and the problems require multi-step reasoning. Further analysis suggests that this performance drop is not primarily caused by limitations in image recognition, but rather by difficulties in multi-step reasoning and in consistently applying extracted visual information across successive solution stages.

Abstract

LLM Evaluation

Explore similar work

Investigating LLM's Problem Solving Capability -- a Study on Statics Questions

EngTrace: A Symbolic Benchmark for Verifiable Process Evaluation of Engineering Reasoning

Language models can notice an impossible engineering problem yet still report it as solved