ThermoQA: A Three-Tier Benchmark for Evaluating Thermodynamic Reasoning in Large Language Models

Kemal Düzkar

cs.AIMar 25, 2026

ThermoQA: A Three-Tier Benchmark for Evaluating Thermodynamic Reasoning in Large Language Models

Authors: Kemal Düzkar

Organizations: Olivenet Kyrenia, Cyprus

Abstract

We present ThermoQA, a benchmark of 293 open-ended engineering thermodynamics problems in three tiers: property lookups (110 Q), component analysis (101 Q), and full cycle analysis (82 Q). Ground truth is computed programmatically from CoolProp 7.2.0, covering water, R-134a, and variable-cp air. Six frontier LLMs are evaluated across three independent runs each. The composite leaderboard is led by Claude Opus 4.6 (94.1%), GPT-5.4 (93.1%), and Gemini 3.1 Pro (92.5%). Cross-tier degradation ranges from 2.8 pp (Opus) to 32.5 pp (MiniMax), confirming that property memorization does not imply thermodynamic reasoning. Supercritical water, R-134a refrigerant, and combined-cycle gas turbine analysis serve as natural discriminators with 40-60 pp performance spreads. Multi-run sigma ranges from +/-0.1% to +/-2.5%, quantifying reasoning consistency as a distinct evaluation axis. Dataset and code are open-source at https://huggingface.co/datasets/olivenet/thermoqa

Figures & tables

Benchmark	Size	Format	Scoring	Real Fluids	Cycles ‡	Exergy	Multi-run
GPQA ( 2023 )	448 †	MCQ	Exact match	✗	✗	✗	✗
SciBench ( 2023 )	789 †	Open	Final answer	✗	✗	✗	✗
UTQA ( 2025 )	50	MCQ	Exact match	✗	✗	✗	✗
Loubet et al. ( 2025a )	22	Open calc.	Expert grading	✗	✓ ∗	✗	✓
EngiBench ( 2025 )	1293 †	Mixed	Binary/rubric	✗	✗	✗	✗
ThermoQA (ours)	293	Open calc.	Step-weighted	✓	✓	✓	✓

Table 1: Comparison with existing thermodynamics-related LLM benchmarks. ThermoQA is the first to combine open-ended numerical calculation with real-fluid coverage, full cycle analysis, and exergy.

Model	Provider	API Model String	Reasoning
Claude Opus 4.6	Anthropic	claude-opus-4-6	Adaptive
GPT-5.4	OpenAI	gpt-5.4	reasoning_effort=high
Gemini 3.1 Pro	Google	gemini-3.1-pro-preview	thinking_level=HIGH
Grok 4	xAI	grok-4.20-beta-0309	Native
DeepSeek-R1	DeepSeek	deepseek-reasoner	Native
MiniMax M2.5	MiniMax	MiniMax-M2.5	Inline

Table 2

Rank	Model	Tier 1	Tier 2	Tier 3	T1 $\to$ T3 Drop	Composite
1	Claude Opus 4.6	96.4 $\pm$ 0.9	92.1 $\pm$ 0.1	93.6 $\pm$ 0.5	$-$ 2.8 pp	94.1%
2	GPT-5.4	97.8 $\pm$ 0.8	90.8 $\pm$ 0.5	89.7 $\pm$ 0.1	$-$ 8.1 pp	93.1%
3	Gemini 3.1 Pro	97.9 $\pm$ 0.5	90.8 $\pm$ 1.2	87.5 $\pm$ 1.5	$-$ 10.4 pp	92.5%
4	DeepSeek-R1	90.5 $\pm$ 0.2	89.2 $\pm$ 2.5	81.0 $\pm$ 2.2	$-$ 9.5 pp	87.4%
5	Grok 4	91.8 $\pm$ 1.2	87.9 $\pm$ 0.7	80.4 $\pm$ 0.8	$-$ 11.4 pp	87.3%
6	MiniMax M2.5	85.2 $\pm$ 0.6	76.2 $\pm$ 1.1	52.7 $\pm$ 1.5	$-$ 32.5 pp	73.0%

Table 2: Final leaderboard (3-run mean

\pm

std). Composite = question-weighted average.

Figure 1: Per-tier performance for all six models (3-run mean with

\pm\sigma

error bars). Tier difficulty increases left to right. Note the ranking reshuffle: Gemini leads Tier 1 but Opus leads Tiers 2–3.

Figure 2: Cross-tier degradation: performance drop from Tier 1 (property lookups) to Tier 3 (cycle analysis). Smaller is better.

Figure 3: Natural discriminators. (a) Supercritical water properties in Tier 1: 45–90% spread. (b) R-134a refrigerant in Tier 2: 44–63%. Faded bars show the gap to overall tier score.

Figure 4: Tier 2 component heatmap (3-run mean scores). The compressor column is uniformly the hardest; pump is near-trivial for all models.

Figure 5: Tier 3 scores by cycle type. Ideal Brayton variants (BRY-I, BRY-A, BRY-RG) are solved near-perfectly by frontier models. Variable-

c_{p}

variants (BRY-AV, BRY-RV), VCR, and CCGT provide meaningful discrimination.

Figure 6: Performance by analysis depth. (a) Tier 2: counter-intuitive improvement at Depth C for frontier models. (b) Tier 3: peak at Depth B, drop at Depth C.

Figure 7: Standard deviation across 3 independent runs. DeepSeek-R1 shows the highest variance on Tiers 2–3; GPT-5.4 is the most consistent on Tier 3.

Figure 8: Token usage and efficiency across all three tiers. (a) Mean output tokens per question (including thinking tokens where exposed by the API). Opus invests up to 53K tokens on Tier 3; Gemini and Grok 4 stay below 3K. (b) Tokens per percentage point on Tier 3 (lower = more efficient), showing a 20

\times

gap between Opus and Gemini.

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Step	Description	Weight	Category
$h_{1}$ , $h_{2s}$ , $h_{2}$ , $T_{2}$	Gas compressor states	1	State points
$h_{3}$ , $h_{4s}$ , $h_{4}$ , $T_{4}$	Combustion & turbine states	1	State points
$h_{5}$	Gas turbine exhaust	1	State points
$h_{6}$ , $h_{7s}$ , $h_{7}$	Steam cycle states	1	State points
$h_{8}$ , $h_{9s}$ , $h_{9}$	Condenser/pump states	1	State points
$\dot{m}_{\text{steam}}$	HRSG mass balance	3	Component

Appendix

Table 3: Weight scheme for a CCGT Depth A question (23 steps). The weighted score emphasizes engineering outcomes over intermediate state lookups.

	Opus	GPT-5.4	Gemini	Grok 4	DeepSeek	MiniMax
Opus	—	✓	✓	✓	✓	✓
GPT-5.4		—	$\times$	✓	✓	✓
Gemini			—	✓	✓	✓
Grok 4				—	$\times$	✓
DeepSeek					—	✓

Appendix

Table 13

LLM Evaluation

Large language model evaluation measures what models can do, how reliably they do it, and whether benchmark scores predict real-world performance.

See all 2,408 papers

TPS-CalcBench: A Benchmark and Diagnostic Evaluation Framework for LLM Analytical Calculation Competence in Hypersonic Thermal Protection System Engineering
Apr 20, 2026Jinglai Zheng, Chuhan Qiao, Haiming HuangSynthetic Benchmark Generation LLM Safety Evaluation
School of Civil Engineering, Beijing Jiaotong University, Beijing 100044, China
Stan: An LLM-based thermodynamics course assistant
Mar 4, 2026Eric M. Furst, Vasudevan VenkateshwaranGenerative AI in Education
Department of Chemical and Biomolecular Engineering, Allan P. Colburn Laboratory, University of Delaware, Newark, DE 19716, USA · W. L. Gore & Associates Inc., 100 Airport Road, Elkton, MD 21921, USA
Classroom Final Exam: An Instructor-Tested Reasoning Benchmark
Feb 23, 2026Chongyang Gao, Diji Yang, Shuyan Zhou +4MLLM Reasoning LLM Evaluation
Northwestern University · UC Santa Cruz · Duke University +3

Apr 20, 2026J/K move · Enter open · S save

TPS-CalcBench: A Benchmark and Diagnostic Evaluation Framework for LLM Analytical Calculation Competence in Hypersonic Thermal Protection System Engineering

Jinglai Zheng, Chuhan Qiao, Haiming Huang

School of Civil Engineering, Beijing Jiaotong University, Beijing 100044, China

Synthetic Benchmark Generation LLM Safety Evaluation LLM Mathematical Reasoning Computational Fluid Dynamics

Open paper AskPDF arXiv

Deploying LLMs as reasoning assistants in safety-critical aerospace engineering requires stricter evaluation criteria than general scientific benchmarks. In hypersonic thermal protection system (TPS) design, inaccurate stagnation-point heat flux or boundary-layer calculations may cause catastrophic design margin violations. Models with numerically reasonable but physically invalid answers are more dangerous than those declining to respond. Current scientific benchmarks only test abstract math and basic physics, evaluate final answers solely, ignore engineering reasoning processes, and cannot detect such critical failures. We propose TPS-CalcBench, the first diagnostic benchmark for closed-form analytical calculations in hypersonic aerodynamics and high-temperature gas dynamics that experienced TPS engineers conduct without simulations. Our contributions include domain-oriented task taxonomy with 4 difficulty levels and 8 categories from Anderson's textbook, dual-track evaluation measuring result accuracy and reasoning quality via an 8-dimension rubric and calibrated judge with human audit to identify right answer wrong reasoning issues, human-AI data pipeline producing 420 high-confidence core items and 810 noise-controlled pre-gating items from 4560 raw data, noise-sensitivity analysis measuring data quality impacts on model ranking, and three diagnostic intervention methods: DFA-TPS fine-tuning, RAG-EQ retrieval grounding and PA-CoT process-aware prompting. Tests on 13 models from 7 groups show wide performance differences (KPI 12.6-87.9), hidden formula selection defects, data-driven rank changes and effective intervention improvements, establishing a complete diagnose-evaluate-intervene framework for safety-critical engineering LLM deployment assessment.

Abstract

LLM Evaluation

Explore similar work

TPS-CalcBench: A Benchmark and Diagnostic Evaluation Framework for LLM Analytical Calculation Competence in Hypersonic Thermal Protection System Engineering

Stan: An LLM-based thermodynamics course assistant

Classroom Final Exam: An Instructor-Tested Reasoning Benchmark