AeroTherm-GPT: A Verification-Centered LLM Framework for Thermal Protection System Engineering Workflows

Chuhan Qiao; Jinglai Zheng; Jie Huang; Buyue Zhao; Fan Li; Haiming Huang

cs.AIApr 2, 2026

AeroTherm-GPT: A Verification-Centered LLM Framework for Thermal Protection System Engineering Workflows

Authors: Chuhan Qiao, Jinglai Zheng, Jie Huang, Buyue Zhao, Fan Li, Haiming Huang

Organizations: School of Civil Engineering, Beijing Jiaotong University, Beijing, 100044, China · Beijing Research Institute of Telemetry, Beijing, 100076, China

Abstract

Integrating Large Language Models (LLMs) into hypersonic thermal protection system (TPS) design is bottlenecked by cascading constraint violations when generating executable simulation artifacts. General-purpose LLMs, treating generation as single-pass text completion, fail to satisfy the sequential, multi-gate constraints inherent in safety-critical engineering workflows. To address this, we propose AeroTherm-GPT, the first TPS-specialized LLM Agent, instantiated through a Constraint-Closed-Loop Generation (CCLG) framework. CCLG organizes TPS artifact generation as an iterative workflow comprising generation, validation, CDG-guided repair, execution, and audit. The Constraint Dependency Graph (CDG) encodes empirical co-resolution structure among constraint categories, directing repair toward upstream fault candidates based on lifecycle ordering priors and empirical co-resolution probabilities. This upstream-priority mechanism resolves multiple downstream violations per action, achieving a Root-Cause Fix Efficiency of 4.16 versus 1.76 for flat-checklist repair. Evaluated on HyTPS-Bench and validated against external benchmarks, AeroTherm-GPT achieves 88.7% End-to-End Success Rate (95% CI: 87.5-89.9), a gain of +12.5 pp over the matched non-CDG ablation baseline, without catastrophic forgetting on scientific reasoning and code generation tasks.

Figures & tables

Figure 1: Overview of the proposed approach. An engineering requirement enters the CCLG workflow (generation $\to$ validation $\to$ CDG-guided repair $\to$ execution $\to$ audit), producing an executable and traceable simulation artifact. AeroTherm-GPT is a TPS-oriented instantiation of this framework; HyTPS-Bench provides the workflow-aligned evaluation setting.

Figure 2: End-to-end architecture of AeroTherm-GPT. Solid-line modules (constraint-aware RAG, TPS-specific SFT, CDG-aware VER loop, constraint-aware DPO, reference executor) are fully implemented and evaluated in all experiments. Dashed-line extensions (probabilistic constraint bounds, multi-fidelity CDG, edge-deployed variants) represent directions for future work and are not evaluated here.

Figure 3: Constraint Dependency Graph (CDG) structure and regime-conditioned calibration. Top-left (Zone A): five-tier CDG with 23 nodes organized by lifecycle ordering (Unit $\rightarrow$ Physics $\rightarrow$ Numerical $\rightarrow$ Execution $\rightarrow$ Audit); directed edges represent empirical co-resolution probabilities calibrated from 4,206 repair episodes. Top-right: regime-conditioned adjacency matrix $W(c)$ across four thermal contexts (nominal, moderate, high-flux, extreme), with edge weights stratified by flight regime; mean absolute calibration error $=0.08$ on 20% held-out split ( $n=841$ ). Bottom: per-regime CDG visualizations ( $c_{1}$ – $c_{4}$ ) illustrating how edge weight distributions shift across operating conditions.

Figure 4: Core computational mechanisms of AeroTherm-GPT. Zone B (top): PRM-guided constraint tree search (CDG priority, $K=4$ candidates scored by $V_{\theta}(s)$ ), terminating at $\text{Ready}(s)=1$ or budget exhaustion; right panel compares repair strategy trajectories. Zone C (middle): Transformer backbone with RAG cross-attention, frozen encoder, PRM head ( $V_{\theta}(s)\in[0,1]$ ), and LoRA adaptation ( $r=32,\alpha=64$ ). Zone D (bottom): Three-tier DPO preference construction (binary, self-evolution, severity-graded) with implicit reward distribution.

Repair Strategy	Root-Cause Fix Efficiency	Avg Iter.	Rework Red.
Random	1.13	3.17	0.0%
Flat Checklist	1.76	2.43	23.3%
CDG Topological Order	3.67	1.58	50.2%
CDG + Severity Weighting	4.16	1.38	56.5%

Table 1: Upstream-Priority Fix Efficiency (RCFE, measured via HyTPS-Bench). CDG-aware repair targets upstream violations first, reducing iterations and downstream rework.

Component	Training	Evaluation	Shared	Leakage control
Constraint assets (schema)	Yes	Yes	Yes (API)	Shared schema/API only; no privileged repair traces or oracle outputs
Reference Executor	No	Yes	Yes	External scoring implementation, identical for all systems
Repair logs (CDG training)	Yes	No	No	Separate calibration pool; excluded from benchmark test construction
HyTPS-Bench test set	No	Yes	Yes	Held out at benchmark-instance and template-family level
Validator rules	No	Yes (post)	Yes	Applied uniformly post-generation to every system
Synthetic instance parameters	Subset	Remaining	N/A	Held-out parameter ranges with material-class-disjoint split

Table 2: Data leakage and evaluation integrity control. Each component’s role in training vs. evaluation is explicitly documented.

Item	HyTPS-Bench Value
Size	$\sim$ 2,500 JSON instances
Task mixture (T1–T4)	25%/25%/30%/20%
Sources	Public lit. (NASA NTRS, AIAA); de-identified snippets; synthetic
Multilingual	EN/CN, $q\approx 0.2$
Input length	0.5–2 pages (500–1500 tokens)
Token stats	$\mu_{\text{tok}}\approx 900$ , $p95_{\text{tok}}\approx 1500$

Table 3: Dataset composition and baseline difficulty facts for HyTPS-Bench. The zero-shot SI-G results confirm the benchmark is not saturated by existing models.

Method	Retrieval	Validator feedback	Exec. backend	Iterative repair	CDG ordering	Domain SFT+DPO
Zero-Shot LLMs	$\times$	$\times$	$\times$	$\times$	$\times$	$\times$
Reasoning (w/ think)	$\times$	$\times$	$\times$	$\times$	$\times$	$\times$
RAG + Few-Shot	✓	$\times$	$\times$	$\times$	$\times$	$\times$
GPT-4o+Refl.	$\times$	✓	✓	✓	$\times$	$\times$
Tool-Agent	✓	✓	✓	✓	$\times$	$\times$
Flat-VER	✓	✓	✓	✓	$\times$	$\times$

Table 4: Baseline capability alignment. “Flat-VER + SFT + DPO” and “CDG-SFT Agent” are symmetric ablation baselines: the former adds SFT+DPO to flat repair (isolating CDG’s contribution); the latter adds CDG ordering to SFT-only training (isolating DPO’s contribution). AeroTherm-GPT combines both.

Model	Core		Task Perf.			Over.	Cons. (%)
	EESR	SI-G	MP	TD	PP-A		U/P	N
1. Baseline LLMs (w/o thinking)
GPT-5 (min)	0.350	0.290	0.641	0.573	0.521	0.475	62.3	28.5
Claude-4-Sonnet (w/o)	0.420	0.480	0.493	0.712	0.643	0.550	68.2	32.1
Claude-4.1-Opus (w/o)	0.450	0.550	0.551	0.741	0.691	0.597	70.5	33.7
Qwen3-32B (w/o)	0.250	0.230	0.387	0.331	0.293	0.298	51.7	22.4

Table 5: Main results on HyTPS-Bench. Core: EESR = End-to-End Success Rate, SI-G(T3) = Simulation Input Generation. Task Perf.: MP = Material Property QA (T1), TD = Technical Document Analysis (T2), PP-A = Post-Processing + Lifecycle Audit (T4). Over. = overall; U/P = unit/physics compliance; N = numerical compliance (%).

Figure 5: Ablation Study — Incremental EESR Contribution per Module with Violation Rate Trajectories. The waterfall chart decomposes the EESR gain from each framework component, while the overlaid lines track the corresponding reduction in unit, physics, and numerical violation rates.

Figure 6: Per-Gate Pass Rates — Five Methods $\times$ Five Verification Gates. The radar chart visualizes the compliance rate across unit, physics, numerical, execution, and audit gates, highlighting AeroTherm-GPT’s near-ceiling performance across all categories.

Figure 7: Stratified EESR by Task Complexity. Performance degrades gracefully with task complexity, from single-layer nominal tasks to multi-layer extreme-regime tasks and non-standard materials. Error bars denote 95% confidence intervals.

Model	Fund. Knowledge (Q1–Q30, %)	Formula Inv. (F1–F6, %)	Auto. Prog. (P1–P4, %)	TCR (%)
DeepSeek-R1 [ 54 ]	100.0	83.3	100.0	97.5
DeepSeek-V3 [ 54 ]	93.3	66.7	25.0	82.5
GPT-4o [ 54 ]	76.7	66.7	25.0	70.0
Qwen2.5 [ 54 ]	76.7	50.0	25.0	67.5
AeroTherm-GPT (ours)	96.7	100.0	100.0	97.5

Table 6: Results on HyperKFA-Bench [ 54 ] (hypersonic benchmark). Fund. Knowledge: accuracy on 30 hypersonic QA questions; Formula Inv.: correctness rate on 6 quantitative formula tasks (relative error

<2\%

); Auto. Prog.: code execution success rate on 4 programming tasks; TCR: overall Task Completion Rate. Baseline results are taken directly from the original publication; AeroTherm-GPT is evaluated zero-shot without TPS-specific scaffolding. AeroTherm-GPT achieves high TCR (97.5%), matching DeepSeek-R1’s 97.5% while substantially outperforming GPT-4o (70.0%) and Qwen2.5 (67.5%). The formula invocation advantage (+33.3 pp over DeepSeek-R1) reflects domain SFT’s internalization of hypersonic governing equations and unit-consistent numerical procedures.

Figure 8: Backbone Transfer — Zero-Shot vs Full Pipeline EESR. Marker size indicates active parameter count. The linear fit ( $r=0.96$ ) demonstrates that the CCLG pipeline’s performance gains are robust across diverse architectures (dense vs. MoE) and parameter scales.

Benchmark	$n$	Metric	Llama-3 70B	Aero- Therm	$\Delta$	GPT-4o
SciBench	695	Acc	35.8	42.6	+6.8	44.3
GPQA-Diamond	198	Acc	33.1	38.9	+5.8	49.4
MMLU-STEM	3,014	5s-Acc	79.2	80.7	+1.5	87.4

Table 7: Cross-benchmark validation results (Track B). Panel (a): scientific reasoning; AeroTherm-GPT’s SFT provides meaningful gains on SciBench (+6.8) and GPQA (+5.8) with minimal MMLU-STEM forgetting (+1.5). Panel (b): code generation; the VER iterative repair mechanism improves pass@1 across all base models. AeroTherm-GPT exhibits the largest VER repair gain (+6.8 pp avg), suggesting that DPO alignment improves responsiveness to structured error feedback even outside the TPS domain. GPT-4o retains absolute leadership on general tasks.

Failure Pattern	Share of all tasks	Share within failed tasks	Impact	Mitigation
CDG weight miscalibration	5.2%	46.0%	Medium	Recalibrate on diverse cases
Physics bounds too restrictive	3.1%	27.4%	High	Extend bounds
Missing constraint assets	2.4%	21.2%	Medium	Expand asset library
VER convergence exceeds budget	0.6%	5.3%	Low	Optimize repair ordering
Total	11.3%	100% ∗

Table 8: Failure pattern frequency and mitigation strategies identified during HyTPS-Bench evaluation. “Share of all tasks” corresponds to the 11.3% overall failure rate; “Share within failed tasks” normalizes by the number of failed tasks only. The four patterns account for essentially all observed failures. ∗ Percentages are rounded to one decimal place; the sum is 99.9% due to rounding.

Gate	First Pass	Flat Repair (iter. 8)	CDG Repair (iter. 1)
Unit	$\times$	$\times$	✓
Physics	$\times$	$\times$	✓
Numerical	$\times$	$\times$	✓
Execution	$\times$	$\times$	✓
Audit	✓	✓	✓
EESR	$\times$	$\times$	✓

Table 9: Gate-level verification outcomes for the end-to-end case study. First pass: four gates fail due to unit transcription error. Flat repair: eight iterations of symptom-patching fail to resolve the upstream unit fault; all four downstream gates remain open. CDG-guided repair: upstream unit fix in iteration 1 resolves all dependent gates simultaneously (RCFE

\,{=}\,3

Iter.	Strategy	Action taken	Gates resolved
1	Flat	Reduce $\Delta t$ by $10\times$ (Fo violation $\to$ tighten time step)	None
2	Flat	Reduce $\Delta t$ further $5\times$ ; halve $\Delta x$ (instability persists)	None
3	Flat	Clamp $k_{1}$ heuristically to plausible range (physics gate open)	None
4	Flat	Relax physics bounds tolerance (bounds still violated)	None
5	Flat	Reduce heat flux by 20% (reduce thermal load to ease convergence)	None
6	Flat	Switch time-integration to implicit Euler (execution gate still open)	None

Table 10: Repair action trace for the end-to-end case study. Both strategies have identical tool access (same VER loop, RAG, and execution backend); the only difference is repair ordering. Flat-checklist repair exhausts the 8-iteration budget with symptom-directed but execution-focused actions, never reaching the upstream unit fault. CDG-guided repair identifies the unit constraint as the highest-priority root-cause node and resolves all four failing gates in a single action.(U, P, N, E = Unit, Physics, Numerical, Execution)

Figure 9: End-to-end CDG verification on the 1-D composite wall benchmark. (a) The pipeline extracts 14 constraints (dimensional, conservation, boundary, interface) and an 83-LOC Python solver from the NL requirement. Pass 1 triggers a cascade failure at the Unit gate ( $k_{1}$ specified in W/cm K instead of W/m K); the fault propagates through Physics, Numerical, and Execution gates (purple arcs). The CDG repair agent traces the root cause via the constraint dependency graph, applies a single unit-scale correction, and Pass 2 clears all five gates (EESR = 5/5). Inset: fault DAG and per-gate confidence lift from Pass 1 to Pass 2. (b) Repair trajectory comparison. Flat repair (left) exhausts 8 iterations with symptom-directed actions targeting the Execution and Numerical gates—each locally rational given the observed divergence—without ever identifying the upstream Unit-gate fault as the root cause, terminating at budget exhaustion (EESR $\times$ ). CDG repair (right) resolves the entire cascade in a single iteration by targeting the root gate directly (EESR ✓). Bottom strip: aggregate efficiency metrics.

Figure 10: End-to-end TPS thermal analysis case study. (a) Two-layer stack geometry. (b) CDG constraint subgraph for this instance (RCFE = 3) with comparison against system-level benchmark statistics.

Figure 11: Spatial temperature distribution in the TPS. (a) Temperature field over 0–35 mm depth and 30 s duration; significant heating is confined to the outermost 10 mm. (b) Depth profiles at $t=10$ , 20 and 30 s; FD results match the semi-infinite analytical reference to within 0.02 % at the surface. (c) Pointwise relative error for $x\leq 10$ mm: maximum 0.1 % ( $t=10$ s) and 0.5 % ( $t=20$ s), well below the 2 % threshold.

Figure 12: Transient thermal response of a two-layer TPS to a triangular heat pulse (peak 800 kW/m² at $t=10$ s). (a) Temperature histories at the ceramic–substructure interface ( $x=15$ mm) and back wall ( $x=35$ mm); the 523 K structural limit is not exceeded. (b) Validation of the finite-difference solver against the semi-infinite analytical solution at $x=0$ ; relative error stays below 3 %, with late-time growth attributed to finite-thickness effects.

Method	EESR (5/5)	EESR-4 (4/5)	Macro-Gate	Best-Gate
GPT-5 (high)	0.480	0.612	0.646	0.782
Flat-VER	0.468	0.601	0.729	0.865
CDG-SFT Agent	0.784	0.851	0.871	0.941
AeroTherm-GPT	0.887	0.941	0.965	0.991

Table 11: Sensitivity of method rankings to EESR strictness. EESR-4: any 4 of 5 gates pass; Macro-Gate: macro-average of five gate pass rates; Best-Gate: highest individual gate (ceiling reference). Rankings are stable across all relaxations, confirming the advantage is not an artifact of the strict 5-gate product.

Method	EESR lang (pass@1)	EESR scaffold (gain)	EESR (full)
Flat-VER	0.219	0.249	0.468
Flat-VER + SFT + DPO	0.521	0.241	0.762
CDG-SFT Agent	0.312	0.472	0.784
AeroTherm-GPT	0.521	0.366	0.887

Table 12: EESR decomposition separating intrinsic language capability (EESR lang : pass@1 without VER) from scaffold contribution (EESR scaffold : incremental gain from VER). AeroTherm-GPT’s advantage is present in both components: its EESR lang (

+30.2

pp over Flat-VER) reflects genuine first-pass improvement from domain SFT+DPO alignment, while EESR scaffold captures the structural CDG repair benefit. The “Flat-VER + SFT + DPO” row shows that without CDG, domain training raises EESR lang substantially (+30.2 pp) but scaffold gain is modest; CDG-SFT Agent shows the inverse pattern (higher scaffold gain, lower first-pass). AeroTherm-GPT achieves the best balance across both dimensions.

LLM Agent Workflow Optimization

LLM agent workflow optimization studies how to design and run multi-step agents for better task success, reliability, latency, and cost.

See all 256 papers

TPS-CalcBench: A Benchmark and Diagnostic Evaluation Framework for LLM Analytical Calculation Competence in Hypersonic Thermal Protection System Engineering
Apr 20, 2026Jinglai Zheng, Chuhan Qiao, Haiming HuangSynthetic Benchmark Generation LLM Safety Evaluation
School of Civil Engineering, Beijing Jiaotong University, Beijing 100044, China
Mnemosyne: Agentic Transaction Processing for Validating and Repairing AI-generated Workflows
Jun 30, 2026Edward Y. Chang, Longling GengAgentic Workflows AI Agent Safety
Stanford University
SafePilot: A Framework for Assuring LLM-enabled Cyber-Physical Systems
Mar 23, 2026Weizhe Xu, Mengyu Liu, Fanxin KongLLM Safety Cyber-Physical Systems
University of Notre Dame, Notre Dame, IN, USA · Washington State University, Richland, WA, USA

Apr 20, 2026J/K move · Enter open · S save

TPS-CalcBench: A Benchmark and Diagnostic Evaluation Framework for LLM Analytical Calculation Competence in Hypersonic Thermal Protection System Engineering

Jinglai Zheng, Chuhan Qiao, Haiming Huang

School of Civil Engineering, Beijing Jiaotong University, Beijing 100044, China

Synthetic Benchmark Generation LLM Safety Evaluation LLM Mathematical Reasoning Computational Fluid Dynamics

Open paper AskPDF arXiv

Deploying LLMs as reasoning assistants in safety-critical aerospace engineering requires stricter evaluation criteria than general scientific benchmarks. In hypersonic thermal protection system (TPS) design, inaccurate stagnation-point heat flux or boundary-layer calculations may cause catastrophic design margin violations. Models with numerically reasonable but physically invalid answers are more dangerous than those declining to respond. Current scientific benchmarks only test abstract math and basic physics, evaluate final answers solely, ignore engineering reasoning processes, and cannot detect such critical failures. We propose TPS-CalcBench, the first diagnostic benchmark for closed-form analytical calculations in hypersonic aerodynamics and high-temperature gas dynamics that experienced TPS engineers conduct without simulations. Our contributions include domain-oriented task taxonomy with 4 difficulty levels and 8 categories from Anderson's textbook, dual-track evaluation measuring result accuracy and reasoning quality via an 8-dimension rubric and calibrated judge with human audit to identify right answer wrong reasoning issues, human-AI data pipeline producing 420 high-confidence core items and 810 noise-controlled pre-gating items from 4560 raw data, noise-sensitivity analysis measuring data quality impacts on model ranking, and three diagnostic intervention methods: DFA-TPS fine-tuning, RAG-EQ retrieval grounding and PA-CoT process-aware prompting. Tests on 13 models from 7 groups show wide performance differences (KPI 12.6-87.9), hidden formula selection defects, data-driven rank changes and effective intervention improvements, establishing a complete diagnose-evaluate-intervene framework for safety-critical engineering LLM deployment assessment.

Abstract

LLM Agent Workflow Optimization

Explore similar work

TPS-CalcBench: A Benchmark and Diagnostic Evaluation Framework for LLM Analytical Calculation Competence in Hypersonic Thermal Protection System Engineering

Mnemosyne: Agentic Transaction Processing for Validating and Repairing AI-generated Workflows

SafePilot: A Framework for Assuring LLM-enabled Cyber-Physical Systems