Chain of Physics: Supporting Fault Diagnosis with Engineering-Inspired Large Language Model Reasoning
Authors: Wenqi Zhou, Leonardo Baldo, Kai Goebel, Manuel Arias Chao
Organizations: Institute for Data Science, Zurich University of Applied Science, Winterthur, 8401, Switzerland · Fragum Global, Mountain View, CA 94040, USA · Luleå University of Technology, Luleå, 971 87, Sweden · Faculty of Aerospace Engineering, Delft University of Technology, HS 2926 Delft, The Netherlands
Fault diagnosis in complex systems requires linking observed changes in physical quantities to the failure mechanisms that produce them. Experienced engineers perform this task by combining sensor information with knowledge of system behavior and fault propagation. Large Language Models (LLMs) can support this process, but their diagnostic reasoning depends on the availability of asset-specific engineering knowledge and on how this knowledge is represented and used during inference. This work introduces Chain of Physics (CoP), an LLM-based fault-diagnosis framework built around a Constrained Knowledge Graph (CKG) generated automatically from asset-specific reference documentation. CoP addresses failure-sensor association from textual engineering knowledge. The CKG stores diagnostic chains connecting failure modes, changes in physical quantities, and monitoring sensors. During inference, semantically similar graph paths are retrieved for each question and used as both information and in-context examples for the reasoning process. Candidate failure modes are extracted from the question and analysed independently through dedicated LLM calls, which generate plausible failure-to-observation chains. A final decision module compares these chains with the available answer choices. CoP is evaluated on the FailureSensorIQ benchmark across four industrial asset classes and three LLMs. Compared with direct prompting and conventional Retrieval-Augmented Generation (RAG), CoP improves diagnostic accuracy across all evaluated models, with gains ranging from 7.8 to 19.6 percentage points and from 11.7 to 18.2 percentage points, respectively. The results show that structured engineering knowledge can improve LLM-based diagnostic reasoning when graph retrieval is combined with in-context examples of fault propagation.
Figures & tables
Figure 1: Overview of CoP. Once per asset, constrained extraction turns reference documentation into a CKG whose paths are complete failure-to-observation chains. For each diagnostic question, retrieved CKG paths provide both asset-specific knowledge and reasoning examples. These guide failure-specific reasoning, which produces new chains for the candidate failure modes. A separate answer-selection stage makes the decision based on those chains.
Group
Representative studies
Key characteristic
Distinction from CoP
Reasoning procedure without persistent KG
Gong et al. Gong et al. (2026)
Structured prompting and agent-based architectures, including multi-agent debate and planning.
No reusable, offline-built engineering-knowledge representation.
KG-based symptom / solution retrieval
Lin and Miao Lin and Miao (2025) ; Nie et al. Nie et al. (2026) ; Liu et al. Liu et al. (2025)
Graph retrieval links symptoms and faults to causes and maintenance actions.
Do not explicitly impose CoP’s constraining schema.
Graph-grounded retrieval and reasoning
Gu et al. Gu et al. (2026) ; Xie et al. Xie et al. (2024) ; Miao et al. Miao et al. (2026)
Retrieved causal paths or subgraphs provide structured, query-specific context to support subsequent LLM reasoning.
Evaluated in individual application domains, without a common cross-asset benchmark or not reproducible.
Causal / mechanism-explicit KG
Guo et al. Guo et al. (2026)
Weighted causal graph of fault-propagation paths derived from monitoring data.
Constructed from operational measurements instead of reference technical documentation.
Automatic knowledge construction from documentation
Zhou et al. Zhou et al. (2026) ; Wang et al. Wang et al. (2026) ; Marandi et al. Marandi et al. (2025)
Automatic extraction of structured knowledge from technical documents. Marandi et al. also reuse the resulting representation for graph traversal and LLM-based reasoning.
Use different ontologies or functional representations, without CoP’s predefined constraining schema.
Comparable evaluation design
Ma et al. Ma et al. (2026) (FDRKG-LLM)
Comparison of a KG-grounded LLM with direct prompting and multiple conventional RAG baselines.
Single equipment class, with open-ended generation metrics instead of multi-asset multiple-choice accuracy.
Table 1: Representative studies grouped by methodological similarity and their relationship to CoP.
Figure 2: Complete CoP methodology. The framework consists of an offline stage, performed once for each asset, and an online stage executed for each diagnostic question. Offline, the asset-specific reference documentation is divided into chunks and processed through a grounding gate. The accepted chains are converted into triples using the fixed relations results in , leads to , and is monitored by , and subsequently merged into the CKG. Online, the clarified question is used independently to retrieve and expand relevant CKG paths and to extract the failure modes under consideration. The retrieved paths guide the generation of new diagnostic chains for each failure mode, after which the Judge maps the resulting reasoning to the available answer choices.
Asset
Reference documentation
Questions
Aero gas turbine
Gas Turbine Theory , 7th ed. Saravanamuttoo et al. (2017)
336
Electric motor
Noninvasive Methods for Condition Monitoring and Electrical Fault Diagnosis of Induction Motors Sheikh et al. (2017)
234
Fan
Improving Fan System Performance: A Sourcebook for Industry U.S. Department of Energy and Air Movement and Control Association International (2003)
200
Power transformer
Transformers: Basics, Maintenance, and Diagnostics U.S. Department of the Interior, Bureau of Reclamation (2005)
544
Total
1,314
Table 2: Industrial assets considered in the evaluation, reference documentation used for CKG construction, and number of single-correct FailureSensorIQ questions Constantinides et al. (2025) .
Table 3: Outcome of the offline CKG construction for the four evaluated assets.
Figure 4: Excerpt from the aero gas turbine CKG. Orange nodes denote initiating conditions, blue nodes denote physical-variable changes, and green nodes denote monitoring sensors. The relations results in , leads to , and is monitored by distinguish initial effects, physical propagation, and observation. The explicit chain structure supports step-by-step inspection by domain experts.
Figure 5: Example of the CoP inference process for a fan mounting fault. The input question is first assessed by the clarification module and is left unchanged because no ambiguity is detected. The target failure mode is extracted directly from the clarified question, while the CKG retrieval stage independently recovers complete stored chains associated with the most relevant graph entities. These paths provide examples of the expected failure–physical effect–sensor reasoning structure. The reasoning stage then generates new plausible chains for the mounting fault, including paths leading to structural and lateral vibration measurements. The Judge receives the question, answer choices, and all inferred chains, and selects vibration as the final answer.
Model
Direct
RAG
CoP
Claude Opus 4.7
51.98
50.61
68.80
Llama 3.1 8B
30.75
26.86
38.51
Qwen 2.5 7B
30.52
35.24
50.15
Table 4: Overall diagnostic accuracy on the 1,314 single-correct FailureSensorIQ questions.
Table 5: Ablation study of the CoP inference procedure.
Table 6: Diagnostic accuracy (%) by asset class, model, and inference configuration. CKG denotes graph retrieval with direct answer generation; CoP denotes the complete inference procedure, including clarification, separate failure-specific reasoning, and final judgement. The number of evaluated questions is reported once for each asset. Bold values indicate the highest accuracy within each row.
Figure 8: Accuracy (%) by question type for direct prompting, RAG, and the three CoP configurations. (FM → S: failure mode to sensor; S → FM: sensor to failure mode; n : number of questions of each type.)
Model
Method
AGT
CMP
EG
EM
FAN
IGT
PT
PMP
RICE
ST
Overall
Llama 3.1 8B
Gong et al.
41.96
40.45
41.88
44.44
46.00
52.50
36.95
51.97
47.92
38.01
43.34
CoP w/o CKG
41.67
45.91
45.30
46.58
50.50
52.08
38.79
51.97
48.81
39.77
45.14
Qwen 2.5 7B
Gong et al.
41.67
47.73
52.99
53.85
54.00
55.00
43.38
52.63
56.85
43.27
49.34
CoP w/o CKG
45.54
51.82
58.55
50.00
57.50
57.50
50.74
53.95
62.20
46.78
53.28
Questions
336
220
234
234
200
240
544
152
336
171
2,667
Table 7: Accuracy (%) on the 2,667 single-correct FailureSensorIQ questions without graph retrieval. Gong et al. denotes the reimplemented multi-agent framework of Gong et al. (2026) with all agents instantiated using the same model.
We introduce FailureSensorIQ, a novel Multi-Choice Question-Answering (MCQA) benchmarking system designed to assess the ability of Large Language Models (LLMs) to reason and understand complex, domain-specific scenarios in Industry 4.0. Unlike traditional QA benchmarks, our system focuses on multiple aspects of reasoning through failure modes, sensor data, and the relationships between them across various industrial assets. Through this work, we envision a paradigm shift where modeling decisions are not only data-driven using statistical tools like correlation analysis and significance tests, but also domain-driven by specialized LLMs which can reason about the key contributors and useful patterns that can be captured with feature engineering. We evaluate the Industrial knowledge of over a dozen LLMs-including GPT-4, Llama, and Mistral-on FailureSensorIQ from different lens using Perturbation-Uncertainty-Complexity analysis, Expert Evaluation study, Asset-Specific Knowledge Gap analysis, ReAct agent using external knowledge-bases. Even though closed-source models with strong reasoning capabilities approach expert-level performance, the comprehensive benchmark reveals a significant drop in performance that is fragile to perturbations, distractions, and inherent knowledge gaps in the models. We also provide a real-world case study of how LLMs can drive the modeling decisions on 3 different failure prediction datasets related to various assets. We release: (a) expert-curated MCQA for various industrial assets, (b) FailureSensorIQ benchmark and Hugging Face leaderboard based on MCQA built from non-textual data found in ISO documents, and (c) LLMFeatureSelector, an LLM-based feature selection scikit-learn pipeline. The software is available at https://github.com/IBM/FailureSensorIQ.
Christodoulos Constantinides, Dhaval Patel, Shuxin Lin +3
In so-called "Industry 4.0," integrated sensors, analytics tools, and other digital technologies supplement industrial workflows, potentiating enhanced visibility and automation. Prognostics and health management (PHM) and natural language processing (NLP) practitioners alike have explored the application of large language models (LLMs) to industrial asset management, but many prior works' focus on exam-like benchmarks has limited their assessment of LLMs' ability to automate practical operations. Herein, we introduce CoffeeBench, a fault diagnostics task for LLM agents, using a simulated appliance. The LLMs we evaluate struggle to consistently identify faults from the sensor information alone, but the best-performing ones can iteratively refine the fault-option space to eventually obtain a successful diagnosis. Our subsequent analyses, therefore, focus on: differentiating models by diagnostic efficiency (RQ1); examining the lagging performance of the three worst models (RQ2); and identifying viable heuristics for selective prediction (RQ3), including that initial diagnostic uncertainty and LLM-judged soundness of reasoning are high-signal predictors of diagnostic success. In addition to our benchmark results (Section 3.1), we provide practical insights for improving the performance of the open-weight models on CoffeeBench, such as by explicitly tracking episode state on their behalf to reduce timeouts (Section 3.2) or applying a deliberation loop to refine diagnostic hypotheses (Section 3.3).
Large language models are increasingly used as integration layers above specialized tools, but a stronger component does not necessarily produce a stronger combined system. Across five diagnostic datasets (bearing vibration, process monitoring, semiconductor equipment), we study whether an LLM can reliably use external diagnostic information; paired repeat calls separate advice effects from output instability. In all five, conflicting external information overturned initially correct LLM judgments. Among the four datasets with direct integration comparisons, none showed a consistent advantage for implicit LLM integration over the stronger standalone source. On a Tennessee Eastman confirmation set whose protocol was fixed before evaluation, unaided accuracy was 64.67%, implicit LLM-specialist integration 77.43%, and the specialist alone 83.33%. Specialist information improved the LLM by 12.8 points (95% interval 9.7 to 15.9), yet the integrated output stayed 5.9 points below the specialist (95% interval -12.0 to -0.7). A two-source selector oracle reached 92.76%, indicating complementarity that the integrated output did not fully realize. The integrated output missed 140 of 295 specialist corrections (47.5%) but lost 15 of 99 initially correct LLM judgments (15.2%). The deficit remained under prompt and specialist sensitivity analyses. Among CWRU cases solved under both evidence presentations, task-aligned physical evidence yielded lower estimates of susceptibility to incorrect advice in six of seven models (five intervals excluding zero); higher reasoning effort gave no reliable reduction in five models, and a separate four-model TEP analysis gave no clear evidence that it resolves the integration problem. Source quality and integration quality should be evaluated separately: an integration layer should be compared with its stronger standalone component, not only with the unaided LLM.