We introduce FailureSensorIQ, a novel Multi-Choice Question-Answering (MCQA) benchmarking system designed to assess the ability of Large Language Models (LLMs) to reason and understand complex, domain-specific scenarios in Industry 4.0. Unlike traditional QA benchmarks, our system focuses on multiple aspects of reasoning through failure modes, sensor data, and the relationships between them across various industrial assets. Through this work, we envision a paradigm shift where modeling decisions are not only data-driven using statistical tools like correlation analysis and significance tests, but also domain-driven by specialized LLMs which can reason about the key contributors and useful patterns that can be captured with feature engineering. We evaluate the Industrial knowledge of over a dozen LLMs-including GPT-4, Llama, and Mistral-on FailureSensorIQ from different lens using Perturbation-Uncertainty-Complexity analysis, Expert Evaluation study, Asset-Specific Knowledge Gap analysis, ReAct agent using external knowledge-bases. Even though closed-source models with strong reasoning capabilities approach expert-level performance, the comprehensive benchmark reveals a significant drop in performance that is fragile to perturbations, distractions, and inherent knowledge gaps in the models. We also provide a real-world case study of how LLMs can drive the modeling decisions on 3 different failure prediction datasets related to various assets. We release: (a) expert-curated MCQA for various industrial assets, (b) FailureSensorIQ benchmark and Hugging Face leaderboard based on MCQA built from non-textual data found in ISO documents, and (c) LLMFeatureSelector, an LLM-based feature selection scikit-learn pipeline. The software is available at https://github.com/IBM/FailureSensorIQ.
Figures & tables
Figure 1Figure 2
Figure 4: Real-world knowledge capacities assessed by ACC@Consist.
Figure 4Table 5Figure 6
Method/Model
4-Mav-17B-128E
4-scout-17b-16e
3-405b
3.3-70b
3.1-8b
Direct Prompt
55.83
53.96
51.26
41.69
40.04
COT@Inductive
56.88
54.22
53.17
49.46
40.27
COT@Expert
56.96
53.06
55.38
50.96
42.11
COT@Standard
57.29
52.53
55.57
51.18
45.74
Plan@Solve
56.47
55.46
54.89
50.36
46.46
Average
56.90
53.82
54.75
50.49
43.65
Table 6: Performance Comparison across Different Reasoning Models for llama models
Figure 8
Asset
Task: (Failure or
Top-5 recommended features
Max
Energy Pred.)
1
2
3
4
5
Corr.
Electrical Transf.
Magnetic Oil Gauge
7.83
12.52
20.33
0.25
0.31
35.88
Air Compressor
Bearings
0.07
1.94
4.51
2.75
0.01
36.03
Air Compressor
Water Pump
15.28
21.38
13.62
15.87
14.52
21.38
Air Compressor
Radiator
31.85
31.83
31.78
86.88
25.16
86.88
Air Compressor
Valve
1.66
52.64
14.42
1.43
0.30
52.64
Table 8: Absolute value Correlations between the top-5 recommended sensors and the target variable
Appendix figures & tables15 assets
Supplementary material from the paper’s appendix.
Appendix
Task
Question Template
FM2Sensor-Sel.
For {asset_class} , if a failure event {failure_mode} occurs, which sensors out of the choices are the most relevant regarding the occurrence of the failure event?
FM2Sensor-El.
For {asset_class} , if a failure event {failure_mode} occurs, which sensors out of the choices are not relevant regarding the occurrence of the failure event?
Sensor2FM-Sel.
In the context of {asset_class} , which failure modes are the most relevant when {sensor} shows abnormal readings?
Sensor2FM-El.
In the context of {asset_class} , which failure events are not relevant when the sensor {sensor} shows an abnormal reading?
Appendix
Table 9: Examples of question templates for sensor selection and failure mode identification. Each template is rephrased multiple times for different tasks.
Figure 8: Knowledge Graph Interaction
Figure 9: Pipeline for Preparing Perturb Dataset
Figure 10: Input Prompt. Ground truth is B, Exhaust Pressure.
Figure 11: Reasoning Trace by o1.
Figure 12: Reasoning Trace by Mistral-Large-Instruct-2407.
Figure 13: Reasoning Trace by Llama-3.3-70B.
Model
Acc. Total
Acc. Sel.
Acc El.
Acc Perturb.
Acc. Consist.
o1
60.40
61.06
67.89
49.76
44.17
o3-mini
58.46
56.81
69.84
53.28
47.47
llama-4-mav.-17b-128e
55.83
44.47
71.90
49.12
40.83
llama-4-scout-17b-16e
53.96
44.47
63.53
29.36
24.11
gpt-4.1
53.47
56.38
59.17
45.74
39.93
o1-preview
52.31
49.57
62.5
47.51
40.68
Appendix
Table 10: Performance comparison of different models across total, selection, elimination, perturbation, and consistency accuracies.
Model
Electric Motor
Steam Turbine
Aero Gas Turbine
Industr. Gas Turbine
Pump
Compressor
o1
68.8
47.95
50.89
70.83
58.55
56.36
o3-mini
63.25
49.71
51.49
68.33
52.63
55.45
llama-4-maverick-17b-128e
66.24
47.37
47.32
69.17
55.26
52.27
llama-4-scout-17b-16e
60.26
40.94
47.02
67.5
50.0
46.36
gpt-4.1
58.12
43.27
43.75
69.17
52.63
46.36
o1-preview
65.38
43.86
42.56
62.92
50.0
46.82
Appendix
Table 11: FailureSensorIQ performance per asset (Part A).
Model
Recipr. Intern. Comb. Engine
Electric Generator
Fan
Power Transformer
o1
65.48
70.94
58.0
57.35
o3-mini
63.39
65.81
61.0
54.78
llama-4-maverick-17b-128e
61.01
62.39
56.5
48.71
llama-4-scout-17b-16e
66.96
59.4
60.5
45.04
gpt-4.1
61.01
58.12
57.0
48.9
o1-preview
58.33
62.82
53.5
44.85
Appendix
Table 12: FailureSensorIQ performance per asset (Part B).
CoT Stype
Trigger Statement
Standard
Let me think step by step
Expert
Let me think step by step as a reliability engineer
Inductive
Let’s use step by step inductive reasoning, given the domain specific nature of the question
Appendix
Table 13: CoT Style
Plan-Solve
Trigger Statement
prompt-3
Let’s first prepare relevant information and make a plan. Then, let’s answer the question step by step (pay attention to commonsense and logical coherence).
Appendix
Table 14: Plan-Solve
Figure 22
Figure 17: LLMFeatureSelect input prompt and ChatGPT response.
Figure 18: Screenshot from the quiz used for the user study.
Industrial maintenance environments increasingly rely on AI systems to assist operators in understanding asset behavior, diagnosing failures, and evaluating interventions. Although large language models (LLMs) enable fluent natural-language interaction, deployed maintenance assistants routinely produce generic explanations that are weakly grounded in telemetry, omit verifiable provenance, and offer no testable support for counterfactual or action-oriented reasoning that undermine trust in safety-critical settings. We present IndustryAssetEQA, a neurosymbolic operational intelligence system that combines episodic telemetry representations with a Failure Mode Effects Analysis Knowledge Graph (FMEA-KG) to enable Embodied Question Answering (EQA) over industrial assets. We evaluate on four datasets covering four industrial asset types, including rotating machinery, turbofan engines, hydraulic systems, and cyber-physical production systems. Compared to LLM-only baselines, IndustryAssetEQA improves structural validity by up to 0.51, counterfactual accuracy by up to 0.47, and explanation entailment by 0.64, while reducing severe expert-rated overclaims from 28% to 2% (approximately 93% reduction). Code, datasets, and the FMEA-KG are available at https://github.com/IBM/AssetOpsBench/tree/IndustryAssetEQA/IndustryAssetEQA.
Chathurangi Shyalika, Dhaval Patel, Amit Sheth
Artificial Intelligence Institute, University of South Carolina USA · Software Innovation Lab IBM Yorktown USA · Artificial Intelligence Institute, University of South Carolina Indian AI Research Organization
Monitoring complex industrial assets relies on engineer-authored symbolic rules that trigger based on sensor conditions and prompt technicians to perform corrective actions. The bottleneck is not detection but response: translating rules into maintenance steps requires asset-specific knowledge gained through years of practice. We investigate whether LLMs can serve as decision support for this rule-to-action step and introduce \ours{}, a benchmark of 6{,}690 expert-validated multiple-choice questions from 118 rule-action pairs across 16 asset types. We contribute (i) a symbolic-to-MCQA pipeline normalizing rules to Disjunctive Normal Form with embedding-based distractor sampling, (ii) five variants probing distinct failure modes (Pro, Pert, Verbose, Aug, Rationale), and (iii) a benchmark of 29 LLMs and 4 embedding baselines. A human evaluation (9 practitioners, mean 45.0%) confirms \ours{} requires specialist knowledge beyond operational experience. Three findings stand out. The frontier has closed: the top three LLMs lie within one Macro point, with Bradley-Terry Elo placing claude-opus-4-6 30 points above the next model. Yet \ours{},Pro exposes brittleness, with every model losing 13--60% relative accuracy under distractor expansion. \ours{},Aug exposes pattern-matching: under condition inversion, frontier models still select the original answer 49--63% of the time. The deployment bottleneck is not capability but calibration: frontier models handle template-style fault detection but break under structural perturbation.
Devin Yasith De Silva, Dhaval Patel, Christodoulos Constantinides +7
2Rensselaer Polytechnic Institute · 1IBM · 3Independent Contributor
In so-called "Industry 4.0," integrated sensors, analytics tools, and other digital technologies supplement industrial workflows, potentiating enhanced visibility and automation. Prognostics and health management (PHM) and natural language processing (NLP) practitioners alike have explored the application of large language models (LLMs) to industrial asset management, but many prior works' focus on exam-like benchmarks has limited their assessment of LLMs' ability to automate practical operations. Herein, we introduce CoffeeBench, a fault diagnostics task for LLM agents, using a simulated appliance. The LLMs we evaluate struggle to consistently identify faults from the sensor information alone, but the best-performing ones can iteratively refine the fault-option space to eventually obtain a successful diagnosis. Our subsequent analyses, therefore, focus on: differentiating models by diagnostic efficiency (RQ1); examining the lagging performance of the three worst models (RQ2); and identifying viable heuristics for selective prediction (RQ3), including that initial diagnostic uncertainty and LLM-judged soundness of reasoning are high-signal predictors of diagnostic success. In addition to our benchmark results (Section 3.1), we provide practical insights for improving the performance of the open-weight models on CoffeeBench, such as by explicitly tracking episode state on their behalf to reduce timeouts (Section 3.2) or applying a deliberation loop to refine diagnostic hypotheses (Section 3.3).