PHMForge: Evaluating LLM Agents on Industrial Prognostics through MCP-Native, Algorithm-Grounded Tools

Yusheng Li; Tianjun Feng; Yunfeng Chen; Chun-Yi Tsai; Yihan Sun; Ayan Das; Kaoutar El Maghraoui; Shuxin Lin

cs.AIApr 2, 2026

PHMForge: Evaluating LLM Agents on Industrial Prognostics through MCP-Native, Algorithm-Grounded Tools

Authors: Yusheng Li, Tianjun Feng, Yunfeng Chen, Chun-Yi Tsai, Yihan Sun, Ayan Das, Kaoutar El Maghraoui, Shuxin Lin, +1 more

Organizations: Columbia University · Georgia Institute of Technology · IBM, New York

Abstract

LLM agents are beginning to invoke industrial asset-management tools through the Model Context Protocol (MCP), yet whether they can act reliably on this substrate for safety-critical Prognostics and Health Management (PHM) is unanswered. Prior benchmarks conflate protocol fluency with reasoning, instrumentation failures with agent failures, and tool use with tool retrieval. We introduce PHMForge, an evaluation environment that closes each conflation. PHMForge ships 99 SME-authored scenarios across eight industrial asset classes spanning rotating equipment, aero-engines, and lithium-ion cells, on public datasets including NASA PCoE, served through 39 MCP-native tools wrapping published PHM algorithms (C-MAPSS, ISO~10816, Arrhenius capacity-fade models, time-series foundation models). Krippendorff's $α\in [0.74,\,0.82]$ on a 30-scenario stratified rotating-equipment/aero-engine sample; the battery extension is single-rater. Across three agentic frameworks and six LLM backbones, the strongest configuration reaches 80.8% pass@1, with the residual gap concentrated in orchestration and tool-sequencing errors. Crucially, an architectural ablation shows that replacing MCP execution with text-based Retrieval-Augmented Generation (RAG) over telemetry-equivalent evidence collapses Remaining Useful Life pass-all-3 from 100% to 20% (5/5 vs.\ 1/5) on the battery class, exposing the structural limits of static retrieval for prognostic computation. Trajectory decomposition shows orchestration errors dominate failures across backbones, while schema-invalid tool calls concentrate in smaller open-weight models. Frontier LLMs are stronger at calling tools than at planning when to call them. PHMForge is open-sourced with deterministic evaluators, a public leaderboard, and a datasheet.

Figures & tables

Figure 1: PHMForge evaluation pipeline with full state tracking. Each scenario begins from an SME-authored industrial query and a curated dataset context. The PHMForge-Agent (e.g., ReAct, ReActXen) then executes a tool-calling loop over algorithm-grounded MCP tools, followed by a deterministic verifier that evaluates whether all required checks are satisfied.

Benchmark	PHM-	Multi-	Agentic	MCP-	Tool-	SME-	Real	Det.	#Scen.
	Domain	Asset	Agentic	Native	Retrieval	Authored	Alg.	Eval	#Scen.
ITFormer [ 23 ]	$\checkmark$	$\times$	$\times$	$\times$	$\times$	$\times$	$\times$	$\checkmark$	110k
PDMBench [ 29 ]	$\checkmark$	$\checkmark$	$\times$	$\times$	$\times$	$\times$	$\times$	$\checkmark$	–
PHM-Bench [ 27 ]	$\checkmark$	$\times$	$\times$	$\times$	$\times$	$\times$	$\times$	$\checkmark$	–
MLE-Bench [ 2 ]	$\times$	–	$\checkmark$	$\times$	$\times$	$\times$	$\times$	$\checkmark$	75
MCP-Bench [ 24 ]	$\times$	–	$\checkmark$	$\checkmark$	$\times$	$\times$	$\times$	$\checkmark$	250

Table 1: PHMForge in context with eight peer benchmarks. PHM-Domain . Tools and scenarios are physically grounded. Multi-Asset . Covers

\geq

3 asset classes. Agentic . Evaluates multi-turn tool orchestration. MCP-Native . Tools served through MCP. Tool-Retrieval . Agent must locate tools, not just invoke them. SME-Authored . Scenarios authored by domain experts. Real Alg. Tools wrap published algorithms rather than stubbed mocks. Det. Eval . Deterministic. 1 1 1 PHMForge’s deterministic evaluator applies to the automated ReAct/ReActXen harness on the 25-scenario stratified subset; the frontier evaluation (Claude Code, 99 scenarios) was conducted manually under the same scenario-level scoring rubric.

Figure 2: PHMForge benchmark composition. 99 scenarios across 5 task categories and 8 asset classes (top); representative tools from the Prognostics and Maintenance servers (bottom).

Figure 3: Representative PHMForge scenarios across the five task categories. Each card pairs an SME-authored natural-language query with a representative subset of the required tool set (full counts indicated, e.g., “3 of 7”) and the validation criterion. Queries embed domain terminology (HPT, RUL, IEC, NEMA) as plant managers, technicians, and safety officers actually use it. Each scenario also includes 2–4 distractor tools

\mathcal{T}_{\tau}^{-}

, omitted from the cards for clarity.

Dimension	Krippendorff’s $\alpha$	95% CI
Realism	0.78	[0.71, 0.85]
Difficulty calibration	0.74	[0.66, 0.81]
Ground-truth correctness	0.82	[0.75, 0.88]
Pooled	0.78	[0.73, 0.83]

Table 2: Inter-annotator agreement (Krippendorff’s

\alpha

) on the 30-scenario stratified sample drawn from the 75 rotating-equipment and aero-engine scenarios. The 24 lithium-ion battery scenarios are not covered by this IAA study (see Appendix E.6 ). CIs are 95% bootstrap intervals (1,000 resamples, full details in Appendix E ).

Framework + Model	Pass@1	RUL	Fault	Health	Cost	Safety
		(5)	(5)	(10)	(2)	(3)
ReAct + Llama 4 Maverick	80.0%	60.0%	100.0%	80.0%	50.0%	100.0%
ReAct + Mistral Medium 2505	64.0%	0.0%	40.0%	90.0%	100.0%	100.0%
ReAct + GPT-OSS 120B	56.0%	0.0%	100.0%	50.0%	50.0%	100.0%
ReAct + Compact-LLM	44.0%	20.0%	40.0%	60.0%	50.0%	33.3%
ReAct + Mistral Small 3.1 24B	44.0%	40.0%	60.0%	30.0%	50.0%	66.7%

Table 3: Framework-and-model Pass@1 across PHM task categories on a 25-scenario stratified subset (5 RUL + 5 Fault + 10 Health + 2 Cost + 3 Safety) preserving all 5 task categories. † Partial coverage on Cost-Benefit and Safety/Policy categories; ReActXen’s reflection loop on Llama-4-Maverick exceeded the compute budget on those scenarios. Best per column in bold .

Framework + Model	Pass@1	RUL	Fault	Health	Cost	Safety
		(20)	(20)	(39)	(7)	(13)
Claude Code + Opus 4.6	80.8%	85.0%	75.0%	79.5%	71.4%	92.3%
Claude Code + Sonnet 4.5	64.6%	70.0%	60.0%	69.2%	42.9%	61.5%

Table 4: Frontier-model Pass@1 across PHM task categories on the full 99-scenario benchmark (39 Health + 20 RUL + 20 Fault + 13 Safety + 7 Cost). Manual evaluation under scenario-level scoring; Claude Code is the interactive CLI agent paired with each frontier backbone. Best per column in bold .

Asset	Metric	Agent	Baseline	$\Delta$	Primary Error Mode	Rate
RUL Prediction
Turbofan	MAE $\downarrow$ (cy)	14.8	10.5	$+4.3\,\uparrow$	Train-test contamination	27%
	RMSE $\downarrow$ (cy)	19.2	13.8	$+5.4\,\uparrow$
Bearing	MAE $\downarrow$ (%)	11.3	8.2	$+3.1\,\uparrow$	Invalid range predictions	15%
	RMSE $\downarrow$ (%)	14.7	10.9	$+3.8\,\uparrow$
Lithium-Ion	MAE $\downarrow$ (cy) ∗	13.5	28.4	$-14.9\,\downarrow$	Capacity regeneration	—

Table 5: Task-specific quality metrics for the strongest configuration (Claude Code + Opus 4.6). The agent matches reference baselines on Safety/Policy and Cost-Benefit (

|\Delta|\leq 4

pp) but lags on Motor fault discrimination (

|\Delta|\geq 11

pp).

\downarrow

\uparrow

indicates direction of improvement; baselines from PHM-Bench [ 27 ] for RUL and Fault Classification or SME consensus for Decision tasks. ∗ Lithium-Ion RUL evaluated under leave-one-battery-out (LOO) protocol with the TTM fine-tuned predictor; baseline is linear regression under the same protocol (Appendix F.2 ).

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

Tool Name	Description	Key Parameters
Data Loading
load_dataset	Load dataset from PDMBench data directory	dataset_name (str), split (str, default= "train" )
load_ground_truth	Load ground truth RUL values or fault labels	dataset_name (str), file (str, optional)
Model Training
train_rul_model	Train RUL prediction model with Adam optimizer	dataset (str), model_type (str: mlp/lstm/transformer), epochs (int, default=50)
train_fault_classifier	Train multi-class fault classification model	dataset (str), model_type (str), epochs (int)

Appendix

Table 6: Complete tool inventory for the Prognostics Server ( prognostics-server ), providing 15 tools for data loading, model training, prediction, metric computation, and engine health analysis.

Tool Name	Description	Key Parameters
Cost-Benefit Analysis
calculate_maintenance_cost	Compute annual preventive maintenance costs including downtime	preventive_cost (float), frequency_per_year (int), downtime_hours (float), hourly_rate (float)
calculate_failure_cost	Estimate expected annual cost of unplanned failures	failure_probability (float), repair_cost (float), downtime_hours (float), hourly_rate (float), consequential_cost (float)
optimize_maintenance_schedule	Find cost-optimal RUL threshold for scheduling maintenance	rul_estimate (float), preventive_cost (float), failure_cost (float), safety_margin (float)
Safety & Policy Evaluation
assess_safety_risk	Classify risk level (low/medium/high/critical) using RPN analysis	failure_mode (str), severity (int, 1–10), probability (int, 1–10), detectability (int, 1–10)

Appendix

Table 7: Complete tool inventory for the Maintenance Server ( maintenance-server ), providing 7 tools for cost-benefit analysis, safety/policy evaluation, and web search.

Tool Name	Description	Key Parameters
Data Access
fetch_cycle_data	Full per-cycle telemetry with all sensor channels	battery_id (str), cycle (int)
fetch_cycle_summary	5-feature lightweight projection (capacity, temp, voltage, duration)	battery_id (str), cycle_range (tuple)
fetch_impedance_data	EIS records with $R_{e}$ / $R_{ct}$ trend statistics	battery_id (str)
Diagnostics
capacity_soh_calculator	Deterministic SOH against first-cycle reference	battery_id (str), cycle (int)

Appendix

Table 8: Complete tool inventory for the Battery Prognostics Server ( battery-prognostics-server ), providing 17 tools for the lithium-ion battery storage asset class. See Appendix F for methodology details.

Dataset	Equipment Class	Records	Features	Assets	Classes	Scenarios	Splits	Primary Task
Turbofan Engines
CMAPSS FD001	Aircraft engine	33,727	26	200	—	6	✓	RUL Prediction
CMAPSS FD002	Aircraft engine	87,750	26	519	—	3	✓	RUL Prediction
CMAPSS FD003	Aircraft engine	41,316	26	200	—	2	✓	RUL Prediction
CMAPSS FD004	Aircraft engine	102,463	26	497	—	3	✓	RUL Prediction
Azure	Aircraft engine	876,905	11	100	5	1	—	RUL Prediction

Appendix

Table 9: Dataset characteristics for the 19 PHMForge benchmark datasets. Records is the total row count across all files. Assets denotes unique units, bearings, machines, or cells. Classes indicates distinct fault types or labels. Scenarios is the number of benchmark scenarios using each dataset. Datasets with pre-computed train/val/test splits are marked

\checkmark

Asset Class	# Scenarios	Task Categories Covered
Turbofan Engines	14	RUL, Health, Cost-Benefit, Safety
Aero-Engines	30	Health Analysis (4 cognitive levels)
Bearings	22	RUL, Fault Classification
Electric Motors	3	Fault Classification
Induction Motors	2	Fault Classification
Gearboxes	3	Fault Classification

Appendix

Table 10: Distribution of 99 scenarios across asset classes.

Stage	# Scenarios (cum.)	Task Categories Introduced
Stage 1	1	RUL Prediction
Stage 2	10	+ Fault Classification
Stage 3	20	+ Cost-Benefit, Safety/Policy
Stage 4	40	(multi-asset expansion)
Stage 5	75	+ Engine Health Analysis (4 cognitive levels)
Stage 6	99	+ BESS lithium-ion (24 scenarios across 5 categories)

Appendix

Table 11: Scenario counts and task-category coverage at each expansion stage. Cumulative scenario counts are shown.

Predictor	Overall	B0005	B0006	B0018
TTM fine-tuned (LOO)	13.5	14.0	16.0	10.6
Linear regression	28.4	71.6	6.7	6.9
Chronos fine-tuned (LOO)	31.8	69.2	19.4	6.9
TTM zero-shot	33.5	32.5	41.7	26.2
Empirical (Arrhenius)	37.4	101.3	5.2	5.7
LSTM trained from scratch	45.4	16.8	37.4	82.0

Appendix

Table 12: Six-model RUL benchmark on NASA PCoE Li-ion cells under LOO protocol. MAE in cycles; lower is better. Best per column in bold.

Category	n	Path A Fuzzy	Path A Explicit	Path B Fuzzy	Path B Explicit
Health Analysis	9	4/9	7/9	8/9	9/9
RUL Prediction	5	1/5	1/5	5/5	5/5
Fault Classification	5	2/5	4/5	3/5	4/5
Safety/Policy	3	2/3	3/3	3/3	3/3
Cost-Benefit Analysis	2	0/2	1/2	0/2	1/2
Total	24	9/24	16/24	19/24	22/24

Appendix

Table 13: Per-category pass-all-3 on the lithium-ion battery asset class (n=24) under semantic-aware scoring with Claude Opus 4.6 as orchestrator. Path A is text-RAG; Path B is MCP tool execution. Both paths leave headroom on Cost-Benefit and on fuzzy queries.

Path	Query	Pass-all-3 (Wilson 95% CI)	Mean pass@1 (Wilson 95% CI)
Path A RAG	Fuzzy	9/24 (37.5%) [21.1%, 57.4%]	35/72 (48.6%) [37.4%, 59.9%]
Path A RAG	Explicit	16/24 (66.7%) [46.6%, 82.2%]	53/72 (73.6%) [62.4%, 82.5%]
Path B MCP	Fuzzy	19/24 (79.2%) [59.1%, 91.2%]	58/72 (80.6%) [69.8%, 88.2%]
Path B MCP	Explicit	22/24 (91.7%) [73.0%, 98.8%]	66/72 (91.7%) [82.7%, 96.4%]

Appendix

Table 14: BESS scenario stability over the three timestamped semantic runs used to compute Table 13 . Pass-all-3 counts a scenario only if it passes in all three runs; mean pass@1 pools the 72 single-run trials. Wilson 95% confidence intervals are shown in brackets. McNemar paired tests compare Path A and Path B on per-scenario pass-all-3 outcomes: fuzzy

b{=}0

c{=}10

, exact

p{=}0.002

; explicit

b{=}1

c{=}7

, exact

p{=}0.07

Framework + Model	Steps	Tokens	Time (s)	Pass@1
ReAct + Llama 4 Maverick	7.7	33,017	42	80.0%
ReAct + Mistral Medium 2505	8.0	37,259	49	64.0%
ReAct + GPT-OSS 120B	7.6	35,157	26	56.0%
ReAct + Compact-LLM	7.7	31,947	37	44.0%
ReAct + Mistral Small 3.1 24B	8.2	42,161	47	44.0%
ReAct + Llama 3.3 70B	6.1	24,635	312	36.0%

Appendix

Table 15: Per-configuration process metrics. Average steps taken, total tokens consumed, and execution time per scenario across the 25-scenario stratified subset. ReActXen consistently consumes more time per scenario than ReAct on the same backbone due to its reflection loop, but uses a similar token budget. Llama-3.3-70B’s long per-scenario time reflects WatsonX inference latency rather than agent overhead.

Explore similar work

Can LLM Agents Efficiently Identify Appliance Faults? Insights From Diagnosing a Simulated Appliance
Oct 13, 2026Lucas H. McCabe, Sarah Lukens, H. Howie HuangLLM Agents Fault Diagnosis
LMI Consulting · The George Washington University
From paper to benchmark: agentic, framework-based reproduction of under-specified methods in machine health intelligence
May 27, 2026Raffael Theiler, Ludovico Comito, David Leko +3Agentic Code Generation ML Reproducibility
EPFL · University of Roma “La Sapienza”
Agent Delivery Engineering Predictive Reliability Framework
Jul 8, 2026Dexing LiuAI Agent Reliability Multi-Agent LLM Systems
Shanghai Qijing Digital Technology Co., Ltd · Shanghai Qijing Digital Technology Co., Ltd.

Oct 13, 2026J/K move · Enter open · S save

Can LLM Agents Efficiently Identify Appliance Faults? Insights From Diagnosing a Simulated Appliance

Lucas H. McCabe, Sarah Lukens, H. Howie Huang

LMI Consulting · The George Washington University

LLM Agents Fault Diagnosis AI Agent Benchmarks Selective Prediction

Open paper AskPDF arXiv

In so-called "Industry 4.0," integrated sensors, analytics tools, and other digital technologies supplement industrial workflows, potentiating enhanced visibility and automation. Prognostics and health management (PHM) and natural language processing (NLP) practitioners alike have explored the application of large language models (LLMs) to industrial asset management, but many prior works' focus on exam-like benchmarks has limited their assessment of LLMs' ability to automate practical operations. Herein, we introduce CoffeeBench, a fault diagnostics task for LLM agents, using a simulated appliance. The LLMs we evaluate struggle to consistently identify faults from the sensor information alone, but the best-performing ones can iteratively refine the fault-option space to eventually obtain a successful diagnosis. Our subsequent analyses, therefore, focus on: differentiating models by diagnostic efficiency (RQ1); examining the lagging performance of the three worst models (RQ2); and identifying viable heuristics for selective prediction (RQ3), including that initial diagnostic uncertainty and LLM-judged soundness of reasoning are high-signal predictors of diagnostic success. In addition to our benchmark results (Section 3.1), we provide practical insights for improving the performance of the open-weight models on CoffeeBench, such as by explicitly tracking episode state on their behalf to reduce timeouts (Section 3.2) or applying a deliberation loop to refine diagnostic hypotheses (Section 3.3).

Abstract

Explore similar work

Can LLM Agents Efficiently Identify Appliance Faults? Insights From Diagnosing a Simulated Appliance

From paper to benchmark: agentic, framework-based reproduction of under-specified methods in machine health intelligence

Agent Delivery Engineering Predictive Reliability Framework