Moving Generality from Model to Modeler: LLM Agents for PHM on a Deployment-Style Benchmark
Organizations: Laboratoire Génie Industriel, CentraleSupélec, Université Paris-Saclay, Gif-sur-Yvette, France · Chair on Risk and Resilience of Complex Systems, Laboratoire Génie Industriel, CentraleSupélec, Université Paris-Saclay, Gif-sur-Yvette, France
Abstract
Coding agents built on large language models (LLMs) now help develop many models published as general solutions. Should generality instead reside in the agent that builds a model for each new task? We examine this question in prognostics and health management (PHM), where every model must work on unseen equipment. Our deployment-style benchmark, NextUnit-Bench, covers 24 public datasets split into 58 folds that hold out units or operating conditions; each agent has two hours to deliver an executable predictor, then run on the hidden units. First, frontier agents outperform AutoML, which reaches less than half the best agent's score, and faithful implementations of published methods on four datasets, of which only 6 of 30 beat the strongest agent's own design; published accuracies, where reproducible, rest on the papers' own conventions, and a single convention, capping the remaining life of test engines, can halve the reported error. Second, the agents' advantage is a design made on each machine's data; once made, a plain-text plan of a few hundred words carries most of the gap between two frontier agents, and given a weaker agent's plan, the strongest agent falls to that agent's level, often to identical predictions. Before touching the data, however, it matters little whose plan an agent carries out, and domain toolkits appear to help neither: what sets the strongest apart is how much it learns from the data. Third, such designs reach only implementers that can execute them: given the strongest agent's designs, a locally deployable open-weight model delivers half as often, even when they are lightened, and scores no higher than without them. Generality, we argue, is not disappearing but moving, from the model to the modeler.
Figures & tables
| Dataset | Asset | Task | Units | Folds | Mode | Metric |
| Condition classification | ||||||
| Paderborn [ 16 ] | Rolling bearing | 4-class condition | 32 | 3 | Batch | Accuracy |
| CWRU [ 17 ] | Rolling bearing | 4-class fault | 40 a | 4 | Batch | Accuracy |
| SEU [ 18 ] | Gearbox | 5-class fault | 10 a | 2 | Batch | Accuracy |
| UCI hydraulic [ 19 ] | Hydraulic rig | Valve condition | 3 a | 3 | Batch | Balanced accuracy |
| Tennessee Eastman [ 20 ] | Chemical process b | Fault class | 2100 | 1 | Batch | Accuracy |
| Agent | , floor | , without | , 21 | Fail | Worse | |
|---|---|---|---|---|---|---|
| (24) | shortcut sets | unit sets | (%) | (%) | ||
| Astra | 43.0 (1) | 33.8 (2) | 40.4 (1) | 37.9 (1) | 0.0 | 8.6 |
| [32.4, 53.8] | [15.5, 50.0] | [29.4, 52.0] | [28.3, 46.8] | |||
| Kimi | 39.7 (2) | 37.7 (1) | 38.4 (2) | 34.7 (2) | 1.7 | 8.6 |
| [29.2, 51.0] | [26.6, 49.5] | [26.6, 50.7] | [25.1, 44.1] | |||
| DeepSeek | 36.1 (3) | 30.9 (3) | 34.7 (3) | 31.0 (3) | 0.0 | 10.3 |
| Carried out by | |||
| Plan by | Data seen | Luna | Astra |
| Astra | yes | 36.8 (37.8) | 44.2 a |
| Luna | yes | 31.0 a | 31.5 (31.5) |
| Astra | no | 27.6 | 29.5 (30.0) |
| Luna | no | 26.0 (23.8) | 28.0 (28.6) |
| a The agent’s own run, without a plan. | |||
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
| Dataset | Method | Year | (method) | (Astra’s design) | Beat |
|---|---|---|---|---|---|
| C-MAPSS (FD004) | Dual-task LSTM [ 60 ] | 2019 | 21.0 | 44.7 | 0/1 |
| DA-TCN [ 39 ] | 2021 | 23.4 | 0/1 | ||
| Two-stream Bi-LSTM [ 41 ] | 2022 | 21.4 | 0/1 | ||
| DSFormer [ 40 ] | 2024 | 9.4 | 0/1 | ||
| MSBLS [ 42 ] | 2024 | 9.3 | 0/1 | ||
| N-CMAPSS | CART-Net [ 61 ] | 2025 | 43.2 | 38.1 | 1/1 |
| Model | Access | Harness, runs |
| GPT-6 Astra | API | Codex 0.156.1 a , Sep–Oct 2026 |
| GPT-6 Luna | API | Codex 0.156.1 a , Sep–Oct 2026 |
| Kimi K3 | API | Codex 0.156.1, Sep 2026 |
| DeepSeek V4.1-Flash | API | Codex 0.156.1, Sep 2026 |
| Qwen3.6-35B-A3B | 4-bit (IQ4_XS) | OpenCode 1.19.0, Oct 2026 |
| Gemma 4 26B-A4B | 4-bit (IQ4_XS) | OpenCode 1.19.0, Oct 2026 |
| Toolkit | Agent | Datasets (folds) | |
| AWS predictive-maintenance skill | Luna | 24 (58) | 2.4 [ 8.7, +3.5] |
| tsfresh features | Luna | 24 (58) | +1.2 [ 3.4, +5.7] |
| Predictive-maintenance server | Luna | 9 (25) | 1.0 [ 9.7, +8.8] |
| PyOD | Luna | 4 (10) | +2.2 [ 5.5, +12.1] |
| ProgPy | Luna | 15 (35) | 0.9 [ 4.9, +3.2] |
| Own feature components | Luna | 24 (58) | +1.7 [ 3.3, +7.4] |
| Original | Random | Better of | AutoGluon | ||
| Method | (RF+CNN)/2 | forest | 1D CNN | the two | + features |
| Astra | 43.6 (1) | 42.3 (1) | 42.2 (1) | 37.0 (1) | 37.1 (1) |
| Kimi | 41.5 (2) | 40.8 (2) | 39.9 (2) | 34.2 (2) | 32.8 (2) |
| DeepSeek | 36.9 (3) | 35.9 (3) | 36.3 (3) | 30.8 (3) | 29.8 (3) |
| Luna | 32.1 (4) | 30.3 (4) | 31.5 (4) | 25.1 (4) | 27.2 (4) |
| Qwen3.6 | 14.7 | 13.7 | 15.9 | 12.0 | 13.1 |
LLM Agent Evaluation
Large language model agent evaluation measures whether tool-using systems complete tasks reliably, safely, and efficiently.
See all 1,313 papers