Predictive maintenance in Industry 4.0 refers to using data from sensors, machines, and production systems to estimate when equipment is likely to fail, so maintenance can be planned before a breakdown occurs [1]. However, a model that predicts maintenance may work perfectly in the lab but fail unexpectedly when applied to real factory data [2]. To solve this "reliability" gap, we evaluated six deep learning architectures across more than 700 experimental runs. We focused on the two dominant approaches in the field: Recurrent Neural Networks (RNNs), which process data step-by-step, like reading a sentence [3], and Transformers, a recent dominant approach, which look at the entire sequence at once to spot important connections [4]. We examined whether Transformers still outperform recurrent neural networks (RNNs) when the data includes noise [5]. We found that while Transformers excelled at tracking stable, slow-moving processes, they tend to overreact to chaotic data, mistakenly taking sensor noise for meaningful signals [6]. We also found that the hybrid method that combines a Long Short-Term Memory (LSTM) layer with a Transformer layer is more resilient to noisy data from factory shops [7]. Functioning as a noise filter, the LSTM smooths out data volatility, allowing the Transformer to focus on the bigger picture without being distracted [8]. The hybrid model did not just improve accuracy; it proved to be significantly more consistent than complex models, delivering reliable predictions regardless of how chaotic the underlying system became.
Figures & tables
Fig. 1: ACF Comparison. (a) Manufacturing data show high inertia. (b) IoT data show limited temporal dependence.
Fig. 2: Predictability Comparison. (a): The tight diagonal relationship indicates a predictable system. (b): The scattered cloud indicates a noisy, chaotic system.
Fig. 3: Stability Comparison. (a): Manufacturing data show high state persistence. (b): IoT data show state degradation.
Fig. 4: Architecture Comparison. Data flow (Bottom-to-Top) for the six evaluated models. (a) CNN (b) LSTM (c) CNN-LSTM (d) Pure Attention. (e) Our proposed hybrid (LSTM+Attention). (f) Parallel “Wide & Deep”.
Model
R2 Score (Mean ± Std)
RMSE
MAE
Transformer
0.644 ± 0.002
2.392
1.938
LSTM-Trans
0.644 ± 0.002
2.393
1.943
LSTM
0.643 ± 0.002
2.396
1.943
CNN-LSTM
0.642 ± 0.003
2.399
1.946
Tri-Hybrid
0.640 ± 0.003
2.406
1.951
CNN
0.633 ± 0.005
2.428
1.967
TABLE I: Manufacturing Dataset Results (20 Runs)
Fig. 5: Stability Analysis. The boxplots reveal that LSTM or Attention-based models have significantly lower variance than the Convolutional baseline.
Fig. 6: Learning Curves. Comparison of training (Blue) and validation (Orange) loss. All models show stable convergence with minimal overfitting.
Fig. 7: Prediction Analysis (Best Run). While all models track well in their best attempts, the CNN lacks the consistency of memory-based models across repeated trials.
Machine
Model
R2 (Mean ± Std)
RMSE
MAE
M001
LSTM
0.450 ± 0.004
8.054
7.612
(Chaotic)
Tri-Hybrid
0.448 ± 0.006
8.066
7.614
LSTM-Trans
0.448 ± 0.005
8.067
7.591
CNN-LSTM
0.446 ± 0.006
8.083
7.615
Transformer
0.443 ± 0.005
8.102
7.653
ResNet
0.437 ± 0.008
8.143
7.679
TABLE II: Model Performance by Machine Type
Fig. 8: Analysis of Chaotic Regime (M001). (a) Only LSTM-based models maintain tight variance; others show significant instability. (b) In the “Best Run” scenario, performance is visually identical, highlighting the need for stability metrics.
Model
R2 Score (Mean ± Std)
RMSE
MAE
LSTM-Trans
0.472 ± 0.004
7.823
7.323
CNN-LSTM
0.472 ± 0.003
7.825
7.330
Tri-Hybrid
0.472 ± 0.004
7.828
7.334
LSTM
0.471 ± 0.002
7.833
7.347
Transformer
0.471 ± 0.004
7.837
7.350
ResNet
0.470 ± 0.004
7.843
7.345
TABLE III: Global Dataset Results
Fig. 9: Analysis of Global Test. (a) The mixed dataset increases variance for all models, but the LSTM-Transformer remains the most stable. (b) The “Best Run” plots are virtually identical, confirming that capacity is not the bottleneck—consistency is.
School of Software Engineering, Sun Yat-Sen University, China · Agency for Science, Technology and Research (A*STAR), Singapore · School of the Art Institute of Chicago, Chicago +1