Delay-Guided Attention for Quality Prediction in Multistage Manufacturing
Authors: Jiarui Zhang, Zhaoyang Guan, Zhixuan Xia, Naichen Shi
Organizations: Department of Engineering Sciences and Applied Mathematics, Northwestern University, Evanston, IL, USA · Department of Industrial Engineering and Management Sciences, Northwestern University, Evanston, IL, USA
Quality prediction in multistage manufacturing is challenging when inter-stage lags vary over time, as measurements taken at the same timestamp may reflect different process states. Existing approaches often preserve production order but fuse timestamp-aligned features or model delayed dependencies implicitly, without identifying the upstream history relevant to each downstream observation. We propose Delay-Guided Attention for Multi-stage Forecasting (DAMF), which explicitly estimates inter-stage lag distributions and uses them to guide cross-stage information transfer for final quality prediction. For each adjacent stage pair, a lag predictor compares the current downstream representation with candidate lagged upstream representations. An occurrence-magnitude factorization separately estimates the probability of a positive lag and its conditional magnitude, yielding a sample- and stage-pair-specific distribution over discrete lag states without requiring plant-specified delays at inference. DelayAlignment incorporates this soft distribution as a temporal prior into differentiable cross-stage attention, while gated residual fusion regulates information propagation across production stages. For interpretation, segment-wise Viterbi decoding converts pointwise lag distributions into temporally coherent lag paths, whereas forecasting retains the soft distributions for alignment. On the real-world LiquidSugar dataset, DAMF improves final quality prediction over the compared baselines. Evaluations on two controlled-delay benchmarks show more accurate and stable lag recovery, and ablation studies support the contributions of lag prediction, delay-guided attention, and structured temporal decoding.
Figures & tables
Figure 1: Schematic illustration of the sequential multi-stage Liquid Sugar production process.
Figure 2: Overall architecture of DAMF. Edge-wise lag distributions guide DelayAlignment and gated residual fusion across successive processing stages.
Figure 3: Illustration of an inter-stage lag. An upstream process feature observed at t−k corresponds to a downstream feature observed at t , giving a lag of k sampling steps.
Figure 4: Transition-score structure for lag-path decoding under the experimental parameter settings. Darker cells indicate higher scores and smaller penalties.
Figure 5: Lag-guided alignment and fusion for an adjacent-stage pair. DelayAlignment combines content-based matching with the estimated lag distribution to compute attention weights over upstream candidates. The resulting upstream message is combined with the current-stage representation through gated residual fusion.
Figure 6: Lag-recovery comparison on the two controlled-delay benchmarks. Each panel reports one lag-recovery metric for MNet, WCC, and DAMF. MNet uses the hidden-lag output of MultistageNet, WCC combines window-based cross-correlation with a conventional Viterbi decoder, and DAMF denotes the proposed lag-recovery pipeline and is highlighted in red.
Figure 7: Representative lag-recovery examples on the controlled-delay benchmarks. The blue curves denote the ground-truth lag paths, while the orange curves and shaded regions represent the mean decoded paths and their ranges across random seeds, respectively. Each benchmark contains both block-shaped and bump-shaped delay patterns.
Setting
Occ F1 ↑
Lag MAE ↓
±1 Acc ↑
Large-lag Recall ↑
Collapse to 0 ↓
Full lag-prediction objective
0.91 ± 0.02
0.65 ± 0.13
0.90 ± 0.02
0.76 ± 0.05
0.03 ± 0.03
Occurrence + positive CE only
0.90 ± 0.02
0.82 ± 0.15
0.87 ± 0.02
0.68 ± 0.05
0.20 ± 0.09
w/o relaxed-TopN loss
0.89 ± 0.04
0.94 ± 0.15
0.86 ± 0.02
0.76 ± 0.13
0.10 ± 0.08
w/o positive-lag CE loss
0.64 ± 0.11
2.77 ± 0.08
0.60 ± 0.01
0.14 ± 0.02
0.65 ± 0.04
w/o large-occurrence recall loss
0.83 ± 0.07
1.32 ± 0.23
0.82 ± 0.03
0.62 ± 0.16
0.23 ± 0.17
w/o selector loss
0.86 ± 0.05
0.91 ± 0.39
0.85 ± 0.05
0.65 ± 0.15
0.12 ± 0.09
Table 8
Setting
Acc ↑
±1 Acc ↑
Lag MAE ↓
Large-lag Recall ↑
Collapse to 0 ↓
Collapse to small ↓
Raw argmax
0.65 ± 0.04
0.79 ± 0.04
1.03 ± 0.12
0.55 ± 0.04
0.02 ± 0.02
0.44 ± 0.03
Viterbi
0.76 ± 0.05
0.90 ± 0.02
0.65 ± 0.13
0.76 ± 0.05
0.03 ± 0.03
0.22 ± 0.03
Table 2: Effect of Viterbi decoding on lag recovery. The best numerical result for each metric is shown in bold.
Figure 8: Path-level comparison between raw pointwise argmax predictions and Viterbi decoding on representative test regions. The left column shows raw argmax predictions before decoding, whereas the right column shows the lag paths after Viterbi decoding. The blue curve denotes the ground-truth lag path, the orange curve denotes the prediction mean across random seeds, and the shaded region denotes the corresponding prediction range.
Figure 9: Sensitivity analysis of the lag prior weight. The four panels report RMSE, Δ RMSE, MAE, and R2 under different prior-weight settings.
Appendix figures & tables2 assets
Supplementary material from the paper’s appendix.
Appendix
Figure A1: Lag recovery results of the edge-level lag predictor on two controlled-delay datasets. The blue line denotes the ground-truth lag path, the orange line denotes the mean Viterbi-decoded lag path across seeds, and the shaded region denotes the corresponding range across seeds 42, 43, and 44.
Figure B1: Effect of Viterbi decoding on lag-path recovery across five test regions. The left and right columns show the raw point-wise lag predictions and the corresponding Viterbi-decoded lag paths, respectively. Solid colored curves and shaded bands denote the mean and range across three random seeds, while blue curves represent the ground-truth lag paths. Vertical dashed lines delimit the evaluation intervals, and the reported MAE is computed within each interval.
In data-driven predictive maintenance (PdM), feature extraction is usually treated as fixed preprocessing: a descriptor set is chosen once and reused while the downstream model or forecasting horizon changes. This paper isolates the representation-learning stage and presents a quantile-led feature-extraction framework based on a dual-stage MLP-QRNN hierarchy. QRNN1 learns a broad ten-quantile conditional distribution for each sensor channel, while skip-connected QRNN2 refines a retained mid-tail quantile set into compact, channel-resolved, distribution-aware features. A fixed thirteen-pipeline ablation spans 1-hour, 70-hour, and 30-day regimes across 72 machines in 9 industrial facilities, with the downstream temporal classifier held fixed within each regime. Increasing the retained mid-tail set from two to four quantiles improves 30- and 60-minute F1-score, reaching 75.92% and 72.44% with attention enabled. The results also show that representations do not transfer reliably beyond their design horizon unless feature capacity, temporal embedding, activation strategy, and sensor breadth are scaled with the forecasting task. The unmodified short-horizon extractor falls to 42.90% F1 at 70 hours, whereas horizon-conditioned extractors reach 60.38% at 70 hours and 79.97% at 30 days. The framework therefore supports treating PdM feature extraction as a horizon-dependent representational stage rather than fixed preprocessing.
David J Poland, Daniele Ravi, Na Helian
Department of Computer Science, University of Hertfordshire, College Lane, Hatfield, AL10 9AB, United Kingdom
Long-horizon predictive maintenance requires models to distinguish slowly evolving degradation from normal operating-regime variation over planning windows measured in days rather than hours. This paper evaluates whether an explicit conditional-quantile representation provides an informative classifier interface for this problem. The proposed TQRNN30d framework combines a dual-stage quantile regression neural network (QRNN) feature extractor with a multi-stream temporal fusion classifier. Each hourly word of 81-channel machine behaviour is mapped to a 324-dimensional quantile-state representation, and 720 ordered hourly words form the 30-day document supplied to the long-horizon model. The classifier fuses quantile states with dynamic covariates, channel-level static metadata, and a 168-hour latent-history stream using gated residual processing, causal recurrent encoding, and metadata-conditioned cross-modal attention. A bounded instability-aware signal derived from sustained one-word-ahead prediction-error divergence provides auxiliary memory modulation at the longest horizon. Evaluation uses a machine-disjoint 43/14/15 train/validation/test allocation across 72 machines in nine manufacturing facilities. At 30 days, TQRNN30d achieves 79.97% F1, 80.18% recall, 81.82% precision, 82.39% accuracy, and 0.820 ROC-AUC. It leads all 18 evaluated baselines at the 7-, 14-, and 30-day fixed-threshold comparisons, with the largest F1 advantage at 14 days. The results support held-out-machine performance within the observed homogeneous nine-facility fleet, but do not establish unseen-site, cross-equipment, or cross-sector generalisation.
David J Poland, Daniele Ravi, Na Helian
Department of Computer Science, University of Hertfordshire, Hatfield AL10 9AB, U.K.
Flight delay prediction has become a key focus in air traffic management (ATM), as delays reflect inefficiencies in the system. This paper proposes LLM4Delay, a large language model (LLM)-based framework for predicting flight delays from the perspective of air traffic controllers monitoring aircraft after they enter the terminal maneuvering area (TMA). LLM4Delay is designed to integrate textual aeronautical information, including flight data, weather reports, and aerodrome notices, together with multiple trajectories that model airspace conditions, forming a comprehensive delay-relevant context. By jointly leveraging comprehensive textual and trajectory contexts via instance-level projection, an effective cross-modality adaptation strategy that maps multiple instance-level trajectory representations into the language modality, the framework improves delay prediction accuracy. LLM4Delay demonstrates superior performance compared to existing ATM frameworks and prior time-series-to-language adaptation methods. This highlights the complementary roles of textual and trajectory data while leveraging knowledge from both the pretrained trajectory encoder and the pretrained LLM. The proposed framework enables continuous updates to predictions as new information becomes available, indicating potential operational relevance.