Uncertainty-Guided Label Rebalancing for CPS Safety Monitoring

John Ayotunde; Qinghua Xu; Guancheng Wang; Lionel C. Briand

cs.LGMar 26, 2026

Uncertainty-Guided Label Rebalancing for CPS Safety Monitoring

Authors: John Ayotunde, Qinghua Xu, Guancheng Wang, Lionel C. Briand

Organizations: Lero Research Ireland Centre for Software Research, University of Limerick, Castletroy, Limerick, Ireland. · the University of Ottawa, Canada, and Lero Research Ireland Centre for Software Research, University of Limerick, Ireland.

Abstract

Safety monitoring is essential for Cyber-Physical Systems (CPSs). However, unsafe events are rare in real-world CPS operations, creating an extreme class imbalance that degrades safety predictors. Standard rebalancing techniques perform poorly on time-series CPS telemetry, either generating unrealistic synthetic samples or overfitting on the minority class. Meanwhile, behavioral uncertainty in CPS operations, defined as the degree of doubt or uncertainty in CPS decisions , is often correlated with safety outcomes but unexplored in safety monitoring. To that end, we propose U-Balance, a supervised approach that leverages behavioral uncertainty to rebalance imbalanced datasets prior to training a safety predictor. U-Balance first trains a GatedMLP-based uncertainty predictor that summarizes each telemetry window into distributional kinematic features and outputs an uncertainty score. It then applies an uncertainty-guided label rebalancing (uLNR) mechanism that probabilistically relabels $\textit{safe}$ -labeled windows with unusually high uncertainty as $\textit{unsafe}$ , thereby enriching the minority class with informative boundary samples without synthesizing new data. Finally, a safety predictor is trained on the rebalanced dataset for safety monitoring. We evaluate U-Balance on a large-scale UAV benchmark with a 46:1 safe-to-unsafe ratio. Results confirm a moderate but significant correlation between behavioral uncertainty and safety. We then identify uLNR as the most effective strategy to exploit uncertainty information, compared to direct early and late fusion. U-Balance achieves a 0.806 F1 score, outperforming the strongest baseline by 14.3 percentage points, while maintaining competitive inference efficiency. Ablation studies confirm that both the GatedMLP-based uncertainty predictor and the uLNR mechanism contribute significantly to U-Balance's effectiveness.

Figures & tables

Fig. 2: Examples of uncertainty and safety in UAV operations.

Fig. 3: Overview of the Uncertainty Predictor (①).

Fig. 5: Overview of the Safety Predictor (③)

Strategy	Precision	Recall	F1	p -value	$\mathbf{\hat{A}_{12}}$
uLNR (ours)	0.792 $\pm$ 0.057	0.822 $\pm$ 0.033	0.806 $\pm$ 0.031	—	—
Plain	0.873 $\pm$ 0.052	0.498 $\pm$ 0.025	0.633 $\pm$ 0.022	$<$ 0.001	1.000 (L)
Early Fusion	0.851 $\pm$ 0.030	0.520 $\pm$ 0.017	0.645 $\pm$ 0.014	$<$ 0.001	1.000 (L)
Late Fusion	0.848 $\pm$ 0.036	0.526 $\pm$ 0.016	0.648 $\pm$ 0.009	$<$ 0.001	1.000 (L)

TABLE I: Comparison of uncertainty integration strategies, including uLNR (ours), Plain (without uncertainty), Early Fusion , and Late Fusion .

	Effectiveness						Efficiency
Method	Precision	Recall	F1	$\Delta$ F1	p -value	$\mathbf{\hat{A}_{12}}$	#Params	Lat.(s)
U-Balance	0.792 $\pm$ 0.057	0.822 $\pm$ 0.033	0.806 $\pm$ 0.031	—	—	—	285.9K	0.0045
Deep Learning Methods
TFT	0.827 $\pm$ 0.078	0.484 $\pm$ 0.030	0.608 $\pm$ 0.029	+0.197	$<$ 0.001	1.000 (L)	106.7K	0.0026
TimeMoE	0.909 $\pm$ 0.026	0.522 $\pm$ 0.012	0.663 $\pm$ 0.010	+0.143	$<$ 0.001	1.000 (L)	113.4M	0.0045
Superialist	0.115 $\pm$ 0.002	0.366 $\pm$ 0.012	0.175 $\pm$ 0.004	+0.631	$<$ 0.001	1.000 (L)	4.1K	0.3429

TABLE II: Effectiveness and efficiency of U-Balance and baselines.

\Delta

F1 shows the difference in F1 between U-Balance and each baseline. #Params is the number of model parameters. ”–” denotes non-parametric models. Lat. (s) represents the per-sample inference time in seconds.

Preprocess	Architecture	Precision	Recall	F1	p -value	$\mathbf{\hat{A}_{12}}$
Yes	GatedMLP	0.792 $\pm$ 0.057	0.822 $\pm$ 0.033	0.806 $\pm$ 0.031	—	—
Yes	Plain MLP	0.713 $\pm$ 0.058	0.757 $\pm$ 0.053	0.732 $\pm$ 0.042	$<$ 0.001	0.944 (L)
No	Bi-LSTM	0.753 $\pm$ 0.090	0.690 $\pm$ 0.126	0.707 $\pm$ 0.066	$<$ 0.001	0.902 (L)
No	Bi-GRU	0.727 $\pm$ 0.099	0.688 $\pm$ 0.087	0.697 $\pm$ 0.051	$<$ 0.001	0.972 (L)
No	Bi-RNN	0.769 $\pm$ 0.080	0.652 $\pm$ 0.116	0.695 $\pm$ 0.068	$<$ 0.001	0.938 (L)

TABLE III: Ablation study of the uncertainty predictor under different preprocessing settings and model architectures.

$\bm{\tau}$	Labels flipped	Flip ratio	Final ratio	Precision	Recall	F1
0.5	7557	11.0%	12.2%	0.467	0.907	0.616
1.0	6602	9.6%	10.8%	0.500	0.900	0.643
1.5	5456	8.0%	9.2%	0.547	0.896	0.679
2.0	4146	6.1%	7.3%	0.608	0.878	0.718
2.5	2653	3.9%	5.1%	0.738	0.849	0.790
3.0	1140	1.7%	2.9%	0.792	0.822	0.806

TABLE IV: Ablation study of the flip threshold

\tau

in uLNR . Labels flipped is the number of safe-labelled windows relabelled as unsafe. Flip ratio is the proportion of training samples relabelled. Final ratio is the proportion of unsafe samples in the training set after relabelling.

Strategy	Precision	Recall	F1	p -value	$\mathbf{\hat{A}_{12}}$
uLNR	0.792 $\pm$ 0.057	0.822 $\pm$ 0.033	0.806 $\pm$ 0.031	—	—
Data-level
SMOTE	0.559 $\pm$ 0.170	0.581 $\pm$ 0.090	0.556 $\pm$ 0.102	$<$ 0.001	1.000 (L)
Borderline-SMOTE	0.524 $\pm$ 0.163	0.594 $\pm$ 0.051	0.540 $\pm$ 0.094	$<$ 0.001	1.000 (L)
T-SMOTE	0.898 $\pm$ 0.028	0.541 $\pm$ 0.014	0.675 $\pm$ 0.009	$<$ 0.001	1.000 (L)
ADASYN	0.623 $\pm$ 0.189	0.613 $\pm$ 0.052	0.596 $\pm$ 0.099	$<$ 0.001	1.000 (L)

TABLE V: Comparison of uLNR against SOTA rebalancing strategies.

Class-Imbalanced Learning

Class-imbalanced learning develops classifiers that recognize rare classes without sacrificing performance on common ones.

See all 256 papers

Oct 2, 2026

Positive-Unlabeled Learning for Agent Safety False Alarm Auditing

Safety monitors help safeguard language-model agents interacting with external tools and environments, but conservative monitoring can generate many false alarms, consuming extensive review resources and weakening trust in alerts. Because false and genuine alarms often remain interleaved in native monitor scores, obtaining a reliable cutoff still requires substantial manual verification. In practice, a small set of verified-safe non-alarmed trajectories may be available while alarms remain unlabeled, naturally casting false-alarm auditing as a positive-unlabeled (PU) ranking problem. The key challenge is monitor-induced selection, since observed safe references are accepted by the monitor, while the hidden safe alarms of interest are precisely those it incorrectly flags, making the observed positives poorly representative of the positives to be recovered. To address this challenge, we propose a two-stage framework in which Trust-aware PU Supervision adapts safe references toward the alarm domain and protects plausible false alarms from excessive negative pressure, while Reliability-gated Rank Distillation consolidates consistent ordering preferences from multiple PU reference models into a single student. Consensus-guided Structural Refinement then improves the student ranking using hierarchical safe-reference support, alarm relations, and predicted reference consensus. The framework requires no alarm safety labels for fitting and leaves the underlying monitor unchanged. Across mainstream safety monitors, our method achieves a macro AUPRC of

0.6444

, outperforming eight evaluated PU baselines by 5.27--16.98 absolute percentage points; compared with PULDA, the strongest evaluated PU baseline, it recovers 33.3% more false alarms at a 5% review budget.

Xichen Yan, Chongyang Gao, Kezhen Chen +3

Jinan University · Northwestern University · Analogy AI, Inc. +3

AI Agent Safety Positive-Unlabeled Learning AI Agent Monitoring

Ask PDF

Nov 2, 2024

Uncertainty measurement for complex event prediction in safety-critical systems

Complex events originate from other primitive events combined according to defined patterns and rules. Instead of using specialists' manual work to compose the model rules, we use machine learning (ML) to self-define these patterns and regulations based on incoming input data to produce the desired complex event. Complex events processing (CEP) uncertainty is critical for embedded and safety-critical systems. This paper exemplifies how we can measure uncertainty for the perception and prediction of events, encompassing embedded systems that can also be critical to safety. Then, we propose an approach (ML_CP) incorporating ML and sensitivity analysis that verifies how the output varies according to each input parameter. Furthermore, our model also measures the uncertainty associated with the predicted complex event. Therefore, we use conformal prediction to build prediction intervals, as the model itself has uncertainties, and the data has noise. Also, we tested our approach with classification (binary and multi-level) and regression problems test cases. Finally, we present and discuss our results, which are very promising within our field of research and work.

Maria J. P. Peixoto, Akramul Azim

Department of Electrical, Computer and Software Engineering, Ontario Tech University, Oshawa, Ontario, Canada.

Uncertainty Quantification Sensitivity Analysis

Ask PDF

Jul 7, 2026

Modeling Normal Is All You Need: Joint Latent Clustering for Anomaly Detection in Multimodal Cyber-Physical Systems

Faults on a cyber-physical system (CPS) are too rare and unrepresentative to characterise, or even to select a model on, so detection must instead model normal behaviour; the standard point-adjusted evaluation, however, rewards detectors that never do. CPS normal behaviour is the union of many imbalanced, curved, thin-fringed operating regimes rather than a single blob; we state this structure as ten assumptions (A1-A10), abbreviated Massive, Implicit, Imbalanced Multimodality (MIIM). We model the normal law with a jointly learned latent representation plus explicit Gaussian-mixture mode clustering, scored in the latent rather than by a global density or a reconstruction residual, and evaluate under a deliberately fair protocol: raw point-wise metrics with no point adjustment, a trivial-detector difficulty split, prevalence-matched F1, and train-normal-only calibration. On three real CPS datasets (WADI, HAI, SKAB), the detector wins both the combined column and the difficult correlation/dynamics-fault column on all three, reaching difficult-subset AUROC 0.831 on HAI, 0.726 on WADI, and 0.610 on SKAB. The margin is largest on the two multimodal datasets the MIIM assumptions target and slimmest on the near-unimodal one, tracking multimodality as the thesis predicts, and it holds against three deep detectors (USAD, TranAD, GDN) re-computed with the same raw metrics, all of which collapse on the difficult subset. The methodological contributions are the MIIM assumption set, the difficulty-stratified fair protocol, and a latent-only score that drops reconstruction because a flexible decoder rebuilds the hard faults faithfully.

Alexander Apartsin, Yehudit Aperstein

Holon Institute of Technology (HIT), Holon, Israel · Afeka Academic College of Engineering, Tel Aviv, Israel

Multimodal Anomaly Detection Industrial Anomaly Detection Fault Detection Cyber-Physical Systems Anomaly Detection Time-Series Anomaly Detection+1

Ask PDF

Abstract

Class-Imbalanced Learning

Explore similar work

Positive-Unlabeled Learning for Agent Safety False Alarm Auditing

Uncertainty measurement for complex event prediction in safety-critical systems

Modeling Normal Is All You Need: Joint Latent Clustering for Anomaly Detection in Multimodal Cyber-Physical Systems