Physics-Aware Machine Unlearning for Cyber-Physical Systems

Deep reinforcement learning (DRL) offers powerful control for industrial cyber-physical systems (ICPSs), but its "black-box" exploration risks violating strict hardware safety limits. Typically, these constraints are managed through complex reward shaping. In this work-in-progress paper, we embed a differentiable physics model directly into the proximal policy optimization (PPO) actor loss function. By simulating short-horizon future trajectories during training, the policy is penalized for anticipated safety violations independent of the task-reward signal. Evaluated on a simulated 1-degree-of-freedom helicopter testbed with strict pitch constraints, our physics-informed soft regularizations substantially reduce constraint violations while maintaining reliable target tracking.

Reinforcement learning (RL) enables the synthesis of control policies directly from data, making it highly appealing for complex cyber-physical systems (CPSs) and robotics. A persistent challenge, however, is ensuring strict, hard safety constraints during the active learning phase. In real-world physical systems, violating mechanical limits can cause irreversible damage, necessitating that exploration remains strictly within safe operational regions. We propose a generalized framework that combines the adaptive, high-performance nature of deep reinforcement learning (DRL) with the formal safety guarantees of model predictive control (MPC). Using a mathematical model of the system dynamics, offline MPC computations define a feasible state-action space, representing all safe combinations of system states and control inputs that guarantee constraint satisfaction. During training and deployment, the RL agent's instantaneous actions are projected onto this globally verified feasible set via a safety filter. We systematically evaluate our generalized approach on a non-linear 1-DoF laboratory testbed, demonstrating successful exploration and stable policy convergence on physical hardware.

Power System State Estimation (PSSE) converts geographically distributed measurements into the voltage magnitudes and phase angles needed for grid monitoring and control. Learned estimators can perform this mapping rapidly, but model-aware False Data Injection Attacks (FDIAs) may corrupt their inputs while retaining AC plausibility and residual-based stealth. Physics-Informed Neural Networks (PINNs) limit candidate states through power-flow consistency; however, their robustness depends on balancing supervised and physics losses whose scales and gradient contributions evolve during training. This paper proposes a PINN that jointly learns homoscedastic uncertainty parameters and uses them to adapt the two objectives. The formulation assigns trainable log-uncertainties to the active-power, reactive-power, voltage, and angle losses while safeguarding against an underweighted physics objective. The estimator is trained only on clean steady-state data and is evaluated, without adversarial retraining, under baseline state-distortion and stricter residual-profile-matching regimes on the IEEE~118-bus system. Accuracy is measured against the uncompromised system state. Relative to a fixed-weight PINN, dynamic weighting reduces overall Mean Absolute Error (MAE) by $55\%$ while also improving voltage- and angle-estimation accuracy. The results show that learning the physics/data balance from clean data improves robustness to unseen FDIAs.