Homomorphic Advantage Operator: Stabilizing Reinforcement Learning Under Fully Homomorphic Encryption Constraints

Abid Mohamed Nadhir; Ahmad Al Hanbali; Beggas Mounir

cs.AIOct 1, 2026

Homomorphic Advantage Operator: Stabilizing Reinforcement Learning Under Fully Homomorphic Encryption Constraints

Authors: Abid Mohamed Nadhir, Ahmad Al Hanbali, Beggas Mounir

Organizations: El Oued University, El Oued, Algeria · King Fahd University of Petroleum and Minerals (KFUPM), Dhahran, Saudi Arabia · Interdisciplinary Research Center For Smart Mobility and Logistics, KFUPM, Dhahran, Saudi Arabia

Abstract

Privacy-preserving machine learning presents significant deployment challenges on the cloud for intelligent systems with confidential data. Fully Homomorphic Encryption (FHE) offers a compelling solution for secure computation, preserving data confidentiality of cloud computations. However, applying FHE to reinforcement learning (RL) requires replacing non-linear operations with polynomial approximations, which diverge catastrophically due to a unique recursive error phenomenon known as the Bellman drift. This article introduces the Homomorphic Advantage Operator (HAO), a stabilization framework designed to prevent polynomial approximation divergence in FHE-based deep RL. HAO adapts the zero-mean centering projection from advantage-based value estimation directly to temporal-difference (TD) targets. This linear projection annihilates the uniform state-value baseline that drives the Bellman drift, maintaining per-state action rankings while requiring zero additional non-linear multiplicative depth and avoiding expensive ciphertext bootstrapping. The proposed HAO framework was evaluated using a three-tier experimental methodology, including a tabular Markov Decision Process (MDP), an encrypted CartPole environment using real CKKS cryptographic operations, and a 20-node logistics routing benchmark with dense continuous features. The results demonstrate that the proposed HAO strictly bounds network pre-activations within the safe polynomial approximation domain. The proposed HAO RL agents achieved 0% boundary breaches across all random seeds used, whereas regularization alone (L2 weight decay and gradient clipping) breached the bound on 3 of 5 seeds and the unstabilized baseline did so in 83.8% of episodes. Finally, HAO agents improve optimal policy accuracy by 18.0 percentage points in tabular domains and remain stable when DP-SGD-style Gaussian noise is added to the clipped gradients.

Figures & tables

#	Solution	Reference	HE	Paradigm	Mechanism Summary
Training-Time Stabilization (Supervised)
1	Boundary Loss	Araki et al. [ 3 ]	LHE	Supervised	Exponential penalty on $\|z\|>B$ ; prevents explosion during training
2	Selective Grad. Clipping	Araki et al. [ 3 ]	LHE	Supervised	Clips gradients excluding BatchNorm $\gamma$ , $\beta$ ; preserves normalization
3	SLAF	Pulido-Gaytan & Tchernykh [ 27 ]	LHE	Supervised	Trainable polynomial coefficients; learns task-specific activations
4	HE-Friendly Training	Baruch et al. [ 4 ]	LHE	Supervised	Knowledge distillation + polynomial BN folding
5	Weight Clamping / L2	Standard technique	Any	Supervised	Constrains weight norms; bounds pre-activations indirectly

Table 1: Taxonomy of existing solutions for polynomial activation and privacy-preserving learning under HE. PHE = Partially HE, LHE = Leveled HE (no bootstrapping), FHE = Fully HE (with bootstrapping).

Figure 1: High-level overview of the HAO framework for privacy-preserving reinforcement learning. The system integrates three interdependent components: (1) CKKS-encrypted state transmission and polynomial forward evaluation, (2) the HAO centering projection that neutralizes Bellman drift, and (3) clipped weight updates with optional DP-SGD-style Gaussian noise.

Panel of Figure 2 — Figure 2: HAO client-server architecture for privacy-preserving RL. The client encrypts private states via CKKS, the server computes an encrypted forward pass with polynomial activations, and the client applies HAO centering after decryption. Weight updates can be perturbed with DP-SGD-style Gaussian noise. The HAO centering (gold) is the core contribution: it removes the $V(s)$ Bellman drift without any ciphertext–ciphertext multiplication.

Experiment	Environment	Architecture	Training Parameters	Evaluated Modes
1: Tabular MDP	10 states, 4 actions	1 Hidden Layer (244 params)	2000 epochs, 5 random seeds	Standard, Full HAO, Centering Only
2: Encrypted CartPole	Hybrid client-server using real TenSEAL CKKS operations	1 Hidden Layer (114 params)	250 episodes	Standard vs. HAO LHE
3: Logistics Ablation	20-node routing, 8D continuous states, 6 actions, $\alpha=0.05$	1 Hidden Layer (486 params)	1000 eps, 5 seeds, shift at ep 500	Full HAO, Centering Only, Reg Only, No Stab., HAO+DP

Table 2: Summary of Experimental Setup Methodology.

Figure 3: Pre-activation growth over 2000 epochs. Centering successfully bounds polynomial inflation and stabilizes training.

Figure 4: Gradient norm dynamics on a logarithmic scale over 2000 epochs. The uncentered baseline exhibits exponential gradient explosion upon breaching the polynomial boundary, whereas HAO centering compresses gradient variance.

Mode	Accuracy (%)	Max $\\|z\\|_{\infty}$	Grad Norm
Standard	$68.0\pm$ 8.4	$1.83\pm 0.20$	$5.04\pm 0.34$
Full HAO	86.0 $\pm$ 5.5	0.74 $\pm$ 0.07	0.06 $\pm$ 0.00
Centering Only	86.0 $\pm$ 5.5	0.70 $\pm$ 0.08	0.01 $\pm$ 0.00

Table 3: Tabular MDP (2000 epochs, 5 seeds)

Figure 5: Trapping region (pre-activation trajectory) for the hybrid TenSEAL CKKS CartPole agent. The network utilizes a single hidden layer architecture.

Figure 6: Optimization health (gradient norm) over 250 episodes (log scale). Standard LHE diverges exponentially.

Mode	Max $\\|z\\|_{\infty}$	Max $\\|\nabla\theta\\|$	Max.	Avg.
			Reward	Q $\cdot$ P time (s)
Standard LHE	$28.68$	$25,003$	$28.0$	N/A
HAO LHE	3.61	2.88	93.0	1.64

Table 4: CKKS Encrypted CartPole (250 episodes). Q

\cdot

P time is the mean time per episode of the encrypted centering step.

Figure 7: Pre-activation growth over 1,000 episodes. The HAO framework strictly bounds pre-activations below

B=5.0

, whereas uncentered baseline agents diverge under the environment shift.

Figure 8: Agent performance (moving average reward) over 1,000 episodes in the 20-node Logistics environment. Full HAO maintains optimal performance post-shift.

Mode	Max $\\|z\\|_{\infty}$	Breach %	FHE-Deploy?
Full HAO	3.89	0.0%	Yes
Centering Only	7.79	54.5%	No
Regularization Only	6.98	23.8%	No
No Stabilization	7.82	83.8%	No
HAO + DP Moderate	4.12	0.0%	Yes
HAO + DP Strong	4.01	0.0%	Yes

Table 5: Realistic Logistics (

\gamma=0.99

\alpha=0.05

, 5 seeds)

Figure 9: Pre-activation robustness under DP noise. HAO successfully bounds the trajectory despite strong DP noise.

Figure 10: Agent performance with DP noise injected. The privacy guarantee trades off only a marginal reduction in reward output.

$\alpha\rightarrow$	$\mathbf{0.01}$	$\mathbf{0.02}$	$\mathbf{0.05}$	$\mathbf{0.08}$	$\mathbf{0.10}$	$\mathbf{0.25}$
Reg. Only	2.14	4.91	Breach	Breach	Breach	Breach
Full HAO	1.25	1.87	2.95	4.88	Breach	Breach

Table 6: Maximum Pre-Activation (

\|z\|_{\infty}

) Across Reward Scale (

\alpha

) Sweeps. Values

>5.0

trigger divergence.

Explore similar work

Revisiting ML Training under Fully Homomorphic Encryption: Convergence Guarantees, Differential Privacy, and Efficient Algorithms
May 27, 2026Yvonne Zhou, Mingyu Liang, Ivan Brugere +5Homomorphic Encryption Convergence
University of Maryland, College Park, MD 20742 · J.P. Morgan AI Research, New York, NY, 10017 · AlgoCRYPT CoE
HEAT: Faster Fully Homomorphic Inference via Approximations-Weights Co-Adaptation
Sep 1, 2026Alessandro Zirilli, Davide Marincione, Evgenios M. Kornaropoulos +2Homomorphic Encryption Newton
Sapienza University of Rome · George Mason University · Paradigma
Decision-Aware Quadratic ReLU Replacement for HE-Friendly Inference
May 21, 2026Rui Li, Wenyuan Wu, Weijie MiaoHomomorphic Encryption Rectified Linear Unit
Chongqing Key Laboratory of Secure Computing for Biology, Chongqing Institute of Green and Intelligent Technology, Chinese Academy of Sciences, 266 Fangzheng Avenue, Beibei District, Chongqing 400714, China · Department of Industrial and Systems Engineering, The Hong Kong Polytechnic University, Hung Hom, Hong Kong, China

May 27, 2026, cs.LGJ/K move · Enter open · S save

Revisiting ML Training under Fully Homomorphic Encryption: Convergence Guarantees, Differential Privacy, and Efficient Algorithms

Yvonne Zhou, Mingyu Liang, Ivan Brugere, Danial Dervovic+4

University of Maryland, College Park, MD 20742 · J.P. Morgan AI Research, New York, NY, 10017 · AlgoCRYPT CoE

Homomorphic Encryption Convergence Loss Function

Open paper AskPDF arXiv

We present the first theoretical convergence analysis of machine learning training under fully homomorphic encryption (FHE), combined with a differentially private (DP) training algorithm tailored to encrypted computation. Our approach improves computational efficiency over standard differentially private gradient descent (DP-GD) while achieving comparable utility. In particular, we prove convergence of approximate gradient descent using polynomial approximations of activation and loss functions, which are required for FHE compatibility. To preserve privacy in downstream tasks, we integrate differential privacy without relying on costly per-sample gradient clipping, enabling scalable encrypted learning. We also provide data-independent hyperparameter selection and theoretically grounded strategies for polynomial approximation which can be of independent interest. Together, these contributions advance the feasibility of efficient, private, and secure machine learning on sensitive data.

Abstract

Explore similar work

Revisiting ML Training under Fully Homomorphic Encryption: Convergence Guarantees, Differential Privacy, and Efficient Algorithms

HEAT: Faster Fully Homomorphic Inference via Approximations-Weights Co-Adaptation

Decision-Aware Quadratic ReLU Replacement for HE-Friendly Inference