When to Intervene? State-Aware Sparse Manipulation in Federated Reinforcement Learning

Shutong Zheng; Sijia Chen

cs.LGOct 8, 2026

When to Intervene? State-Aware Sparse Manipulation in Federated Reinforcement Learning

Authors: Shutong Zheng, Sijia Chen

Organizations: Sun Yat-sen University · The Hong Kong University of Science and Technology (Guangzhou)

Abstract

Federated reinforcement learning (FRL) enables distributed agents to collaboratively train decision-making policies, but its decentralized training process also exposes global policy learning to Byzantine manipulation. Existing poisoning attacks primarily focus on how to construct malicious updates, while trajectory-level intervention timing remains largely implicit. In sequential decision making, however, where an intervention is applied can alter subsequent trajectories and learning signals. Through controlled experiments, we find that changing the selected trajectory states materially alters attack efficacy even when the malicious-update construction is fixed. We therefore identify when as a distinct attack dimension and introduce the Viability-constrained Behavioral Steering Attack (V-BSA), which uses local policy uncertainty to select sparse intervention states and applies envelope-constrained behavioral steering. Across discrete-action benchmarks, V-BSA achieves substantial degradation against robust aggregators and ensemble defenses with only a fraction of the interventions used by dense poisoning, while revealing task- and aggregation-dependent boundaries. Overall, our results highlight intervention timing as a distinct dimension of sequential robustness in FRL. The code is available at https://github.com/Yodeesy/V-BSA

Figures & tables

Figure 1: Conceptual comparison of intervention timing in FRL poisoning. (a) Prior work leaves trajectory-level intervention timing implicit, effectively defaulting to dense, state-agnostic intervention across rollout steps. (b) Ours ( V-BSA ) treats when as an explicit attack dimension: compromised agents maintain nominal execution on routine states and selectively intervene at states identified by policy-entropy gating.

Figure 2: Pilot evaluation of intervention timing. Terminal degradation

D

on LunarLander under FedPG-BR with fixed update construction across schedules (

\rho

). Entropy gating induces substantially higher degradation than dense and budget-matched baselines.

Figure 3: Overview of V-BSA . The attack factorizes into trajectory-level when selection and parameter-space how construction: an entropy gate selects sensitive states $\mathcal{D}_{i,\mathrm{when}}^{t}$ , from which a targeted update is synthesized, norm-matched, and constrained by the local empirical envelope $\mathcal{B}_{i}^{t}$ before server-side aggregation.

Aggregation Rule	Ensemble ( $K=5,M=10$ )			Non-Ensemble ( $N=30,M=10$ )
Aggregation Rule	LunarLander	Acrobot	CartPole	LunarLander	Acrobot	CartPole
FedAvg	79.90	120.50	3.00	$-57.19$	351.17	1.63
FedPG-BR	218.75	194.80	110.63	288.01	70.63	114.27
Coordinate-wise Median	175.83	151.17	8.80	434.35	284.87	42.80
Trimmed Mean	173.96	323.97	66.40	193.38	312.97	5.87

Table 1: Mean degradation

D

across aggregations and tasks (higher is more damaging). Positive values indicate successful terminal policy degradation induced by V-BSA .

Panel of Figure 4 — Figure 4: Evaluation dynamics on Ensemble-LunarLander ( $K=5,M=10$ ). Returns of Clean vs. V-BSA across four aggregation rules, with shaded bands denoting standard deviation across random seeds.

Figure 5: Mechanistic audit of when selection (LunarLander, epoch 261). (a) Normalized policy entropy $h(s)$ versus behavioral susceptibility $L(s)$ . (b) Susceptibility distributions for non-gated versus gated states across five client groups.

Figure 6: Timing ablation under matched budgets (LunarLander

\times

FedPG-BR). Fixing all how parameters, entropy gating achieves

1.89\times

higher mean degradation than budget-matched random selection across evaluated dynamics.

Figure 7: Temporal countermeasures against V-BSA on LunarLander. Terminal degradation

D

across dynamics

d_{0},d_{1},d_{2}

under no defense, the epoch auditor, and the temporal accumulator. The dashed line marks the preregistered resilience threshold (

3\sigma=49.7

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

Environment	State Dim. $d_{s}$	Action Space $\mathcal{A}$	Episode Horizon	Task Criterion
LunarLander-v2	8	Discrete(4)	1000	Return $\geq 200$
CartPole-v1	4	Discrete(2)	500	Return $=500$
Acrobot-v1	6	Discrete(3)	500	Return $\geq-100$
LunarLanderContinuous-v2	8	Box(2)	1000	Return $\geq 200$

Appendix

Table 2: Environment specifications. Benchmark tasks covering binary-action, multi-action, and continuous-control settings.

Component	Specification / Value
Policy Architecture & Optimization
Network Architecture	MLP: 2 hidden layers, 64 units each, Tanh activations
Output Layer	Discrete: categorical logits ( $\mathbb{R}^{\|\mathcal{A}\|}$ ); Continuous: Gaussian mean and log-variance
Optimizer	Adam ( $\beta_{1}=0.9,\beta_{2}=0.999$ ), learning rate $\eta=10^{-3}$
Discount Factor $\gamma$	$0.99$
Rollout Budget	Mini-batch size $b=8$ ; inner steps sampled via geometric schedule ( $B=32$ )

Appendix

Table 3: Hyperparameter configuration. Fixed parameters applied across federated reinforcement learning experiments.

	Dynamics $d_{0}$			Dynamics $d_{1}$			Dynamics $d_{2}$
Aggregation Rule	$J_{\mathrm{ctl}}$	$J_{\mathrm{atk}}$	$D$	$J_{\mathrm{ctl}}$	$J_{\mathrm{atk}}$	$D$	$J_{\mathrm{ctl}}$	$J_{\mathrm{atk}}$	$D$
FedPG-BR	228.6	80.5	148.17	211.9	74.4	137.48	245.4	$-125.2$	370.59
Trimmed Mean	236.2	34.1	202.08	196.7	$-17.0$	213.62	209.7	103.5	106.18
Median	167.1	$-17.7$	184.82	233.3	103.6	129.71	216.5	3.5	212.95
FedAvg	243.3	167.3	76.00	224.7	112.6	112.10	194.5	142.8	51.60

Appendix

Table 4: Discrete LunarLander-v2 evaluation breakdown under ensemble training (

K=5,M=10

). Terminal evaluation returns and degradation

D=J_{\mathrm{ctl}}-J_{\mathrm{atk}}

across evaluation dynamics.

	Dynamics $d_{0}$			Dynamics $d_{1}$			Dynamics $d_{2}$
Aggregation Rule	$J_{\mathrm{ctl}}$	$J_{\mathrm{atk}}$	$D$	$J_{\mathrm{ctl}}$	$J_{\mathrm{atk}}$	$D$	$J_{\mathrm{ctl}}$	$J_{\mathrm{atk}}$	$D$
FedPG-BR	162.7	$-59.2$	221.91	240.4	$-115.1$	355.48	225.1	$-61.5$	286.64
Trimmed Mean	258.4	$-12.3$	270.70	249.1	$-15.5$	264.60	233.9	189.1	44.80
Median	234.4	$-287.2$	521.60	210.3	$-168.1$	378.40	225.8	$-177.2$	403.00
FedAvg	216.3	255.0	$-38.70$	198.2	269.1	$-70.90$	201.5	263.4	$-61.90$

Appendix

Table 5: Discrete LunarLander-v2 evaluation breakdown under non-ensemble training (

N=30,M=10

). Terminal returns and degradation

D

across evaluation dynamics.

Panel of Figure 8 — Figure 8: Ensemble paradigm on CartPole ( $K=5,M=10$ ). Evaluation trajectories across aggregation rules. Solid curves: V-BSA attack; dashed curves: clean control.

Panel of Figure 9 — Figure 9: Ensemble paradigm on Acrobot ( $K=5,M=10$ ). Evaluation trajectories across aggregation rules. Solid curves: V-BSA attack; dashed curves: clean control.

Panel of Figure 10 — Figure 10: Non-ensemble paradigm on CartPole ( $N=30,M=10$ ). Evaluation trajectories across aggregation rules. Solid curves: V-BSA attack; dashed curves: clean control.

Panel of Figure 11 — Figure 11: Non-ensemble paradigm on Acrobot ( $N=30,M=10$ ). Evaluation trajectories across aggregation rules. Solid curves: V-BSA attack; dashed curves: clean control.

Panel of Figure 12 — Figure 12: Non-ensemble paradigm on discrete LunarLander-v2 ( $N=30,M=10$ ). Evaluation trajectories across aggregation rules. Solid curves: V-BSA attack; dashed curves: clean control.

Perturbation Schedule	Dynamics $d_{0}$	Dynamics $d_{1}$	Dynamics $d_{2}$	Median $D$
Fixed Perturbation Scale ( $\lambda=0.03$ )	$-23.12$	$-169.51$	$+29.35$	$-23.12$
Adaptive Scale Servo ( $\lambda\in[0.03,0.12]$ )	$+82.05$	$+82.91$	$-33.72$	$+82.05$

Appendix

Table 6: Continuous control evaluations on LunarLanderContinuous-v2 under FedPG-BR. Terminal degradation

D

across evaluation dynamics under fixed and adaptive perturbation scaling.

Explore similar work

Constraint-Aware Aggregation for Federated Reinforcement Learning in Microgrid Energy Coordination
Jul 14, 2026Usman Haider, Karl MasonFederated Learning Aggregation Federated RL
School of Computer Science, University of Galway, Ireland.
Efficient Federated RLHF via Zeroth-Order Policy Optimization
Apr 20, 2026Deyi Wang, Qining Zhang, Lei YingFederated RL Communication-Efficient Distributed Training
University of Michigan, Ann Arbor
Fed-CausalDiff: Decoupled Synchronization for Federated Do-Simulation and Policy Evaluation
Jun 21, 2026Pengfei Li, Mohammad KhalilPolicy Evaluation Causal Inference
Centre for the Science of Learning & Technology (SLATE), University of Bergen Bergen, Norway

Jul 14, 2026, cs.LGJ/K move · Enter open · S save

Constraint-Aware Aggregation for Federated Reinforcement Learning in Microgrid Energy Coordination

Usman Haider, Karl Mason

School of Computer Science, University of Galway, Ireland.

Federated Learning Aggregation Federated RL Constrained RL Power Systems

Open paper AskPDF arXiv

Federated Reinforcement Learning (FedRL) enables coordination of distributed energy resources without sharing raw local data, but standard aggregation methods such as FedAvg do not account for system-level constraints, often leading to unsafe global behavior. In this work, we study constraint-aware aggregation for federated reinforcement learning in distributed energy coordination. We propose aggregation rules that incorporate both local performance and estimated constraint violation into the server-side update. Among these, a simple penalty-based rule,

w_i \propto R_i - αV_i

, consistently provides the most reliable trade-off between reward and safety, without requiring dual optimization or modifications to local training. \textcolor{black}{We evaluate our approach on DairyGridEnv, a benchmark modeling multiple farms coordinating battery storage under stochastic demand and a shared grid capacity constraint, and further assess robustness using real load-driven demand profiles from Finland and the German FIELD dataset. Across multiple seeds, penalty-based aggregation substantially reduces violations while improving reward relative to FedAvg in both synthetic and real load-driven settings.} A combined reward-violation scheme exposes a tunable trade-off via

λ

, but is less stable. These results demonstrate that lightweight aggregation strategies can substantially improve empirical safety in federated reinforcement learning while preserving standard communication protocols.

Abstract

Explore similar work

Constraint-Aware Aggregation for Federated Reinforcement Learning in Microgrid Energy Coordination

Efficient Federated RLHF via Zeroth-Order Policy Optimization

Fed-CausalDiff: Decoupled Synchronization for Federated Do-Simulation and Policy Evaluation