Exploration via Exploitation: The Blessing of Reward Diversity in Personalized Federated RL
Authors: Jiin Woo, Gauri Joshi, Yuejie Chi
Organizations: Carnegie Mellon University · Department of Electrical and Computer Engineering, Carnegie Mellon University. · Department of Statistics and Data Science, Yale University.
While federated reinforcement learning (RL) has traditionally guaranteed high efficiency in homogeneous environments, collaboration is often hindered when agents possess distinct goals. In this paper, we flip this paradigm, framing reward heterogeneity not as a hurdle to consensus, but as a structural asset that catalyzes exploration. We first propose Personalized Federated Upper Confidence Bound Value Iteration (PF-UCBVI), which achieves optimal linear speedup by decoupling shared dynamics from personalized objectives. To eliminate the risks of explicit exploration, we then introduce Personalized Federated Exploration-Free Value Iteration (PF-EFVI), a purely greedy algorithm that leverages reward diversity to ensure state-action coverage. We prove that PF-EFVI attains logarithmic regret without explicit exploration bonuses under sufficient reward diversity. Our results reveal that agent disagreement shifts the exploration burden from temporal complexity to spatial diversity, enabling efficient collective learning without explicit exploration.
Figures & tables
Figure 1: Two regimes of reward heterogeneity that induce global coverage in Assumption 1 , illustrated on a common two-step MDP with S={x,Y1,Y2} , A={up,down} , M=2 (agent A and agent B). In each row, the right panel plots each agent’s step-1 Q-gap Qgapm=Q1m(x,up)−Q1m(x,down) , whose sign determines the agent’s greedy choice at x , as the assumed kernel P1 varies. Top (Regime I): action-dependent state-independent rewards make V2m take the same value at every state, so each agent’s Q-gap is fixed regardless of P1 (horizontal lines). Both agents persistently select different actions, and the cross-agent average visitation remains 1/2 on each action at both extremes of P1 . Bottom (Regime II): state-dependent action-independent rewards anti-correlated across agents make V2A+V2B take the same value at every state, so the two agents’ Q-gaps are exact mirror images for any P1 (crossing lines). Individual visitations swap between the two extremes (blue and red bars exchange positions), but the cross-agent average remains 1/2 on each action throughout.
Figure 2: Empirical feasibility of Assumption 1 on random tabular MDPs. (a) λmin(P) shows no downward trend in ϵ . (b) λmin becomes positive from M=16 and grows with M , overlapping across ϵ , indicating that reward heterogeneity drives coverage independently of P .
Figure 3: Reward diversity, not ensemble size, drives collective coverage. State visitation dˉh(s) on a single MDP with P=P . (a) M=64 agents with identical rewards: visitation is highly uneven (brightest cells exceed the color scale). (b) M=64 agents with i.i.d. rewards: visits spread nearly uniformly. The shared color scale is capped at the diverse panel’s maximum.
Figure 4: Per-agent regret of PF-UCBVI and PF-EFVI . Top: cumulative regret Regret(k)/M . Bottom: instantaneous regret. PF-UCBVI converges sub-linearly at every M with a visible burn-in. PF-EFVI fails at small M ( λmin=0 for M=1,4 ) but matches or beats PF-UCBVI once M≥16 by avoiding the explicit-exploration burn-in.
Appendix figures & tables1 asset
Supplementary material from the paper’s appendix.
Appendix
M
Algorithm
Cumulative regret / agent
Instantaneous regret / agent
1
PF-UCBVI
87.8±8.8
0.011±0.007
PF-EFVI
205.4±130.2
0.113±0.083
Fed-UCBVI
107.0±28.9
0.016±0.011
4
PF-UCBVI
25.0±5.3
0.002±0.001
PF-EFVI
33.1±16.5
0.010±0.009
Fed-UCBVI
744.9±132.1
0.481±0.089
Appendix
Table 1: Per-agent regret at K=1500 (mean ± std over 8 MDP draws).
Reinforcement learning studies how agents learn sequential decisions from rewards and interaction, balancing exploration, performance, safety, and adaptation.
Personalized Federated Reinforcement Learning (PFRL) takes a decentralized approach to storing and accessing information based on past experiences while keeping each client's data private during the learning of each client's policy. Many current methods for PFRL rely heavily on exploiting existing reinforcement learning reward signals to derive an optimal policy for each client, thereby neglecting exploration in non-stationary or sparse-reward environments. In this work, we introduce a new exploration-driven framework, Exploration-Driven Personalized Federated Reinforcement Learning via Intrinsic Motivation (EDPFRL-IM), that leverages an inherent curiosity-driven exploration at each client to promote local exploration and protect client privacy. Furthermore, to facilitate policy discovery via exploration in previously unexplored state spaces, clients add an intrinsic random network distillation (RND) signal to their extrinsic reward. Additionally, the server does not have access to clients' raw experiences or local gradient estimates; instead, the server sends global exploration priors and collects minimal novelty summaries from each client to enable both diverse and coordinated exploration among clients. Experiments in benchmark environments show that our framework outperforms average PFRL benchmarks in policy personalization and sample efficiency, primarily in delayed and sparse reward systems. Overall, EDPFRL-IM enables the integration of a flexible exploratory learning structure into federated reinforcement learning systems while preserving client privacy.
Md Rafid Islam, Rafsan Jany, Zahid Hasan +1
Department of Electrical and Computer Engineering North South University Dhaka, Bangladesh · Digital Health Research Division Korea Institute of Oriental Medicine Daejeon, South Korea · Department of Electrical and Computer Engineering The University of Alabama in Huntsville Huntsville, AL, USA
Federated reinforcement learning (FedRL) enables multiple agents to collaboratively train a global policy without sharing raw data, making it ideal for privacy-sensitive applications. However, FedRL faces challenges in heterogeneous environments where differing state-transition dynamics lead to non-identical input distributions and imbalanced parameter updates during aggregation. Therefore, this paper develops a personalized observation normalization (PON) method, allowing each agent to locally normalize raw state inputs using a continuously updated running mean and variance. This design ensures consistent scaling of local feature without overshadowing across agents during aggregation. Furthermore, we demonstrate that sharing normalization parameters across agents is ineffective due to the diverse local input distributions, which highlights the necessity of personalized statistics. Experiments on heterogeneous MuJoCo tasks show that our developed PON accelerates training and achieves superior performance compared to baseline methods.
Yiran Pang, Zhen Ni, Xiangnan Zhong
Department of Electrical Engineering & Computer Science, Florida Atlantic University, Boca Raton, FL, USA
Federated Reinforcement Learning (FRL) provides a promising way to speedup training in reinforcement learning when one has access to multiple edge devices that can operate in parallel. Recently, it has been shown that even when these edge devices have access to different dynamic models---such as cars or robots with differing system configurations---an optimal convergence rate that has a linear speedup proportional to the number of devices is achievable. However, this result requires that the algorithm's stepsize be chosen in a manner that depends on the unknown model parameters. Also, it applies only to a discounted setting, which has been argued to fit episodic tasks better than continuing control tasks. In this paper, we obtain finite-time bounds for heterogeneous FRL with average rewards. We show that optimal convergence rate and linear speedup are achievable even with a universal stepsize choice, independent of the underlying dynamics. To achieve our result, we modify the existing one-timescale FRL method to a novel two-timescale variant that additionally incorporates iterate averaging. We remark that this idea also extends to other settings already studied in the literature, such as the homogeneous FRL or the heterogeneous case with a discounted cost function.
Department of Computer Science and Automation, Indian Institute of Science, Bengaluru, India · Department of Electrical and Computer Engineering, Purdue University, IN, USA