cs.LGNov 7, 2026

Exploration via Exploitation: The Blessing of Reward Diversity in Personalized Federated RL

Authors: Jiin Woo, Gauri Joshi, Yuejie Chi

Organizations: Carnegie Mellon University · Department of Electrical and Computer Engineering, Carnegie Mellon University. · Department of Statistics and Data Science, Yale University.

Abstract

While federated reinforcement learning (RL) has traditionally guaranteed high efficiency in homogeneous environments, collaboration is often hindered when agents possess distinct goals. In this paper, we flip this paradigm, framing reward heterogeneity not as a hurdle to consensus, but as a structural asset that catalyzes exploration. We first propose Personalized Federated Upper Confidence Bound Value Iteration (PF-UCBVI), which achieves optimal linear speedup by decoupling shared dynamics from personalized objectives. To eliminate the risks of explicit exploration, we then introduce Personalized Federated Exploration-Free Value Iteration (PF-EFVI), a purely greedy algorithm that leverages reward diversity to ensure state-action coverage. We prove that PF-EFVI attains logarithmic regret without explicit exploration bonuses under sufficient reward diversity. Our results reveal that agent disagreement shifts the exploration burden from temporal complexity to spatial diversity, enabling efficient collective learning without explicit exploration.

Figures & tables

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. Exploration-Driven Personalized Federated Reinforcement Learning via Intrinsic Motivation

    Aug 11, 2026Md Rafid Islam, Rafsan Jany, Zahid Hasan +1Federated RLIntrinsic Motivation

  2. Personalized Observation Normalization for Federated Reinforcement Learning in Simulation Environments with Heterogeneity

    Apr 10, 2026Yiran Pang, Zhen Ni, Xiangnan ZhongFederated RLHeterogeneous Federated Learning

  3. Federated TD Learning in Heterogeneous Environments with Average Rewards: A Two-timescale Approach with Polyak-Ruppert Averaging

    Nov 16, 2026Ankur Naskar, Gugan Thoppe, Abbasali Koochakzadeh +1Federated RLHeterogeneous Federated Learning