Organizations: Department of Computer Science and Automation, Indian Institute of Science, Bengaluru, India · Department of Electrical and Computer Engineering, Purdue University, IN, USA
Federated Reinforcement Learning (FRL) provides a promising way to speedup training in reinforcement learning when one has access to multiple edge devices that can operate in parallel. Recently, it has been shown that even when these edge devices have access to different dynamic models---such as cars or robots with differing system configurations---an optimal convergence rate that has a linear speedup proportional to the number of devices is achievable. However, this result requires that the algorithm's stepsize be chosen in a manner that depends on the unknown model parameters. Also, it applies only to a discounted setting, which has been argued to fit episodic tasks better than continuing control tasks. In this paper, we obtain finite-time bounds for heterogeneous FRL with average rewards. We show that optimal convergence rate and linear speedup are achievable even with a universal stepsize choice, independent of the underlying dynamics. To achieve our result, we modify the existing one-timescale FRL method to a novel two-timescale variant that additionally incorporates iterate averaging. We remark that this idea also extends to other settings already studied in the literature, such as the homogeneous FRL or the heterogeneous case with a discounted cost function.
Figures & tables
Constant
Definition
Heterogeneity and stability
Cd(ϵp)
(1−ϵp1+ϵp)∣S∣−1
CA,inv
max{i∈[N]max∥Ai−1∥,∥A−1∥}
CA(ϵp)
2min{2,Cd(ϵp)}+2ϵp
Cb(ϵp,ϵr)
2ϵr+3Rmaxmin{2,Cd(ϵp)}
Hr(ϵp,ϵr)
[ϵr+Rmaxmin{2,Cd(ϵp)}]2
TABLE I : Constants used in the finite-time analysis.
Fig. 1 : Comparison of our proposed two-timescale federated TD algorithm (left column) with the vanilla single-timescale federated TD algorithm described in Section III (right column). The top row corresponds to ϵp=ϵr=0.1 , while the bottom row corresponds to ϵp=ϵr=0.7 . We consider N∈{2,5,10,20} agents. The horizontal axis shows the number of iterations, while the vertical axis shows the mean-squared error relative to θ1∗ , computed using the Polyak–Ruppert averaged iterate for our algorithm and the raw iterate for the vanilla algorithm. Each curve is averaged over 10 independent runs, and the final 1000 iterations are magnified in the inset plots. Our algorithm uses β=0.3 . Both algorithms exhibit lower errors as N increases, with comparable performance despite our algorithm not requiring model-dependent stepsizes.
Fig. 2 : Effect of the stepsize exponent β on the performance of Algorithm 1 for N=5 agents. We vary β from 0.01 to 0.99 . The left panel corresponds to ϵp=ϵr=0.1 , while the right panel corresponds to ϵp=ϵr=0.7 . Each curve shows the mean-squared error averaged over 10 independent runs of 5000 iterations each. Smaller values of β generally lead to faster error reduction in both settings.
Fig. 3 : Long-term performance of our proposed Algorithm 1 (solid line) and the vanilla single-timescale federated TD algorithm (dotted line) for N=10 agents with ϵp=ϵr=0.7 . Each curve is averaged over 5 independent runs of 200,000 iterations. The errors eventually level off, consistent with the residual error due to heterogeneity predicted by our theoretical results. Our algorithm achieves a substantially lower error than the vanilla baseline in this experiment.
While federated reinforcement learning (RL) has traditionally guaranteed high efficiency in homogeneous environments, collaboration is often hindered when agents possess distinct goals. In this paper, we flip this paradigm, framing reward heterogeneity not as a hurdle to consensus, but as a structural asset that catalyzes exploration. We first propose Personalized Federated Upper Confidence Bound Value Iteration (PF-UCBVI), which achieves optimal linear speedup by decoupling shared dynamics from personalized objectives. To eliminate the risks of explicit exploration, we then introduce Personalized Federated Exploration-Free Value Iteration (PF-EFVI), a purely greedy algorithm that leverages reward diversity to ensure state-action coverage. We prove that PF-EFVI attains logarithmic regret without explicit exploration bonuses under sufficient reward diversity. Our results reveal that agent disagreement shifts the exploration burden from temporal complexity to spatial diversity, enabling efficient collective learning without explicit exploration.
Jiin Woo, Gauri Joshi, Yuejie Chi
Carnegie Mellon University · Department of Electrical and Computer Engineering, Carnegie Mellon University. · Department of Statistics and Data Science, Yale University.
Federated Reinforcement Learning (FRL) enables collaborative policy learning across distributed agents with heterogeneous environments. While recent methods based on variance reduction, divergence penalization, and momentum optimization improve FRL under heterogeneous settings, they still primarily synchronize policy or value-network parameters and do not explicitly address distributional mismatch among heterogeneous clients. Therefore, we propose FedGuide, a FRL framework that uses diffusion priors as behavior models to provide personalized data supported distributions for heterogeneous local policy learning. Instead of directly averaging local policies, FedGuide aggregates those diffusion priors through Optimal-Transport Mixture-of-Experts (OT-MoE), preserving heterogeneous behavior modes in distribution space. It further develops a Distribution Correction Estimation (DICE) value baseline to provide low-variance, return-aware guidance for local policy improvement. Experiments across heterogeneous environments show that FedGuide outperforms representative FRL methods in client-average returns, final-round performance, and worst-round robustness, while maintaining stable learning under stronger heterogeneity.
We study personalized multi-agent average reward TD learning, in which a collection of agents interacts with different environments and jointly learns their respective value functions. We focus on the setting where there exists a shared linear representation, and the agents' optimal weights collectively lie in an unknown linear subspace. Inspired by the recent success of personalized federated learning (PFL), we study the convergence of cooperative single-timescale TD learning in which agents iteratively estimate the common subspace and local heads. We showed that this decomposition can filter out conflicting signals, effectively mitigating the negative impacts of ``misaligned'' signals, and achieving linear speedup. The main technical challenges lie in the heterogeneity, the Markovian sampling, and their intricate interplay in shaping error evolutions. Specifically, not only are the error dynamics of multiple variables closely interconnected, but there is also no direct contraction for the principal angle distance between the optimal subspace and the estimated subspace. We hope our analytical techniques can be useful to inspire research on deeper exploration into leveraging common structures. Experiments are provided to show the benefits of learning via a shared structure to the more general control problem.