cs.LGNov 16, 2026

Federated TD Learning in Heterogeneous Environments with Average Rewards: A Two-timescale Approach with Polyak-Ruppert Averaging

Authors: Ankur Naskar, Gugan Thoppe, Abbasali Koochakzadeh, Vijay Gupta

Organizations: Department of Computer Science and Automation, Indian Institute of Science, Bengaluru, India · Department of Electrical and Computer Engineering, Purdue University, IN, USA

Abstract

Federated Reinforcement Learning (FRL) provides a promising way to speedup training in reinforcement learning when one has access to multiple edge devices that can operate in parallel. Recently, it has been shown that even when these edge devices have access to different dynamic models---such as cars or robots with differing system configurations---an optimal convergence rate that has a linear speedup proportional to the number of devices is achievable. However, this result requires that the algorithm's stepsize be chosen in a manner that depends on the unknown model parameters. Also, it applies only to a discounted setting, which has been argued to fit episodic tasks better than continuing control tasks. In this paper, we obtain finite-time bounds for heterogeneous FRL with average rewards. We show that optimal convergence rate and linear speedup are achievable even with a universal stepsize choice, independent of the underlying dynamics. To achieve our result, we modify the existing one-timescale FRL method to a novel two-timescale variant that additionally incorporates iterate averaging. We remark that this idea also extends to other settings already studied in the literature, such as the homogeneous FRL or the heterogeneous case with a discounted cost function.

Figures & tables

Explore similar work

CardsList
  1. Exploration via Exploitation: The Blessing of Reward Diversity in Personalized Federated RL

    Nov 7, 2026Jiin Woo, Gauri Joshi, Yuejie ChiReinforcement LearningFederated RL

  2. Personalized Multi-Agent Average Reward TD-Learning via Joint Linear Approximation

    Mar 2, 2026Leo Muxing Wang, Pengkun Yang, Lili SuTemporal-Difference LearningMulti-Agent Reinforcement Learning