cs.LGMar 2, 2026

Personalized Multi-Agent Average Reward TD-Learning via Joint Linear Approximation

Authors: Leo Muxing Wang, Pengkun Yang, Lili Su

Organizations: Northeastern University · Tsinghua University

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

We study personalized multi-agent average reward TD learning, in which a collection of agents interacts with different environments and jointly learns their respective value functions. We focus on the setting where there exists a shared linear representation, and the agents' optimal weights collectively lie in an unknown linear subspace. Inspired by the recent success of personalized federated learning (PFL), we study the convergence of cooperative single-timescale TD learning in which agents iteratively estimate the common subspace and local heads. We showed that this decomposition can filter out conflicting signals, effectively mitigating the negative impacts of ``misaligned'' signals, and achieving linear speedup. The main technical challenges lie in the heterogeneity, the Markovian sampling, and their intricate interplay in shaping error evolutions. Specifically, not only are the error dynamics of multiple variables closely interconnected, but there is also no direct contraction for the principal angle distance between the optimal subspace and the estimated subspace. We hope our analytical techniques can be useful to inspire research on deeper exploration into leveraging common structures. Experiments are provided to show the benefits of learning via a shared structure to the more general control problem.

Figures & tables

ParametersAgent 0Agent 1Agent 2Agent 3Agent 4Agent 5
link length 12.21.5232.11.5
link length 2110.8111
Table 1
Table 2
ParametersAgent 0Agent 1Agent 2Agent 3Agent 4Agent 5
pole length0.510.510.51
gravity9.8159.8159.815
force101010101010
start positionπ6\frac{\pi}{6}- π6\frac{\pi}{6}π6\frac{\pi}{6}- π6\frac{\pi}{6}π6\frac{\pi}{6}- π6\frac{\pi}{6}
envnormalmirrorednormalmirrorednormalmirrored
Table 1: CartPole environment setup for each agent
ParametersAgent 0Agent 1Agent 2Agent 3Agent 4Agent 5
link length 1111111
link length 2111111
envnormalmirrorednormalmirrorednormalmirrored
Table 2: Acrobot environment setup for each agent
Figure 1: Value function convergence comparison on Acrobot.
Figure 2: Value function convergence speed comparison between single-timescale and two-timescale settings on Acrobot.
Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 3: Geometry illustration.
Figure 4: Policy Learning Performance Comparison on Acrobot
More in this topic

Temporal-Difference Learning

Temporal-difference learning estimates values by bootstrapping from later predictions, enabling online learning and control from sequential experience.

See all 80 papers

Explore similar work

CardsList

  1. Best of Both Worlds in Federated LSA: Speedup When Possible, Personalization Always

    Oct 8, 2026Safwan Labbi, Paul Mangold, Eric MoulinesFederated LearningHeterogeneous Federated Learning

    CMAP, CNRS, École Polytechnique, Institut Polytechnique de Paris, 91120 Palaiseau, France · Mohamed bin Zayed University of Artificial Intelligence, UAE · LRE EPITA, 94270 Le Kremlin-Bicêtre, France

  2. Collaborative Yet Personalized Policy Training: Single-Timescale Federated Actor-Critic

    May 14, 2026Leo Muxing Wang, Pengkun Yang, Lili SuFederated RLActor-Critic Methods

    Northeastern University · Tsinghua University

  3. Multi-Agent Reinforcement Learning with Submodular Reward

    Mar 6, 2026Wenjing Chen, Chengyuan Qian, Shuo Xing +2Cooperative MARLMulti-Agent Reinforcement Learning

    Department of Computer Science and Engineering, Texas A&M University