Scalar Federated Learning for Linear Quadratic Regulator

Mohammadreza Rostami; Shahriar Talebi; Solmaz S. Kia

eess.SYApr 6, 2026

Scalar Federated Learning for Linear Quadratic Regulator

Authors: Mohammadreza Rostami, Shahriar Talebi, Solmaz S. Kia

Organizations: University of California, Irvine, CA, US · University of California, Los Angeles, CA, US

Abstract

We propose ScalarFedLQR, a communication-efficient federated algorithm for model-free learning of a common policy in linear quadratic regulator (LQR) control of cooperative agents. The method builds on a decomposed projected gradient mechanism, in which each agent communicates only a scalar projection of a local zeroth-order gradient estimate. The server aggregates these scalar messages to reconstruct a global descent direction, reducing per-agent uplink communication from O(d) to O(1), independent of the policy dimension. Crucially, the projection-induced approximation error diminishes as the number of participating agents increases, yielding a favorable scaling law: larger fleets enable more accurate gradient recovery, admit larger stepsizes, and achieve faster linear convergence despite high dimensionality. Under standard regularity conditions for homogeneous agents, all iterates remain stabilizing and the average LQR cost decreases linearly fast. Numerical results demonstrate performance comparable to full-gradient federated LQR with substantially reduced communication, even for heterogeneous (but similar) agents.

Figures & tables

Panel of Figure 1 — Fig. 1 : (a) Normalized optimality gap versus communication round. (b) Recovery percentage ( $1-\text{normalized optimality gap}$ ) versus total transmitted bits under two heterogeneity levels. (c) Recovery percentage versus total transmitted bits for $\epsilon_{1}=\epsilon_{2}=0$ with a fixed bit budget of $6\times 10^{5}$ . (d) Recovery percentage versus total transmitted bits for $\epsilon_{1}=\epsilon_{2}=5\times 10^{-1}$ with a fixed bit budget of $6\times 10^{5}$ .

Explore similar work

Robust Federated Q-Learning with Almost No Communication
Aug 24, 2026Sreejeet Maity, Aritra MitraFederated RL Q-Learning
Department of Electrical and Computer Engineering, North Carolina State University
Provably Efficient Federated Reinforcement Learning with Linear Function Approximation and Logarithmic Communication Cost
Aug 31, 2026Zihang Liang, Haochen Zhang, Lingzhou XueReinforcement Learning Federated RL
Department of Statistics, The Pennsylvania State University
FedQHD: Closed-Form Function-Space Federated Reinforcement Learning
May 27, 2026Yuchen Hou, Yongshan Chen, Zhuowen Zou +4RL Control Federated Learning Aggregation
Northeastern University · University of California, Irvine · The George Washington University

Aug 24, 2026J/K move · Enter open · S save

Robust Federated Q-Learning with Almost No Communication

Sreejeet Maity, Aritra Mitra

Department of Electrical and Computer Engineering, North Carolina State University

Federated RL Q-Learning Federated Learning Byzantine-Robust Federated Learning

Open paper AskPDF arXiv

We consider a federated reinforcement learning setting involving

M

agents, all of whom interact with a common Markov Decision Process (MDP). The agents exchange information via a central server to learn the optimal value function. Our goal is to understand to what extent one can hope for collaborative sample-complexity speedups in such a setting, when a small fraction of the agents are adversarial and can act arbitrarily. To that end, we propose Robust Fed-Q}, a federated Q-learning algorithm that blends ideas from both model-based and model-free RL, along with the median-of-means device from robust statistics. We prove that despite corruption, with high-probability, Robust Fed-Q (i) guarantees exact convergence to the optimal value function in the limit of infinite samples, and (ii) enjoys near-optimal finite-time rates that benefit from collaboration. In addition, our approach requires just

\tilde{O}(1)

rounds of communication to achieve each of the above guarantees, a feature of independent interest in FL where communication is the major bottleneck.

Abstract

Explore similar work

Robust Federated Q-Learning with Almost No Communication

Provably Efficient Federated Reinforcement Learning with Linear Function Approximation and Logarithmic Communication Cost

FedQHD: Closed-Form Function-Space Federated Reinforcement Learning