We study federated model-free learning of a common linear quadratic regulator (LQR) gain under a limited uplink communication budget, with clients that share dynamics and costs but experience different process-noise levels. Conventionally, clients transmit gradient matrices whose dimension equals the number of controller parameters. Exploiting the structure of two-point zeroth-order estimates, we consider an implementation in which clients transmit only one scalar cost-difference coefficient per rollout pair, while the server regenerates the associated perturbation directions from shared seeds. Communication thus scales with the number of rollout pairs rather than the controller dimension, turning the uplink budget into a rollout budget. Because noise heterogeneity produces gradient estimates of widely differing variance and every client carries equal weight in the fleet objective, uniform rollout allocation allows noisy clients to dominate the aggregated update and increases the probability of destabilizing updates. We propose a rollout-allocation scheme that assigns rollout pairs in proportion to the standard deviation of each client's estimates and show that it minimizes the variance of the aggregated update. The scheme is implemented exactly through a greedy integer allocation rule whose weights are estimated from the communicated scalars. We further prove that a single allocation factor quantifies its improvement over uniform allocation in a one-round instability bound, the certified step size, and the error floor of a joint stability and convergence guarantee. Simulations show that, under the same communication budget, personalized allocation reduces the one-step instability rate from 37% to 0.3%.
Figures & tables
Fig. 1: One round of the protocol. The server broadcasts the gain Kk and the rollout counts (dashed). Each client executes its assigned rollout pairs on its own system and uploads one scalar coefficient per pair (solid), subject to a fleet budget of S scalars per round. The server regenerates the perturbation directions from the shared seed, reconstructs the gradient estimate, updates the gain, and reallocates the next round’s counts from the variance of the received coefficients, so that clients with noisier measurements are assigned more rollout pairs.
Fig. 2: Percentage of destabilizing updates after one step from K0 at h=0.8 , under uniform counts (“Equal”) and personalized counts (Algorithm 1 ). (a) Versus the number of rollout pairs per client, S/M , at η=0.13 , for M=10 and M=20 . (b) Versus the step size at S=4M and M=10 . The dotted curve in (b) is obtained by scaling the uniform-count sampling errors by Γk(S)−1/2 and recomputing the failures (Remark 2 ).
Fig. 3: Learning with ten clients at h=0.8 and S=4M , under uniform counts (“Equal”) and personalized counts (Algorithm 1 ). (a) Mean cost gap over 100 runs at η=0.01 ; the dashed and dotted curves use the larger step sizes η=0.02 and 0.03 , which only personalized counts tolerate, and the horizontal line marks the uniform-count floor divided by Γk(S) (Corollary 3 ; labeled Remark 3 in the legend). (b) Fraction of 200 runs in which all 60 updates and all queried gains are stabilizing, versus step size. (c) Fraction of runs that are stable and within 25% of the initial gap after 60 updates, versus step size.
We propose ScalarFedLQR, a communication-efficient federated algorithm for model-free learning of a common policy in linear quadratic regulator (LQR) control of cooperative agents. The method builds on a decomposed projected gradient mechanism, in which each agent communicates only a scalar projection of a local zeroth-order gradient estimate. The server aggregates these scalar messages to reconstruct a global descent direction, reducing per-agent uplink communication from O(d) to O(1), independent of the policy dimension. Crucially, the projection-induced approximation error diminishes as the number of participating agents increases, yielding a favorable scaling law: larger fleets enable more accurate gradient recovery, admit larger stepsizes, and achieve faster linear convergence despite high dimensionality. Under standard regularity conditions for homogeneous agents, all iterates remain stabilizing and the average LQR cost decreases linearly fast. Numerical results demonstrate performance comparable to full-gradient federated LQR with substantially reduced communication, even for heterogeneous (but similar) agents.
Mohammadreza Rostami, Shahriar Talebi, Solmaz S. Kia
University of California, Irvine, CA, US · University of California, Los Angeles, CA, US
We consider a federated reinforcement learning setting involving M agents, all of whom interact with a common Markov Decision Process (MDP). The agents exchange information via a central server to learn the optimal value function. Our goal is to understand to what extent one can hope for collaborative sample-complexity speedups in such a setting, when a small fraction of the agents are adversarial and can act arbitrarily. To that end, we propose Robust Fed-Q}, a federated Q-learning algorithm that blends ideas from both model-based and model-free RL, along with the median-of-means device from robust statistics. We prove that despite corruption, with high-probability, Robust Fed-Q (i) guarantees exact convergence to the optimal value function in the limit of infinite samples, and (ii) enjoys near-optimal finite-time rates that benefit from collaboration. In addition, our approach requires just O~(1) rounds of communication to achieve each of the above guarantees, a feature of independent interest in FL where communication is the major bottleneck.
Sreejeet Maity, Aritra Mitra
Department of Electrical and Computer Engineering, North Carolina State University
Federated learning (FL) is an emerging distributed machine learning paradigm that enables local devices to jointly train a global model while keeping data decentralized and private. We propose a variance-reduction based algorithm, VRA-FedSGD, for FL in the presence of heavy-tailed gradient noise and communication noise, where these noises are prevalent in large-scale machine learning over wireless networks and Internet of Things deployments. VRA-FedSGD employs a momentum variance reduction technique together with a nonlinear mapping to mitigate heavy-tailed gradient noise, and uses a variance-reduced aggregation mechanism to suppress heavy-tailed communication noise. In the mean sense, VRA-FedSGD achieves a convergence rate of {\smallO(K−(p−1)/(2p−1))} for nonconvex objective functions, where p is the tail index of heavy-tailed noise. In the almost sure sense, VRA-FedSGD achieves a convergence rate of O~(K−(1−1/(p−ε))) for strongly convex objective functions, where ε is an arbitrarily small constant. Simulated experiments on a logistic regression problem with real-world data verify the effectiveness of VRA-FedSGD.
Shengchao Zhao, Yongchao Liu
School of Mathematics, China University of Mining and Technology, Xuzhou, China · School of Mathematical Sciences, Dalian University of Technology, Dalian, China