This paper studies model-free federated learning of a common LQR controller for a fleet of similar systems that communicate with a server over an uplink subject to limited bandwidth, packet dropout, and additive noise. We propose ResiFedLQR, a scalar communication scheme in which each agent sends one real number per packet, with redundancy, instead of full gradient estimates. The scalars can be gradient coordinates, projections of the gradient onto random directions reproduced at the server from a shared seed, or the rollout costs themselves. For the projection encoding, we prove that, with high probability, every iterate stabilizes all agents and the fleet-average cost converges linearly to a neighborhood of the optimum whose radius separates heterogeneity, sampling, and communication effects, and we derive a rule for choosing the number of transmissions from the channel parameters and the fleet size. We then compare the encodings with full-gradient transmission under the same uplink budget and identify the signal-to-noise regime and power constraint under which each is preferable. Numerical experiments on a heterogeneous fleet evaluate stability and convergence under varying noise levels and dropout probabilities.
Figures & tables
Fig. 1 : Federated LQR over an unreliable uplink. The server broadcasts the common gain Kt over a reliable downlink. Each agent runs rollouts on its own unknown system, forms a local gradient estimate, and returns messages over an uplink subject to packet dropout and additive noise.
GL
OL
CL
Scalar sent
v⊤g~t,n
rollout cost Jt,s(n)
[g~t,n]i
Packets/round
Nsc (design choice)
ns (= rollouts)
dNF
Local computation
ZO estimate
none
ZO estimate
Encoding error
(d−1)∥g~t,n∥2/Nt,nrec
none
none
Channel-noise 2nd moment
dσch2/Nt,nrec
d2σch2/(r2Nt,nrec)
≈d2σch2/Nt,nrec
A lost packet removes
one projection
one rollout
one copy of one coordinate
TABLE I : Comparison of the scalar encodings of Definition 1 .
Fig. 2 : Fraction of realizations losing closed-loop stability versus channel noise for gradient-level (GL) and oracle-level (OL) encoding, at equal rollouts ( ns=5 ) and equal packets ( Nsc=ns per agent and round), in the setting of Section VI ( d=9 , r=0.1 , pdrop=0.2 , 20 realizations per noise level).
Fig. 3 : Empirical resilience regions ( 52 ) over stepsize and dropout probability for σch∈{1,2,3,4} at (NF,Nsc)=(8,72) .
σch
1
2
3
4
FedLQR
52%
12%
0%
0%
ResiFedLQR
100%
79%
52%
38%
TABLE II : Fraction of the 10×10(η,pdrop) grid in the empirical resilience region ( 52 ).
Fig. 4 : Empirical failure probability pfail over stepsize and dropout probability. Top: FedLQR ; bottom: ResiFedLQR . Darker colors indicate smaller pfail ; solid and dashed curves mark pfail=0 and 0.5 .
Fig. 5 : Normalized optimality gap ( 53 ) versus round under increasing uplink channel noise, at equal uplink budget Nsc=dNF and pdrop=0.2 . Lines are medians over 75 realizations and shaded bands are interquartile ranges, truncated at the panel edges; the dotted line marks G0 .
σch=1
σch=2
(NF,Nsc)
F
R
F
R
(1,9)
0.73 (15)
0.60 (0)
0.75 (44)
0.58 (0)
(2,18)
0.60 (1)
0.62 (0)
0.62 (30)
0.64 (0)
(4,36)
0.67 (0)
0.59 (0)
0.60 (11)
0.59 (0)
(6,54)
0.60 (0)
0.60 (0)
0.63 (1)
0.60 (0)
(8,72)
0.59 (0)
0.59 (0)
0.57 (1)
0.60 (0)
TABLE III : Median of GT over 75 realizations at pdrop=0.2 ( G0=0.93 ); in parentheses, the number of realizations with maxt≤TGt>5 . F: FedLQR ; R: ResiFedLQR .
Fig. 6 : Normalized optimality gap ( 53 ) versus cumulative uplink bits for σch=1 and pdrop=0.2 . Lines are medians over 75 realizations; the vertical line marks the budget of 5.76×106 bits and the dotted line marks G0 .
Fig. 7 : Normalized mean-square reconstruction error of one transmission round versus normalized channel noise, for d=9 , NF=2 , and pdrop=0.2 at equal budget. Lines: ( 47 ) and ( 48 ); markers: Monte Carlo estimates from 4×104 realizations. The dashed line marks the crossover.
Fig. 8 : Fraction of realizations losing closed-loop stability versus channel noise for bundled full-gradient transmission ( FedLQR ), CL encoding (element-wise) without and with a transmit gain of d , and GL encoding ( ResiFedLQR ), at (NF,Nsc)=(18,162) , pdrop=0.2 , η=10−6 , and T=2000 , with 20 realizations per noise level. (a) Fleet of Section VI-A ( M=10 ). (b) The same fleet extended to M=100 agents.
pdrop=0.1
0.3
0.5
0.7
η
F
G
F
G
F
G
F
G
5×10−7
0
0
3
0
2
0
0
0
1×10−6
5
0
4
0
5
0
8
1
2×10−6
7
0
6
2
7
2
8
3
4×10−6
8
2
8
2
8
6
8
6
TABLE IV : Number of realizations, out of 8 , losing closed-loop stability for M=100 , (NF,Nsc)=(18,162) , and σch=16 . F: bundled full-gradient transmission; G: GL encoding.
Fig. 9 : (a) Gap after T=2×104 rounds versus heterogeneity level γ : median and interquartile range over 20 realizations; dashed: gain minimizing Javg . (b) Fraction of realizations losing stability versus channel noise for d∈{9,36,81} at equal budget.
Problem
M , d
number of agents; policy dimension d=nunx
J(n) , Javg
agent and fleet-average costs ( 2 ), ( 4 )
S , Sc
gains stabilizing all agents ( 5 ); sublevel set of K0 ( 27 )
ϵ1,ϵ2
heterogeneity bounds ( 3 )
Ft , Fˉ
optimality gap Javg(Kt)−Javg⋆ ; certified radius ( 42 )
We propose ScalarFedLQR, a communication-efficient federated algorithm for model-free learning of a common policy in linear quadratic regulator (LQR) control of cooperative agents. The method builds on a decomposed projected gradient mechanism, in which each agent communicates only a scalar projection of a local zeroth-order gradient estimate. The server aggregates these scalar messages to reconstruct a global descent direction, reducing per-agent uplink communication from O(d) to O(1), independent of the policy dimension. Crucially, the projection-induced approximation error diminishes as the number of participating agents increases, yielding a favorable scaling law: larger fleets enable more accurate gradient recovery, admit larger stepsizes, and achieve faster linear convergence despite high dimensionality. Under standard regularity conditions for homogeneous agents, all iterates remain stabilizing and the average LQR cost decreases linearly fast. Numerical results demonstrate performance comparable to full-gradient federated LQR with substantially reduced communication, even for heterogeneous (but similar) agents.
Mohammadreza Rostami, Shahriar Talebi, Solmaz S. Kia
University of California, Irvine, CA, US · University of California, Los Angeles, CA, US
We study federated model-free learning of a common linear quadratic regulator (LQR) gain under a limited uplink communication budget, with clients that share dynamics and costs but experience different process-noise levels. Conventionally, clients transmit gradient matrices whose dimension equals the number of controller parameters. Exploiting the structure of two-point zeroth-order estimates, we consider an implementation in which clients transmit only one scalar cost-difference coefficient per rollout pair, while the server regenerates the associated perturbation directions from shared seeds. Communication thus scales with the number of rollout pairs rather than the controller dimension, turning the uplink budget into a rollout budget. Because noise heterogeneity produces gradient estimates of widely differing variance and every client carries equal weight in the fleet objective, uniform rollout allocation allows noisy clients to dominate the aggregated update and increases the probability of destabilizing updates. We propose a rollout-allocation scheme that assigns rollout pairs in proportion to the standard deviation of each client's estimates and show that it minimizes the variance of the aggregated update. The scheme is implemented exactly through a greedy integer allocation rule whose weights are estimated from the communicated scalars. We further prove that a single allocation factor quantifies its improvement over uniform allocation in a one-round instability bound, the certified step size, and the error floor of a joint stability and convergence guarantee. Simulations show that, under the same communication budget, personalized allocation reduces the one-step instability rate from 37% to 0.3%.
Mohammadreza Rostami, Shahriar Talebi, Solmaz S. Kia
University of California, Irvine, CA, US · University of California, Los Angeles, CA, US
We consider a federated reinforcement learning setting involving M agents, all of whom interact with a common Markov Decision Process (MDP). The agents exchange information via a central server to learn the optimal value function. Our goal is to understand to what extent one can hope for collaborative sample-complexity speedups in such a setting, when a small fraction of the agents are adversarial and can act arbitrarily. To that end, we propose Robust Fed-Q}, a federated Q-learning algorithm that blends ideas from both model-based and model-free RL, along with the median-of-means device from robust statistics. We prove that despite corruption, with high-probability, Robust Fed-Q (i) guarantees exact convergence to the optimal value function in the limit of infinite samples, and (ii) enjoys near-optimal finite-time rates that benefit from collaboration. In addition, our approach requires just O~(1) rounds of communication to achieve each of the above guarantees, a feature of independent interest in FL where communication is the major bottleneck.
Sreejeet Maity, Aritra Mitra
Department of Electrical and Computer Engineering, North Carolina State University