Personalized Rollout Allocation for Federated LQR Policy Optimization under Heterogeneous Process Noise

Mohammadreza Rostami; Shahriar Talebi; Solmaz S. Kia

cs.MAOct 31, 2026

Personalized Rollout Allocation for Federated LQR Policy Optimization under Heterogeneous Process Noise

Authors: Mohammadreza Rostami, Shahriar Talebi, Solmaz S. Kia

Organizations: University of California, Irvine, CA, US · University of California, Los Angeles, CA, US

Abstract

We study federated model-free learning of a common linear quadratic regulator (LQR) gain under a limited uplink communication budget, with clients that share dynamics and costs but experience different process-noise levels. Conventionally, clients transmit gradient matrices whose dimension equals the number of controller parameters. Exploiting the structure of two-point zeroth-order estimates, we consider an implementation in which clients transmit only one scalar cost-difference coefficient per rollout pair, while the server regenerates the associated perturbation directions from shared seeds. Communication thus scales with the number of rollout pairs rather than the controller dimension, turning the uplink budget into a rollout budget. Because noise heterogeneity produces gradient estimates of widely differing variance and every client carries equal weight in the fleet objective, uniform rollout allocation allows noisy clients to dominate the aggregated update and increases the probability of destabilizing updates. We propose a rollout-allocation scheme that assigns rollout pairs in proportion to the standard deviation of each client's estimates and show that it minimizes the variance of the aggregated update. The scheme is implemented exactly through a greedy integer allocation rule whose weights are estimated from the communicated scalars. We further prove that a single allocation factor quantifies its improvement over uniform allocation in a one-round instability bound, the certified step size, and the error floor of a joint stability and convergence guarantee. Simulations show that, under the same communication budget, personalized allocation reduces the one-step instability rate from $37\%$ to $0.3\%$ .

Figures & tables

Panel of Figure 1 — Fig. 1: One round of the protocol. The server broadcasts the gain $K_{k}$ and the rollout counts (dashed). Each client executes its assigned rollout pairs on its own system and uploads one scalar coefficient per pair (solid), subject to a fleet budget of $S$ scalars per round. The server regenerates the perturbation directions from the shared seed, reconstructs the gradient estimate, updates the gain, and reallocates the next round’s counts from the variance of the received coefficients, so that clients with noisier measurements are assigned more rollout pairs.

Fig. 2: Percentage of destabilizing updates after one step from

K_{0}

h=0.8

, under uniform counts (“Equal”) and personalized counts (Algorithm 1 ). (a) Versus the number of rollout pairs per client,

S/M

, at

\eta=0.13

, for

M=10

and

M=20

. (b) Versus the step size at

S=4M

and

M=10

. The dotted curve in (b) is obtained by scaling the uniform-count sampling errors by

\Gamma_{k}(S)^{-1/2}

and recomputing the failures (Remark 2 ).

Fig. 3: Learning with ten clients at

h=0.8

and

S=4M

, under uniform counts (“Equal”) and personalized counts (Algorithm 1 ). (a) Mean cost gap over

100

runs at

\eta=0.01

; the dashed and dotted curves use the larger step sizes

\eta=0.02

and

0.03

, which only personalized counts tolerate, and the horizontal line marks the uniform-count floor divided by

\Gamma_{k}(S)

(Corollary 3 ; labeled Remark 3 in the legend). (b) Fraction of

200

runs in which all

60

updates and all queried gains are stabilizing, versus step size. (c) Fraction of runs that are stable and within

25\%

of the initial gap after

60

updates, versus step size.

Linear Quadratic Regulator

Linear Quadratic Regulator (LQR) designs feedback controllers that minimize quadratic costs for systems with linear dynamics.

See all 33 papers

Scalar Federated Learning for Linear Quadratic Regulator
Apr 6, 2026Mohammadreza Rostami, Shahriar Talebi, Solmaz S. KiaFederated Learning Linear Quadratic Regulator
University of California, Irvine, CA, US · University of California, Los Angeles, CA, US
Robust Federated Q-Learning with Almost No Communication
Aug 24, 2026Sreejeet Maity, Aritra MitraFederated RL Q-Learning
Department of Electrical and Computer Engineering, North Carolina State University
Federated learning with heavy-tailed gradient noise and communication noise: a variance-reduction based algorithm
Jun 21, 2026Shengchao Zhao, Yongchao LiuStochastic Optimization Nonconvex Stochastic Optimization
School of Mathematics, China University of Mining and Technology, Xuzhou, China · School of Mathematical Sciences, Dalian University of Technology, Dalian, China

Apr 6, 2026J/K move · Enter open · S save

Scalar Federated Learning for Linear Quadratic Regulator

Mohammadreza Rostami, Shahriar Talebi, Solmaz S. Kia

University of California, Irvine, CA, US · University of California, Los Angeles, CA, US

Federated Learning Linear Quadratic Regulator Multi-Agent Control

Open paper AskPDF arXiv

We propose ScalarFedLQR, a communication-efficient federated algorithm for model-free learning of a common policy in linear quadratic regulator (LQR) control of cooperative agents. The method builds on a decomposed projected gradient mechanism, in which each agent communicates only a scalar projection of a local zeroth-order gradient estimate. The server aggregates these scalar messages to reconstruct a global descent direction, reducing per-agent uplink communication from O(d) to O(1), independent of the policy dimension. Crucially, the projection-induced approximation error diminishes as the number of participating agents increases, yielding a favorable scaling law: larger fleets enable more accurate gradient recovery, admit larger stepsizes, and achieve faster linear convergence despite high dimensionality. Under standard regularity conditions for homogeneous agents, all iterates remain stabilizing and the average LQR cost decreases linearly fast. Numerical results demonstrate performance comparable to full-gradient federated LQR with substantially reduced communication, even for heterogeneous (but similar) agents.

Abstract

Linear Quadratic Regulator

Explore similar work

Scalar Federated Learning for Linear Quadratic Regulator

Robust Federated Q-Learning with Almost No Communication

Federated learning with heavy-tailed gradient noise and communication noise: a variance-reduction based algorithm