On the Gradient Domination of the LQG Problem

Kasra Fallah; Leonardo F. Toso; James Anderson

math.OCJul 11, 2025

On the Gradient Domination of the LQG Problem

Authors: Kasra Fallah, Leonardo F. Toso, James Anderson

Organizations: The authors are with the Department of Electrical Engineering at Columbia University, New York, USA.

Abstract

We consider solutions to the linear quadratic Gaussian (LQG) regulator problem via policy gradient (PG) methods. Although PG methods have demonstrated strong theoretical guarantees in solving the linear quadratic regulator (LQR) problem, despite its nonconvex landscape, their theoretical understanding in the LQG setting remains limited. Notably, the LQG problem lacks gradient dominance in the classical parameterization, i.e., with a dynamic controller, which hinders global convergence guarantees. In this work, we study a PG method for the LQG problem by adopting an alternative parameterization of the set of stabilizing controllers and employing a lifting argument. We refer to this parameterization as a history representation of the control input as it is parameterized by past input and output data from the previous p time-steps. Under this parameterization, the infinite-history LQG optimization reduces to an equivalent static state-feedback LQR problem, allowing us to recover gradient dominance and smoothness. For finite history length p, we establish model-based and model-free convergence guarantees up to a geometrically decaying history-truncation error.

Figures & tables

Work	Process noise	Meas. noise	Partial obs.	Param.	Per-iter. stability	Gradient dominance	PG conv.
Fazel et al. [ 2 ] ; follow-ups [ 21 , 4 ]	✗	✗	✗	standard	✓ [ 2 ]	✓	✓
Mohammadi et al. [ 13 ] ; Tang et al. [ 15 ]	✓	✓	✓	standard	N/A	✗	✗
Zhao et al. [ 12 ] ; Xie & Ni [ 23 ]	✗	✗	✓	history representation	✗	✓	✓
This work	✓	✓	✓	lifted history	sufficiently small step size (Thm. 1 )	✓ on $\tilde{\mathcal{K}}_{p}$	✓ under Assms. 2 and 4

TABLE I : Policy gradient methods for linear quadratic control. ✓ = holds; ✗ = does not hold.

Fig. 1 : Mean-square history-truncation error as a function of the history length

p

. The empirical error

\mathbb{E}\|e_{t,p}\|_{2}^{2}

decays geometrically with

p

, consistent with the bound in Lemma 1. The shaded region is the

95\%

confidence interval.

Fig. 2 : Approximation error between the finite- and infinite-history costs as a function of the history length

p

. The quantity

|J_{p}(K)-J_{\infty}(K)|

is evaluated at the fixed stabilizing controller

K=0.75K^{\star}

and exhibits the geometric decay predicted by Lemma 3 .

Fig. 3 : Approximation error between the finite- and infinite-history policy gradients as a function of the history length

p

. The error

\|\nabla J_{p}(K)-\nabla J_{\infty}(K)\|_{F}

, evaluated at

K=0.75K^{\star}

, decreases geometrically with

p

, in agreement with Lemma 3 .

Fig. 4 : Model-based optimality gap for varying history length $p$ .

\displaystyle C_{8}J_{\gamma}(\widetilde{K}^{+})\leq J_{\gamma_{+}}(\widetilde{K}^{+})\leq C_{9}J_{\gamma}(\widetilde{K}^{+}).

Algorithm 2 Discount-Annealing Initialization for PG-LQG

Fig. 6 : Optimality gap of Algorithm 1 over the number of iterations.

Policy Gradient Methods

Policy gradient methods optimize stochastic policies by estimating how parameter changes affect expected reward.

See all 164 papers

Wasserstein Policy Gradient for Entropy-Regularized Linear-Quadratic Control
Aug 7, 2026Zhaoyu Zhu, Rui Gao, Shuang LiMaximum Entropy RL Policy Gradient Methods
University of Illinois Urbana-Champaign · The University of Texas at Austin · The Chinese University of Hong Kong, Shenzhen
Sample Complexity of Linear Quadratic Regulator Without Initial Stability
Feb 20, 2025Amirreza Neshaei Moghaddam, Alex Olshevsky, Bahman GharesifardRL Control Sample Complexity
LQG solution for POMDP without estimating states: A minimum variance approach
Jul 13, 2026Ranjan Sarkar, Prabhat K. MishraStochastic Optimal Control Feedback Control
Department of Artificial Intelligence at Indian Institute of Technology Kharagpur, West Bengal, India

Aug 7, 2026J/K move · Enter open · S save

Wasserstein Policy Gradient for Entropy-Regularized Linear-Quadratic Control

Zhaoyu Zhu, Rui Gao, Shuang Li

University of Illinois Urbana-Champaign · The University of Texas at Austin · The Chinese University of Hong Kong, Shenzhen

Maximum Entropy RL Policy Gradient Methods Linear Quadratic Regulator

Open paper AskPDF arXiv

Wasserstein policy gradient (WPG) updates state-conditional action laws by transport in the action space. We study entropy-regularized discounted linear-quadratic (LQ) control. A Bellman verification argument shows that the unrestricted problem has a linear-Gaussian optimal policy, and the discounted-occupancy-weighted statewise Wasserstein gradient is tangent to this policy class. WPG therefore reduces exactly to a finite-dimensional ODE for the feedback gain and action covariance. We prove that this ODE is globally well posed and converges exponentially from every admissible initialization. For each fixed LQ problem, the exponent has a positive limit as the entropy temperature tends to zero and contains no perturbative factor of the form

\exp(-c/τ)

, while retaining the usual dependence on the conditioning of the control problem.

Abstract

Policy Gradient Methods

Explore similar work

Wasserstein Policy Gradient for Entropy-Regularized Linear-Quadratic Control

Sample Complexity of Linear Quadratic Regulator Without Initial Stability

LQG solution for POMDP without estimating states: A minimum variance approach