We consider solutions to the linear quadratic Gaussian (LQG) regulator problem via policy gradient (PG) methods. Although PG methods have demonstrated strong theoretical guarantees in solving the linear quadratic regulator (LQR) problem, despite its nonconvex landscape, their theoretical understanding in the LQG setting remains limited. Notably, the LQG problem lacks gradient dominance in the classical parameterization, i.e., with a dynamic controller, which hinders global convergence guarantees. In this work, we study a PG method for the LQG problem by adopting an alternative parameterization of the set of stabilizing controllers and employing a lifting argument. We refer to this parameterization as a history representation of the control input as it is parameterized by past input and output data from the previous p time-steps. Under this parameterization, the infinite-history LQG optimization reduces to an equivalent static state-feedback LQR problem, allowing us to recover gradient dominance and smoothness. For finite history length p, we establish model-based and model-free convergence guarantees up to a geometrically decaying history-truncation error.
Figures & tables
Work
Process noise
Meas. noise
Partial obs.
Param.
Per-iter. stability
Gradient dominance
PG conv.
Fazel et al. [ 2 ] ; follow-ups [ 21 , 4 ]
✗
✗
✗
standard
✓ [ 2 ]
✓
✓
Mohammadi et al. [ 13 ] ; Tang et al. [ 15 ]
✓
✓
✓
standard
N/A
✗
✗
Zhao et al. [ 12 ] ; Xie & Ni [ 23 ]
✗
✗
✓
history representation
✗
✓
✓
This work
✓
✓
✓
lifted history
sufficiently small step size (Thm. 1 )
✓ on K~p
✓ under Assms. 2 and 4
TABLE I : Policy gradient methods for linear quadratic control. ✓ = holds; ✗ = does not hold.
Fig. 1 : Mean-square history-truncation error as a function of the history length p . The empirical error E∥et,p∥22 decays geometrically with p , consistent with the bound in Lemma 1. The shaded region is the 95% confidence interval.
Fig. 2 : Approximation error between the finite- and infinite-history costs as a function of the history length p . The quantity ∣Jp(K)−J∞(K)∣ is evaluated at the fixed stabilizing controller K=0.75K⋆ and exhibits the geometric decay predicted by Lemma 3 .
Fig. 3 : Approximation error between the finite- and infinite-history policy gradients as a function of the history length p . The error ∥∇Jp(K)−∇J∞(K)∥F , evaluated at K=0.75K⋆ , decreases geometrically with p , in agreement with Lemma 3 .
Fig. 4 : Model-based optimality gap for varying history length p .
C8Jγ(K+)≤Jγ+(K+)≤C9Jγ(K+).
Algorithm 2 Discount-Annealing Initialization for PG-LQG
Fig. 6 : Optimality gap of Algorithm 1 over the number of iterations.
Wasserstein policy gradient (WPG) updates state-conditional action laws by transport in the action space. We study entropy-regularized discounted linear-quadratic (LQ) control. A Bellman verification argument shows that the unrestricted problem has a linear-Gaussian optimal policy, and the discounted-occupancy-weighted statewise Wasserstein gradient is tangent to this policy class. WPG therefore reduces exactly to a finite-dimensional ODE for the feedback gain and action covariance. We prove that this ODE is globally well posed and converges exponentially from every admissible initialization. For each fixed LQ problem, the exponent has a positive limit as the entropy temperature tends to zero and contains no perturbative factor of the form exp(−c/τ), while retaining the usual dependence on the conditioning of the control problem.
Zhaoyu Zhu, Rui Gao, Shuang Li
University of Illinois Urbana-Champaign · The University of Texas at Austin · The Chinese University of Hong Kong, Shenzhen
Inspired by REINFORCE, we introduce a novel receding-horizon algorithm for the Linear Quadratic Regulator (LQR) problem with unknown dynamics. Unlike prior methods, our algorithm avoids reliance on two-point gradient estimates while maintaining the same order of sample complexity. Furthermore, it eliminates the restrictive requirement of starting with a stable initial policy, broadening its applicability. Beyond these improvements, we introduce a refined analysis of error propagation through the contraction of the Riccati operator under the Riemannian distance. This refinement leads to a better sample complexity and ensures improved convergence guarantees.
Amirreza Neshaei Moghaddam, Alex Olshevsky, Bahman Gharesifard
This paper investigates the control of discrete-time linear time-invariant (LTI) systems subject to incomplete and corrupted measurements. Specifically, we focus on designing a Linear Quadratic Gaussian (LQG) controller without relying on explicit state estimation. By leveraging minimum variance duality, our approach allows the current control input to be represented as a linear function of available measurements and previously applied inputs, successfully reducing the task to a tractable deterministic optimization problem. We provide theoretical justification for this framework and demonstrate its practical effectiveness through numerical experiments.
Ranjan Sarkar, Prabhat K. Mishra
Department of Artificial Intelligence at Indian Institute of Technology Kharagpur, West Bengal, India