Learning Stabilizing Latent Feedback Controllers via Multi-Step Backward Losses
Authors: Paul Lutkus, Lars Lindemann, Stephen Tu
Organizations: Thomas Lord Department of Computer Science, University of Southern California · Automatic Control Laboratory, ETH Zürich · Ming-Hsieh Department of Electrical and Computer Engineering, University of Southern California
Can learned latent dynamics models support the design of feedback controllers that provably stabilize the original system? For linear time-invariant systems, we show that learning linear autoencoders and latent dynamics with multi-step prediction-reconstruction losses \unicodex2014 which we term "backward losses" as they encourage latent trajectories to reconstruct full state trajectories \unicodex2014 provably recovers the structure needed for stabilization. Specifically, a sufficiently long reconstruction horizon ensures that a minimum H∞-norm controller designed using the learned latent model stabilizes the original system through output-feedback via the corresponding encoder, provided the latent dimension is large enough to capture all non-stable modes of the underlying system. We also show why both reconstruction and a multi-step horizon are essential: forward prediction losses that require only low prediction error along latent trajectories admit global loss minimizers that discard unstable modes regardless of horizon length, while single-step reconstruction losses can yield latent models whose stabilizing controllers fail to stabilize the original system. Finally, experiments on a non-linear flexible-arm system illustrate the practical implications of our results for learning non-linear autoencoders and latent dynamics.
Figures & tables
Figure 1: Stabilization of a flexible arm using linear latent output feedback. Only Lbwd , the multi-step prediction-reconstruction loss, leads to a stabilizing latent output-feedback controller. In (A), (B), the light gray arm denotes the shared initial state, and opacity denotes the position of the arm in time (darker denotes later). In (B), the × denotes the arm impacting the ground. Panels (C) and (D) are linearizations about the upright equilibrium.
Figure 2: For the single-step backward loss Lbwd(1) , the set of stabilizing gains for the learned latent system separates from the set of output-feedback stabilizing gains for the LTI system ( 5.10 ) as non-normality grows. In the right panel, the spectral radius ρ(Aη+BKEˉ1) of the resulting output-feedback closed-loop matrix is shown.
Figure 3: For the multi-step backward loss Lbwd(T) , the set of stabilizing gains for the learned latent system converges to the set of output-feedback stabilizing gains for LTI system ( 5.10 ) as T grows (here the non-normality parameter η is held fixed). In the right panel, the spectral radius ρ(Aη+BKEˉT) of the resulting output-feedback closed-loop matrix is shown, indicating stable closed-loop behavior for T⩾2 .
Figure 4: Local geometry of the flexible arm described in ( 6.1 ). Section i , of length ℓ , connects joints i and i+1 , and the gravitation force mˉg acts at its center of mass. The solid dots indicate torsional springs of stiffness ki:=κ⋅(K0)ii . The torque τi combines the input, spring, and damping contributions at joint i , with Cd:=c0M+c1Keff .
Figure 5: Linearized flexible arm controlled by linear latent models ( m=4 ). Panels (a, b) show results from training on the population losses Lbwd(T) (blue, λreg=1 ) and Lfwd(T) (orange, most favorable λreg ). (a) Subspace distance Δ of the minimizer with the lowest objective among 8 random initializations (error bars span the 25 %- 75 % interquartile range over the 8 initializations). (b) Number of the 8 initializations whose latent H∞ gain K makes A+BKE Schur stable. (c) The ratio ∥xt∥/∥x0∥ for both the closed-loop of the latent LQR gain (cost J ) learned from Lbwd(40) with Rorth , and of full state LQR (cost J⋆ ). The lines and shaded area illustrate the median and 25 %- 75 % interquartile range over 32 initial states x0∼N(0,In) . The panels (d, e) are as (a, b), but for the empirical losses of N trajectories at T=40 . The black dash-dotted line in (b, e) signifies Algorithm 1 with known (A,B) , which stabilizes at every T and N . Note that series are slightly offset horizontally for legibility.
Table 1: A comparison of stabilization performance for latent H∞ versus latent LQR control design, for both population losses (top, with T varying) and finite-sample losses (bottom, with T=40 fixed, N varying). The subspace distance Δ shown is for the lowest-objective model over 8 initialization (same as Figure 5 (a)) for the population loss, and the median over 5 datasets (each the better of 2 initializations) for the finite-sample loss. Metrics for Lfwd(T) and Lfwd(T) are shown for the value of λreg that produces the largest number of stabilizing models over all T .
Figure 6: MLP (green) and linear (blue) latent models trained on data from the non-linear arm. Solid lines are medians, and error bars are the 25% to 75% interquartile range. All models are trained using the empirical backwards objective Lbwd(T) with latent dimension m=4 (with the modifications to Jfwd(1) discussed in the beginning of Section 6 ). Panels (a)–(d) compare the linearization of the MLP model about the unstable equilibrium q=0 to the linear model. In (a), we compare the subspace distance of row(JE(0)) , where JE(x):=dxdE(x) . Panels (b) and (c) compare relative errors of the unstable eigenvalue of Jaz(0) and induced input gain, against L0 and H0 (shown only for models that capture the unstable mode, i.e. Δ<0.5 ). For panel (c), let w be the left eigenvector of Jaz(0) for the unstable eigenvalue, scaled so that wTJE(0)D0=1 . Thus, h^:=wTgz(0) is invariant to the learned latent representation, and equals H0 when the unstable mode is recovered exactly (see Appendix I for more details). Panel (d) displays ρ(A+BKJE(0)) for the latent LQR gain K , and Algorithm 1 with known (A,B) (dash-dotted). At T=1 , the linear model is degenerate ( E≈0 ) and has no stabilizing latent LQR gain. MLP values for T⩽7 lie above the axis. For the models selected at T=40 , (e) displays for each data set (dot) the number of the 8 initial states from which its controller stabilizes the non-linear arm, under u=KE(x) (filled) and u=KJE(0)x (hollow; the same law for the linear model). Data sets with the same count are stacked vertically. Each initial state releases the arm from rest with its tip displaced along the buckling mode (random sign), plus a random bend that leaves the tip in place. The same 8 initial states are scaled to each tip displacement.
Figure 7: Training on empirical Lbwd(40) with three unstable modes, from the standard initialization (random orthogonal L , grey) and from L=ρ^Im held for the first 500 Adam steps (blue), for each of the 16 random initializations of one data set. The dashed line marks the switch from Adam to L-BFGS. (a) Objective relative to the lowest one found, f⋆ . (b) Real parts of the latent eigenvalues (dotted lines are σ+(A)={1.15,1.12,1.09} ). (c) ∥Prow(E)W+∥F2 , where the columns of W+ are the unit left eigenvectors of the unstable modes (equals 3 when all three modes are captured).
initialization
T=10
T=20
T=30
T=40
random orthogonal L
0/80
0/80
0/80
0/80
L=1.2Im
10/80
71/80
78/80
72/80
L=1.2Im , held for 500 steps
80/80
80/80
80/80
80/80
L=ρ^Im , held for 500 steps
80/80
80/80
80/80
80/80
Table 2: Flexible arm with three unstable modes ( m=m+=3 ): We count runs (over 5 data sets with 16 random initializations each) that capture all three modes i.e., Δ(G+R(AT);row(E))<0.5 (in every run, Δ⩽0.19 or Δ⩾0.82 ).
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 5
Figure 6
Figure 7 , Table 2
population
finite sample
plant
linearized arm
linearized arm
non-linear arm, coordinates x/ε
three-mode linearized arm, mass coordinates
data
none (exact expectations)
5 data sets, N∈{5,20,100,500,1000} , nested
5 data sets, N=2000 , ε=0.01 ( 0.03 and 0.1 in Table 5 )
5 data sets, N=2000
initializations
8
2 per data set and N
MLP: 8 per data set, one continued; linear: 8 per data set
16 per data set and strategy, all counted
Appendix
Table 3: Protocols of the experiments of Section 6 . The data are regenerated at every horizon with the same initial states.
loss
regularizer
N
λmin
rA
ru
rh
H∞ , non-linear arm, tip [cm]
2
10
20
30
40
60
80
Lbwd(T)
Rorth
pop.
1
0.034
2.2⋅10−4
5.5⋅10−5
1.00
1.00
1.00
1.00
1.00
1.00
1.00
100
1
0.057
2.3⋅10−4
6.8⋅10−5
1.00
1.00
1.00
1.00
1.00
1.00
1.00
500
1
0.039
2.3⋅10−4
6.8⋅10−5
1.00
1.00
1.00
1.00
1.00
1.00
1.00
1000
1
0.035
2.2⋅10−4
6.2⋅10−5
1.00
1.00
1.00
1.00
1.00
1.00
1.00
Rvr , N(0,I)
pop.
0.99
0.034
2.2⋅10−4
8.8⋅10−5
1.00
1.00
1.00
1.00
1.00
1.00
1.00
Appendix
Table 4: Linear models at T=40 ( λreg as in Figure 5 ): for the population losses (pop.), the lowest-objective initialization; for the empirical losses, the reported models of the 5 data sets with N trajectories ( Δ and stabilization of A+BKE : Table 1 ). λmin is the smallest eigenvalue of EET , and rA:=∥E^A−L^E^∥F , where (E^,L^,H^):=(S−1E,S−1LS,S−1H) with S:=(EET)1/2 is the same model with orthonormal encoder rows. ru:=∥eTA−λ^ueT∥ and rh:=∣eTB−wTH∣ , where λ^u is the eigenvalue of L nearest λu=1.129 , w is its left eigenvector, and e:=ETw , scaled so that ∥e∥=1 ; both residuals are invariant under any invertible change of latent coordinates. We report ru and rh for models with Δ<0.5 . For the empirical losses, λmin , rA , ru and rh are medians over the data sets. Last columns: fraction of the 8 initial states (averaged over the data sets) from which the H∞ gain stabilizes the non-linear arm, per tip displacement.
ε
T
dev. [%]
fallen [%]
model
captured
Δ
λmin
rA
ru
rh
stable
0.01
10
0.17
0
linear
0/5
1
1
0.026
–
–
0/5
MLP
4/5
0.094
0.99
2
0.11
0.012
4/5
20
2.3
0
linear
5/5
0.015
1
0.037
5.9⋅10−3
1.1⋅10−3
5/5
MLP
5/5
0.019
0.99
2
0.059
2.9⋅10−3
5/5
40
54
47
linear
5/5
0.013
1.4⋅10−4
4.6
0.17
0.07
1/5
MLP
5/5
0.011
2.3⋅10−3
2.6
0.03
2.2⋅10−3
5/5
Appendix
Table 5: Non-linear arm data at scale ε , and the linearization (E,L,H):=(JE(0),Jaz(0),gz(0)) of the models selected at horizon T (for the linear model, its parameters). Data: deviation from linearized rollouts, and fraction of trajectories in which a section passes 90∘ . Models: data sets (of 5 ) that capture the unstable mode, medians over the data sets of Δ , λmin and rA , medians over the capturing data sets of ru and rh (all as in Table 4 ), and data sets whose latent LQR gain stabilizes A+BKE (stabilization of the non-linear arm: Table 6 ).
ε
T
controller
tip displacement [cm]
2
10
20
30
40
60
80
0.01
10
MLP, u=KE(x)
0.65
0.60
0.50
0.40
0.38
0.20
0.20
MLP, u=KJE(0)x
0.70
0.70
0.70
0.75
0.78
0.80
0.80
linear, u=KEx
0.00
0.00
0.00
0.00
0.00
0.00
0.00
20
MLP, u=KE(x)
1.00
1.00
1.00
1.00
0.78
0.25
0.25
MLP, u=KJE(0)x
1.00
1.00
1.00
1.00
1.00
1.00
1.00
Appendix
Table 6: Non-linear arm data at scale ε (cf. Table 5 ): fraction of the 8 initial states, averaged over the 5 data sets, from which the controllers of the models selected at horizon T stabilize the non-linear arm, per tip displacement (panel (e) of Figure 6 shows ε=0.01 , T=40 per data set).
Institute for Bioengineering, School of Engineering and Centre for Engineering Biology, University of Edinburgh, Edinburgh, United Kingdom · Electrical and Computer Engineering and Applied Mathematics, University of Washington, Seattle, WA, USA · Electrical and Computer Engineering and Computer Science & Engineering, University of Washington, Seattle, WA, USA