In this paper, we present a Koopman autoencoder-based least-squares policy iteration (KAE-LSPI) algorithm in reinforcement learning (RL). The KAE-LSPI algorithm is based on reformulating the so-called least-squares fixed-point approximation method in terms of extended dynamic mode decomposition (EDMD), thereby enabling automatic feature learning via the Koopman autoencoder (KAE) framework. The approach is motivated by the lack of a systematic choice of features or kernels in linear RL techniques. We compare the KAE-LSPI algorithm with two previous works, the classical least-squares policy iteration (LSPI) and the kernel-based least-squares policy iteration (KLSPI), using stochastic chain walk and inverted pendulum control problems as examples. Unlike previous works, no features or kernels need to be fixed a priori in our approach. Empirical results show the number of features learned by the KAE technique remains reasonable compared to those fixed in the classical LSPI algorithm. The convergence to an optimal or a near-optimal policy is also comparable to the other two methods.
Figures & tables
Fig. 1 : Architecture of the KAE. In our case, the activation function tanh is used only between the two last layers in the encoder part, otherwise we use the with the rectified linear unit (ReLU): f(x)=max(0,x) .
20 States
50 States
Learned features k
15
45
Encoder layers
[128,64,32]
[256,128,64]
Decoder layers
[32,64,128]
[64,128,256]
Learning rate
10−4
10−4
Batch size
256
256
Epochs
300
500
TABLE I : KAEhyperparameters for the chain walk problem
Fig. 2 : Policy evolution across iterations for (a) LSPI. (b) KLSPI. (c) KAE-LSPI. Policies are represented in the radial direction. The white gap between states 1 and 20 indicates the absence of transitions between them. (d) KLSPIdictionary size per iteration.
Fig. 3 : Policies by iteration for (a) LSPI. (b) KLSPI. (c) KAE-LSPI. (d) KLSPIdictionary size per iteration.
Fig. 4 : (a) LSPIperformance. (b) KAE-LSPIperformance. The red and black lines represent the worst and the best performances, respectively.
Reinforcement learning control uses reward-driven learning to make decisions in dynamic systems, from robots and vehicles to energy and infrastructure.