Reinforcement learning (RL) under partial observability typically re- lies on recurrent networks to encode temporal information, with the recurrent module trained jointly with the policy or value function. This raises a natural question: is learning the recurrent network actually necessary? In this paper, we investigate whether a frozen recurrent network can provide sufficient memory for solving par- tially observable Markov decision processes (POMDPs). Motivated by the theoretical foundations of Reservoir Computing (RC) and Generalized Reservoir Computing (GRC), we study a simple archi- tecture that combines a fixed recurrent network with a trainable nonlinear output module. We evaluate this approach across tasks with varying memory demands, observation modalities, and action spaces. Surprisingly, frozen recurrent networks support recall over hundreds of steps and achieve competitive control performance across diverse POMDPs, while using less than half the number of trainable parameters required by their fully trainable counterparts. These results suggest that learning the recurrent memory module is not always necessary for solving POMDPs. Frozen recurrent networks therefore provide a simple, parameter-efficient, and sur- prisingly strong baseline for partially observable RL. Our source code is available at https://github.com/Aphlios/FRN-POMDP
Figures & tables
Figure 1. Overview of our PQN agent with frozen recurrence (left) and its nonlinear MLP Readout (right). We encode vector and visual observations differently. For visual control tasks, we train a Convolutional Neural Network (CNN) to extract and compress high-dimensional features. We optimize all trainable parameters using the PQN objective. The right panel shows the architecture of our simple MLP in detail. We also illustrate the architecture of the frozen recurrent network in Fig. 7 . Overview of our PQN agent with frozen recurrence (left) and its nonlinear MLP Readout (right). We encode vector and visual observations differently. For visual control tasks, we train a Convolutional Neural Network (CNN) to extract and compress high-dimensional features. We optimize all trainable parameters using the PQN objective. The right panel shows the architecture of our simple MLP in detail. We also illustrate the architecture of the frozen recurrent network in Fig.~\ref{fig:frozen-recurrent-layer}.
Model
Total
Trainable
Frozen
Reduction
PQN
GRU
1,119,746
1,119,746
0
—
Frozen GRU
1,119,746
331,778
787,968
70.37%
LRU
858,114
858,114
0
—
Frozen LRU
858,114
331,778
526,336
61.34%
PPO
Table 1. Parameter counts for two types of recurrent agents we used. All hidden layers have the same width. Reduction is relative to the fully trainable agent. Parameter counts for two type of recurrent agents we used. All hidden layers have same width. Reduction is relative to the fully trainable agent.
Figure 2. Evaluation results on MemoryChain. The shaded regions report the range of success rates across 5 seeds and 10 evaluation episodesper seed. Evaluation results on MemoryChain. The shaded regions report the range of success rates across 5 seeds, and 10 evaluation per seed.
Figure 3. POPGym Results: (a) learning curves on seven selected environments, with 10 seeds per environment; (b) final evaluation returns over all 18 environments, with 10 seeds per environment and 10 evaluation episodes per seed. POPGym Results: (a) learning curves on seven selected environments, with 10 seeds per environment; (b) final evaluation returns over all 18 environments, with 10 seeds per environment and 10 evaluation episodes per seed.
Figure 4. MuJoCo Playground Results: (a) learning curves across seven position only (P) and velocity only (V) settings, with means and confidence intervals over 10 seeds; (b) final evaluation return averaged across the seven settings over 10 seeds. Each seed is evaluated on 10 episodes after training. MuJoCo Playground Results: (a) learning curves across seven position only (P) and velocity-only (V) settings, with means and confidence intervals over 10 seeds; (b) final evaluation return averaged across the seven settings over 10 seeds. Each seed is evaluated on 10 episodes after training
Figure 5. POPGym Arcade Results: (a) learning curves on four partially observable tasks with means and confidence intervals over 5 seeds; (b) final evaluation return averaged across the four tasks over 5 seeds. Each seed is evaluated on ten episodes. POPGym Arcade Results: (a) learning curves on four partially observable tasks with means and confidence intervals over 5 seeds; (b) final evaluation return averaged across the four tasks over 5 seeds. Each seed is evaluated on ten episodes.
Figure 6. Comparison with trainable or frozen mixer. Bars show final evaluation return across 18 POPGym tasks, with 10 seeds and 10 evaluation episodes per seed. Comparison with trainable or frozen mixer. Bars show final evaluation return across 18 POPGym tasks, with 10 seeds and 10 evaluation episodes per seed.
Figure 7. Frozen recurrent layer and mixer. (a) Each frozen recurrent unit is followed by a mixer. The overlapping outlines represent recurrent layers stacked to form a recurrent network. (b) The mixer simply applies Linear, LayerNorm, and LeakyReLU. Blue denotes trainable parameters. Frozen recurrent layer and mixer. (a) Each frozen recurrent unit is followed by a mixer. The overlapping outlines represent recurrent layers stacked to form a recurrent network. (b) The mixer simply applies Linear, LayerNorm, and LeakyReLU. Blue denotes trainable parameters.
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
Figure A2. POPGym results for DeepESN, SRC, and ParalESN and their trainable variants. (a) Learning curves on the selected seven tasks over ten seeds. (b) Final evaluation returns across all 18 POPGym tasks over ten seeds and ten evaluation episodes per seed The left panel contains seven training-return plots and a shared legend for DeepESN, SRC, and ParalESN and their trainable variants. The right panel compares final evaluation returns averaged across eighteen POPGym settings, with bootstrap confidence intervals. Filled markers show trainable models, and hollow markers show standard models.
Figure A3. Detailed final evaluation returns for LRU, GRU, Frozen GRU , Frozen LRU and MLP on POPGym. Each entry shows the mean ± sample standard deviation over 10 seeds, with 10 evaluation episodes per seed A heatmap reports final evaluation returns for MLP and variants of GRU and LRU across eighteen POPGym settings. Each cell contains a mean and standard deviation. Columns distinguish trainable models from variants with frozen recurrent cores.
Figure A4. Detailed final evaluation returns for DeepESN, SRC, ParalESN, and their trainable variants on POPGym. Each entry shows the mean ± sample standard deviation over 10 seeds, with 10 evaluation episodes per seed. A heatmap compares final evaluation returns for DeepESN, SRC, and ParalESN and their trainable variants across eighteen POPGym settings. Each cell contains a mean and standard deviation.
Figure A5. Detailed final evaluation returns for GRU, LRU, and their frozen variants on POPGym Arcade (top) and MuJoCo Playground (bottom). Each entry shows the mean ± sample standard deviation over 5 and 10 seeds, respectively, with 10 evaluation episodes per seed. The top heatmap reports final evaluation returns for four GRU and LRU configurations on four POPGym Arcade tasks. The bottom heatmap reports the same configurations on seven MuJoCo Playground settings. Each cell shows a mean and standard deviation, and the panels use separate color scales.
The family of linear recurrent neural networks has shown strong performance as recurrent memory units in partially observable reinforcement learning. We provide a theoretical justification for their empirical effectiveness by constructing and studying two linear filters: (i) the first exactly reproduces the pre-softmax logits of the belief vector in a hidden Markov model (HMM) under a deterministic transition matrix, thereby serving as a sufficient statistic for optimal policy learning, (ii) the second achieves vanishing state-decoding error under a nearly deterministic transition matrix, thus reducing state ambiguity to near zero. The results extend to action-controlled HMMs, where the corresponding linear filters become time-varying with action-dependent dynamics. We illustrate our main results through numerical experiments and further show that the constructed linear filter serves as a strong feature extractor in a small reinforcement learning game.
Yike Zhao, Onno Eberhard, Malek Khammassi +2
EPFL, Lausanne, Switzerland · Max Planck Institute for Intelligent Systems, Tübingen, Germany
Streaming reinforcement learning has emerged as an online learning paradigm that conforms to the restrictions of natural learning agents that process data incrementally, i.e. with a batch size of 1 and no replay buffer. While streaming RL has recently been shown to scale with deep function approximation with full observability, partially observable settings have remained out of reach. Truncated backpropagation through time collapses to a one-step gradient horizon under the streaming setting, and exact real-time recurrent learning is prohibitively expensive. We close this gap using recurrent trace units, a diagonal recurrent architecture that enables exact RTRL with linear time and memory complexity in the parameter count, and show that they integrate cleanly into existing streaming algorithms across both discrete and continuous control. On a MemoryChain diagnostic with chain lengths from 2 to 128, our method sustains performance where streaming TBPTT(1) baselines using feedforward, GRU, and RTU networks collapse. On five POPGym tasks and on partially observable MuJoCo continuous control, the streaming approach is competitive with batched PPO on POPGym and recovers a substantial fraction of batched performance on masked MuJoCo, despite using no replay buffer or batched updates.
Noah Farr, Aryaman Reddi, Carlo D'Eramo +1
Technical University of Darmstadt · Zuse School ELIZA · Center for Artificial Intelligence and Data Science, University of Würzburg +1
Partially observable environments pose a fundamental challenge in deep reinforcement learning, requiring agents to compress temporal information from observations and maintain a memory to make effective decisions. While there exist many approaches ranging from gated recurrence to attention mechanisms and model-based RL, the search for effective representational techniques that can capture long-term dependencies remains an active area of research. In this work we revisit Unitary recurrent networks (uRNNs) [Arjovsky et al., 2016, Jing et al., 2017], that demonstrated superior gradient flow and associative recall, expressing the recurrence and the hidden state in a complex vector space. Their norm preserving unitary dynamics enable information propagation through long sequences. To this end, we propose three different versions of uRNNs as drop-in replacements for recurrent PPO architectures, and demonstrate that the simple recurrence and the added degree of freedom from the phase of the complex representations enable significant gains over baselines on several memory-improvable tasks, including continuous control. We further explore how to preserve the phase information of the complex hidden state for a phase-aware policy by drawing a parallel to how quantum states are measured. With our methods reaching up to 2-3 × the reward in environments like rocksample and Craftax compared to the baselines, this work points towards an exciting new direction of representations for RL and the problem of partial observability. Code is available at: https://github.com/Sathya98/qurl
Sathya Kamesh Bhethanabhotla, Efstratios Gavves, André Biedenkapp