cs.LGOct 9, 2026
SaveDoes Learning Recurrent Networks Matter in Partially Observable RL?
Organizations: University of Macau Macau SAR, China
Abstract
Reinforcement learning (RL) under partial observability typically re- lies on recurrent networks to encode temporal information, with the recurrent module trained jointly with the policy or value function. This raises a natural question: is learning the recurrent network actually necessary? In this paper, we investigate whether a frozen recurrent network can provide sufficient memory for solving par- tially observable Markov decision processes (POMDPs). Motivated by the theoretical foundations of Reservoir Computing (RC) and Generalized Reservoir Computing (GRC), we study a simple archi- tecture that combines a fixed recurrent network with a trainable nonlinear output module. We evaluate this approach across tasks with varying memory demands, observation modalities, and action spaces. Surprisingly, frozen recurrent networks support recall over hundreds of steps and achieve competitive control performance across diverse POMDPs, while using less than half the number of trainable parameters required by their fully trainable counterparts. These results suggest that learning the recurrent memory module is not always necessary for solving POMDPs. Frozen recurrent networks therefore provide a simple, parameter-efficient, and sur- prisingly strong baseline for partially observable RL. Our source code is available at https://github.com/Aphlios/FRN-POMDP
Figures & tables
Figure 1. Overview of our PQN agent with frozen recurrence (left) and its nonlinear MLP Readout (right). We encode vector and visual observations differently. For visual control tasks, we train a Convolutional Neural Network (CNN) to extract and compress high-dimensional features. We optimize all trainable parameters using the PQN objective. The right panel shows the architecture of our simple MLP in detail. We also illustrate the architecture of the frozen recurrent network in Fig. 7 . Overview of our PQN agent with frozen recurrence (left) and its nonlinear MLP Readout (right). We encode vector and visual observations differently. For visual control tasks, we train a Convolutional Neural Network (CNN) to extract and compress high-dimensional features. We optimize all trainable parameters using the PQN objective. The right panel shows the architecture of our simple MLP in detail. We also illustrate the architecture of the frozen recurrent network in Fig.~\ref{fig:frozen-recurrent-layer}.
| Model | Total | Trainable | Frozen | Reduction |
| PQN | ||||
| GRU | 1,119,746 | 1,119,746 | 0 | — |
| Frozen GRU | 1,119,746 | 331,778 | 787,968 | 70.37% |
| LRU | 858,114 | 858,114 | 0 | — |
| Frozen LRU | 858,114 | 331,778 | 526,336 | 61.34% |
| PPO | ||||
Table 1. Parameter counts for two types of recurrent agents we used. All hidden layers have the same width. Reduction is relative to the fully trainable agent. Parameter counts for two type of recurrent agents we used. All hidden layers have same width. Reduction is relative to the fully trainable agent.
Figure 2. Evaluation results on MemoryChain. The shaded regions report the range of success rates across 5 seeds and 10 evaluation episodesper seed. Evaluation results on MemoryChain. The shaded regions report the range of success rates across 5 seeds, and 10 evaluation per seed.
Figure 3. POPGym Results: (a) learning curves on seven selected environments, with 10 seeds per environment; (b) final evaluation returns over all 18 environments, with 10 seeds per environment and 10 evaluation episodes per seed. POPGym Results: (a) learning curves on seven selected environments, with 10 seeds per environment; (b) final evaluation returns over all 18 environments, with 10 seeds per environment and 10 evaluation episodes per seed.
Figure 4. MuJoCo Playground Results: (a) learning curves across seven position only (P) and velocity only (V) settings, with means and confidence intervals over 10 seeds; (b) final evaluation return averaged across the seven settings over 10 seeds. Each seed is evaluated on 10 episodes after training. MuJoCo Playground Results: (a) learning curves across seven position only (P) and velocity-only (V) settings, with means and confidence intervals over 10 seeds; (b) final evaluation return averaged across the seven settings over 10 seeds. Each seed is evaluated on 10 episodes after training
Figure 5. POPGym Arcade Results: (a) learning curves on four partially observable tasks with means and confidence intervals over 5 seeds; (b) final evaluation return averaged across the four tasks over 5 seeds. Each seed is evaluated on ten episodes. POPGym Arcade Results: (a) learning curves on four partially observable tasks with means and confidence intervals over 5 seeds; (b) final evaluation return averaged across the four tasks over 5 seeds. Each seed is evaluated on ten episodes.
Figure 6. Comparison with trainable or frozen mixer. Bars show final evaluation return across 18 POPGym tasks, with 10 seeds and 10 evaluation episodes per seed. Comparison with trainable or frozen mixer. Bars show final evaluation return across 18 POPGym tasks, with 10 seeds and 10 evaluation episodes per seed.
Figure 7. Frozen recurrent layer and mixer. (a) Each frozen recurrent unit is followed by a mixer. The overlapping outlines represent recurrent layers stacked to form a recurrent network. (b) The mixer simply applies Linear, LayerNorm, and LeakyReLU. Blue denotes trainable parameters. Frozen recurrent layer and mixer. (a) Each frozen recurrent unit is followed by a mixer. The overlapping outlines represent recurrent layers stacked to form a recurrent network. (b) The mixer simply applies Linear, LayerNorm, and LeakyReLU. Blue denotes trainable parameters.
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
Figure A2. POPGym results for DeepESN, SRC, and ParalESN and their trainable variants. (a) Learning curves on the selected seven tasks over ten seeds. (b) Final evaluation returns across all POPGym tasks over ten seeds and ten evaluation episodes per seed The left panel contains seven training-return plots and a shared legend for DeepESN, SRC, and ParalESN and their trainable variants. The right panel compares final evaluation returns averaged across eighteen POPGym settings, with bootstrap confidence intervals. Filled markers show trainable models, and hollow markers show standard models.
Figure A3. Detailed final evaluation returns for LRU, GRU, Frozen GRU , Frozen LRU and MLP on POPGym. Each entry shows the mean sample standard deviation over 10 seeds, with 10 evaluation episodes per seed A heatmap reports final evaluation returns for MLP and variants of GRU and LRU across eighteen POPGym settings. Each cell contains a mean and standard deviation. Columns distinguish trainable models from variants with frozen recurrent cores.
Figure A4. Detailed final evaluation returns for DeepESN, SRC, ParalESN, and their trainable variants on POPGym. Each entry shows the mean sample standard deviation over 10 seeds, with 10 evaluation episodes per seed. A heatmap compares final evaluation returns for DeepESN, SRC, and ParalESN and their trainable variants across eighteen POPGym settings. Each cell contains a mean and standard deviation.
Figure A5. Detailed final evaluation returns for GRU, LRU, and their frozen variants on POPGym Arcade (top) and MuJoCo Playground (bottom). Each entry shows the mean sample standard deviation over 5 and 10 seeds, respectively, with 10 evaluation episodes per seed. The top heatmap reports final evaluation returns for four GRU and LRU configurations on four POPGym Arcade tasks. The bottom heatmap reports the same configurations on seven MuJoCo Playground settings. Each cell shows a mean and standard deviation, and the panels use separate color scales.