Grid Cells as Spectral Successor Features: A Reusable Basis for Continual Reinforcement Learning
Organizations: McGill University and Mila – Quebec AI Institute · Université Laval and Mila – Quebec AI Institute
Abstract
Continual reinforcement learning requires agents to adapt when the task changes while reusing what they know about the environment. Successor features separate dynamics from task-specific reward, but their effectiveness depends on the choice of base features. Taking the identification of entorhinal grid cells with eigenvectors of the successor representation as a design principle, we build this basis from eigenvectors of the transition matrix of a fixed reference policy. With such a basis, the successor features at a state-action pair are a linear combination of the base features at that same pair, which is not true in general, and the coefficients are known in closed form: each successor feature is its base feature multiplied by , where is its eigenvalue. The long-term return is thus predicted from the same features as the immediate reward, no successor-feature network or replay buffer is needed, and adapting to a new task reduces to a linear regression on the reward. The agent acts greedily on the action values of the reference policy, which is one step of policy improvement over it and suffices to reach a single goal. In a two-goal gridworld, with the uniform random walk as reference policy, whose eigenvectors are the low-frequency Laplacian eigenvectors, agents built this way maintain near-maximal return across every change of goal, whereas SimpleSF and DQN lose most of their return at each change.
Figures & tables
| DQN | SimpleSF | Offline | Online | |
| Spectral-SF | Spectral-SF | |||
| Input | pixels | pixels | pixels | cell index |
| Base features | — | learned CNN | frozen CNN | eigenvectors of |
| (pretrained) | observed graph | |||
| Successor features | — | learned by TD | , exact | , estimated |
| Learned online | all weights | CNN, head, | graph, |
Appendix figures & tables3 assets
Supplementary material from the paper’s appendix.
Appendix
| All agents | |
| Discount, exploration | ; for k steps, then linear to at k steps, never reset |
| Task sequence | two goals, k steps each, repeated: M steps; seeds |
| DQN | |
| Network | CNN (4 convolutional layers, 32 filters of ), then dense layers of , and units |
| Optimisation | Adam, learning rate ; batch ; one gradient step every steps, from k steps into each task |
| Target | -step double Q-learning; squared TD error with its gradient clipped to , equivalent to a Huber loss with threshold ; soft target network, , applied every steps |
Reinforcement Learning
Reinforcement learning studies how agents learn sequential decisions from rewards and interaction, balancing exploration, performance, safety, and adaptation.
See all 1,862 papers