Permanent and Transient Representations for Continual Reinforcement Learning

Nishanth Anand; Doina Precup

cs.LGNov 5, 2026

Permanent and Transient Representations for Continual Reinforcement Learning

Authors: Nishanth Anand, Doina Precup

Organizations: School of Computer Science, McGill University and Mila · School of Computer Science, McGill University, Mila, and Google DeepMind

Abstract

Continual Reinforcement Learning (CRL) agents struggle to adapt to new situations while retaining prior knowledge, leading to a stability-plasticity trade-off. One approach to solving this problem is by using complementary learning systems: one for slow, long-term learning and another for quick, transient adaptation. Anand & Precup (2023) instantiated this idea using permanent and transient value functions, but used the same features in both systems. Intuitively, long-term learning should rely more on parametric representations for broad generalization, while transient learning would be best served by non-parametric representations for situation-specific adaptation. In this paper, we explore this idea both theoretically and empirically. By introducing a novel non-parametric approximation for estimating the transient value function in complex tasks, we demonstrate that our method enables online learning and outperforms competitive baselines on image-based tasks and Craftax-Classic, underscoring the effectiveness of our system-level decomposition.

Figures & tables

Panel of Figure 1 — Figure 1: Results in 1D chain environment. (a) TD learning using both features results in the lowest RMSVE. (b) The separate variant of PT-TD learning has a lower initial RMSVE following task changes, and its permanent component converges to the correct value after just one task.

Panel of Figure 2 — Figure 2: Environments used in the experiments.

Panel of Figure 3 — Figure 3: (a) 2D grid prediction results. PT-TD-Separate variant has the lowest RMSVE in both online and other tasks (shown in solid and dotted lines, respectively). (b) Control results. The PTQ-Separate variant adapts faster when the tasks change.

Panel of Figure 4 — Figure 4: (a) Transient Approximator. The observation is mapped to value function in four steps: (1) tokenization; (2) hashing; (3) binning; (4) value estimation. (b) Individual Table. Within each table, the value of the query observation is weighted by $\rho$ , and the remaining mass ( $1-\rho$ ) is distributed among its neighbours to form the estimate.

Panel of Figure 5 — Figure 5: Image Task Results. Our algorithm adapts rapidly when the task changes, whether task boundaries are available or hidden, while other methods are slow.

Panel of Figure 6 — Figure 6: Craftax-Classic Results. Our algorithm outperforms other continual learning algorithms, including ablations, in both two-parallel and 1024-parallel environments.

Figure 7: Chain MDP with the two feature sets

Figure 8: Value function heatmap for the prediction task

Figure 9: Image Task (500) Results.

Parameter	Value
Hyperparameters
ALG_NAME	PQN
NUM_MINIBATCHES	4
NUM_EPOCHS	1
NORM_INPUT	True
NORM_TYPE	layer_norm

Table 3: Hyperparameters for PQN for Image task.

Parameter	Value
TRANSIENT_TABLE_SIZE	128
NUM_TABLES	2
NUM_HASHES	128
NUM_SLOTS	4
TRANSIENT_LR	0.3
DECAY	0.95

Table 4: PT-Ours HPs.

Parameter	Value
TRANSIENT_LR	0.003
DECAY	1.0

Table 5: PT-NN HPs.

Parameter	Value
TRANSIENT_TABLE_SIZE	512
NUM_TABLES	2
NUM_HASHES	128
TRANSIENT_LR	0.5
DECAY	1.0

Table 6: PT-IHT HPs.

Parameter	Value
TRANSIENT_TABLE_SIZE	128
NUM_TABLES	2
NUM_HASHES	128
NUM_SLOTS	4
TRANSIENT_LR	0.3
DECAY	0.95

Table 7: T-Only HPs.

Panel of Figure 10 — Figure 10: Hyperparameter tuning results.

Parameter	Value
NUM_TABLES	2
TRANSIENT_TABLE_SIZE	2048
NUM_SLOTS	32
NUM_HASHES	128
CROP_SIZE	7
$\rho$	0.85

Table 8: PT-Ours HPs for Craftax.

Parameter	Value
NUM_TABLES	16
TRANSIENT_TABLE_SIZE	65536
NUM_SLOTS	32
NUM_HASHES	128
CROP_SIZE	7
TRANSIENT_LR	0.8

Table 9: PT-IHT HPs for Craftax.

Parameter	Value
ALG_NAME	PQN
TOTAL_TIMESTEPS	$5\times 10^{8}$
TOTAL_TIMESTEPS_DECAY	$5\times 10^{8}$
NUM_ENVS	1024
NUM_STEPS	32
EPS_START	0.1

Table 10: PQN HPs for Craftax

Figure 11: Parameter Ablation Studies. Rows represent different parameters tested: Minibatch updates (MB),

\rho

, table configuration, decay parameter, and transient learning rates (v1 and v2). Columns represent standard metrics: Episode Returns, Score, and Episode Lengths.

Panel of Figure 12 — Figure 12: All achievements in the craftax online experiment.

Panel of Figure 13 — Figure 13: All achievements in the craftax 250M experiment.

Complementary Learning Systems

See all 8 papers

Representation Stability in a Minimal Continual Learning Agent
Feb 23, 2026Vishnu SubramanianContinual Learning Lifelong Learning Agents
AI/ML Computational Science Associate Manager, Accenture
To Retain or to Adapt? Generalizing Continual Learning
Jul 6, 2026Giulia Lanzillotta, Mandana Samiei, Doina Precup +2Continual Learning Online Continual Learning
ETH AI Center, Department of Computer Science, ETH Zürich · Mila - Quebec AI Institute, School of Computer Science, McGill University · Mila - Quebec AI Institute +1
Position: Deployed Reinforcement Learning should be Continual
Jun 1, 2026Parnian Behdin, Kevin Roice, Golnaz MesbahiNon-Stationary RL Reinforcement Learning
Alberta Machine Intelligence Institute (Amii), Edmonton, Canada.

Feb 23, 2026J/K move · Enter open · S save

Representation Stability in a Minimal Continual Learning Agent

Vishnu Subramanian

AI/ML Computational Science Associate Manager, Accenture

Continual Learning Lifelong Learning Agents Stability-Plasticity Dilemma Representation Drift Online Continual Learning

Open paper AskPDF arXiv

Continual learning systems are increasingly deployed in environments where retraining or reset is infeasible, yet many approaches emphasize task performance rather than the evolution of internal representations over time. In this work, we study a minimal continual learning agent designed to isolate representational dynamics from architectural complexity and optimization objectives. The agent maintains a persistent state vector across executions and incrementally updates it as new textual data is introduced. We quantify representational change using cosine similarity between successive normalized state vectors and define a stability metric over time intervals. Longitudinal experiments across eight executions reveal a transition from an initial plastic regime to a stable representational regime under consistent input. A deliberately introduced semantic perturbation produces a bounded decrease in similarity, followed by recovery and restabilization under subsequent coherent input. These results demonstrate that meaningful stability plasticity tradeoffs can emerge in a minimal, stateful learning system without explicit regularization, replay, or architectural complexity. The work establishes a transparent empirical baseline for studying representational accumulation and adaptation in continual learning systems.

Abstract

Complementary Learning Systems

Explore similar work

Representation Stability in a Minimal Continual Learning Agent

To Retain or to Adapt? Generalizing Continual Learning

Position: Deployed Reinforcement Learning should be Continual