Full-Gradient Successor Feature Representations

Ritish Shrirao; Aditya Priyadarshi; Raghuram Bharadwaj Diddigi

cs.LGApr 1, 2026

Full-Gradient Successor Feature Representations

Authors: Ritish Shrirao, Aditya Priyadarshi, Raghuram Bharadwaj Diddigi

Organizations: Dept. of CSE, IIIT Bangalore · Dept. of DSAI, IIIT Bangalore

Abstract

Successor Features (SF) combined with Generalized Policy Improvement (GPI) provide a robust framework for transfer learning in Reinforcement Learning (RL) by decoupling environment dynamics from reward functions. However, standard SF learning methods typically rely on semi-gradient Temporal Difference (TD) updates. When combined with non-linear function approximation, semi-gradient methods lack robust convergence guarantees and can lead to instability, particularly in the multi-task setting where accurate feature estimation is critical for effective GPI. Inspired by Full Gradient DQN, we propose Full-Gradient Successor Feature Representations Q-Learning (FG-SFRQL), an algorithm that optimizes the successor features by minimizing the full Mean Squared Bellman Error. Unlike standard approaches, our method computes gradients with respect to parameters in both the online and target networks. We provide a theoretical proof of almost-sure convergence for FG-SFRQL and demonstrate empirically that minimizing the full residual leads to superior sample efficiency and transfer performance compared to semi-gradient baselines in both discrete and continuous domains.

Figures & tables

Parameter	4-Room	Reacher	PointMaze
Training steps / task	10,000	100,000	30,000
Number of tasks	6	4	8
Batch size	64	64	512
Replay buffer size	200,000
Discount factor ( $\gamma$ )	0.95
Exploration rate ( $\epsilon$ )	0.60

TABLE I : Experimental Hyperparameters

Fig. 1 : Cumulative training reward across tasks. FG-SFDQN consistently achieves faster learning and higher cumulative returns than baselines across all environments. Resets to zero indicate task transitions.

Fig. 2 : Ablation of averaging for FG-SFDQN. Curves plot cumulative reward collected on the active task over training steps. FG-SFDQN (Alg. 1 ) achieves much higher cumulative returns compared to its averaging variants with $N=5,10,20$ .

Fig. 3 : Final evaluation performance across environments. FG-SFDQN (Alg. 1) consistently achieves higher returns compared to all baselines. Color scheme: DQN (blue), SFDQN (orange), FG-SFDQN (green), FG-SFDQN Alg. 2 (red), FG-SFDQN Alg. 3 (purple), FGDQN (brown).

Algorithm	Mean (ms)	Variance (ms)
DQN	1.364644	0.004691
SFDQN	11.064082	0.038130
FG-SFDQN (alg1)	17.913484	0.943214
FG-SFDQN (alg2)	16.985953	4.479506
FG-SFDQN (alg3)	18.615328	2.947520
FG-DQN	1.398863	0.001512

TABLE II : Computational overhead comparison

Explore similar work

Robust Successor Features
Sep 25, 2026Erik Nikulski, Yamen Habib, Vicenç Gomez +3Reinforcement Learning Generalization in RL
Department of Engineering Universitat Pompeu Fabra
On the Convergence of Semi-Gradient TD: A Mean Field Perspective
Oct 14, 2026Haruki Settai, Tadashi Kozuno, Shinji ItoTemporal-Difference Learning Q-Learning
The University of Tokyo · Sun Intelligence · The University of Tokyo & RIKEN
Chunk-Guided Q-Learning
Mar 14, 2026Gwanwoo Song, Kwanyoung Park, Youngwoon LeeReinforcement Learning Q-Learning
Department of Artificial Intelligence, Yonsei University · UC Berkeley

Sep 25, 2026J/K move · Enter open · S save

Robust Successor Features

Erik Nikulski, Yamen Habib, Vicenç Gomez, Anders Jonsson+2

Department of Engineering Universitat Pompeu Fabra

Reinforcement Learning Generalization in RL Successor Representations Robust RL Sequential Decision Making

Open paper AskPDF arXiv

Generalization in Reinforcement Learning (RL) refers to the ability to execute close-to-optimal policies in unseen tasks after the agent has been trained on a different set of tasks. Building on the seminal work of the successor representation and further adaptations with function approximation, Transfer in RL has traditionally focused on generalizing to tasks that only differ in the reward function. A decade after the introduction of the successor representation, Robust RL emerged simultaneously from several articles in the field of operations research. In Robust RL, the transition kernel is unknown, and the goal is to maximize the expected reward under this uncertainty. Our work unifies these two paradigms through robust successor features, which generalize across both the reward function and the transition kernel, under the assumption that tasks are linear Markov Decision Processes. We derive a bound on Generalized Policy Improvement (GPI) that explicitly quantifies how performance degrades with the mismatch between transition kernels, recovering existing successor-feature guarantees when dynamics are shared. Finally, the generalization capabilities of robust successor features are validated on several grid-based benchmarks and compared to previous alternatives that focus solely on either the reward or the transition kernel.

Abstract

Explore similar work

Robust Successor Features

On the Convergence of Semi-Gradient TD: A Mean Field Perspective

Chunk-Guided Q-Learning