A Unified Framework for Zero-Shot Reinforcement Learning

Jacopo Di Ventura; Jan Felix Kleuker; Aske Plaat; Thomas Moerland

cs.LGOct 23, 2025

A Unified Framework for Zero-Shot Reinforcement Learning

Authors: Jacopo Di Ventura, Jan Felix Kleuker, Aske Plaat, Thomas Moerland

Organizations: Leiden University

Abstract

Zero-shot reinforcement learning (RL) has emerged as a setting for developing general agents, capable of solving downstream tasks without additional training or planning at test-time. While conventional RL optimizes policies for fixed rewards, zero-shot RL requires learning representations that enable immediate adaptation to arbitrary reward functions. As the field matures, the growing diversity of approaches demands a foundational framework reconciling different perspectives under a common unifying structure. In this work, we introduce a formal, unified framework for zero-shot RL, allowing for rigorous comparisons across methods. We propose a taxonomy organizing the algorithmic landscape along two levels: representation, distinguishing between compositional and direct methods based on their exploitation of action-value function decompositions; and learning paradigm, differentiating between reward-free and pseudo reward-free training. Additionally, we propose a unified view of existing error bounds, decomposing the total error into three primary contributing components: inference, reward, and approximation, serving as a foundation for more grounded comparisons of zero-shot methods.

Figures & tables

Panel of Figure 1 — Figure 1: Taxonomy of zero-shot RL methods, organized along two axes that jointly determine how each method recovers the optimal action-value function $Q_{r}^{\ast}=\mathcal{F}(\mu,r)$ from a learned representation $\mu$ and a reward function $r$ . The first split ( representation ) separates direct methods from compositional methods: direct methods define $\mu$ as the value function itself $\mu=Q^{*}_{r}$ , so the decomposition operator reduces to the identity, $\mathcal{F}=\mathrm{Id}_{\mu}$ ; compositional methods instead define $\mu$ as some intermediate representation that requires a non-trivial operator, $\mathcal{F}\neq\mathrm{Id}_{\mu}$ , to reconstruct task-specific value functions. The second split ( learning paradigm ) separates pseudo reward-free training (dashed boxes), where the target $\mu^{r}(s,a)$ is conditioned on the reward function, from reward-free training (solid boxes), where the target $\mu^{\pi}(s,a)$ is learned independently of any reward. Because direct representations are reward-conditioned by construction, a reward-free direct method is structurally infeasible, marked by the grayed-out node ( $-$ ).

Figure 2: Error terms of Eq. 25 versus representation dimension

d

in Four Rooms (log scale; median and IQR over seeds). Terms absent by construction are omitted. Details in Appendix D .

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Method	Representation	Reward	Paradigm	Policy Extraction
SF+GPI ( Barreto et al., 2017 )	Compositional	Free	Online	GPI
USF ( Borsa et al., 2019 )	Compositional	Pseudo	Online	Instant
USF Lap. ( Wu et al., 2019 )	Compositional	Pseudo	Offline	Instant
FB ( Touati and Ollivier, 2021 ; Touati et al., 2023 )	Compositional	Pseudo	Offline	Instant
FB-AWARE ( Cetin et al., 2025 )	Compositional	Pseudo	Offline	Instant
VISR ( Hansen et al., 2020 )	Compositional	Pseudo	Online	Instant

Appendix

Table 1: Feature matrix of a non-exhaustive list of zero-shot RL methods

Figure 3: Visualization of the environment used to obtain empirical results on error bounds. Images after the first to the left show the predefined features $\phi$ used for SF-based methods. Rewards are normally distributed around a goal with $\sigma=1.5$ .

Figure 4: Empirical measurement of error bounds as the number of tasks grows. Results averaged over 50 seeds. For USF and FB this translates in a bigger pool of sampled reward embeddings on which to train. For SF+GPI the successor features of each sampled policy simply are stored. Results are obtained with dataset of size D = 10000 and latent dimension w = 9 (4x4 blocks).

Figure 5: Empirical measurement of error bounds as the number of transitions in the training dataset grows. Results averaged over 50 seeds. Results are obtained with latent dimension w = 104 (1x1 blocks) and n_tasks = 16 to isolate the approximation error.

Figure 6: What the terms translate to in practice: the fraction of optimal value the extracted policy gives up, across the same three sweeps. Lines are medians over 50 seeds, bands the inter-quartile range; all methods see identical tasks, and transitions at each setting.

Figure 7: The same measurement with the task set held fixed at n_tasks = 4 , over 50 seeds ordered by the distance from the test task to its nearest training task,

\min_{i}\|w_{r}-w_{i}\|

. Fixing

n

removes the averaging effect of the codebook sweep and leaves only the per-draw one. A : SF&GPI books this distance explicitly as

\varepsilon_{\text{inference}}

, while USF and FB, which carry no inference term, show it inside

\varepsilon_{\text{approx}}

instead. B : the value lost, for the same runs. Lines are per-bin medians, bands the inter-quartile range. Latent dimension w = 36 (2x2 blocks), D = 10000 .

Zero-Shot RL

See all 17 papers

Improving Zero-Shot Offline RL via Behavioral Task Sampling
Apr 28, 2026Nazim Bendib, Nicolas Perrin-Gilbert, Olivier SigaudZero-Shot RL Offline RL
Institut des Systèmes Intelligents et de Robotique, ISIR, Sorbonne Université, Paris, France.
Can We Really Learn One Representation to Optimize All Rewards?
Feb 11, 2026Chongyi Zheng, Royina Karegoudra Jayanth, Benjamin EysenbachZero-Shot RL Representation Learning
Princeton University
Sparse-TD-JEPA: Improving Zero-Shot Reinforcement Learning With Sparse Latent Task Representations
Oct 9, 2026Duaa Nakshbandi, Zhiyu An, Wan DuZero-Shot RL Reinforcement Learning
University of California, Merced

Apr 28, 2026J/K move · Enter open · S save

Improving Zero-Shot Offline RL via Behavioral Task Sampling

Nazim Bendib, Nicolas Perrin-Gilbert, Olivier Sigaud

Institut des Systèmes Intelligents et de Robotique, ISIR, Sorbonne Université, Paris, France.

Zero-Shot RL Offline RL Multi-Task RL Reinforcement Learning Generalization in RL Goal-Conditioned RL+1

Open paper AskPDF arXiv

Offline zero-shot reinforcement learning (RL) aims to learn agents that optimize unseen reward functions without additional environment interaction. The standard approach to this problem trains task-conditioned policies by sampling task vectors that define linear reward functions over learned state representations. In most existing algorithms, these task vectors are randomly sampled, implicitly assuming this adequately captures the structure of the task space. We argue that doing so leads to suboptimal zero-shot generalization. To address this limitation, we propose extracting task vectors directly from the offline dataset and using them to define the task distribution used for policy training. We introduce a simple and general reward function extraction procedure that integrates into existing offline zero-shot RL algorithms. Across multiple benchmark environments and baselines, our approach improves zero-shot performance by an average of 20%, highlighting the importance of principled task sampling in offline zero-shot RL.

Abstract

Zero-Shot RL

Explore similar work

Improving Zero-Shot Offline RL via Behavioral Task Sampling

Can We Really Learn One Representation to Optimize All Rewards?

Sparse-TD-JEPA: Improving Zero-Shot Reinforcement Learning With Sparse Latent Task Representations