Zero-shot reinforcement learning (RL) has emerged as a setting for developing general agents, capable of solving downstream tasks without additional training or planning at test-time. While conventional RL optimizes policies for fixed rewards, zero-shot RL requires learning representations that enable immediate adaptation to arbitrary reward functions. As the field matures, the growing diversity of approaches demands a foundational framework reconciling different perspectives under a common unifying structure. In this work, we introduce a formal, unified framework for zero-shot RL, allowing for rigorous comparisons across methods. We propose a taxonomy organizing the algorithmic landscape along two levels: representation, distinguishing between compositional and direct methods based on their exploitation of action-value function decompositions; and learning paradigm, differentiating between reward-free and pseudo reward-free training. Additionally, we propose a unified view of existing error bounds, decomposing the total error into three primary contributing components: inference, reward, and approximation, serving as a foundation for more grounded comparisons of zero-shot methods.
Figures & tables
Figure 1: Taxonomy of zero-shot RL methods, organized along two axes that jointly determine how each method recovers the optimal action-value function Qr∗=F(μ,r) from a learned representation μ and a reward function r . The first split ( representation ) separates direct methods from compositional methods: direct methods define μ as the value function itself μ=Qr∗ , so the decomposition operator reduces to the identity, F=Idμ ; compositional methods instead define μ as some intermediate representation that requires a non-trivial operator, F=Idμ , to reconstruct task-specific value functions. The second split ( learning paradigm ) separates pseudo reward-free training (dashed boxes), where the target μr(s,a) is conditioned on the reward function, from reward-free training (solid boxes), where the target μπ(s,a) is learned independently of any reward. Because direct representations are reward-conditioned by construction, a reward-free direct method is structurally infeasible, marked by the grayed-out node ( − ).
Figure 2: Error terms of Eq. 25 versus representation dimension d in Four Rooms (log scale; median and IQR over seeds). Terms absent by construction are omitted. Details in Appendix D .
Appendix figures & tables6 assets
Supplementary material from the paper’s appendix.
Appendix
Method
Representation
Reward
Paradigm
Policy Extraction
SF+GPI ( Barreto et al., 2017 )
Compositional
Free
Online
GPI
USF ( Borsa et al., 2019 )
Compositional
Pseudo
Online
Instant
USF Lap. ( Wu et al., 2019 )
Compositional
Pseudo
Offline
Instant
FB ( Touati and Ollivier, 2021 ; Touati et al., 2023 )
Compositional
Pseudo
Offline
Instant
FB-AWARE ( Cetin et al., 2025 )
Compositional
Pseudo
Offline
Instant
VISR ( Hansen et al., 2020 )
Compositional
Pseudo
Online
Instant
Appendix
Table 1: Feature matrix of a non-exhaustive list of zero-shot RL methods
Figure 3: Visualization of the environment used to obtain empirical results on error bounds. Images after the first to the left show the predefined features ϕ used for SF-based methods. Rewards are normally distributed around a goal with σ=1.5 .
Figure 4: Empirical measurement of error bounds as the number of tasks grows. Results averaged over 50 seeds. For USF and FB this translates in a bigger pool of sampled reward embeddings on which to train. For SF+GPI the successor features of each sampled policy simply are stored. Results are obtained with dataset of size D = 10000 and latent dimension w = 9 (4x4 blocks).
Figure 5: Empirical measurement of error bounds as the number of transitions in the training dataset grows. Results averaged over 50 seeds. Results are obtained with latent dimension w = 104 (1x1 blocks) and n_tasks = 16 to isolate the approximation error.
Figure 6: What the terms translate to in practice: the fraction of optimal value the extracted policy gives up, across the same three sweeps. Lines are medians over 50 seeds, bands the inter-quartile range; all methods see identical tasks, and transitions at each setting.
Figure 7: The same measurement with the task set held fixed at n_tasks = 4 , over 50 seeds ordered by the distance from the test task to its nearest training task, mini∥wr−wi∥ . Fixing n removes the averaging effect of the codebook sweep and leaves only the per-draw one. A : SF&GPI books this distance explicitly as εinference , while USF and FB, which carry no inference term, show it inside εapprox instead. B : the value lost, for the same runs. Lines are per-bin medians, bands the inter-quartile range. Latent dimension w = 36 (2x2 blocks), D = 10000 .
Offline zero-shot reinforcement learning (RL) aims to learn agents that optimize unseen reward functions without additional environment interaction. The standard approach to this problem trains task-conditioned policies by sampling task vectors that define linear reward functions over learned state representations. In most existing algorithms, these task vectors are randomly sampled, implicitly assuming this adequately captures the structure of the task space. We argue that doing so leads to suboptimal zero-shot generalization. To address this limitation, we propose extracting task vectors directly from the offline dataset and using them to define the task distribution used for policy training. We introduce a simple and general reward function extraction procedure that integrates into existing offline zero-shot RL algorithms. Across multiple benchmark environments and baselines, our approach improves zero-shot performance by an average of 20%, highlighting the importance of principled task sampling in offline zero-shot RL.
Nazim Bendib, Nicolas Perrin-Gilbert, Olivier Sigaud
Institut des Systèmes Intelligents et de Robotique, ISIR, Sorbonne Université, Paris, France.
As unsupervised pretraining becomes increasingly ubiquitous in reinforcement learning, a more thorough theoretical understanding of these methods becomes of equal importance to their empirical success. We focus on the setting of unsupervised learning via interaction, where the forward-backward (FB) representation learning serves as a prototypical and popular example. In this paper, we shed light on FB by formally contextualizing the method within a broader class of recent methods that use regression to obtain a low-rank approximation of a successor measure ratio. Our analysis clarifies when FB representations can exist and how the low-rank approximation converges in practice. Building upon the theory, we propose a variant of FB that is both more amenable to theoretical understanding and simpler to optimize in practice. Experiments in didactic settings, as well as in 10 state-based and image-based continuous control domains, demonstrate that our method converges to desired representations with 105× smaller errors than FB, achieving +24% improved zero-shot performance on average. We also demonstrate that zero-shot policies inferred by our algorithm provide an efficient initialization if the user prefers further fine-tuning on downstream tasks. Our project website is available at https://chongyi-zheng.github.io/onestep-fb.
Chongyi Zheng, Royina Karegoudra Jayanth, Benjamin Eysenbach
Successor-feature-based zero-shot reinforcement learning methods pre-train a task encoder and successor-feature predictor on offline, reward-free data, then recover a policy at test time by solving a closed-form linear regression over a small labeled dataset. We identify an underexplored limitation of this approach: when the task encoder produces dense representations, the inferred reward signal becomes diffuse across the state space, reducing the contrast between goal-relevant states and the background and providing a weaker signal for the policy to follow. We address this limitation by introducing an l1 sparsity penalty that concentrates reward-predictive structure into a smaller effective subspace of the task encoder's latent space. The penalty is applied only during training and requires no architectural change or modification to the closed-form inference procedure. We instantiate this approach as Sparse-TD-JEPA, a single-line of code addition based on Temporal-Difference Joint-Embedding Predictive Architecture (TD-JEPA) that significantly improved its performance in zero-shot reinforcement learning. We evaluate Sparse-TD-JEPA on 65 tasks across the DeepMind Control (DMC) Suite and OGBench, covering locomotion, navigation, trajectory stitching, puzzle solving, and robotic manipulation. Sparse-TD-JEPA improves over TD-JEPA in all 13 domain-level comparisons. On DMC, it increases the average return from 661 +- 6 to 714 +- 3. On OGBench, it increases the average success rate from 38.0 +- 0.8 to 48.4 +- 2.3, with improvements of 13.4-18.0 percentage points on the evaluated antmaze navigation and stitching domains. Overall, Sparse-TD-JEPA achieves the highest mean performance among established zero-shot baselines in nine of the 13 domains. Analysis further shows that sparsity produces a sharper and more task-informative inferred reward landscape.