Forward-backward (FB) representations provide a powerful framework for learning the successor representation (SR) in continuous spaces by enforcing a low-rank factorization. However, a fundamental spectral mismatch often exists between the high-rank transition dynamics of continuous environments and the low-rank bottleneck of the FB architecture, making accurate low-rank representation learning difficult. In this work, we analyze temporal abstraction as a mechanism to mitigate this mismatch. By characterizing the spectral properties of the transition operator, we show that temporal abstraction acts analogously to a low-pass filter that suppresses high-frequency spectral components. This suppression reduces the effective rank of the induced SR while preserving a formal bound on the resulting value function error. Empirically, we show that this alignment is a key factor for stable FB learning, particularly at high discount factors where bootstrapping becomes error-prone. Our results identify temporal abstraction as a principled mechanism for shaping the spectral structure of the underlying MDP and enabling effective long-horizon representations in continuous control.
Figures & tables
Figure 1: Q-function via Successor Representation (SR). The SR enables rapid value inference for arbitrary goals (e.g., star marker ). Low-rank structure in SR is desirable for navigation, as it preserves topological features (e.g., rooms) while suppressing transient dynamics. Top: In discrete MDPs, the SR can be computed from the transition matrix. Bottom: In continuous domains, FB ( FB ) learning approximates the SR, where the embedding dimension controls the rank of the approximation. Low-rank structure can arise through (1) explicit constraints ( Low-Rank column; e.g., SVD or small embeddings), (2) long horizons ( High γ column), or (3) temporal abstraction ( Temp. Abs. column; e.g., action repetition). In continuous settings, temporal abstraction provides the spectral alignment needed for effective bootstrapping, whereas high γ or overly restrictive bottlenecks can impair representation learning, leading to Q-functions with many erroneous local maxima.
Figure 2: Effect of temporal abstraction and discount factor on effective rank. Effective rank decreases by increasing k or γ in both discrete (top) and continuous (bottom) . Entropy decreases more smoothly as k increases, suggesting a more stable reduction in effective rank as compared to increasing γ .
Figure 3: Continuous Navigation Environments: Four-Rooms , Maze , Large-Maze , Cube , and Scene
Figure 4: Effect of temporal abstraction on performance. Ablation over temporal abstraction (k), embedding dimension (d), and discount factor ( γ ) using a continuous four-rooms environment. Addition of temporal abstraction ( k>1 ) boosts performance, whereas increasing d or γ alone does not.
Figure 5: Ablations: Bellman error. Increasing the embedding dimension (a) or discount factor (b) without using temporal abstraction ( k=1 ) leads to an increase in the Bellman error. Increasing k (c) does not increase the Bellman error.
Figure 6: Ablations: Input type and effective discount factor (a): Image and State inputs use CNN and RBF encodings, respectively. (b): A higher return (larger radius) for a similar task horizon can be achieved by combining a lower nominal γ with a higher k . The effective horizon (y-axis) is computed as (1−γ1/k)1 , expressing the repeat-MDP horizon in environment-frame units.
Figure 7: Ablation: Temporal Abstraction vs. Embedding Dimension vs. Discount Factor Temporal abstraction offers a considerable boost in performance even with a moderate number of steps k . Performance is highest in the magenta region around k∈[5,10] across moderate γ and d . After the introduction of temporal abstraction, the performance is less sensitive to variations in the embedding dimension (a) than to changes in the discount factor (b) , especially as γ→1 .
Appendix figures & tables14 assets
Supplementary material from the paper’s appendix.
Appendix
Training steps (gradient update)
1e6
Reward type
Sparse
Hidden Layers (Forward, Backward, Actor)
[256, 256]
Ensemble (Forward)
2
Learning Rate (Backward, Actor)
1e-6
Learning Rate (Forward)
1e-5
Batch size (state)
512
Appendix
Table 1: Relevant hyperparameters used for training FB representation.
Figure 8: Tabular layouts. Single-Room (left) is an open area; Four-Rooms (right) is divided by an internal wall cross with door openings. Black cells are walls, white cells are free states, and colored cells mark the six goal regions.
Figure 9: Realization decomposition. Exact shared-subspace error ∥M−Mπ∥2 (solid) equals the spectral rank floor σd+1(Mπ) (dashed) plus the excess ϵreal (dotted), mirroring Definition 3.1 and the decomposition in Eq. ( 3 ). Within each panel, rows group the dimensions d∈{16,32,64} and columns the two environments (top row γ=0.95 , bottom row γ=0.99 ). Curves are medians over the six jointly fitted rewards.
Environment
Quantity
d=16
d=32
d=64
Single-Room
ϵreal
2.18
1.51
1.19
σd+1(Mπ)
2.50
1.99
1.41
Four-Rooms
ϵreal
2.20
1.58
1.57
σd+1(Mπ)
2.76
2.05
1.18
Appendix
Table 2: Realization error: median ϵreal and spectral floor σd+1(Mπ) (over rewards, discounts, and repeat factors).
Environment
k=2
k=3
k=5
k=9
Single-Room
1.3%
1.9%
2.9%
4.5%
Four-Rooms
2.1%
2.9%
4.5%
6.3%
Appendix
Table 3: Action-repeat value error, normalized by ∥Q⋆∥∞ , at effective discount β=0.99 .
Environment
k=1
k=5
k=9
k=11
Single-Room
1.405
1.407
1.409
1.409
Four-Rooms
1.396
1.399
1.403
1.404
Appendix
Table 4: Spectral SR bound: median Ratio = Bound/Exact over ranks and discounts.
Figure 10: Relationship between the main hyperparameters and episodic return. This plot gives an overview of the effect of different combinations of embedding dimension ( d ), discount factor ( γ ), and temporal abstraction k over all experiments. Noticeably, k=1 or γ=0.999 leads to poor performance in most combinations.
Figure 11: Ablation: Training plots. Increasing embedding dimension d or discount factor γ without increasing the temporal abstraction does not yield a meaningful increase in performance.
Figure 12: Successor Representation (SR) and its Q-Function - Discrete and Continuous This is a more complete picture of Figure 1 . The Q-functions are derived from the same SR that is presented here. The star marker marks the starting state for SR and the goal state for the Q-function.
Figure 13: Ablations: Normalized Bellman error. Bellman errors are normalized by the Q values. The relative Bellman error decreases sharply as the discount factor is increased (b). However, this decrease in relative Bellman error does not translate to better performance (episodic return) as discussed in Section 5 .
Figure 14: Scatter plot of visited states during training. Increasing the action repetition does not significantly affect the coverage of the space in the Large-Maze environment. A similar coverage pattern holds in Four-Rooms and Maze environments. (omitted for brevity)
Figure 15: (a): Temporal abstraction in the Cube environment. Unlike the maze environments, the Cube task requires only a small amount of temporal abstraction. Nevertheless, applying action repetition ( k=3 ) substantially improves the performance of the FB algorithm, consistent with the trend observed in navigation tasks. (b): Offline play data versus longer training. Without offline play data, the FB algorithm requires substantially longer training (5M steps) to achieve competitive performance in the Cube environment.
Figure 16: (a): Temporal abstraction in the Scene environment. Adding temporal abstraction via action repetition improves overall performance; however, too many repetition steps can hinder the precise control needed by the agent. (b): Different tasks of the Scene environment. Depending on the task, a larger or smaller number of action-repetition steps can be optimal.
Figure 17: Action Repetition vs Action Chunking. Action repetition or action chunking offers a significant boost in return in Large-Maze (a) and Cube (b) environments.
Learning a compact model of the world from interaction data is central to sample-efficient deep reinforcement learning. Spectral representation methods have become the leading paradigm for representation learning in continuous control by taking a matrix view of the transition kernel, with state-action pairs on one side and next states on the other, and learning a low-rank factorization through self-supervised contrastive objectives. We take this view one step further. The transition kernel is naturally a three-mode tensor over states, actions, and next states, and a CP decomposition gives one feature map per mode. We propose FaStR, which fits this decomposition with a noise contrastive objective, producing separate state, action, and next-state encoders that together form a single spectral representation. The factored form yields a smaller hypothesis class, and the sample size needed for representation learning shrinks by a factor that scales with the smaller of the state and action dimensions. Empirically, FaStR delivers its largest gains on high-dimensional locomotion tasks whose dynamics align with the factored structure, and the learned state encoder transfers intact across actuator shift while only the action encoder is retrained.
Junyi Wu, Dan Li
Department of Industrial and Systems Engineering University of Washington Seattle, WA, USA
As unsupervised pretraining becomes increasingly ubiquitous in reinforcement learning, a more thorough theoretical understanding of these methods becomes of equal importance to their empirical success. We focus on the setting of unsupervised learning via interaction, where the forward-backward (FB) representation learning serves as a prototypical and popular example. In this paper, we shed light on FB by formally contextualizing the method within a broader class of recent methods that use regression to obtain a low-rank approximation of a successor measure ratio. Our analysis clarifies when FB representations can exist and how the low-rank approximation converges in practice. Building upon the theory, we propose a variant of FB that is both more amenable to theoretical understanding and simpler to optimize in practice. Experiments in didactic settings, as well as in 10 state-based and image-based continuous control domains, demonstrate that our method converges to desired representations with 105× smaller errors than FB, achieving +24% improved zero-shot performance on average. We also demonstrate that zero-shot policies inferred by our algorithm provide an efficient initialization if the user prefers further fine-tuning on downstream tasks. Our project website is available at https://chongyi-zheng.github.io/onestep-fb.
Chongyi Zheng, Royina Karegoudra Jayanth, Benjamin Eysenbach
Learning compact state representations in Markov Decision Processes (MDPs) has proven crucial for addressing the curse of dimensionality in large-scale reinforcement learning (RL) problems. Existing principled approaches leverage structural priors on the MDP by constructing state representations as linear combinations of the state-graph Laplacian eigenvectors. When the transition graph is unknown or the state space is prohibitively large, the graph spectral features can be estimated directly via sample trajectories. In this work, we prove an upper bound on the approximation error of linear value function approximation under the learned spectral features. We show how this error scales with the algebraic connectivity of the state-graph, grounding the approximation quality in the topological structure of the MDP. We further bound the error introduced by the eigenvector estimation itself, leading to an end-to-end error decomposition across the representation learning pipeline. Additionally, our expression of the Laplacian operator for the RL setting, although equivalent to existing ones, prevents some common misunderstandings, of which we show some examples from the literature. Our results hold for general (non-uniform) policies without any assumptions on the symmetry of the induced transition kernel. We validate our theoretical findings with numerical simulations on gridworld environments.