Spectral Alignment in Forward-Backward Representations via Temporal Abstraction

Seyed Mahdi B. Azad; Jasper Hoffmann; Iman Nematollahi; Hao Zhu; Abhinav Valada; Joschka Boedecker

cs.LGMar 20, 2026

Spectral Alignment in Forward-Backward Representations via Temporal Abstraction

Authors: Seyed Mahdi B. Azad, Jasper Hoffmann, Iman Nematollahi, Hao Zhu, Abhinav Valada, Joschka Boedecker

Organizations: Department of Computer Science, University of Freiburg, Germany

Abstract

Forward-backward (FB) representations provide a powerful framework for learning the successor representation (SR) in continuous spaces by enforcing a low-rank factorization. However, a fundamental spectral mismatch often exists between the high-rank transition dynamics of continuous environments and the low-rank bottleneck of the FB architecture, making accurate low-rank representation learning difficult. In this work, we analyze temporal abstraction as a mechanism to mitigate this mismatch. By characterizing the spectral properties of the transition operator, we show that temporal abstraction acts analogously to a low-pass filter that suppresses high-frequency spectral components. This suppression reduces the effective rank of the induced SR while preserving a formal bound on the resulting value function error. Empirically, we show that this alignment is a key factor for stable FB learning, particularly at high discount factors where bootstrapping becomes error-prone. Our results identify temporal abstraction as a principled mechanism for shaping the spectral structure of the underlying MDP and enabling effective long-horizon representations in continuous control.

Figures & tables

Figure 1: Q-function via Successor Representation (SR). The SR enables rapid value inference for arbitrary goals (e.g., star marker ). Low-rank structure in SR is desirable for navigation, as it preserves topological features (e.g., rooms) while suppressing transient dynamics. Top: In discrete MDPs, the SR can be computed from the transition matrix. Bottom: In continuous domains, FB ( FB ) learning approximates the SR, where the embedding dimension controls the rank of the approximation. Low-rank structure can arise through (1) explicit constraints ( Low-Rank column; e.g., SVD or small embeddings), (2) long horizons ( High $\gamma$ column), or (3) temporal abstraction ( Temp. Abs. column; e.g., action repetition). In continuous settings, temporal abstraction provides the spectral alignment needed for effective bootstrapping, whereas high $\gamma$ or overly restrictive bottlenecks can impair representation learning, leading to Q-functions with many erroneous local maxima.

Panel of Figure 2 — Figure 2: Effect of temporal abstraction and discount factor on effective rank. Effective rank decreases by increasing $k$ or $\gamma$ in both discrete (top) and continuous (bottom) . Entropy decreases more smoothly as $k$ increases, suggesting a more stable reduction in effective rank as compared to increasing $\gamma$ .

Panel of Figure 3 — Figure 3: Continuous Navigation Environments: Four-Rooms , Maze , Large-Maze , Cube , and Scene

Panel of Figure 4 — Figure 4: Effect of temporal abstraction on performance. Ablation over temporal abstraction (k), embedding dimension (d), and discount factor ( $\gamma$ ) using a continuous four-rooms environment. Addition of temporal abstraction ( $k>1$ ) boosts performance, whereas increasing $d$ or $\gamma$ alone does not.

Panel of Figure 5 — Figure 5: Ablations: Bellman error. Increasing the embedding dimension (a) or discount factor (b) without using temporal abstraction ( $k=1$ ) leads to an increase in the Bellman error. Increasing $k$ (c) does not increase the Bellman error.

Panel of Figure 6 — Figure 6: Ablations: Input type and effective discount factor (a): Image and State inputs use CNN and RBF encodings, respectively. (b): A higher return (larger radius) for a similar task horizon can be achieved by combining a lower nominal $\gamma$ with a higher $k$ . The effective horizon (y-axis) is computed as $\frac{1}{(1-\gamma^{1/k})}$ , expressing the repeat-MDP horizon in environment-frame units.

Panel of Figure 7 — Figure 7: Ablation: Temporal Abstraction vs. Embedding Dimension vs. Discount Factor Temporal abstraction offers a considerable boost in performance even with a moderate number of steps $k$ . Performance is highest in the magenta region around $k\in[5,10]$ across moderate $\gamma$ and $d$ . After the introduction of temporal abstraction, the performance is less sensitive to variations in the embedding dimension (a) than to changes in the discount factor (b) , especially as $\gamma\to 1$ .

Appendix figures & tables14 assets

Supplementary material from the paper’s appendix.

Appendix

Training steps (gradient update)	1e6
Reward type	Sparse
Hidden Layers (Forward, Backward, Actor)	[256, 256]
Ensemble (Forward)	2
Learning Rate (Backward, Actor)	1e-6
Learning Rate (Forward)	1e-5
Batch size (state)	512

Appendix

Table 1: Relevant hyperparameters used for training FB representation.

Figure 8: Tabular layouts. Single-Room (left) is an open area; Four-Rooms (right) is divided by an internal wall cross with door openings. Black cells are walls, white cells are free states, and colored cells mark the six goal regions.

Figure 9: Realization decomposition. Exact shared-subspace error

\|\widehat{M}-M^{\pi}\|_{2}

(solid) equals the spectral rank floor

\sigma_{d+1}(M^{\pi})

(dashed) plus the excess

\epsilon_{\mathrm{real}}

(dotted), mirroring Definition 3.1 and the decomposition in Eq. ( 3 ). Within each panel, rows group the dimensions

d\in\{16,32,64\}

and columns the two environments (top row

\gamma=0.95

, bottom row

\gamma=0.99

). Curves are medians over the six jointly fitted rewards.

Environment	Quantity	$d=16$	$d=32$	$d=64$
Single-Room	$\epsilon_{\mathrm{real}}$	$2.18$	$1.51$	$1.19$
	$\sigma_{d+1}(M^{\pi})$	$2.50$	$1.99$	$1.41$
Four-Rooms	$\epsilon_{\mathrm{real}}$	$2.20$	$1.58$	$1.57$
	$\sigma_{d+1}(M^{\pi})$	$2.76$	$2.05$	$1.18$

Appendix

Table 2: Realization error: median

\epsilon_{\mathrm{real}}

and spectral floor

\sigma_{d+1}(M^{\pi})

(over rewards, discounts, and repeat factors).

Environment	$k=2$	$k=3$	$k=5$	$k=9$
Single-Room	$1.3\%$	$1.9\%$	$2.9\%$	$4.5\%$
Four-Rooms	$2.1\%$	$2.9\%$	$4.5\%$	$6.3\%$

Appendix

Table 3: Action-repeat value error, normalized by

\|Q^{\star}\|_{\infty}

, at effective discount

\beta=0.99

Environment	$k=1$	$k=5$	$k=9$	$k=11$
Single-Room	$1.405$	$1.407$	$1.409$	$1.409$
Four-Rooms	$1.396$	$1.399$	$1.403$	$1.404$

Appendix

Table 4: Spectral SR bound: median Ratio

=

Bound/Exact over ranks and discounts.

Figure 10: Relationship between the main hyperparameters and episodic return. This plot gives an overview of the effect of different combinations of embedding dimension ( $d$ ), discount factor ( $\gamma$ ), and temporal abstraction $k$ over all experiments. Noticeably, $k=1$ or $\gamma=0.999$ leads to poor performance in most combinations.

Panel of Figure 11 — Figure 11: Ablation: Training plots. Increasing embedding dimension $d$ or discount factor $\gamma$ without increasing the temporal abstraction does not yield a meaningful increase in performance.

Figure 12: Successor Representation (SR) and its Q-Function - Discrete and Continuous This is a more complete picture of Figure 1 . The Q-functions are derived from the same SR that is presented here. The star marker marks the starting state for SR and the goal state for the Q-function.

Panel of Figure 13 — Figure 13: Ablations: Normalized Bellman error. Bellman errors are normalized by the Q values. The relative Bellman error decreases sharply as the discount factor is increased (b). However, this decrease in relative Bellman error does not translate to better performance (episodic return) as discussed in Section 5 .

Panel of Figure 14 — Figure 14: Scatter plot of visited states during training. Increasing the action repetition does not significantly affect the coverage of the space in the Large-Maze environment. A similar coverage pattern holds in Four-Rooms and Maze environments. (omitted for brevity)

Panel of Figure 15 — Figure 15: (a): Temporal abstraction in the Cube environment. Unlike the maze environments, the Cube task requires only a small amount of temporal abstraction. Nevertheless, applying action repetition ( $k=3$ ) substantially improves the performance of the FB algorithm, consistent with the trend observed in navigation tasks. (b): Offline play data versus longer training. Without offline play data, the FB algorithm requires substantially longer training (5M steps) to achieve competitive performance in the Cube environment.

Panel of Figure 16 — Figure 16: (a): Temporal abstraction in the Scene environment. Adding temporal abstraction via action repetition improves overall performance; however, too many repetition steps can hinder the precise control needed by the agent. (b): Different tasks of the Scene environment. Depending on the task, a larger or smaller number of action-repetition steps can be optimal.

Panel of Figure 17 — Figure 17: Action Repetition vs Action Chunking. Action repetition or action chunking offers a significant boost in return in Large-Maze (a) and Cube (b) environments.

Explore similar work

Factorized Spectral Representations for Reinforcement Learning
Jul 15, 2026Junyi Wu, Dan LiTensor Decomposition Representation Learning for RL
Department of Industrial and Systems Engineering University of Washington Seattle, WA, USA
Can We Really Learn One Representation to Optimize All Rewards?
Feb 11, 2026Chongyi Zheng, Royina Karegoudra Jayanth, Benjamin EysenbachUnsupervised Learning Representation Learning
Princeton University
Impact of Connectivity on Laplacian Representations in Reinforcement Learning
Mar 9, 2026Tommaso Giorgi, Pierriccardo Olivieri, Keyue Jiang +2Reinforcement Learning Representation Learning for RL
Baker Hughes, Florence, Italy · Politecnico di Milano · University College London, AI Center (UCL) +1

Jul 15, 2026J/K move · Enter open · S save

Factorized Spectral Representations for Reinforcement Learning

Junyi Wu, Dan Li

Department of Industrial and Systems Engineering University of Washington Seattle, WA, USA

Tensor Decomposition Representation Learning for RL State Representation Learning Sample-Efficient RL

Open paper AskPDF arXiv

Learning a compact model of the world from interaction data is central to sample-efficient deep reinforcement learning. Spectral representation methods have become the leading paradigm for representation learning in continuous control by taking a matrix view of the transition kernel, with state-action pairs on one side and next states on the other, and learning a low-rank factorization through self-supervised contrastive objectives. We take this view one step further. The transition kernel is naturally a three-mode tensor over states, actions, and next states, and a CP decomposition gives one feature map per mode. We propose FaStR, which fits this decomposition with a noise contrastive objective, producing separate state, action, and next-state encoders that together form a single spectral representation. The factored form yields a smaller hypothesis class, and the sample size needed for representation learning shrinks by a factor that scales with the smaller of the state and action dimensions. Empirically, FaStR delivers its largest gains on high-dimensional locomotion tasks whose dynamics align with the factored structure, and the learned state encoder transfers intact across actuator shift while only the action encoder is retrained.

Abstract

Explore similar work

Factorized Spectral Representations for Reinforcement Learning

Can We Really Learn One Representation to Optimize All Rewards?

Impact of Connectivity on Laplacian Representations in Reinforcement Learning