Reframing Object-Centric Reinforcement Learning: From Absolute to Relative Representations

Cedric Derstroff; Jannis Blüml; Janosch Moos; Michael Girstl; Zihan Ye; Debora Clever; Kristian Kersting

cs.LGOct 20, 2026

Reframing Object-Centric Reinforcement Learning: From Absolute to Relative Representations

Authors: Cedric Derstroff, Jannis Blüml, Janosch Moos, Michael Girstl, Zihan Ye, Debora Clever, Kristian Kersting

Organizations: The Hessian Center for Artificial Intelligence (hessian.AI) · Technische Universität Darmstadt · Centre for Cognitive Science · German Research Center for Artificial Intelligence (DFKI)

Abstract

Object-centric reinforcement learning agents typically receive each object's position in a fixed global frame, even though their decisions usually depend on where objects are relative to one another. These absolute views often fail to capture the local context that is most relevant to the agent's behavior. As a result, an agent must relearn the same response at every location where a configuration can occur, and its performance collapses when the scene is merely shifted. In this work, we propose a shift toward Object-Relative Coordinates fixed to Anchor (ORCA), where entities are described in a relative frame of reference. To operationalize this idea, we introduce Object-Relative inductive-Bias Transformers (ORBiT), a transformer-based architecture for processing anchor-relative inputs in reinforcement learning. Our results suggest these relative representations improve performance for both MLP and transformer agents. Furthermore, we demonstrate that ORBiT outperforms the pixel-based PPO baseline, achieving an aggregate HNS IQM greater than 1.0 over 13 Atari games. ORBiT learns the game of Foosball from scratch, dominating all baselines. In our controlled Tag environment, we further show that ORCA extends to rotations, that ORBiT generalizes to unseen numbers of distractors, and that these gains require a decision-relevant anchor.

Figures & tables

Figure 1: With ORCA inputs, the agent only needs to learn strategies for similar situations once, enabling direct generalization. When the player (Asterix ) is in the same row as the target collectible (cauldron ), e.g., the cauldron $(1,5)$ is right of Asterix $(1,0)$ , the absolute coordinates of the objects should not influence the agent’s decision. The agent should behave the same even if the cauldron is at $(3,9)$ , which is also on the right of Asterix $(3,4)$ , but on a different platform. [words=first]Object-Relative Coordinates fixed to Anchor (ORCA) performs this transformation in object space, simplifying the agent’s decision.

Panel of Figure 2 — Figure 2: Comparison of absolute and relative representations in Asterix . Left: Absolute representations encode object positions with respect to a fixed origin in the environment (e.g., top-left corner). Right: Relative representations encode object positions with respect to the anchor object (e.g., the agent’s location).

Figure 3: ORBiT architecture: a versatile transformer with ORCA inputs . The absolute coordinate object set

\mathcal{O}_{t}

is transformed into the anchor-relative set

\tilde{\mathcal{O}}_{t}

. These ORCA inputs are embedded into the working dimension of the transformer. The transformer’s output sequence is aggregated (mean-pooled) into a single vector

\hat{h}_{t}

before being passed through the linear critic and policy heads. OCT uses

\mathcal{O}_{t}

directly within the otherwise identical architecture.

Figure 4: ORCA improves both architectures in Atari. The gain in HNS IQM is larger for the transformer (OCT/ORBiT,

\Delta{+}0.56

) than for the MLP (PPObj,

\Delta{+}0.15

). Both absolute-coordinate models fall below the pixel baseline (PPO

{}^{\text{\tiny CNN}}

); only ORBiT approaches human level (dotted). Aggregated end performance (HNS IQM) with 95% confidence intervals. Higher is better. Full per-game numbers in Appendix H .

Figure 5: ORBiT enhances robustness and retains the highest performance in most settings. The transformer alone explains the full performance in OCT and ORBiT for shuffle , which is not applicable for the CNN-based PPO

{}^{\text{\tiny CNN}}

that operates on image data. In HackAtari , ORBiT retains its performance similar to PPO

{}^{\text{\tiny CNN}}

, and translate tests Corollary 1 . The IQM of the retained performance

\varrho

when evaluating zero-shot under translated inputs (left, shifts the absolute reference frame while keeping relative structure) and shuffle (middle, permutes the object list while preserving the scene). Bars compare absolute coordinates and player-centric (ORCA) representations across architectures: OCT (transformer over objects), PPO

{}^{\text{\tiny CNN}}

(pixel CNN; no relative variant), and PPObj (MLP over objects). Higher is better. IQM over

5\cdot 10

episodes.

Panel of Figure 6 — Figure 6: ORBiT consistently dominates in the Foosball environment (a). (b) Both PPO ${}^{\text{\tiny MLP}}$ and OCT are decisively outperformed by ORBiT; they score only 330 and 73 goals out of 50k. With ORCA, ORBiT learns effective policies, while OCT (absolute coordinates) remains far behind PPO ${}^{\text{\tiny MLP}}$ . Full per-match numbers are in Appendix H .

Panel of Figure 7 — Figure 7: In Tag , ORBiT outperforms absolute coordinates, but the anchor choice matters. The ORCA observations compute all object observations relative to an anchor object. (a) For example, selecting the player as the anchor transforms absolute (left) into relative (right) representations via a kinematic transformation that sets the player as a static origin (aligned with its heading) and applies the player’s linear velocity $v$ and angular velocity $\omega$ in reverse to all objects. The headings, shown as shaded cones, match the direction of motion in the absolute frame but not in the player’s frame: every object inherits the player’s reversed velocity, plus a sideways velocity due to its rotation. Even static landmarks thus appear to move, and the target’s velocity points past the player, although the target heads almost straight at it. The figure is a schematic, with arrow lengths and turn arcs not drawn to scale. (b) The performance results vary strongly for different anchor choices. Choosing an important game object, such as the player, clearly dominates a randomly moving anchor (distractor).

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

Hyperparameter	Value
ALE version	0.8.1
Gymnasium version	0.29.1
Environment version	v5
Frameskip	4
Buffer Window Size	4
Observation Mode	RGB

Appendix

Table 1: Key hyperparameters regarding the used Atari environments . We are using Gymnasium and ALE , following best practices from Machado et al. (2018a) and the community.

Hyperparameter	Value
Number of Transformer Blocks ( $\ell$ )	$3$
Embedding Dimension ( $d$ )	$128$
Number of Attention Heads	8
Include Width and Height	False

Appendix

Table 2: Key hyperparameters used for ORBiT .

Hyperparameter	Value
Gym version	0.23.1
Task	FoosballMixedSelfplay
DistToBallReward	True
WinReward	150
LossPenalty	150
StagnationPenalty	150

Appendix

Table 3: Key hyperparameters regarding the used Foosball environments .

Hyperparameter	Value (Atari)	Value (Foosball)
Seed	{0, 1, 2, 3, 4}
Learning Rate ( $\alpha$ )	$2.5\cdot 10^{-4}$	$3\cdot 10^{-4}$
Total Timesteps † /Max Epochs ‡	${}^{\dagger}10^{7}$	${}^{\ddagger}6\cdot 10^{7}$
Number of Environments	$10$	$4{,}096$
Batch Size ( $B$ )	$1{,}280$	$4{,}096$
Minibatch Size ( $b$ )	$320$	$4{,}096$

Appendix

Table 4: Key hyperparameters used for training , following Huang et al. (2022a) ; Huang et al. (2022b) ; Moos et al. (2024) . We adopt these parameters without additional tuning to ensure a fair comparison across all methods.

Component	Specification (Atari)	Specification (Foosball)
System	NVIDIA DGX v.5.1.0	NVIDIA DGX v.7.5.0
GPUs	16 $\!\times\!$ NVIDIA V100 32GB	8 $\!\times\!$ NVIDIA A100 40GB
CPUs	Intel Xeon Platinum 8174	AMD EPYC 7742
System Memory	1.58 TB	2.05 TB
Operating System	Ubuntu 20.04 LTS	Ubuntu 24.04 LTS
CUDA Version	12.4	13.0

Appendix

Table 5: Hardware configuration used for our experiments. We used NVIDIA DGX systems and followed standardized training pipelines.

Game	HackAtari Modification
Alien	last_egg
Amidar	pig_enemies
Boxing	switch_position
Breakout	gravity
Freeway	stop_all_cars_tunnel
Frostbite	reposition_floes_easy

Appendix

Table 6: The HackAtari modifications used to evaluate generalization.

Table 7: Extended Foosball results: Goals scored by every agent against every other agent. ORBiT easily defeats the PPO

{}^{\text{\tiny MLP}}

agents on all seed combinations, as well as the OCT without ORCA. The values show the number of goals scored by the first agent (top) against the second agent (left) out of 2,000 total goals; e.g., 7 in (a) means PPO

{}^{\text{\tiny MLP}}

scored 7 goals while ORBiT scored 1,993. Each agent plays 1,000 goals and then switches sides. The bottom row shows the sum of goals scored by the first agent (

\Sigma_{s}

), while the last column shows the number of goals conceded by the second agent (

\Sigma_{c}

Game	ORCA		Absolute Coordinates		Pixel
Game	ORBiT	PPObj	OCT	PPObj	PPO ${}^{\text{\tiny CNN}}$
Alien	$1511\pm 727$	$1658\pm 530$	$474\pm 341$	$748\pm 605$	$1644\pm 649$
translate	$1686\pm 782$	$1613\pm 545$	$242\pm 297$	$584\pm 323$	$277\pm 223$
shuffle	$1639\pm 713$	$310\pm 186$	$494\pm 314$	$228\pm 89$	—
last_egg	$1584\pm 703$	$1741\pm 603$	$100\pm 303$	$60\pm 240$	$1908\pm 1275$
Amidar	$195\pm 88$	$347\pm 78$	$105\pm 78$	$255\pm 124$	$578\pm 128$

Appendix

Table 8: Per-game results on the Atari environments ; mean and standard deviation over 5 random seeds. Higher is better. Games marked with ∗ do not have a translate variant as the game screen content covers the entire frame and a lossless translation is not possible.

Method	Seed					IQM
Method	0	1	2	3	4	IQM
OCT	3980	1800	12680	4470	1800	3435
ORBiT	4700	1800	4620	4710	1800	3727

Appendix

Table 9: The higher average score of OCT in Kangaroo depends on a single seed. The other seeds result in very similar performance, yielding a comparable IQM as well. IQM over all episodes.

	ORCA		Absolute Coordinates
Configuration	ORBiT	PPObj	OCT	PPObj
Base ( $m=5$ , Anchor: Player)	22.4 [21.9, 22.9]	21.6 [18.1, 22.0]	1.8 [1.2, 2.9]	5.6 [1.6, 7.4]
$m^{\prime}=0$ distractors	21.9 [21.1, 22.4]	—	2.4 [1.0, 4.8]	—
$m^{\prime}=1$ distractors	22.1 [21.7, 22.5]	—	2.1 [1.0, 4.0]	—
$m^{\prime}=10$ distractors	22.5 [21.8, 23.3]	—	1.5 [1.1, 2.4]	—
$k^{\prime}=0$ landmarks	21.9 [20.7, 22.4]	—	1.1 [0.8, 1.6]	—

Appendix

Table 10: Per-configuration results in Tag : catches IQM with 95% CI over 5 seeds, each averaged over 100 episodes. Higher is better. Extended results for Figure 7 . PPObj cannot be evaluated beyond its

N_{\max}

object slots, i.e., 1 Player, 1 Target, 2 Landmarks, and 5 Distractors. Landmarks are always spaced evenly along the box diagonal and in orientation, so changing their number moves all of them. The number of distractors and landmarks is changed only during testing, without retraining.

Panel of Figure 8 — Figure 8: The perturbed environment variants to test the generalizability of ORBiT and ORCA. In the shuffled object list variant (a), the order of $\mathcal{O}_{t}$ (or $\tilde{\mathcal{O}}_{t}$ ) is randomized before it is passed to the agent. The random translation (b) shifts the actual game content within the video frame without cutting any content. The HackAtari framework (c) features a variety of variants of the Atari games from level changes to logic changes, e.g., hiding the opponent in Pong .

Reinforcement Learning

Reinforcement learning studies how agents learn sequential decisions from rewards and interaction, balancing exploration, performance, safety, and adaptation.

See all 1,742 papers

Trajectory-Relative Hindsight Distillation for Agentic Reinforcement Learning
Aug 7, 2026Haoyu Zheng, Yun Zhu, Qing Wang +1Agentic RL Credit Assignment in RL
Zhejiang University · Tencent · Shanghai AI Laboratory
Better Slots, Better Worlds: Representation Quality & Robustness in Object-Centric World Models
Aug 12, 2026Shukrullo Nazirjonov, Sai Prasanna, Anna Manasyan +1World Models World Model-Based Planning
University of Tuebingen · Max Planck Institute for Intelligent Systems
FOCI Policy: Focus on Object-Centric Interactions for Relational Manipulation Policies
Sep 8, 2026Ze Fu, Pinhao Song, Yutong Hu +1Generalization in Robotic Manipulation Robotic Manipulation
KU Leuven, Dept. Mechanical Engineering, Research unit Robotics, Automation and Mechatronics · KU Leuven, Dept. Electrical Engineering, Research unit Processing Speech and Images

Aug 7, 2026J/K move · Enter open · S save

Trajectory-Relative Hindsight Distillation for Agentic Reinforcement Learning

Haoyu Zheng, Yun Zhu, Qing Wang, Wenqiao Zhang

Zhejiang University · Tencent · Shanghai AI Laboratory

Agentic RL Credit Assignment in RL Sparse-Reward RL Policy Learning RL for LLM Agents

Open paper AskPDF arXiv

Recent agentic reinforcement learning methods use hindsight to complement sparse outcome rewards. However, a completed rollout can yield many such signals, leaving their appropriate allocation across turns unclear. We introduce TRIAL, a trajectory-relative hindsight distillation framework with a unified turn-aligned scoring protocol. For each decision turn, TRIAL extracts an outcome view of that decision's realized consequence and evaluates the same response under ordinary and hindsight-conditioned contexts. The signed log-probability gap determines the direction and local strength of token-level supervision, while turn-level magnitudes are normalized jointly over the realized trajectory. The resulting allocation multipliers have an eligible-token-weighted mean of one, redistributing dense supervision across turns while fixing its average multiplier. Experiments on WebShop and ALFWorld with different backbones show that TRIAL outperforms GRPO across all eight combinations of backbone, environment, and evaluation metric, while achieving the best or tied-best performance among six methods on six of them. On WebShop with Qwen3-1.7B, TRIAL improves the success rate from 56.4% to 75.2% and the task score from 78.7% to 85.7%. Controlled ablations further show that trajectory-relative turn allocation provides substantial gains beyond those of dense hindsight distillation alone.

Abstract

Reinforcement Learning

Explore similar work

Trajectory-Relative Hindsight Distillation for Agentic Reinforcement Learning

Better Slots, Better Worlds: Representation Quality & Robustness in Object-Centric World Models

FOCI Policy: Focus on Object-Centric Interactions for Relational Manipulation Policies