SlotVLA: Towards Modeling of Object-Relation Representations in Robotic Manipulation
Authors: Taisei Hanyu, Nhat Chung, Huy Le, Toan Nguyen, Yuki Ikebe, Anthony Gunderman, Duy Nguyen Ho Minh, Khoa Vo, +5 more
Organizations: University of Arkansas, USA · FPT Software AI Center, Vietnam · University of Stuttgart, Germany · German Research Center for Artificial Intelligence, Germany · Max Planck Research School for Intelligent Systems, Germany · Aalborg University, Denmark · Carnegie Mellon University, USA · University of Liverpool, UK
Inspired by how humans reason over discrete objects and their relationships, we explore whether compact object-centric and object-relation representations can form a foundation for multitask robotic manipulation. Most existing robotic multitask models rely on dense embeddings that entangle both object and background cues, raising concerns about both efficiency and interpretability. In contrast, we study object-relation-centric representations as a pathway to more structured, efficient, and explainable visuomotor control. Our contributions are two-fold. First, we introduce LIBERO+, a fine-grained benchmark dataset designed to enable and evaluate object-relation reasoning in robotic manipulation. Unlike prior datasets, LIBERO+ provides object-centric annotations that enrich demonstrations with box- and mask-level labels as well as instance-level temporal tracking, supporting compact and interpretable visuomotor representations. Second, we propose SlotVLA, a slot-attention-based framework that captures both objects and their relations for action decoding. It uses a slot-based visual tokenizer to maintain consistent temporal object representations, a relation-centric decoder to produce task-relevant embeddings, and an LLM-driven module that translates these embeddings into executable actions. Experiments on LIBERO+ demonstrate that object-centric slot and object-relation slot representations drastically reduce the number of required visual tokens, while providing competitive generalization. Together, LIBERO+ and SlotVLA provide a compact, interpretable, and effective foundation for advancing object-relation-centric robotic manipulation.
Figures & tables
Fig. 1: Comparison of visuomotor tokenization strategies. (a) Dense tokenizers generate hundreds of tokens across the scene, leading to computationally costly representations. (b) Object-centric tokenizer yields NK tokens, each representing an object. (c) Our object–relation-centric tokenizer yields NS object tokens and NR relation tokens, producing structured and efficient representations. Plot (2) shows that our method achieves higher success rates with fewer tokens compared to baselines on LIBERO-Goal.
Fig. 2: Overview of the LIBERO+ dataset.
Statistics
LIBERO+
L-Object
L-Goal
L-Spatial
L-Long
# Tasks
10
10
10
10
# Object Layouts
1
1
10
9
# Objects
12
7
11
29
# TR Objects
2–3
2–3
3–4
3–4
# Total Frames
72,063
54,779
47,253
84,896
TABLE I: Statistics of LIBERO+. TR corresponds to Task-relevant objects.
Fig. 3: Overall framework of our proposed model. Stage-1 trains the Task-aware Object-Centric Encoder with slot attention and task-aware filtering. Stage-2 freezes Stage-1 parameters and introduces the Relation-Centric Encoder, enabling relational reasoning for final action decoding.
Fig. 4: Slot decomposition result. Task query: “ Put the bowl on the stove ”. Task-relevant slots correctly bind to objects, while irrelevant slots scatter.
LIBERO-Goal
LIBERO-Spatial
LIBERO-Object
LIBERO-Long
OpenVLA
OC
ORC
OpenVLA
OC
ORC
OpenVLA
OC
ORC
OpenVLA
OC
ORC
Average
0.77
0.77
0.86
0.72
0.48
0.60
0.70
0.90
0.91
0.56
0.12
0.31
No. Token
256
4 ( 64 × )
20 ( 13 × )
256
4 ( 64 × )
28 ( 9 × )
256
4 ( 64 × )
28 ( 9 × )
256
4 ( 64 × )
28 ( 9 × )
GFLOPs
2,112
561 ( 4 × )
697 ( 3 × )
2,112
568 ( 4 × )
723 ( 3 × )
2,112
568 ( 4 × )
723 ( 3 × )
2,112
568 ( 4 × )
723 ( 3 × )
Task 1
0.60
0.90
0.70
0.82
0.70
0.65
0.75
0.85
0.95
0.75
0.50
0.20
Task 2
0.95
0.50
0.75
0.95
0.90
0.20
0.70
0.80
0.80
0.90
0.00
0.00
TABLE II: Benchmark on LIBERO+ consisting of four subsets from LIBERO. Highest results are bolded , second-highest are underlined . No. Token indicates the number of tokens used. The tasks, numbered from 1 to 10, are specific to each subset and sorted alphabetically.
LIBERO-Goal
LIBERO-Spatial
LIBERO-Object
LIBERO-Long
OC
ORC
OC
ORC
OC
ORC
OC
ORC
Language
✗
✓
✗
✓
✗
✓
✗
✓
✗
✓
✗
✓
✗
✓
✗
✓
Average
0.77
0.77
0.72
0.86
0.53
0.48
0.60
0.60
0.76
0.90
0.91
0.91
0.11
0.07
0.12
0.31
Task 1
0.75
0.90
0.60
0.70
0.75
0.70
0.80
0.65
0.95
0.85
0.75
0.95
0.05
0.15
0.50
0.20
Task 2
1.00
0.50
0.40
0.75
0.70
0.90
0.20
0.20
0.40
0.80
0.85
0.80
0.00
0.00
0.00
0.00
Task 3
0.95
0.90
0.70
1.00
0.15
0.00
0.75
0.40
1.00
1.00
0.85
1.00
0.25
0.00
0.10
0.40
TABLE III: Ablation study on Task-Aware Slot Filtering. Object-centric slots (OC) and object–relation-centric slots (ORC) are compared. ✓ indicates that filtering is included, while ✗ indicates that filtering is not included. The tasks, numbered from 1 to 10 are specific to each subset and sorted alphabetically.
Method
4 tokens
8 tokens
16 tokens
OC
0.77
0.65
0.77
ORC
0.86
0.74
0.72
TABLE IV: Ablation study on number of object tokens.
Fig. 5: Trajectory demonstration in simulation from exocentric views. Task query: “Put the bowl on the stove”.
Method
Temporal Consistency
✗
✓
OC
0.38
0.77
ORC
0.40
0.86
TABLE V: Ablation study on the effect of temporal consistency.
Visual foundation models provide strong perceptual features for robotics, but their dense representations lack explicit object-level structure, limiting robustness and controllability in manipulation tasks. We propose STORM (Slot-based Task-aware Object-centric Representation for robotic Manipulation), a lightweight object-centric adaptation module that augments frozen visual foundation models with a small set of task-aware slots for robotic manipulation. Rather than fully tuning large backbones on the task, STORM employs an efficient two-stage training strategy: few layers of object-centric representation are first trained on top of the frozen backbone through visual--semantic pretraining using language embeddings, then jointly adapted with a downstream manipulation policy for task alignement. This staged learning prevents degenerate slot formation and preserves semantic consistency while aligning perception with task objectives. Experiments on object discovery benchmarks and robotic manipulation tasks show that STORM improves control performance and generalization to visual shifts (distractors, textures, lighting) compared to directly using frozen or fine-tuned foundation model features, or existing object-centric representations. STORM serves not only as an efficient mechanism for refining generic foundation model features, but also as a novel way of injecting beneficial structural and semantic bias into policy learning.
The generalization capabilities of robotic manipulation policies are heavily influenced by the choice of visual representations. Existing approaches typically rely on representations extracted from pre-trained encoders, using two dominant types of features: global features, which summarize an entire image via a single pooled vector, and dense features, which preserve a patch-wise embedding from the final encoder layer. While widely used, both feature types mix task-relevant and irrelevant information, leading to poor generalization under distribution shifts, such as changes in lighting, textures, or the presence of distractors. In this work, we explore an intermediate structured alternative: Slot-Based Object-Centric Representations (SBOCR), which group dense features into a finite set of object-like entities. This representation permits to naturally reduce the noise provided to the robotic manipulation policy while keeping enough information to efficiently perform the task. We benchmark a range of global and dense representations against intermediate slot-based representations, across a suite of simulated and real-world manipulation tasks ranging from simple to complex. We evaluate their generalization under diverse visual conditions, including changes in lighting, texture, and the presence of distractors. Our findings reveal that SBOCR-based policies outperform dense and global representation-based policies in generalization settings, even without task-specific pretraining. These insights suggest that SBOCR is a promising direction for designing visual systems that generalize effectively in dynamic, real-world robotic environments.
Alexandre Chapin, Bruno Machado, Emmanuel Dellandréa +1
Robotic manipulation policies rely on pre-trained vision models that give either a global scene embedding or a dense patch grid. Both mix task-relevant and task-irrelevant features. Object-centric slot representations are a structured alternative: they group features into a few per-object slots. We test what this structure buys on ManiSkill3 PickCube-v1, with a frozen encoder and a held-out-seed evaluation. Holding the policy, goal token, rendering, and calibration fixed and changing only the encoder, a frozen object-centric SPOT representation (DINO ViT-B/16 + Slot Attention) reaches 55.0±2.9% success, 22.4% above a dense DINO global-feature baseline (32.6 ± 1.5%), with the same trainable policy and no encoder fine-tuning. More tokens alone do not help: a dense patch grid with 16x the tokens performs no better than the global feature. Adding an explicit 2D spatial goal and native-resolution rendering raises the full system to 68.7±4.2%, just below a privileged 3D-oracle upper bound (71.7±4.1%). An automated kinematic failure taxonomy then separates spatial-precision (Near-Miss) failures from object-tracking (No-Grasp) failures: spatial grounding reduces Near-Miss while leaving No- Grasp unchanged. The same taxonomy transfers to the harder StackCube-v1 and points to occlusion as the main bottleneck.