Organizations: Southeast University · Shanghai Innovation Institute · Shanghai Jiao Tong University · Institute of Automation, Chinese Academy of Sciences
Vision-language-action (VLA) models have emerged as a powerful paradigm for robotic manipulation, enabling robots to generate actions from visual observations and language instructions. However, most existing VLA models still operate on image-level visual representations inherited from pretrained vision-language backbones, while lacking fine-grained object understanding for manipulation. As a result, their robustness can degrade under out-of-distribution object layouts, clutter, and background variations. To address these limitations, we introduce ObjDream, an object-centric VLA framework centered on object-level understanding and future object state prediction. ObjDream uses a spatio-temporal object encoder to organize visual observations into temporal object slots, where persistent object identity and dynamic object state are represented through separate channels. These structured object representations are inserted into the token sequence of the vision-language model (VLM), where identity-guided routing modulates VLM attention to the corresponding object tokens. In addition, an action-conditioned object dynamics module predicts compact future object states relevant to manipulation during training, explicitly supervising object state evolution under robot actions without requiring future prediction at inference. Experiments on LIBERO, LIBERO-Plus, and RoboTwin 2.0 show that ObjDream improves generalization under diverse distribution shifts and scene perturbations. Real-world robot experiments further demonstrate that ObjDream transfers to physical manipulation settings with novel object arrangements and background changes.
Figures & tables
Figure 1: Overview of ObjDream. We introduce ObjDream, an object-centric VLA framework for robust manipulation. Unlike prior VLA models (a), which primarily rely on image-level understanding and entangled object representations, ObjDream (b) introduces explicit object-level representations and future object state prediction. This object-centric formulation improves manipulation robustness, leading to stronger performance than baseline methods (c).
Figure 2: Overview of the ObjDream architecture. The model integrates three main components: (1) a spatio-temporal object encoder that constructs factorized object slots by separately representing persistent object identity and time-varying object state; (2) an object-level dynamics module that decodes current object states and predicts action-conditioned future object states; and (3) a flow-matching action expert that predicts robot action chunks. ObjDream enhances object-level understanding through factorized object representations and future object state modeling, enabling more generalizable language-conditioned manipulation under object and scene variations.
Figure 3: Experimental overview. We evaluate ObjDream on LIBERO, LIBERO-Plus, a fixed 8 -task subset of RoboTwin 2.0, and four real-world manipulation tasks, covering standard and perturbed evaluation settings.
Method
LIBERO
LIBERO-Plus
Spatial
Object
Goal
Long
Avg
Cam.
Robot
Layout
Lang.
Light
BG
Noise
Overall
General VLA models
OpenVLA ( Kim et al., 2024 )
84.7
88.4
79.2
53.7
76.5
0.8
3.5
28.5
23.0
8.1
34.8
15.2
15.6
OpenVLA-OFT ( Kim et al., 2025 )
97.6
98.4
97.9
94.5
97.1
56.4
31.9
74.2
79.5
88.7
93.3
75.8
69.6
UniVLA ( Bu et al., 2025 )
96.5
96.8
95.6
92.0
95.2
1.8
46.2
31.9
69.6
69.0
81.0
21.2
42.9
π0 ( Black et al., 2024 )
96.8
98.8
95.8
85.2
94.2
13.8
6.0
68.9
58.8
85.0
81.4
79.0
53.6
Table 1: Performance on standard LIBERO and zero-shot robustness on LIBERO-Plus. All LIBERO-Plus results are evaluated without fine-tuning on perturbed environments. Best results are shown in bold, and second-best results are underlined.
Figure 4: Success rates on the fixed 8 -task RoboTwin 2.0 subset under Easy and Hard settings.
Variant
LIBERO
LIBERO-Plus
Long
Cam.
Robot
Layout
Lang.
Light
BG
Noise
Overall
Backbone baseline
95.4
66.4
74.7
80.0
87.1
91.7
92.4
82.8
81.2
w/o ObjRel & ObjDyn
95.2
65.6
73.7
78.9
87.4
95.6
96.9
85.0
82.1
w/o ObjDyn
96.2
66.6
75.0
80.6
88.2
96.2
96.4
87.3
83.2
w/o ObjRel
96.6
67.4
76.1
80.9
88.2
97.6
97.8
88.9
84.1
w/o Fact. Injection
96.2
65.9
75.3
81.0
89.0
94.4
93.5
85.5
82.5
Table 2: Ablation study of object-centric components. Success rates (%) on LIBERO-Long and LIBERO-Plus. Overall is the trial-weighted LIBERO-Plus average. Best results are shown in bold, and second-best results are underlined.
Eval.
Method
Button Pressing
Block Pick-and-Place
Two-Fruit Placement
Tabletop Sweeping
Avg. Success
ID
π0.5 ( Physical Intelligence et al., 2025 )
90
70
40
65
66
ControlVLA ( Li et al., 2025a )
65
40
70
65
60
ObjDream (Ours)
80
65
60
90
74
OOD
π0.5 ( Physical Intelligence et al., 2025 )
65
60
40
45
53
ControlVLA ( Li et al., 2025a )
40
10
35
40
31
ObjDream (Ours)
65
50
50
55
55
Table 3: Real-robot manipulation results. Success rates (%) on four real-world tasks under ID and OOD evaluation. Best and second-best results within each setting are bold and underlined.
Appendix figures & tables6 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 5: Automatic offline annotation pipeline for object encoder pretraining. Qwen3-VL-8B extracts task-relevant object phrases from demonstration instructions and initial observations, while a separate robot gripper label is used for robot annotation. Grounding DINO initializes the corresponding object regions, SAM2 propagates segmentation masks across frames, and DINOv2 extracts object-specific features from the initial grounded regions. Masks, geometry, visibility, and phrase–region correspondences are automatically constructed from this pipeline without manual frame-wise object annotation. The resulting annotations supervise encoder pretraining, while the DINOv2 features provide visual identity references.
Subset
Method
Center error (%) ↓
Box IoU ↑
Mask IoU ↑
Unclamped
Current state persistence
6.37
0.634
0.564
Future prediction
1.60
0.690
0.616
Displacement
Current state persistence
13.91
0.287
0.245
Future prediction
2.03
0.575
0.542
Appendix
Table 4: Future object state prediction accuracy on LIBERO. Current state persistence repeats the current object state at each future target. Bold indicates the better value within each comparison.
Figure 6: Qualitative future object mask predictions on LIBERO. Rows show three representative manipulation tasks. The decoder predicts all future steps h=1,…,Hf for non-robot objects; for clarity, we visualize only h=1,4,8 . The predicted masks are overlaid on the corresponding future RGB frames, which are used only for visualization and are not provided to the decoder.
Figure 7: Qualitative future object state predictions on RoboTwin 2.0. Rows show representative tasks from the fixed eight-task subset. The decoder predicts all future steps for non-robot objects; for clarity, we visualize masks and bounding boxes only at h=1,4,8 . The predictions are overlaid on the corresponding future RGB frames, which are used only for visualization and are not provided to the decoder.
Figure 8: Qualitative future object state predictions on real-world tasks. Rows show button pressing, two-fruit placement, block pick-and-place, and tabletop sweeping. The decoder predicts all future steps for non-robot objects; for clarity, we visualize masks and bounding boxes only at h=1,4,8 . The predictions are overlaid on the corresponding future RGB frames, which are used only for visualization and are not provided to the decoder.
Subset
Conditioning
Center error (%) ↓
Box IoU ↑
Mask IoU ↑
Unclamped
No Action
1.90
0.659
0.609
Mismatched Action
2.02
0.660
0.587
Native Predicted Action
1.60
0.690
0.616
Displacement
No Action
2.35
0.546
0.534
Mismatched Action
2.71
0.532
0.486
Native Predicted Action
2.03
0.575
0.542
Appendix
Table 5: Sensitivity of future object state prediction to action conditioning on LIBERO. Native Predicted Action uses the original auxiliary action proposal; No Action removes the position-aware action embedding; and Mismatched Action replaces it with the position-aware embedding of a predicted action chunk from another sample within the same task. All other model components and prediction targets are kept fixed.