For robotic agents operating in dynamic environments, learning visual state representations from streaming video observations is essential for sequential decision making. Recent self-supervised learning methods have shown strong transferability across vision tasks, but they do not explicitly address what a good visual state should encode. We argue that effective visual states must capture what-is-where by jointly encoding the semantic identities of scene elements and their spatial locations, enabling reliable detection of subtle dynamics across observations. To this end, we propose CroBo, a visual state representation learning framework based on a global-to-local reconstruction objective. Given a reference observation compressed into a compact bottleneck token, CroBo learns to reconstruct heavily masked patches in a local target crop from sparse visible cues, using the global bottleneck token as context. This learning objective encourages the bottleneck token to encode a fine-grained representation of scene-wide semantic entities, including their identities, spatial locations, and configurations. As a result, the learned visual states reveal how scene elements move and interact over time, supporting sequential decision making. We evaluate CroBo on diverse vision-based robot policy learning benchmarks, where it achieves state-of-the-art performance. Reconstruction analyses and perceptual straightness experiments further show that the learned representations preserve pixel-level scene composition and encode what-moves-where across observations. Project page available at: https://seokminlee-chris.github.io/CroBo-ProjectPage.
Figures & tables
Figure 1: Understanding what-moves-where requires knowing what-is-where . Robots interact with dynamic environments through compact state representations derived from visual observations. To reason scene dynamics, the robot must understand which objects move and detect how their locations change between observations. This requires each state to capture what-is-where information in the scene so that subtle spatial differences between observations can be recognized.
Figure 2: Learning a visual state that captures pixel-level scene composition. We present a state representation learning (SRL) method that captures the entire scene composition, including object identities, locations, and their spatial relationship. Conceptually, the global state becomes the bottleneck token that contains the contextual information of the scene. By enforcing reconstruction of arbitrary cropped views from such contextual information, the model is encouraged to encode pixel-level scene composition information in the global state.
Figure 3: Overview of CroBo. Given a global source view xg and a local target view xl cropped from xg , the encoder maps the source view to a single bottleneck token (i.e., [CLS] token) and the target view to a few visible patch tokens under heavy masking (e.g., 90%). The decoder reconstructs the masked target patches from the visible target patch tokens together with source bottleneck token. Because only a few target patches remain visible, the decoder must rely on the bottleneck token to provide the missing scene context, encouraging it to capture fine-grained what-is-where scene composition.
Franka Kitchen
DeepMind Control Suite
Model
x Knob on
x Light on
Sdoor open
Ldoor open
Micro open
walker/stand
walker/walk
reacher/easy
finger/spin
MAE †
12.0 ± 3.3
24.3 ± 4.2
71.5 ± 4.3
12.8 ± 3.9
10.0 ± 2.8
-
-
-
-
DINO †
27.0 ± 3.2
44.3 ± 6.5
77.0 ± 5.0
16.5 ± 2.5
28.5 ± 4.8
-
-
-
-
DINOv2
25.0 ± 1.4
46.6 ± 5.8
87.8 ± 2.7
17.6 ± 2.4
21.8 ± 3.5
81.2 ± 3.7
38.2 ± 2.1
87.6 ± 3.8
67.4 ± 0.8
SiamMAE †
16.8 ± 4.4
36.5 ± 7.0
68.0 ± 7.9
17.3 ± 3.7
13.5 ± 4.8
-
-
-
-
CropMAE
33.0 ± 3.1
65.0 ± 5.1
89.6 ± 2.1
22.6 ± 2.2
25.0 ± 2.9
59.9 ± 4.5
33.8 ± 3.7
75.4 ± 2.8
69.9 ± 0.8
Table 1: Experimental results on vision-based robot policy learning. We evaluate imitation learning performance on Franka Kitchen [ 23 ] and DeepMind Control Suite (DMC) [ 50 ] . The agents are trained on frozen visual representations extracted from a ViT-S/16 encoder pre-trained on the Kinetics-400 [ 41 ] , except for DINOv2. Success rates (%) are reported for Franka Kitchen and normalized scores are reported for DMC. The best results are bold , second-best are underlined . † denotes numbers taken from [ 37 ] . We report the gains of our method over the previous state-of-the-art method.
Arch.
Method
x Knob1 on
x Light on
Sdoor open
Ldoor open
Micro open
5-task Avg.
ViT-S/16
CroBo (Ours)
57.6 ± 8.1
81.6 ± 2.1
98.6 ± 1.4
36.8 ± 3.8
50.4 ± 5.8
65.0 ± 4.2
ViT-B/16
SiamMAE †
18.0 ± 2.0
34.0 ± 2.0
80.7 ± 3.1
18.7 ± 1.2
19.3 ± 6.1
34.1 ± 2.9
RSP †
24.7 ± 3.1
51.7 ± 9.1
87.3 ± 2.3
23.3 ± 7.6
26.7 ± 2.3
42.7 ± 4.9
ToBo †
46.7 ± 6.4
78.7 ± 7.6
95.3 ± 1.2
47.3 ± 5.0
37.3 ± 4.6
61.1 ± 5.0
CroBo (Ours)
64.4 ± 2.4
86.2 ± 3.7
98.0 ± 1.0
41.0 ± 6.6
62.8 ± 3.4
70.5 ± 3.4
Δ prev. SOTA
+17.7
+7.5
+2.7
-6.3
+25.5
+9.4
Table 2: Scalability of our method. Success rates (%) are reported for ViT-S/16, ViT-B/16, and ViT-L/16 backbones pre-trained on Kinetics-400 [ 41 ] for 100 epochs. While ViT-S results are provided for CroBo, ViT-B and ViT-L scales include comparisons with SiamMAE [ 24 ] , RSP [ 37 ] , and ToBo [ 42 ] . Δ denotes the performance margin relative to the best-performing baseline in each scale. Best results are in bold and second best are underlined . CroBo results are averaged over 10 trials, while baseline results and those marked with † are from [ 42 ] .
Figure 4: Reconstruction of CroBo. We visualize image reconstructions on CLEVR [ 39 ] , DAVIS [ 49 ] , MOSEv2 [ 16 ] and Franka Kitchen [ 23 ] . Using the bottleneck token of the reference view as context, CroBo reconstructs a highly masked (90%) target view cropped from the reference view. The results show that the bottleneck token from the reference view provides sufficient information to recover overall scene structure, including object identity, location, and spatial relationships. (e.g., in column 1, the two cyan spheres are not visible in the masked input, yet their positions are correctly reconstructed using the reference bottleneck token).
Figure 5: Perceptual straightness of representation dynamics in video. (a) Local curvature of representation trajectories measured on DAVIS videos. Curvature is computed as the angle between consecutive representation differences, averaged across videos. Lower curvature indicates smoother temporal dynamics. CroBo consistently exhibits lower curvature than prior models such as DINOv2 and CropMAE, suggesting more temporally coherent representations that better preserve what-moves-where across frames. (b) Representation trajectories across video frames visualized using PCA. Each point corresponds to the representation of a frame, and colors indicate temporal progression. CroBo produces smooth and locally linear trajectories that follow the natural evolution of the scene compared to other models.
Table 3: Ablation studies on target view design and masking ratio. Models are pre-trained on Kinetics-400 for 100 epochs and evaluated on Franka Kitchen (success rate %). Our method is highlighted in gray .
Appendix figures & tables3 assets
Supplementary material from the paper’s appendix.
Appendix
Table 4: Implementation details. We follow MAE [ 31 ] for the decoder, RSP [ 42 ] for pre-training, and CropMAE [ 19 ] for augmentation, except for synchronized horizontal flipping.
Figure 6: Reconstruction results of CroBo. We present image reconstruction examples on CLEVR [ 39 ] , DAVIS [ 49 ] , MOSEv2 [ 16 ] and Franka Kitchen [ 23 ] . Given the bottleneck token extracted from the reference view as context, CroBo reconstructs a heavily masked (90%) target view spatially cropped from the reference view. These reconstructions indicate that the bottleneck token captures sufficient information to recover the overall scene structure, including object identities, spatial locations, and their relationships.
Figure 7: Perceptual straightness of representation trajectories in video. We visualize representation trajectories across video frames using PCA, where each point denotes a frame and colors indicate temporal order. CroBo produces smooth and locally linear trajectories that follow the natural evolution of the scene, whereas DINOv2 and CropMAE exhibit more irregular and entangled trajectories. Videos are drawn from MOSEv2 (first and second rows) [ 16 ] and Franka Kitchen (third row) [ 23 ] .
We present V-JEPA 2.1, a family of self-supervised models that learn dense, high-quality visual representations for both images and videos while retaining strong global scene understanding. The approach combines four key components. First, a dense predictive loss uses a masking-based objective in which both visible and masked tokens contribute to the training signal, encouraging explicit spatial and temporal grounding. Second, deep self-supervision applies the self-supervised objective hierarchically across multiple intermediate encoder layers to improve representation quality. Third, multi-modal tokenizers enable unified training across images and videos. Finally, the model benefits from effective scaling in both model capacity and training data. Together, these design choices produce representations that are spatially structured, semantically coherent, and temporally consistent. Empirically, V-JEPA 2.1 achieves state-of-the-art performance on several challenging benchmarks, including 7.71 mAP on Ego4D for short-term object-interaction anticipation and 40.8 Recall@5 on EPIC-KITCHENS for high-level action anticipation, as well as a 20-point improvement in real-robot grasping success rate over V-JEPA-2 AC. The model also demonstrates strong performance in robotic navigation (5.687 ATE on TartanDrive), depth estimation (0.307 RMSE on NYUv2 with a linear probe), and global recognition (77.7 on Something-Something-V2). These results show that V-JEPA 2.1 significantly advances the state of the art in dense visual understanding and world modeling.
Pretrained dense visual features from Vision Transformers (ViTs) are powerful yet have been underutilized in robot learning. Modern robot policies either compress each observation into a single global token, or rely on visual backbones trained from scratch, sacrificing both fine-grained spatial detail and the benefits of large-scale visual pre-training. While there exist policies that do operate on dense patch features like large vision-language-action models (VLAs), they tend to be heavy and slow, inheriting the full cost of a billion-parameter vision-language model (VLM) backbone. We close this gap with Patch Policy, a minimal architectural extension that enables transformer-based policies to consume dense pre-trained patch tokens directly without the computational overhead of a full VLM. At its core is a block-causal attention mask that preserves the temporal causality of standard policies while letting the model attend over many patch tokens per observation, alongside other state information. Patch Policy is lightweight, fast, and highly effective. Across four simulated and three real-world environment suites, our method achieves a 40% relative improvement over policies using state-of-the-art global-pooled representations. Furthermore, it surpasses fine-tuned OpenVLA-OFT by 18% while using roughly 0.7% of the parameters. We believe Patch Policy provides a pipeline for the robotics community to readily leverage continuing progress in visual representation learning, without sacrificing the training efficiency or inference speed required for high-frequency, reactive control. Videos can be viewed at https://patch-policy.github.io
Gaoyue Zhou, Zichen Jeff Cui, Ada Langford +3
1Courant Institute, New York University · 3AMI Labs · 2Meta-FAIR
Predictive representations for robot manipulation should connect scene geometry with the state changes induced by actions. Our intuition is that understanding what an action will change starts with understanding where things are: complementary RGB and depth observations ground a common scene representation from which to infer those changes. Guided by this intuition, we introduce DepState-JEPA, a dual-stream RGB--Depth joint-embedding predictive architecture that connects spatial understanding with action-conditioned prediction through a shared latent state, DepState. Built on V-JEPA~2.1, the framework uses depth both as a complementary observation stream and to ground both modalities in metric 3D space, aggregating shared information into DepState while retaining modality-private features for decoding. Ray-conditioned readouts predict local features without target depth; after an action-conditioned transition, the same decoders predict future RGB and depth features using the predicted state and fixed initial private features. Robot-state readouts provide recursive proprioceptive inputs; object-trajectory supervision and calibrated forward kinematics shape state learning through motion targets and joint--TCP consistency. Comparisons show stronger RGB-encoder transfer after RGB-D adaptation, feature completion, conditioned visual forecasting, and manipulation success. Within-model ablations identify contributions from depth content, metric coordinates, and future-state matching, with additional control gains from FK regularization. The project page is available at https://shawnleung-sii.github.io/depstate-jepa/.
Xiujian Liang, Liuyuan Na, Zehao Du +6
Fudan University · Shanghai Innovation Institute (SII) · Shanghai Jiao Tong University +2