Pixel-level Scene Understanding in One Token: Visual States Need What-is-Where Composition

Seokmin Lee; Yunghee Lee; Byeonghyun Pak; Byeongju Woo

cs.CVMar 14, 2026

Pixel-level Scene Understanding in One Token: Visual States Need What-is-Where Composition

Authors: Seokmin Lee, Yunghee Lee, Byeonghyun Pak, Byeongju Woo

Organizations: Agency for Defense Development

Abstract

For robotic agents operating in dynamic environments, learning visual state representations from streaming video observations is essential for sequential decision making. Recent self-supervised learning methods have shown strong transferability across vision tasks, but they do not explicitly address what a good visual state should encode. We argue that effective visual states must capture what-is-where by jointly encoding the semantic identities of scene elements and their spatial locations, enabling reliable detection of subtle dynamics across observations. To this end, we propose CroBo, a visual state representation learning framework based on a global-to-local reconstruction objective. Given a reference observation compressed into a compact bottleneck token, CroBo learns to reconstruct heavily masked patches in a local target crop from sparse visible cues, using the global bottleneck token as context. This learning objective encourages the bottleneck token to encode a fine-grained representation of scene-wide semantic entities, including their identities, spatial locations, and configurations. As a result, the learned visual states reveal how scene elements move and interact over time, supporting sequential decision making. We evaluate CroBo on diverse vision-based robot policy learning benchmarks, where it achieves state-of-the-art performance. Reconstruction analyses and perceptual straightness experiments further show that the learned representations preserve pixel-level scene composition and encode what-moves-where across observations. Project page available at: https://seokminlee-chris.github.io/CroBo-ProjectPage.

Figures & tables

Figure 1: Understanding what-moves-where requires knowing what-is-where . Robots interact with dynamic environments through compact state representations derived from visual observations. To reason scene dynamics, the robot must understand which objects move and detect how their locations change between observations. This requires each state to capture what-is-where information in the scene so that subtle spatial differences between observations can be recognized.

Figure 2: Learning a visual state that captures pixel-level scene composition. We present a state representation learning (SRL) method that captures the entire scene composition, including object identities, locations, and their spatial relationship. Conceptually, the global state becomes the bottleneck token that contains the contextual information of the scene. By enforcing reconstruction of arbitrary cropped views from such contextual information, the model is encouraged to encode pixel-level scene composition information in the global state.

Figure 3: Overview of CroBo. Given a global source view $\mathbf{x}^{g}$ and a local target view $\mathbf{x}^{l}$ cropped from $\mathbf{x}^{g}$ , the encoder maps the source view to a single bottleneck token (i.e., [CLS] token) and the target view to a few visible patch tokens under heavy masking (e.g., 90%). The decoder reconstructs the masked target patches from the visible target patch tokens together with source bottleneck token. Because only a few target patches remain visible, the decoder must rely on the bottleneck token to provide the missing scene context, encouraging it to capture fine-grained what-is-where scene composition.

	Franka Kitchen					DeepMind Control Suite
Model	x Knob on	x Light on	Sdoor open	Ldoor open	Micro open	walker/stand	walker/walk	reacher/easy	finger/spin
MAE $\dagger$	12.0 $\pm$ 3.3	24.3 $\pm$ 4.2	71.5 $\pm$ 4.3	12.8 $\pm$ 3.9	10.0 $\pm$ 2.8	-	-	-	-
DINO $\dagger$	27.0 $\pm$ 3.2	44.3 $\pm$ 6.5	77.0 $\pm$ 5.0	16.5 $\pm$ 2.5	28.5 $\pm$ 4.8	-	-	-	-
DINOv2	25.0 $\pm$ 1.4	46.6 $\pm$ 5.8	87.8 $\pm$ 2.7	17.6 $\pm$ 2.4	21.8 $\pm$ 3.5	81.2 $\pm$ 3.7	38.2 $\pm$ 2.1	87.6 $\pm$ 3.8	67.4 $\pm$ 0.8
SiamMAE $\dagger$	16.8 $\pm$ 4.4	36.5 $\pm$ 7.0	68.0 $\pm$ 7.9	17.3 $\pm$ 3.7	13.5 $\pm$ 4.8	-	-	-	-
CropMAE	33.0 $\pm$ 3.1	65.0 $\pm$ 5.1	89.6 $\pm$ 2.1	22.6 $\pm$ 2.2	25.0 $\pm$ 2.9	59.9 $\pm$ 4.5	33.8 $\pm$ 3.7	75.4 $\pm$ 2.8	69.9 $\pm$ 0.8

Table 1: Experimental results on vision-based robot policy learning. We evaluate imitation learning performance on Franka Kitchen [ 23 ] and DeepMind Control Suite (DMC) [ 50 ] . The agents are trained on frozen visual representations extracted from a ViT-S/16 encoder pre-trained on the Kinetics-400 [ 41 ] , except for DINOv2. Success rates (%) are reported for Franka Kitchen and normalized scores are reported for DMC. The best results are bold , second-best are underlined .

\dagger

denotes numbers taken from [ 37 ] . We report the gains of our method over the previous state-of-the-art method.

Arch.	Method	x Knob1 on	x Light on	Sdoor open	Ldoor open	Micro open	5-task Avg.
ViT-S/16	CroBo (Ours)	57.6 $\pm$ 8.1	81.6 $\pm$ 2.1	98.6 $\pm$ 1.4	36.8 $\pm$ 3.8	50.4 $\pm$ 5.8	65.0 $\pm$ 4.2
ViT-B/16	SiamMAE †	18.0 $\pm$ 2.0	34.0 $\pm$ 2.0	80.7 $\pm$ 3.1	18.7 $\pm$ 1.2	19.3 $\pm$ 6.1	34.1 $\pm$ 2.9
	RSP †	24.7 $\pm$ 3.1	51.7 $\pm$ 9.1	87.3 $\pm$ 2.3	23.3 $\pm$ 7.6	26.7 $\pm$ 2.3	42.7 $\pm$ 4.9
	ToBo †	46.7 $\pm$ 6.4	78.7 $\pm$ 7.6	95.3 $\pm$ 1.2	47.3 $\pm$ 5.0	37.3 $\pm$ 4.6	61.1 $\pm$ 5.0
	CroBo (Ours)	64.4 $\pm$ 2.4	86.2 $\pm$ 3.7	98.0 $\pm$ 1.0	41.0 $\pm$ 6.6	62.8 $\pm$ 3.4	70.5 $\pm$ 3.4
	$\Delta$ prev. SOTA	+17.7	+7.5	+2.7	-6.3	+25.5	+9.4

Table 2: Scalability of our method. Success rates (%) are reported for ViT-S/16, ViT-B/16, and ViT-L/16 backbones pre-trained on Kinetics-400 [ 41 ] for 100 epochs. While ViT-S results are provided for CroBo, ViT-B and ViT-L scales include comparisons with SiamMAE [ 24 ] , RSP [ 37 ] , and ToBo [ 42 ] .

\Delta

denotes the performance margin relative to the best-performing baseline in each scale. Best results are in bold and second best are underlined . CroBo results are averaged over 10 trials, while baseline results and those marked with † are from [ 42 ] .

Figure 4: Reconstruction of CroBo. We visualize image reconstructions on CLEVR [ 39 ] , DAVIS [ 49 ] , MOSEv2 [ 16 ] and Franka Kitchen [ 23 ] . Using the bottleneck token of the reference view as context, CroBo reconstructs a highly masked (90%) target view cropped from the reference view. The results show that the bottleneck token from the reference view provides sufficient information to recover overall scene structure, including object identity, location, and spatial relationships. (e.g., in column 1, the two cyan spheres are not visible in the masked input, yet their positions are correctly reconstructed using the reference bottleneck token).

Figure 5: Perceptual straightness of representation dynamics in video. (a) Local curvature of representation trajectories measured on DAVIS videos. Curvature is computed as the angle between consecutive representation differences, averaged across videos. Lower curvature indicates smoother temporal dynamics. CroBo consistently exhibits lower curvature than prior models such as DINOv2 and CropMAE, suggesting more temporally coherent representations that better preserve what-moves-where across frames. (b) Representation trajectories across video frames visualized using PCA. Each point corresponds to the representation of a frame, and colors indicate temporal progression. CroBo produces smooth and locally linear trajectories that follow the natural evolution of the scene compared to other models.

Table 3: Ablation studies on target view design and masking ratio. Models are pre-trained on Kinetics-400 for 100 epochs and evaluated on Franka Kitchen (success rate %). Our method is highlighted in gray .

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Table 4: Implementation details. We follow MAE [ 31 ] for the decoder, RSP [ 42 ] for pre-training, and CropMAE [ 19 ] for augmentation, except for synchronized horizontal flipping.

Figure 6: Reconstruction results of CroBo. We present image reconstruction examples on CLEVR [ 39 ] , DAVIS [ 49 ] , MOSEv2 [ 16 ] and Franka Kitchen [ 23 ] . Given the bottleneck token extracted from the reference view as context, CroBo reconstructs a heavily masked (90%) target view spatially cropped from the reference view. These reconstructions indicate that the bottleneck token captures sufficient information to recover the overall scene structure, including object identities, spatial locations, and their relationships.

Figure 7: Perceptual straightness of representation trajectories in video. We visualize representation trajectories across video frames using PCA, where each point denotes a frame and colors indicate temporal order. CroBo produces smooth and locally linear trajectories that follow the natural evolution of the scene, whereas DINOv2 and CropMAE exhibit more irregular and entangled trajectories. Videos are drawn from MOSEv2 (first and second rows) [ 16 ] and Franka Kitchen (third row) [ 23 ] .

State Representation Learning

State representation learning develops compact, task-relevant descriptions of observations and histories for prediction, control, and decision-making.

See all 125 papers

V-JEPA 2.1: Unlocking Dense Features in Video Self-Supervised Learning
Mar 15, 2026Lorenzo Mur-Labadia, Matthew Muckley, Amir Bar +6Self-Supervised Visual Representation Learning Visual Representation Learning
FAIR at Meta · Universidad de Zaragoza
Patch Policy: Efficient Embodied Control via Dense Visual Representations
Jul 20, 2026Gaoyue Zhou, Zichen Jeff Cui, Ada Langford +3Visuomotor Policy Learning Robot Learning
1Courant Institute, New York University · 3AMI Labs · 2Meta-FAIR
DepState-JEPA: Learning Depth-Grounded State Representation for Visual Perception and Manipulation
Oct 21, 2026Xiujian Liang, Liuyuan Na, Zehao Du +6State Representation Learning Visual Representation Learning
Fudan University · Shanghai Innovation Institute (SII) · Shanghai Jiao Tong University +2

Mar 15, 2026J/K move · Enter open · S save

V-JEPA 2.1: Unlocking Dense Features in Video Self-Supervised Learning

Lorenzo Mur-Labadia, Matthew Muckley, Amir Bar, Mido Assran+5

FAIR at Meta · Universidad de Zaragoza

Self-Supervised Visual Representation Learning Visual Representation Learning Video Representation Learning Joint-Embedding Predictive Architecture

Open paper AskPDF arXiv

We present V-JEPA 2.1, a family of self-supervised models that learn dense, high-quality visual representations for both images and videos while retaining strong global scene understanding. The approach combines four key components. First, a dense predictive loss uses a masking-based objective in which both visible and masked tokens contribute to the training signal, encouraging explicit spatial and temporal grounding. Second, deep self-supervision applies the self-supervised objective hierarchically across multiple intermediate encoder layers to improve representation quality. Third, multi-modal tokenizers enable unified training across images and videos. Finally, the model benefits from effective scaling in both model capacity and training data. Together, these design choices produce representations that are spatially structured, semantically coherent, and temporally consistent. Empirically, V-JEPA 2.1 achieves state-of-the-art performance on several challenging benchmarks, including 7.71 mAP on Ego4D for short-term object-interaction anticipation and 40.8 Recall@5 on EPIC-KITCHENS for high-level action anticipation, as well as a 20-point improvement in real-robot grasping success rate over V-JEPA-2 AC. The model also demonstrates strong performance in robotic navigation (5.687 ATE on TartanDrive), depth estimation (0.307 RMSE on NYUv2 with a linear probe), and global recognition (77.7 on Something-Something-V2). These results show that V-JEPA 2.1 significantly advances the state of the art in dense visual understanding and world modeling.

Abstract

State Representation Learning

Explore similar work

V-JEPA 2.1: Unlocking Dense Features in Video Self-Supervised Learning

Patch Policy: Efficient Embodied Control via Dense Visual Representations

DepState-JEPA: Learning Depth-Grounded State Representation for Visual Perception and Manipulation