DepState-JEPA: Learning Depth-Grounded State Representation for Visual Perception and Manipulation

Xiujian Liang; Liuyuan Na; Zehao Du; Jiacheng Xu; Weixi Song; Mingyang Sun; Xingjun Ma; Cewu Lu

cs.ROOct 21, 2026

DepState-JEPA: Learning Depth-Grounded State Representation for Visual Perception and Manipulation

Authors: Xiujian Liang, Liuyuan Na, Zehao Du, Jiacheng Xu, Weixi Song, Mingyang Sun, Xingjun Ma, Cewu Lu,

Organizations: Fudan University · Shanghai Innovation Institute (SII) · Shanghai Jiao Tong University · Shandong University ·

Abstract

Predictive representations for robot manipulation should connect scene geometry with the state changes induced by actions. Our intuition is that understanding what an action will change starts with understanding where things are: complementary RGB and depth observations ground a common scene representation from which to infer those changes. Guided by this intuition, we introduce DepState-JEPA, a dual-stream RGB--Depth joint-embedding predictive architecture that connects spatial understanding with action-conditioned prediction through a shared latent state, DepState. Built on V-JEPA~2.1, the framework uses depth both as a complementary observation stream and to ground both modalities in metric 3D space, aggregating shared information into DepState while retaining modality-private features for decoding. Ray-conditioned readouts predict local features without target depth; after an action-conditioned transition, the same decoders predict future RGB and depth features using the predicted state and fixed initial private features. Robot-state readouts provide recursive proprioceptive inputs; object-trajectory supervision and calibrated forward kinematics shape state learning through motion targets and joint--TCP consistency. Comparisons show stronger RGB-encoder transfer after RGB-D adaptation, feature completion, conditioned visual forecasting, and manipulation success. Within-model ablations identify contributions from depth content, metric coordinates, and future-state matching, with additional control gains from FK regularization. The project page is available at https://shawnleung-sii.github.io/depstate-jepa/.

Figures & tables

Figure 1: From Where Things Are to What Actions Change. DepState-JEPA grounds complementary RGB–depth observations in a shared latent state, DepState, while retaining modality-private features for decoding. This depth-grounded state connects spatial queries with action-conditioned prediction of future visual features, robot state, and object motion.

Figure 2: Architectural Overview of DepState-JEPA. RGB–Depth features are grounded, factorized, and aggregated into DepState. Spatial queries read local content; action-conditioned transitions advance the state and reuse the visual decoders with initial private features. Physical readouts predict robot state and object motion, with FK regularizing independent joint/TCP outputs.

Figure 3: Qualitative Evaluation of Frozen Spatial Representations. Task-specific probes trained on frozen encoders produce (a) monocular depth estimates on NYUv2 and (b) semantic segmentation maps on VOC12. RGB inputs and ground-truth annotations provide references. Depth maps share a common metric color scale, and segmentation maps use a consistent class palette.

	MDE on NYUv2				SS on VOC12		SC on LB-Depth

Method	AbsRel $\downarrow$	RMSE $\downarrow$	$\log_{10}\downarrow$	$\delta_{1}\uparrow$	mIoU $\uparrow$	mAcc $\uparrow$	L1 $\downarrow$	Cos. $\uparrow$
LeJEPA ( Balestriero & LeCun, 2025 )	0.137	0.490	0.059	85.7	76.0	85.0	0.352	0.891
V-JEPA 2 ( Assran et al., 2025 )	0.188	0.642	0.079	77.5	63.9	76.8	0.428	0.843
V-JEPA 2.1 ( Mur-Labadia et al., 2026 )	0.098	0.350	0.042	92.5	84.7	91.8	0.298	0.931
3D-JEPA † ( Arnaud et al., 2025 )	Not evaluated: point-cloud input						0.283	0.938

Table 1: Frozen Spatial Transfer and RGB-D Feature Completion. MDE and SS use frozen-encoder probes on NYUv2 and VOC12. SC measures masked-region L1 and Cos against shared references on LingBot-Depth, with equal RGB/depth weighting.

\dagger

: method-specific adaptation.

Figure 4: Spatial Completion and Temporal Prediction in Feature Space. PCA of (a) completion from jointly masked RGB-D observations and (b) multi-step visual forecasts on robot trajectories.

	1 step		8 steps		16 steps		24 steps		32 steps
Method	L1 $\downarrow$	Cos. $\uparrow$	L1 $\downarrow$	Cos. $\uparrow$	L1 $\downarrow$	Cos. $\uparrow$	L1 $\downarrow$	Cos. $\uparrow$	L1 $\downarrow$	Cos. $\uparrow$
V-JEPA 2-AC ( Assran et al., 2025 )	0.242	0.916	0.421	0.792	0.523	0.701	0.580	0.646	0.658	0.598
V-JEPA 2.1-AC ( Mur-Labadia et al., 2026 )	0.171	0.967	0.342	0.874	0.476	0.781	0.554	0.702	0.614	0.636
3D-JEPA-AC † ( Arnaud et al., 2025 )	0.169	0.968	0.336	0.879	0.467	0.789	0.546	0.711	0.602	0.647
LeMuMo-JEPA-AC † ( Cornelissen et al., 2026 )	0.167	0.969	0.331	0.882	0.479	0.778	0.563	0.694	0.624	0.627
DepState-JEPA-AC (Ours)	0.145	0.976	0.307	0.903	0.394	0.851	0.451	0.811	0.491	0.779

Table 2: Single-Step and Multi-Step Prediction in a Common Feature Space. Frozen readouts map predictions to a shared RGB reference space for L1 and Cos . evaluation.

\dagger

: The AC adaptation.

Figure 5: Sim-to-Real Setup and Multi-View RGB-D Rollouts. Left : simulation and real-world setups. Right : RGB-D observations from two viewpoints during real-robot manipulation tasks.

	Online PL		Latent MPC		Sim2Real MPC

Method	DMC $\uparrow$	MW $\uparrow$	DMC $\uparrow$	MW $\uparrow$	Task 1 $\uparrow$	Task 2 $\uparrow$
V-JEPA 2-AC ( Assran et al., 2025 )	682.4 $\pm$ 14.7	52.8 $\pm$ 2.8	736.5 $\pm$ 15.3	66.4 $\pm$ 2.4	45.6 $\pm$ 5.1	41.1 $\pm$ 5.1
V-JEPA 2.1-AC ( Mur-Labadia et al., 2026 )	763.8 $\pm$ 15.4	68.6 $\pm$ 2.3	807.2 $\pm$ 15.0	81.2 $\pm$ 2.6	66.7 $\pm$ 6.7	64.4 $\pm$ 5.1
3D-JEPA-AC † ( Arnaud et al., 2025 )	784.4 $\pm$ 12.6	72.3 $\pm$ 2.6	826.4 $\pm$ 18.2	85.6 $\pm$ 3.2	73.3 $\pm$ 5.8	71.1 $\pm$ 5.1
LeMuMo-JEPA-AC † ( Cornelissen et al., 2026 )	790.4 $\pm$ 16.1	73.2 $\pm$ 2.5	837.5 $\pm$ 13.6	87.8 $\pm$ 2.2	75.6 $\pm$ 6.9	74.4 $\pm$ 8.4

Table 3: JEPA-Based Policy Learning, Planning, and Sim-to-Real Control. DMC reports task-averaged episode return ; MW and Sim2Real report success rate (%). Entries are mean

\pm

std .

Panel of Figure 6 — Figure 6: Ablation Results. L1/Cos : 3 tasks; SR : 50 MW + 2 real (%). Details in App. E.7 .

Appendix figures & tables19 assets

Supplementary material from the paper’s appendix.

Appendix

Symbol	Definition
$x$ , $x_{\mathcal{V}}$	Complete observation and its visible context in the JEPA objective.
$I,D,K$ ; $O_{t}$	Registered RGB, metric depth, and camera intrinsics; $O_{t}=(I_{t},D_{t},K_{t})$ .
$\mathcal{V},\mathcal{M}$	Visible and masked token-index sets.
$\mathcal{V}_{\mathrm{valid}}$ , $\mathcal{M}_{\mathrm{valid}}$	Geometrically valid visible correspondences and masked targets.
$i,m,\ell,t,o$	Token, modality, encoder level, control-time, and object/part-slot indices.
$f_{\theta},p_{\phi},f_{\bar{\theta}}$	Context encoder, predictor, and momentum target encoder.

Appendix

Table 4: Observation, Index, and JEPA Notation.

Symbol	Definition
$\mathbf{p}_{i}=(u_{i},v_{i},1)^{\top}$	Homogeneous coordinates of patch center $i$ .
$d_{i}$ , $\mathbf{x}_{i}$	Metric center depth and back-projected point $\mathbf{x}_{i}=d_{i}K^{-1}\mathbf{p}_{i}$ .
$z_{i}^{m},\widetilde{z}_{i}^{m}$ ; $\widetilde{Z}^{m}$	Fused feature before/after metric encoding; $\widetilde{Z}^{m}$ is the grounded token set.
$\operatorname{PE}_{3D}$	Learned projection of Fourier-encoded metric coordinates.
$F_{s}^{m},F_{p}^{m}$ ; $H^{m},P^{m}$	Shared/private projections and their output features for modality $m$ .
$A,R$	RGB/depth private features: $A=P^{\mathrm{rgb}}$ and $R=P^{\mathrm{dep}}$ .

Appendix

Table 5: Spatial-State Variables, Modules, and Similarity Operators.

Symbol	Definition
$\mathcal{T}_{\phi}$	Action-conditioned temporal Transformer.
$u_{t},q_{t},\dot{q}_{t},\Delta t_{t}$	Joint command, configuration, instantaneous velocity, and control interval.
$\hat{q}_{t+1}$ , $\widehat{\dot{q}}_{t+1}$ , $\hat{T}_{t+1}^{\mathrm{tcp}}$	Predicted endpoint configuration, velocity, and TCP pose.
$D_{q},D_{\mathrm{tcp}}$	Independent joint-configuration and TCP-pose heads.
$\widehat{\Delta\xi}_{t}^{o}$ , $D_{\mathrm{obj}}$	Predicted object/part body increment and its readout head.
$\tilde{T}_{t}^{o}$	Object/part integration origin.

Appendix

Table 6: Temporal, Physical, and Geometric Notation.

Symbol	Definition
$\mathcal{L}_{\mathrm{JEPA}}$	Dense multi-level RGB/depth latent prediction on masked and visible targets.
$\mathcal{L}_{\mathrm{comp}}$	Masked shared-feature completion.
$\mathcal{L}_{\mathrm{spatial}}$	Grouped spatial prediction: dense features plus shared completion.
$\mathcal{L}_{\mathrm{shared}}$ , $\mathcal{L}_{\mathrm{decorr}}$ , $\mathcal{L}_{\mathrm{var}}$	Shared alignment, shared–private decorrelation, and variance floor.
$\mathcal{L}_{\mathrm{fac}}$	Grouped factorization objective.
$\mathcal{L}_{\mathrm{future}}$ , $\mathcal{L}_{\mathrm{state}}$	Future feature and future state matching components.

Appendix

Table 7: Objectives and Schedule Symbols in the Main Method.

Component	Configuration
Multi-level fusion	Per modality: $5632\to 1024\to 512$ , output LN.
Shared / private heads	Independent per modality: $512\to 512\to 384$ / $512\to 256\to 128$ , output LN.
Metric PE	Fourier XYZ: $39\to 128\to 512$ ; shared across modalities; final linear layer initialized to zero.
State aggregator	64 learned queries, width 384, 2 blocks; query self-attention, cross-attention, and FFN.
Ray-query embedding	Fourier ray plus time: $52\to 384\to 384$ .
Spatial decoder	6 blocks, width 384; query self-attention, cross-attention to $S$ , and FFN.

Appendix

Table 8: State-Construction and Prediction Architecture. Arrows specify MLP widths; all new hidden MLP activations are GELU.

Setting	RGB-D Adaptation	Robot Post-training
Initialization	Pretrained encoders; new state and decoding modules	Adapted online/teacher; new temporal predictor and physical heads
Total updates	35,000	20,000; every fourth update is replay
Resolution	256 for 30,000 updates; 384 for the last 5000	384 for robot data and replay
Temporal input	Images / 16-frame videos at 4 fps	Single image per robot state; 16-frame replay videos
Effective / global micro-batch	128 / 16	128 / 16
Peak encoder LR	$2\cdot 10^{-5}$	$2\cdot 10^{-6}$

Appendix

Table 9: Fixed Optimization Protocol. Counts denote optimizer updates, not micro-steps.

Figure 7: Additional Monocular Depth Estimation Results on NYUv2. RGB inputs, ground-truth depth (GT), and frozen-probe predictions for six indoor scenes.

Figure 8: Additional Semantic Segmentation Results on VOC12. Ground-truth and predicted class regions are overlaid on RGB images for six examples.

		Online PL		Latent MPC

Method	Seed	DMC $\uparrow$	MW $\uparrow$	DMC $\uparrow$	MW $\uparrow$
V-JEPA 2-AC	0	671.8	50.8	719.4	64.2
V-JEPA 2.1-AC	0	757.6	66.4	799.1	78.8
3D-JEPA-AC †	0	781.7	72.4	822.8	85.6
LeMuMo-JEPA-AC †	0	787.3	72.2	833.5	87.4

Appendix

Table 10: Control Results for Individual Training Seeds.

	Task 1: Drawer			Task 2: Cup

Method	Pull $\uparrow$	Close $\uparrow$	Full $\uparrow$	Pick $\uparrow$	Place $\uparrow$	Full $\uparrow$
V-JEPA 2-AC	66.7 $\pm$ 3.3	81.1 $\pm$ 1.9	45.6 $\pm$ 5.1	63.3 $\pm$ 3.3	73.3 $\pm$ 3.3	41.1 $\pm$ 5.1
V-JEPA 2.1-AC	86.7 $\pm$ 3.3	90.0 $\pm$ 3.3	66.7 $\pm$ 6.7	83.3 $\pm$ 3.3	87.8 $\pm$ 1.9	64.4 $\pm$ 5.1
3D-JEPA-AC †	90.0 $\pm$ 3.3	91.1 $\pm$ 1.9	73.3 $\pm$ 5.8	87.8 $\pm$ 5.1	88.9 $\pm$ 1.9	71.1 $\pm$ 5.1
LeMuMo-JEPA-AC †	90.0 $\pm$ 3.3	91.1 $\pm$ 1.9	75.6 $\pm$ 6.9	88.9 $\pm$ 5.1	90.0 $\pm$ 3.3	74.4 $\pm$ 8.4

Appendix

Table 11: Atomic and Full-Task Sim2Real Success Rates (%).

		Task 1: Drawer			Task 2: Cup

Method	Seed	Pull $\uparrow$	Close $\uparrow$	Full $\uparrow$	Pick $\uparrow$	Place $\uparrow$	Full $\uparrow$
V-JEPA 2-AC	0	20/30	24/30	14/30	18/30	21/30	11/30
V-JEPA 2.1-AC	0	26/30	27/30	20/30	24/30	26/30	18/30
3D-JEPA-AC †	0	27/30	28/30	23/30	26/30	27/30	21/30
LeMuMo-JEPA-AC †	0	27/30	27/30	22/30	27/30	27/30	22/30

Appendix

Table 12: Sim2Real Successes/Trials for Individual Training Seeds.

Panel of Figure 9 — Figure 9: Multi-View RGB-D Rollout of Drawer Opening and Closing. Two recorded viewpoints show the real-robot execution. Each panel places RGB frames on the left and depth frames on the right; within each modality, time advances from left to right and then top to bottom.

Panel of Figure 10 — Figure 10: Multi-View RGB-D Rollout of Cup Picking and Placing. Two recorded viewpoints show the real-robot execution. Each panel places RGB frames on the left and depth frames on the right; within each modality, time advances from left to right and then top to bottom.

ID	Experimental Setting	L1 $\downarrow$	Cos $\uparrow$	SR $\uparrow$
Ex.1	Replace online depth content with RGB and disable metric XYZ positional encoding.	0.408 $\pm$ 0.021	0.832 $\pm$ 0.017	76.8 $\pm$ 2.9
Ex.2	Retain depth content; replace metric XYZ positional encoding with an unknown-geometry embedding.	0.369 $\pm$ 0.017	0.860 $\pm$ 0.014	80.1 $\pm$ 2.7
Ex.3	Retain metric XYZ positional encoding; replace online depth content with RGB.	0.359 $\pm$ 0.017	0.869 $\pm$ 0.014	81.3 $\pm$ 2.7
Ex.16	Full model: retain both depth content and metric XYZ positional encoding.	0.340 $\pm$ 0.013	0.880 $\pm$ 0.011	83.4 $\pm$ 2.9

Appendix

Table 13: Geometry-Grounded State Construction.

ID	Experimental Setting	L1 $\downarrow$	Cos $\uparrow$	SR $\uparrow$
Ex.4	Disable the private cross-attention residual in the modality decoders; retain private projections and factorization losses.	0.390 $\pm$ 0.019	0.852 $\pm$ 0.016	83.2 $\pm$ 2.7
Ex.5	Add unit-weight alignment of corresponding private features to the existing shared-feature alignment.	0.376 $\pm$ 0.018	0.855 $\pm$ 0.016	81.5 $\pm$ 2.9
Ex.6	Replace projected aggregation keys/values by repeated valid-token means; retain learned queries and state size.	0.440 $\pm$ 0.029	0.799 $\pm$ 0.025	77.0 $\pm$ 1.9
Ex.7	Replace unit-ray query inputs with normalized pixel vectors $(u/W,v/H,1)$ ; retain the query MLP.	0.340 $\pm$ 0.013	0.880 $\pm$ 0.011	83.5 $\pm$ 2.7
Ex.8	Remove completion supervision throughout training/adaptation; retain dense JEPA and temporal objectives.	0.361 $\pm$ 0.018	0.870 $\pm$ 0.014	82.8 $\pm$ 2.9
Ex.16	Full model: private decoding, shared-only alignment, learned aggregation, ray queries, and completion supervision.	0.340 $\pm$ 0.013	0.880 $\pm$ 0.011	83.4 $\pm$ 2.9

Appendix

Table 14: Selective Sharing and Spatial Readout.

ID	Experimental Setting	L1 $\downarrow$	Cos $\uparrow$	SR $\uparrow$
Ex.9	Remove future-state matching; retain future-feature matching and physical supervision.	0.397 $\pm$ 0.024	0.827 $\pm$ 0.021	79.5 $\pm$ 2.8
Ex.10	Remove future-feature matching; retain future-state matching and physical supervision.	0.392 $\pm$ 0.019	0.833 $\pm$ 0.017	82.2 $\pm$ 2.8
Ex.11	Remove object-trajectory supervision; retain the object head and integration interface.	0.344 $\pm$ 0.014	0.877 $\pm$ 0.012	83.2 $\pm$ 2.9
Ex.12	Remove FK consistency regularization; retain independent joint/TCP heads and their direct labels.	0.359 $\pm$ 0.018	0.864 $\pm$ 0.016	82.5 $\pm$ 2.8
Ex.13	Apply FK gradients to joint/TCP heads only via a detached state; keep direct supervision undetached.	0.350 $\pm$ 0.016	0.871 $\pm$ 0.014	83.1 $\pm$ 2.9
Ex.14	Recursively update private memory from predicted states without future observations; add a memory updater.	0.351 $\pm$ 0.017	0.871 $\pm$ 0.015	83.1 $\pm$ 2.9

Appendix

Table 15: Temporal Prediction and Physical Grounding.

	Seed 0			Seed 1			Seed 2

ID	L1 $\downarrow$	Cos $\uparrow$	SR $\uparrow$	L1 $\downarrow$	Cos $\uparrow$	SR $\uparrow$	L1 $\downarrow$	Cos $\uparrow$	SR $\uparrow$
Ex.1	0.407	0.834	74.0	0.388	0.848	79.7	0.430	0.814	76.7
Ex.2	0.367	0.861	77.2	0.353	0.874	82.7	0.387	0.845	80.3
Ex.3	0.358	0.870	78.5	0.342	0.882	84.0	0.377	0.854	81.2
Ex.4	0.389	0.853	80.6	0.372	0.867	86.0	0.409	0.835	82.9

Appendix

Table 16: Ablation L1 , Cos , and SR by Training Seed.

	RGB-D Completion		Drawer Forecasting		Cup Forecasting

ID	L1 $\downarrow$	Cos $\uparrow$	L1 $\downarrow$	Cos $\uparrow$	L1 $\downarrow$	Cos $\uparrow$
Ex.1	0.308 $\pm$ 0.008	0.931 $\pm$ 0.005	0.427 $\pm$ 0.024	0.815 $\pm$ 0.019	0.490 $\pm$ 0.030	0.750 $\pm$ 0.028
Ex.2	0.263 $\pm$ 0.006	0.947 $\pm$ 0.004	0.387 $\pm$ 0.020	0.847 $\pm$ 0.016	0.456 $\pm$ 0.026	0.785 $\pm$ 0.023
Ex.3	0.271 $\pm$ 0.008	0.945 $\pm$ 0.006	0.373 $\pm$ 0.020	0.857 $\pm$ 0.016	0.433 $\pm$ 0.025	0.805 $\pm$ 0.022
Ex.4	0.314 $\pm$ 0.008	0.928 $\pm$ 0.006	0.400 $\pm$ 0.021	0.842 $\pm$ 0.017	0.456 $\pm$ 0.027	0.786 $\pm$ 0.025

Appendix

Table 17: Task-Level Ablation L1 and Cos .

	Meta-World		Drawer			Cup

ID	PL $\uparrow$	MPC $\uparrow$	Pull $\uparrow$	Close $\uparrow$	Full $\uparrow$	Pick $\uparrow$	Place $\uparrow$	Full $\uparrow$
Ex.1	70.9 $\pm$ 3.2	83.8 $\pm$ 2.4	82.2 $\pm$ 3.8	85.6 $\pm$ 1.9	65.6 $\pm$ 5.1	80.0 $\pm$ 3.3	81.1 $\pm$ 1.9	61.1 $\pm$ 5.1
Ex.2	73.6 $\pm$ 2.9	87.1 $\pm$ 2.5	90.0 $\pm$ 3.3	91.1 $\pm$ 1.9	74.4 $\pm$ 5.1	86.7 $\pm$ 3.3	90.0 $\pm$ 3.3	71.1 $\pm$ 5.1
Ex.3	74.9 $\pm$ 2.6	88.1 $\pm$ 2.7	91.1 $\pm$ 3.8	93.3 $\pm$ 3.3	77.8 $\pm$ 5.1	88.9 $\pm$ 3.8	91.1 $\pm$ 1.9	74.4 $\pm$ 5.1
Ex.4	76.9 $\pm$ 2.4	89.7 $\pm$ 2.7	95.6 $\pm$ 1.9	94.4 $\pm$ 1.9	80.0 $\pm$ 6.7	93.3 $\pm$ 3.3	93.3 $\pm$ 0.0	77.8 $\pm$ 5.1

Appendix

Table 18: Controller- and Task-Level Ablation Success Rates (%).

State Representation Learning

State representation learning develops compact, task-relevant descriptions of observations and histories for prediction, control, and decision-making.

See all 124 papers

StereoPatch: Patch-Aligned RGB-Depth Fusion for Spatial Perception in Robot Manipulation
Sep 14, 2026Yanan Zhou, Zhaoyan Qian, James Zhao +1Generalization in Robotic Manipulation Visuomotor Policy Learning
School of Computer Science, The University of Sydney, Australia. · Australian Centre for Robotics, The University of Sydney, Australia.
GeoProp: Grounding Robot State in Vision for Generalist Manipulation
Jul 8, 2026Guoyang Zhao, Quanhao Qian, Gongjie Zhang +5Generalization in Robotic Manipulation Visuomotor Policy Learning
Tongji University · DAMO Academy, Alibaba Group · HuPan Lab +2
FOCUS: From Privileged States to RGB-D with Controlled Modality Switching and Representation Alignment
Oct 8, 2026Filip Grigorov, Kourosh Darvish, Nandita VijaykumarRobotic RL Visual RL
University of Toronto Canada

Sep 14, 2026J/K move · Enter open · S save

StereoPatch: Patch-Aligned RGB-Depth Fusion for Spatial Perception in Robot Manipulation

Yanan Zhou, Zhaoyan Qian, James Zhao, Weiming Zhi

School of Computer Science, The University of Sydney, Australia. · Australian Centre for Robotics, The University of Sydney, Australia.

Generalization in Robotic Manipulation Visuomotor Policy Learning Robotic Manipulation Multimodal Fusion Robotic Perception

Open paper AskPDF arXiv

Recent advances in robot imitation learning have produced visuomotor policies that predict actions directly from visual observations. Yet visually similar scenes can require different actions as target position, object height, or contact geometry changes. Pretrained RGB features may map these geometrically distinct states to similar policy inputs, while simply adding depth requires the policy to learn RGB-depth correspondence from the same limited demonstrations used to learn control. We introduce StereoPatch, a patch-aligned RGB-depth representation that binds registered metric geometry directly to the RGB patches used for action prediction. On a shared 2-D patch grid, asymmetric cross-attention incorporates depth information into the corresponding RGB features before action decoding. The resulting StereoPatch Tokens provide a geometry-aware visual representation that can condition general visuomotor policies without changing their underlying learning objectives. Across six real-robot tasks, StereoPatch achieves higher closed-loop success than appearance-only, geometry-only, raw RGB-D, and late-fusion baselines. Additional experiments across three simulation suites evaluate compatibility across visuomotor policy architectures, spatial generalization, and operating limits. Results suggest that resolving control-relevant geometric ambiguity benefits from aligning depth directly with the visual features used for action prediction, rather than supplying it as an independent modality. Project page: https://aus.bot/research/stereopatch/.

Abstract

State Representation Learning

Explore similar work

StereoPatch: Patch-Aligned RGB-Depth Fusion for Spatial Perception in Robot Manipulation

GeoProp: Grounding Robot State in Vision for Generalist Manipulation

FOCUS: From Privileged States to RGB-D with Controlled Modality Switching and Representation Alignment