DeFM: Learning Foundation Representations from Depth for Robotics

Manthan Patel; Jonas Frey; Mayank Mittal; Fan Yang; Alexander Hansson; Amir Bar; Cesar Cadena; Marco Hutter

cs.ROJan 26, 2026

DeFM: Learning Foundation Representations from Depth for Robotics

Authors: Manthan Patel, Jonas Frey, Mayank Mittal, Fan Yang, Alexander Hansson, Amir Bar, Cesar Cadena, Marco Hutter

Organizations: Robotic Systems Lab (RSL), ETH Zurich, Switzerland · Stanford University, Stanford, USA · UC Berkeley, Berkeley, USA · NVIDIA, Zurich, Switzerland

Abstract

Depth sensors are widely deployed across robotic platforms, and advances in fast, high-fidelity depth simulation have enabled robotic policies trained on depth observations to achieve robust sim-to-real transfer for a wide range of tasks. Despite this, representation learning for depth modality remains underexplored compared to RGB, where large-scale foundation models now define the state of the art. To address this gap, we present DeFM, a self-supervised foundation model trained entirely on depth images for robotic applications. Using a DINO-style self-distillation objective on a curated dataset of 60M depth images, DeFM learns geometric and semantic representations that generalize to diverse environments, tasks, and sensors. To retain metric awareness across multiple scales, we introduce a novel input normalization strategy. We further distill DeFM into compact models suitable for resource-constrained robotic systems. When evaluated on depth-based classification, segmentation, navigation, locomotion, and manipulation benchmarks, DeFM achieves state-of-the-art performance and demonstrates strong generalization from simulation to real-world environments. We release all our pretrained models, which can be adopted off-the-shelf for depth-based robotic learning without task-specific fine-tuning. Webpage: https://de-fm.github.io/

Figures & tables

Fig. 2: Overview of the self-supervised pretraining of DeFM.

Dataset Name	Type	Domain	Size (M)
ImageNet-21k [ 86 ]	MDE	Objects	14.0
SA-1B [ 40 ]	MDE	Objects, Indoor, Outdoor	11.0
Replica [ 88 ]	Synthetic	Indoor	0.3
Hypersim [ 89 ]	Synthetic	Indoor	0.07
Meta GraspNet [ 90 ]	Synthetic	Manipulation	0.3
TartanGround [ 91 ]	Synthetic	Indoor, Outdoor	5.6

TABLE I: Overview of depth pretraining dataset collection. MDE denotes monocular depth estimation.

Fig. 3: Visualization of different depth input normalization methods. Log normalization effectively captures the overall depth range while preserving fine-grained structure in near-field regions (b, d, f). In contrast, standard metric normalization yields weaker contrast and poorly separated gradients (c, e). In our representation, we stack columns (b), (d), and (f) to form a 3-channel normalized depth input, which retains metric awareness across diverse domains.

Fig. 4: Overview of distilling the DeFM-L/14 teacher into CNN backbones. A BiFPN module is added on top of the CNN encoder to produce dense spatial features, which are supervised using the teacher’s spatial tokens. The teacher’s class token provides global supervision to the CNN’s pooled feature representation.

Fig. 5: PCA visualization of the patch features obtained from the DeFM-L/14 encoder when processing depth images of various cups captured by different sensors. The first three PCA components are mapped to the RGB color channels for visualization. Notice the feature consistency of the cup handle (visualized in yellow) across all images, demonstrating that DeFM learns a useful prior for a robotic grasping task. The background is removed by thresholding the first PCA component.

Architecture	Feature	Top-1 KNN (↑)	Top-5 KNN (↑)	Linear (↑)
ViT-L/14	DINOv2	60.28	80.75	68.26
ViT-L/16 (CPE)	C-RADIOv3	55.23	76.96	63.92
ViT-L/16	DINOv3	64.07	83.48	69.55
ViT-L/14	DeFM	63.46	84.79	71.72
DeiT-S/16	Theia	28.17	47.47	33.14
ViT-S/14	DINOv2	47.66	69.11	55.06

TABLE II: Classification evaluations on ImageNet-1k-Depth.

Model	Params (M)	Top-1 KNN (↑)	Top-5 KNN (↑)	Linear (↑)
ViT-S/14	22.1	55.27	78.06	61.54
ViT-L/14	307.0	63.46	84.79	71.72
ResNet-18	11.7	45.71	69.69	50.58
ResNet-34	21.8	48.16	72.72	54.39
ResNet-50	26.2	53.71	77.63	61.54
RegNetY-400MF	4.1	49.51	72.87	50.51

TABLE III: Comparison of distilled CNN architectures on ImageNet-1k-Depth Classification.

Dataset	Domain / Scene Type	# Classes	Range (m)	Sensor
ScanNet [ 96 ]	Indoor room scenes	40	$<10$	Structure Sensor
SUN-RGBD [ 110 ]	Indoor rooms / building scenes	37	$<10$	Kinect v1-v2 / RealSense / Xtion
TartanGround [ 91 ]	Outdoor construction, industry, infrastructure	6	$<100$	Simulated
OFFSED [ 111 ]	Outdoor off-road environments	5	$<30$	ZED Stereo Camera
GraspNet-1B [ 97 ]	Tabletop object grasping	40	$<1$	Azure Kinect

TABLE IV: Summary of segmentation datasets used for downstream evaluation.

Model	Pretraining	Indoor		Outdoor		Manipulation
Model	Pretraining	ScanNet	SUN-RGBD	OFFSED	TartanGround	GraspNet-1B
ViT-L/14	DINOv2	24.46	27.11	53.47	59.24	24.26
ViT-L/16 (CPE)	C-RADIOv3	25.56	28.17	56.53	64.31	25.18
ViT-L/16	DINOv3	28.52	32.74	54.42	62.16	23.89
ViT-L/14	DAv2	24.36	27.13	56.93	60.89	24.93
ViT-L/14	DeFM	31.34	31.26	57.62	67.69	27.85

TABLE V: Semantic Segmentation evaluation on various datasets (mIoU) (↑).

Fig. 6: Qualitative results of semantic segmentation on different datasets.

Input Normalization
Ablation	Indoor		Outdoor		Manipulation
Ablation	ScanNet	SUN-RGBD	OFFSED	TartanGround	GraspNet-1B
$C_{1}$ (Global Log-Scaled)	16.24	13.36	49.93	59.23	16.68
$C_{3}$ (Far-Range Norm)	19.16	16.23	51.34	59.56	15.38
$[C_{1},C_{2},C_{3}]$ (Ours)	19.34	16.09	52.07	61.05	17.05
Pretraining Data

TABLE VI: Ablation studies on input normalization, pretraining data, and CNN distillation architectures. Performance is reported as Semantic Segmentation mIoU (↑).

Model	Params (M)	FLOPs (GFLOPs)	RTX 4090 (ms)	Memory (GB)	Jetson AGX Orin (ms)
Batch Size	-	128	128	128	1
ViT-S/14	22.1	707.48	63.76	0.99	11.92
ViT-L/14	307.0	9962.24	624.91	3.32	72.82
ResNet-18	11.7	256.13	21.06	0.93	8.67
ResNet-34	21.8	493.69	33.08	1.33	13.54
ResNet-50	26.2	631.19	69.39	1.66	17.79

TABLE VII: Comparison of inference times of DeFM models on training and deployment hardware.

Model	Training	Industrial Hangar	Abandoned Cable	Modern City Downtown
ImageNet	77.61	50.00	38.33	65.00
VAE	90.23	89.67	83.00	73.67
DINOv3	83.64	90.00	74.67	72.67
DeFM	90.31	92.67	84.33	79.00

TABLE VIII: Navigation Success Rate (SR %) (

\uparrow

) for embodiment aware navigation.

Fig. 8: Split of navigation failure count (

\downarrow

) for embodiment aware navigation evaluation across the test environments.

Fig. 9: Real world deployments using a Unitree B2W robot in diverse environments- (A) Indoor, (B) Urban, (C) Park, (D) Construction Site.

Fig. 10: Real world deployment in a park environment with a goal point at around $90\text{\,}\mathrm{m}$ .

Panel of Figure 18 — Fig. 11: Rollout of DeFM model for dexterous grasping using the Kuka-Allegro setup. Bottom: The simulated noisy depth images used during training.

Fig. 12: Depth images input to the dexterous grasping policy. During training, images are augmented with speckles, dropout, and stick noise. For evaluations, we also consider depth images augmented with the Kinect noise model [ 113 ] .

Pretraining	Frozen		Fine-Tuned
Pretraining	Training Noise	Kinect Noise	Training Noise	Kinect Noise
Scratch	—	—	0.7774	0.7247
ImageNet	0.6576	0.0043	0.7950	0.7366
VAE	0.6894	0.0219	0.8105	0.0379
DINOv3	0.6532	0.2075	0.8243	0.7837
DeFM	0.8089	0.4860	0.8936	0.8763

TABLE IX: Comparison of different model configurations in simulation for dexterous grasping. Performance is normalized relative to the teacher’s performance and averaged across 2,000 episodes. All models use ResNet-18 architecture.

Fig. 13: PCA visualization for real-world images from different scenes of the DROID dataset [ 99 ] .

Fig. 14: ANYmal climbing a ladder, shown together with noisy depth image (top row) from onboard camera.

Model	CNN-Scratch	ImageNet	VAE	DINOv3	DeFM
SR % ( $\uparrow$ )	90.45	89.29	89.75	89.59	90.14

TABLE X: Comparison of different models on the Ladder Climbing task in terms of Climb Success Rate

Fig. 15: PCA visualization for real-world images collected during deployment of ANYmal ladder-climbing in the wild.

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Setting	Value
Backbone	ViT-L/14, stochastic depth 0.4
Objective	DINO + iBOT + KoLeo (weights 1.0 / 1.0 / 0.1)
Projection heads	separate DINO and iBOT heads, 131,072 prototypes
iBOT masking	probability 0.5, mask ratio $\mathcal{U}[0.1,0.5]$
Centering	Sinkhorn-Knopp
Teacher temperature	0.04 $\rightarrow$ 0.07

Appendix

TABLE XI: Pretraining configuration of DeFM-L/14 (

\dagger

: modified w.r.t. DINOv2).

Augmentation	Global 1	Global 2	Local
Intensity jitter a , $p$	0.8	0.8	0.8
Gaussian blur, $p$	1.0	0.1	0.5
max. radius	2.0	1.5	1.5
Gaussian noise, $p$	0.3	0.2	0.25
std. $\sigma$	0.03	0.02	0.018
Pixel dropout b , $p$	0.3	–	0.2

Appendix

TABLE XII: Depth-specific augmentations, applied after cropping and horizontal flipping;

p

denotes the probability of applying an augmentation to a crop. Color jittering, grayscale conversion, and solarization of DINOv2 are removed.

Setting	Value
Features (ViT)	cls token of last 1 or 4 blocks (+ avg. patch tokens) a
Features (CNN)	pooled feature (+ pooled spatial features) a
Input resolution	$224^{2}$
Augmentations	train: resize 256, random crop, flip;
	test: resize 256, center crop
Head	linear

Appendix

TABLE XIII: Probing protocol for classification.

Dataset	Train / Eval	Resolution ( $H\times W$ )	Epochs (warmup)
ScanNet	19,456 / 5,436	$480\times 640$	25 (3)
SUN-RGBD	5,285 / 5,050	$530\times 730$	25 (3)
TartanGround	800 / 400	$640\times 640$	25 (3)
OFFSED	162 / 41	$360\times 640$	200 (20)
GraspNet-1B	2,000 / 600	$360\times 640$	25 (3)

Appendix

TABLE XIV: Dataset-specific settings for segmentation probing.

Explore similar work

StereoPatch: Patch-Aligned RGB-Depth Fusion for Spatial Perception in Robot Manipulation
Sep 14, 2026Yanan Zhou, Zhaoyan Qian, James Zhao +1Generalization in Robotic Manipulation Visuomotor Policy Learning
School of Computer Science, The University of Sydney, Australia. · Australian Centre for Robotics, The University of Sydney, Australia.
DepthWorld: 3D World Model for Robot Manipulation
Oct 6, 2026Jai Bardhan, Josef Sivic, Vladimir PetrikMulti-View Depth Estimation World Models for Robotics
Czech Institute of Informatics, Robotics and Cybernetics Czech Technical University in Prague
DINOcular: Self-Supervised Visuospatial Representations
Aug 27, 2026Farkhat Almukhamedov, Sami Azirar, Hermann BlumVisual Representation Learning Geometric Representation Learning
Robot Perception and Learning Lab University of Bonn & Lamarr Institute

Sep 14, 2026J/K move · Enter open · S save

StereoPatch: Patch-Aligned RGB-Depth Fusion for Spatial Perception in Robot Manipulation

Yanan Zhou, Zhaoyan Qian, James Zhao, Weiming Zhi

School of Computer Science, The University of Sydney, Australia. · Australian Centre for Robotics, The University of Sydney, Australia.

Generalization in Robotic Manipulation Visuomotor Policy Learning Robotic Manipulation Multimodal Fusion Robotic Perception

Open paper AskPDF arXiv

Recent advances in robot imitation learning have produced visuomotor policies that predict actions directly from visual observations. Yet visually similar scenes can require different actions as target position, object height, or contact geometry changes. Pretrained RGB features may map these geometrically distinct states to similar policy inputs, while simply adding depth requires the policy to learn RGB-depth correspondence from the same limited demonstrations used to learn control. We introduce StereoPatch, a patch-aligned RGB-depth representation that binds registered metric geometry directly to the RGB patches used for action prediction. On a shared 2-D patch grid, asymmetric cross-attention incorporates depth information into the corresponding RGB features before action decoding. The resulting StereoPatch Tokens provide a geometry-aware visual representation that can condition general visuomotor policies without changing their underlying learning objectives. Across six real-robot tasks, StereoPatch achieves higher closed-loop success than appearance-only, geometry-only, raw RGB-D, and late-fusion baselines. Additional experiments across three simulation suites evaluate compatibility across visuomotor policy architectures, spatial generalization, and operating limits. Results suggest that resolving control-relevant geometric ambiguity benefits from aligning depth directly with the visual features used for action prediction, rather than supplying it as an independent modality. Project page: https://aus.bot/research/stereopatch/.

Abstract

Explore similar work

StereoPatch: Patch-Aligned RGB-Depth Fusion for Spatial Perception in Robot Manipulation

DepthWorld: 3D World Model for Robot Manipulation

DINOcular: Self-Supervised Visuospatial Representations