DART: Depth-as-Target Pretraining for Surgical Vision Foundation Models

John J. Han; Adam Schmidt; Muhammad Abdullah Jamal; Jie Ying Wu; Omid Mohareri

cs.CVSep 3, 2026

DART: Depth-as-Target Pretraining for Surgical Vision Foundation Models

Authors: John J. Han, Adam Schmidt, Muhammad Abdullah Jamal, Jie Ying Wu, Omid Mohareri

Organizations: Vanderbilt University 2305 West End Ave Nashville, Tennessee, USA · Intuitive Surgical, Inc. 1020 Kifer Rd, Sunnyvale, California, USA

Abstract

Vision foundation models (VFMs) are valuable in data-scarce domains such as surgery, where a single pretrained backbone can provide rich representations for many downstream tasks. Yet the dominant self-supervised pretraining paradigm uses only RGB images, leaving readily available complementary signals, such as depth maps, unused. This is a particular missed opportunity in surgery, where natural-image VFMs transfer poorly while the scene geometry is rich and informative. With strong off-the-shelf models now able to produce pseudo-labeled dense depth for any image corpus, we hypothesize that such signals can be folded into pretraining to learn better representations. We present DART, an RGB-D pretraining recipe that builds on DINOv2 with a simple modification: a pixel-space depth reconstruction objective applied to masked iBOT patches, supervised by pseudo-labeled depth. Depth is used only during pretraining, so fine-tuning and inference remain RGB-only. We find that this pixel-level reconstruction head improves representation quality rather than disrupting it. We further show that depth, which encodes scene geometry, is more effective as a target than alternative dense signals such as Canny edges, confirming that the gains stem from depth rather than added supervision alone. Across eight surgical benchmarks spanning segmentation, depth estimation, and image-level recognition, DART outperforms both natural-image and in-domain baselines, including a vanilla DINOv2 trained on identical data, improving dense prediction while also strengthening image-level understanding. More broadly, DART shows that freely available geometric pseudo-labels can strengthen foundation model pretraining without extra labels or added inference cost, pointing toward stronger backbones for surgery.

Figures & tables

Figure 1: An overview of our method for a single global crop example. We build on the DINOv2 framework and maintain all components and loss terms. In DINOv2, global crops are 50%-masked and enter the student backbone. The iBOT objective enforces student patch features to match the teacher’s patch at the masked locations. DART adds a lightweight head that regresses pixel-wise depth from the student’s patch features. A reconstruction $\ell_{1}$ loss is minimized to match pseudo-labeled depth maps. Loss is computed only at masked patch locations, similar to iBOT. This simple modification lends itself to a RGB-D pre-training recipe that improves downstream tasks without requiring depth maps at fine-tuning/runtime. Local crops are processed identically to the DINOv2 recipe. “sg” stands for stop-gradient.

Figure 2: Samples from the RGB-D dataset and qualitative results displaying the depth reconstruction quality of DART. Each sample is ordered as RGB, pseudo-labeled depth, and reconstructed depth of DART. Note that the objective of our approach is not for good depth estimation but rather improving representation learning for downstream tasks.

			EndoVis18		CholecISeg		SARRARP50		CholecSeg8k		PhaKIR
Data	Method	Depth	lin.	+ms.	lin.	+ms.	lin.	+ms.	lin.	+ms.	lin.	+ms.
Natural	DINOv2	–	28.8	31.8	48.3	52.6	53.5	62.0	36.7	38.5	10.3	10.4
	DINOv3	–	30.1	32.5	53.5	55.2	58.0	64.0	50.0	43.7	11.6	10.6
	MAE	–	26.0	28.0	37.1	47.1	52.7	55.1	37.9	38.9	7.6	8.8
	MultiMAE	PT	26.6	27.0	40.7	48.7	51.2	56.1	36.0	36.7	8.3	10.1
LEMON	LEMON-FM	–	25.7	31.2	40.6	51.0	52.0	66.6	33.9	42.4	8.7	11.0

Table 1: Semantic segmentation with frozen features , linear (lin.) and multiscale (+ms). Depth marks whether depth is used in pre-training (PT) and/or fine-tuning (FT). Bold is best and underline is second best among the RGB-only fine-tuning settings; the RGB-D fine-tuning row (gray) is excluded from the ranking. Mean over 3 seeds; per-cell std reported in the appendix. CholecISeg = CholecInstanceSeg. Natural images refer to LVD for DINO models and IN-1k for MAE models. We report the mean over 3 seeds; standard deviations are in the appendices due to space constraints.

Figure 3: Qualitative results displaying segmentation results on EndoVis18, CholecInstanceSeg, and CholecSeg8k under the multiscale setting. We only display results for models pretrained on the surgical data. We refer the reader to the appendices for more qualitative results.

Pretrain. data	Method	Depth	Linear	DPT
LEMON	LEMON-FM	–	0.485 $\pm$ 0.005	0.608 $\pm$ 0.006
	MAE	–	0.512 $\pm$ 0.003	0.609 $\pm$ 0.026
	MultiMAE	PT	0.498 $\pm$ 0.004	0.605 $\pm$ 0.007
	DINOv2	–	0.542 $\pm$ 0.003	0.626 $\pm$ 0.009
	Ours	PT	0.557 $\pm$ 0.008	0.653 $\pm$ 0.009
Natural	DINOv2	–	0.506 $\pm$ 0.011	0.613 $\pm$ 0.037

Table 2: Depth estimation on SCARED-C (

\delta_{1}\uparrow

). Mean

\pm

std over 3 seeds. Bold is best and underline is second best across all rows. We exclude MultiMAE RGB-D since it would be a trivial task to regress depth from RGB-D. Depth marks whether depth is used in pre-training (PT). Natural images refer to LVD and IN-1k for DINO and MAE models respectively.

Figure 4: Qualitative results displaying depth estimation results on the SCARED-C dataset under the DPT setting. We only display results for models pretrained on the surgical data. We refer the reader to the appendices for more qualitative results.

		Cholec80	CholecT50 Triplet Recognition
Pretrain. data	Method	F1 Score	AP i	AP v	AP t	AP it	AP iv	AP ivt
LEMON	MAE	45.0 $\pm$ 0.4	36.4 $\pm$ 0.1	23.9 $\pm$ 0.2	15.8 $\pm$ 0.5	5.9 $\pm$ 0.1	9.6 $\pm$ 0.2	3.6 $\pm$ 0.1
	MultiMAE	43.9 $\pm$ 0.3	37.8 $\pm$ 0.3	23.7 $\pm$ 0.2	15.7 $\pm$ 0.4	5.2 $\pm$ 0.2	9.5 $\pm$ 0.1	3.5 $\pm$ 0.3
	DINOv2	70.8 $\pm$ 0.1	58.8 $\pm$ 0.5	34.6 $\pm$ 0.2	25.9 $\pm$ 0.1	10.4 $\pm$ 0.5	15.4 $\pm$ 0.1	6.4 $\pm$ 0.1
	Ours	71.5 $\pm$ 0.1	62.1 $\pm$ 0.2	37.0 $\pm$ 0.1	28.2 $\pm$ 0.2	11.6 $\pm$ 0.1	16.2 $\pm$ 0.5	6.9 $\pm$ 0.1
	LEMON-FM	67.1 $\pm$ 0.1	73.6 $\pm$ 0.4	44.6 $\pm$ 0.2	30.0 $\pm$ 0.1	13.3 $\pm$ 0.4	20.0 $\pm$ 0.1	7.9 $\pm$ 0.3

Table 3: Image-level surgical understanding. Cholec80 phase recognition F1 Score (%) and CholecT50 triplet recognition (AP, %). Mean

\pm

std over 3 seeds. Bold = best and underline = second best per column, across all rows. Natural image datasets are LVD for DINO models and IN-1k for MAE models.

	EndoVis18		CholecISeg		SARRARP50		CholecSeg8k		PhaKIR
Aux. signal	lin.	+ms.	lin.	+ms.	lin.	+ms.	lin.	+ms.	lin.	+ms.
None (DINOv2)	30.4	30.9	48.5	50.9	57.4	54.6	39.9	42.7	10.3	9.5
Canny edges	29.1	30.9	48.6	49.9	56.3	55.3	38.5	43.7	10.6	8.7
Blurred grayscale	28.2	29.2	45.4	50.0	55.9	53.8	38.4	44.2	10.4	9.3
Depth (ours)	31.4	31.2	48.5	47.4	57.8	55.6	41.0	42.7	10.7	10.7

Table 4: Auxiliary reconstruction signal ablation (ViT-S; linear (lin.) and multiscale (+ms.) mIoU, mean over 3 seeds). Bold = best and underline = second best per column. We report mIoU. CholecISeg = CholecInstanceSeg.

Depth decoder	$d_{w}$	EV18 (mIoU)	Cholec80 (F1)
–	0	29.3	62.4
MLP	0.025	29.2	62.2
	0.05	29.6	64.0
	0.1	30.6	65.5
	0.25	7.8	17.2
Transformer	0.05	18.5	37.2

Table 5: Depth decoder and loss-weight sweep. EndoVis18 (EV18) linear (lin.) mIoU as a function of the depth decoder architecture and the depth-loss weight

d_{w}

. The top row (

d_{w}=0

) is the depth-free baseline; the Transformer decoder follows the MAE decoder design.

Figure 5: PCA visualization of the features learned by DINOv2 compared to DART at $640\times 640$ resolution. Generally, DART has smoother features which is reflected in its superior quantitative results. Because we do not incorporate registers in our ViTs, we clip outliers outside of 95% for improved visualization.

Config	MAE	MultiMAE †	DINOv2	DART
Optimization
Optimizer	AdamW
Base LR	$1\mathrm{e}{-4}$	$1\mathrm{e}{-4}$	$1.5\mathrm{e}{-3}$	$1.5\mathrm{e}{-3}$
Weight decay	0.05	0.05	0.04	0.04
$\beta_{1},\beta_{2}$	0.9, 0.95	0.9, 0.95	0.9, 0.999	0.9, 0.999
Batch size	4096	2048	1024	1024

Table S1: Pre-training configurations for in-domain baselines and DART, all trained on the LEMON corpus. “—” denotes parameters not applicable to a given method.

Config	lin. + ms.	DPT	Image-level Tasks
Optimization
Optimizer	AdamW	AdamW	SGD
Learning rate	$1\mathrm{e}{-3}$	1e-3	1e-3
$\beta_{1},\beta_{2}$	0.9, 0.999	0.9, 0.999	–
Weight decay	0.05	0.05	0.0
Batch size	64	64	256

Table S2: Fine-tuning configuration used across downstream segmentation experiments. The backbone is frozen and only the task head is trained, so depth supervision affects evaluation solely through the learned representations. Input resolution is

224\times 224

with patch size

16

. We normalize images based on the pre-training data source, e.g., ImageNet-pretrained models undergoes ImageNet normalization.

Dataset	Task	Procedure	Modality	In/Ex	Cls.	Size
Cholec80	Phase recog.	Cholecystectomy	Manual lap.	In	7	80 videos
CholecT50	Triplet recog.	Cholecystectomy	Manual lap.	In	100 ∗	50 vid. / ${\sim}100$ k fr.
SCARED-C	Depth estimation	Porcine abdomen	Robotic	Ex	—	17,135 RGB-D fr.
EndoVis18	Scene seg.	Porcine nephrectomy	Robotic	Ex	12	19 video seq.
CholecInstanceSeg	Instrument seg.	Cholecystectomy	Manual lap.	In	8	41,933 fr. / 85 proc.
SAR-RARP50	Instrument seg.	Radical prostatectomy	Robotic	In	10	50 vid. / ${\sim}16$ k fr.

Table S3: Downstream evaluation datasets used in this work. “In/Ex” denotes in-vivo versus ex-vivo . “Classes” refers to the number of label categories used in this work. Lap. = laparoscopy.

			EndoVis18		CholecISeg		SARRARP50		CholecSeg8k		PhaKIR
Data	Method	Depth	lin.	+ms.	lin.	+ms.	lin.	+ms.	lin.	+ms.	lin.	+ms.
LEMON	DINOv2 ViT-S	–	30.4	30.9	48.5	50.9	57.4	54.6	39.9	42.7	10.3	9.5
	Ours ViT-S	PT	31.4	31.2	48.5	47.4	57.8	55.6	41.0	42.7	10.7	10.7
	DINOv2 ViT-B	–	32.1	31.8	52.8	56.3	61.5	65.1	41.6	46.1	11.8	12.8
	Ours ViT-B	PT	32.1	32.2	57.2	63.5	63.5	67.0	43.9	50.3	12.3	13.6

Table S4: ViT-S semantic segmentation with frozen features , linear (lin.) and multiscale (+ms), with ViT-B counterparts for reference. Depth marks whether depth is used in pre-training (PT) and/or fine-tuning (FT). Mean over 3 seeds. CholecISeg = CholecInstanceSeg. For these experiments, we use batch size = 256 due to compute constraints.

Dataset	Head	DINOv2	DART-128	DART-256	DART-512
EndoVis18	lin.	30.4	29.6	31.4	29.5
EndoVis18	+ms.	30.9	31.4	31.2	30.8
CholecInstanceSeg	lin.	48.5	47.2	48.5	46.8
CholecInstanceSeg	+ms.	50.9	47.2	47.4	46.4
CholecSeg8k	lin.	39.9	42.9	41.0	39.1
CholecSeg8k	+ms.	42.7	42.5	42.7	40.3

Table S5: Depth-head hidden dimension sweep (ViT-S). Linear (lin.) and multiscale (+ms.) frozen-feature mIoU on four surgical segmentation benchmarks. Hidden dimensions are reported in ascending order; the leftmost column is the depth-free DINOv2 baseline. Best per row in bold .

			EndoVis18		CholecInstanceSeg		SARRARP50
Data	Method	Depth	lin.	+ms.	lin.	+ms.	lin.	+ms.
Natural	DINOv2	–	28.8 $\pm$ 1.0	31.8 $\pm$ 0.7	48.3 $\pm$ 0.2	52.6 $\pm$ 0.5	53.5 $\pm$ 0.4	62.0 $\pm$ 0.2
	DINOv3	–	30.1 $\pm$ 0.5	32.5 $\pm$ 0.6	53.5 $\pm$ 2.7	55.2 $\pm$ 4.1	58.0 $\pm$ 0.8	64.0 $\pm$ 0.4
	MAE	–	26.0 $\pm$ 0.2	28.0 $\pm$ 0.4	37.1 $\pm$ 0.3	47.1 $\pm$ 0.4	52.7 $\pm$ 0.2	55.1 $\pm$ 0.5
	MultiMAE	PT	26.6 $\pm$ 0.3	27.0 $\pm$ 0.5	40.7 $\pm$ 0.3	48.7 $\pm$ 0.4	51.2 $\pm$ 0.2	56.1 $\pm$ 0.1
LEMON	LEMON-FM	–	25.7 $\pm$ 0.4	31.2 $\pm$ 0.3	40.6 $\pm$ 0.4	51.0 $\pm$ 0.1	52.0 $\pm$ 0.2	66.6 $\pm$ 0.3

Table S6: Semantic segmentation with frozen features, mean

\pm

std (part 1 of 2). Linear (lin.) and multiscale (+ms.). Depth marks whether depth is used in pre-training (PT) and/or fine-tuning (FT). Mean and standard deviation over 3 seeds. CholecISeg = CholecInstanceSeg. Natural images refer to LVD for DINO models and IN-1k for MAE models. Remaining datasets in Table S7 .

			CholecSeg8k		PhaKIR
Data	Method	Depth	lin.	+ms.	lin.	+ms.
Natural	DINOv2	–	36.7 $\pm$ 0.5	38.5 $\pm$ 0.5	10.3 $\pm$ 0.6	10.4 $\pm$ 0.3
	DINOv3	–	50.0 $\pm$ 2.9	43.7 $\pm$ 0.4	11.6 $\pm$ 0.3	10.6 $\pm$ 0.1
	MAE	–	37.9 $\pm$ 2.1	38.9 $\pm$ 1.2	7.6 $\pm$ 0.8	8.8 $\pm$ 0.4
	MultiMAE	PT	36.0 $\pm$ 1.1	36.7 $\pm$ 0.4	8.3 $\pm$ 0.5	10.1 $\pm$ 0.3
LEMON	LEMON-FM	–	33.9 $\pm$ 0.3	42.4 $\pm$ 0.8	8.7 $\pm$ 0.5	11.0 $\pm$ 0.2

Table S7: Semantic segmentation with frozen features, mean

\pm

std (part 2 of 2). Continuation of Table S6 for the CholecSeg8k and PhaKIR datasets. Mean and standard deviation over 3 seeds. Column conventions as in Table S6 .

	DAv2		DART
Dataset	H1	H2	H1	H2
EndoVis18	27.1	33.7	34.9	36.8
CholecISeg	46.5	51.0	66.2	68.9
SARRARP50	52.0	59.6	68.7	70.2
CholecSeg8k	36.0	36.8	52.4	50.5
PhaKIR	9.6	10.7	14.5	16.2

Table S8: Quantitative results for frozen Depth Anything v2 ViT-B on dense pixel tasks. H1 is a single linear head; H2 is +m.s. for segmentation and DPT for depth. The first five rows are segmentation (mIoU) and the last is depth estimation (

\delta_{1}

Figure S1: EndoVis18 segmentation qualitative results. From left to right: Ground truth, MAE (IN-1k), MultiMAE (IN-1k), DINOv2 (LVD), DINOv3 (LVD), LEMON-FM, MAE (LEMON), MultiMAE (LEMON), DINOv2 (LEMON), DART (LEMON).

Figure S2: CholecSeg8k segmentation qualitative results. From left to right: Ground truth, MAE (IN-1k), MultiMAE (IN-1k), DINOv2 (LVD), DINOv3 (LVD), LEMON-FM, MAE (LEMON), MultiMAE (LEMON), DINOv2 (LEMON), DART (LEMON).

Figure S3: SCARED-C depth estimation qualitative results. From left to right: RGB, Ground truth, MAE (IN-1k), MultiMAE (IN-1k), DINOv2 (LVD), DINOv3 (LVD), LEMON-FM, MAE (LEMON), MultiMAE (LEMON), DINOv2 (LEMON), DART (LEMON).

Visual Representation Learning

Visual representation learning develops image and video features that capture useful structure for recognition, retrieval, spatial reasoning, and action.

See all 227 papers

Aug 27, 2026

DINOcular: Self-Supervised Visuospatial Representations

We introduce a self-supervised framework for learning joint visuospatial representations from RGB-D observations. While modern vision foundation models are trained almost exclusively on RGB images, many embodied systems have access to explicit depth sensing, which provides geometric information that monocular inputs cannot recover. Our method integrates depth-derived geometric priors with a visual backbone through inter-patch and intra-patch fusion, enabling the model to encode both appearance and spatial structure efficiently. The resulting representation shows promising improvements on 3D awareness while preserving semantic transfer: it outperforms prior methods of comparable scale on multiple 3D geometry benchmarks, and remains competitive when probed for standard RGB-D semantic segmentation tasks.

Farkhat Almukhamedov, Sami Azirar, Hermann Blum

Robot Perception and Learning Lab University of Bonn & Lamarr Institute

Visual Representation Learning Geometric Representation Learning 3D Representation Learning Self-Supervised Visual Representation Learning

Ask PDF

Mar 31, 2026

Scaling Video Pretraining for Surgical Foundation Models

Surgical video understanding is essential for computer-assisted interventions, yet existing surgical foundation models remain constrained by limited data scale, procedural diversity, and inconsistent evaluation, often lacking a reproducible training pipeline. We propose SurgRec, a scalable and reproducible pretraining recipe for surgical video understanding, instantiated with two variants: SurgRec-MAE and SurgRec-JEPA. We curate a large multi-source corpus of 10,535 videos and 214.5M frames spanning endoscopy, laparoscopy, cataract, and robotic surgery. Building on this corpus, we develop a unified pretraining pipeline with balanced sampling and standardize a reproducible benchmark across 16 downstream datasets and four clinical domains with consistent data splits. Across extensive comparisons against SSL baselines and vision-language models, SurgRec consistently achieves superior performance across downstream datasets. In contrast, VLMs prove unreliable for fine-grained temporal recognition, exhibiting both performance gaps and sensitivity to prompt phrasing. Our work provides a reproducible, scalable foundation for the community to build more general surgical video models. All code, models, and data will be publicly released.

Sicheng Lu, Zikai Xiao, Jianhui Wei +7

The Johns Hopkins University, United States · Zhejiang University, China · Zhejiang University-University of Illinois Urbana-Champaign Institute, China +4

Surgical Video Understanding Video Representation Learning

Ask PDF

Jun 5, 2026

DALE-CT: Depth-Aware Foundation Models for Computed Tomography

Recent breakthroughs in self-supervised learning (SSL), such as the Latent-Euclidean Joint-Embedding Predictive Architecture (LeJEPA), alongside successes in integrating visual encoders with language models, have driven the demand for adaptable, high-capacity vision encoders in Computed Tomography (CT). In this work, we explore 2D slice-based architectures as a flexible alternative to native 3D models for processing volumetric CT data. Using the CT-RATE dataset, we trained DALE-CT (Depth-Aware Latent-Euclidean Computed Tomography), a 2D model family built entirely from scratch using LeJEPA, and compared its performance against a continually pre-trained DINOv2 baseline. To enhance representation quality, we developed a novel 3D depth-aware pre-training strategy anchored by dense auxiliary supervision from both automated anatomical masks and human-annotated abnormalities. Under linear probe evaluation with Multiple Instance Learning (MIL) for multi-abnormality detection, the frozen backbone of this dual-supervised model (DALE-CT-2S) achieves a Macro AUROC of 0.833. This performance demonstrates near-parity with state-of-the-art 3D vision-language models, achieved entirely from scratch with significantly less data and no textual supervision. To ensure reproducibility, all training code, evaluation scripts, and model weights have been made publicly available.

Evan W. Damron, Mahmut S. Gokmen, Mitchell A. Klusty +3

College of Medicine Office of Research, University of Kentucky, Lexington, KY 40506 USA

Medical Imaging Foundation Models Self-Supervised Pre-Training Medical Image Anomaly Detection Computed Tomography

Ask PDF

Abstract

Visual Representation Learning

Explore similar work

DINOcular: Self-Supervised Visuospatial Representations

Scaling Video Pretraining for Surgical Foundation Models

DALE-CT: Depth-Aware Foundation Models for Computed Tomography