Scaling Video Pretraining for Surgical Foundation Models

Sicheng Lu; Zikai Xiao; Jianhui Wei; Danyu Sun; Qi Lu; Keli Hu; Yang Feng; Jian Wu

cs.CVMar 31, 2026

Scaling Video Pretraining for Surgical Foundation Models

Authors: Sicheng Lu, Zikai Xiao, Jianhui Wei, Danyu Sun, Qi Lu, Keli Hu, Yang Feng, Jian Wu, +2 more

Organizations: The Johns Hopkins University, United States · Zhejiang University, China · Zhejiang University-University of Illinois Urbana-Champaign Institute, China · Zhejiang Lab, China · Shaoxing University, China · Angelalign, China · Harvard Medical School, United States

Abstract

Surgical video understanding is essential for computer-assisted interventions, yet existing surgical foundation models remain constrained by limited data scale, procedural diversity, and inconsistent evaluation, often lacking a reproducible training pipeline. We propose SurgRec, a scalable and reproducible pretraining recipe for surgical video understanding, instantiated with two variants: SurgRec-MAE and SurgRec-JEPA. We curate a large multi-source corpus of 10,535 videos and 214.5M frames spanning endoscopy, laparoscopy, cataract, and robotic surgery. Building on this corpus, we develop a unified pretraining pipeline with balanced sampling and standardize a reproducible benchmark across 16 downstream datasets and four clinical domains with consistent data splits. Across extensive comparisons against SSL baselines and vision-language models, SurgRec consistently achieves superior performance across downstream datasets. In contrast, VLMs prove unreliable for fine-grained temporal recognition, exhibiting both performance gaps and sensitivity to prompt phrasing. Our work provides a reproducible, scalable foundation for the community to build more general surgical video models. All code, models, and data will be publicly released.

Figures & tables

Method	Domain Focus	Reported Scale
EndoDINO	GI Endoscopy	100K – 10M images
SurgeNetXL	General Surgery	${>}$ 4.7M frames
SurgBench-P	Surgery (11 specialties)	53M frames
EndoMamba	Endoscopy	75K clips / 11.3M frames
SurgVLP	Surgical Lectures	1,400 videos
SurgVISTA	Surgery ( $>$ 20 procedures)	3.6K videos / 3.55M frames

Table 1: Pretraining data scale comparison across recent medical foundation models.

Figure 1: Overview of the proposed medical video pretraining and evaluation framework.

Dataset	Baselines					Ours
Dataset	DINOv3 Surg	DINOv3 Surg	V-MAE Surg	SR-MAE (w/o bal.)	JEPA Surg	SR MAE	SR JEPA
AIxSuture	32.61	39.13	39.13	39.13	39.13	43.48	39.13
AutoLap.	17.74	19.35	19.35	17.74	20.97	22.58	17.74
Cat-21	15.69	15.69	15.69	15.69	15.69	15.69	15.69
Cat-101	17.97	19.53	19.53	21.09	24.22	35.16	23.44
CAT-1k	9.90	14.85	11.88	16.83	19.80	18.81	17.82

Table 2: Main comparison on Acc@1 (%) against SSL baselines. SR and V-MAE denote SurgRec and VideoMAE, respectively. SR-MAE (w/o bal.) refers to SurgRec-MAE trained without balanced sampling. Dataset names are abbreviated (e.g., Cat: Cataract, Colono: Colonoscopic).

Dataset	Reference	Qwen3- VL-8B			LLaVA- NeXT-7B			Qwen2.5- VL-7B
Dataset	SurgRec -MAE	P1	P2	$\boldsymbol{\Delta}$	P1	P2	$\boldsymbol{\Delta}$	P1	P2	$\boldsymbol{\Delta}$
AIxSuture	43.48	28.26	36.96	+8.70	21.74	19.57	-2.17	36.96	39.13	+2.17
AutoLap.	22.58	19.35	22.58	+3.23	11.29	11.29	+0.00	12.90	14.52	+1.61
Cat-21	15.69	7.84	3.92	-3.92	9.80	11.76	+1.96	13.73	13.73	+0.00
Cat-101	35.16	13.28	11.72	-1.56	8.59	8.59	+0.00	10.94	11.72	+0.78
CAT-1k	18.81	4.95	4.95	+0.00	5.94	3.96	-1.98	5.94	5.94	+0.00

Table 3: VLM Acc@1 (%). SurgRec-MAE is the supervised reference. P1: baseline prompt; P2: stricter output constraint.

\Delta

= P2

-

P1 (points). Dataset names are abbreviated as in Table 2 .

Model	Cataract	Robotic	Endoscopy	Laparoscopy	Overall Macro	Worst Domain
VideoMAE Variants
V-MAE (Baseline)	19.09	42.22	62.12	29.05	38.12	19.09
SR-MAE (w/o bal.)	17.87	22.80	56.73	35.50	33.23	17.87
SR-MAE (Ours)	22.23	47.37	71.49	33.11	43.55	22.23
$\Delta$ vs. Base	+3.14	+5.15	+9.37	+4.06	+5.43	+3.14
$\Delta$ vs. w/o bal.	+4.36	+24.57	+14.76	-2.39	+10.32	+4.36

Table 4: Generalization analysis on the 4-domain benchmark (Acc@1, %). Domain scores are macro-averaged; Overall Macro and Worst Domain denote the mean and minimum across domains, respectively.

Explore similar work

SurgMotion: A Video-Native Foundation Model for Universal Understanding of Surgical Videos
Feb 5, 2026Jinlin Wu, Felix Holm, Chuxi Chen +18Joint-Embedding Predictive Architecture Surgical Video Understanding
Center for Artificial Intelligence and Robotics, Hong Kong Institute of Science and Innovation, Chinese Academy of Sciences, Hong Kong, China · State Key Laboratory of Multimodal Artificial Intelligence Systems, Institute of Automation, Chinese Academy of Sciences, Beijing, China · Computer Aided Medical Procedures, Technical University of Munich, Munich, Germany +7
SurgAtlas: A Large-Scale Surgical Video-Language Dataset with 2,391 Hours of Open and Minimally Invasive Surgery
Jun 24, 2026Filippos Bellos, Andre S. Gala-Garza, Miaowei Wang +8Medical VQA Surgical Video Understanding
University of Michigan Ann Arbor, Michigan, USA · University of Edinburgh Edinburgh, United Kingdom · Vanderbilt University Medical Center Nashville, Tennessee, USA +3
An Approach to Enriching Surgical Video Datasets for Fine-Grained Spatial-Temporal Understanding of Vision-Language Models
Apr 1, 2026Lennart Maack, Alexander SchlaeferMedical VQA Surgical Video Understanding
Hamburg University of Technology, Germany

Feb 5, 2026J/K move · Enter open · S save

SurgMotion: A Video-Native Foundation Model for Universal Understanding of Surgical Videos

Jinlin Wu, Felix Holm, Chuxi Chen, An Wang+17

Center for Artificial Intelligence and Robotics, Hong Kong Institute of Science and Innovation, Chinese Academy of Sciences, Hong Kong, China · State Key Laboratory of Multimodal Artificial Intelligence Systems, Institute of Automation, Chinese Academy of Sciences, Beijing, China · Computer Aided Medical Procedures, Technical University of Munich, Munich, Germany+7

Joint-Embedding Predictive Architecture Surgical Video Understanding Video Representation Learning

Open paper AskPDF arXiv

While foundation models have advanced surgical video analysis, current approaches rely predominantly on pixel-level reconstruction objectives that waste model capacity on low-level visual details, such as smoke, specular reflections, and fluid motion, rather than semantic structures essential for surgical understanding. We present SurgMotion, a video-native foundation model that shifts the learning paradigm from pixel-level reconstruction to latent motion prediction. Built on the Video Joint Embedding Predictive Architecture (V-JEPA), SurgMotion introduces three key technical innovations tailored to surgical videos: (1) motion-guided latent masked prediction to prioritize semantically meaningful regions, (2) spatiotemporal affinity self-distillation to enforce relational consistency, and (3) spatiotemporal feature diversity regularization (SFDR) to prevent representation collapse in texture-sparse surgical scenes. To enable large-scale pretraining, we curate SurgMotion-15M, the largest surgical video dataset to date, comprising 3,658 hours of video from 50 sources across 13 anatomical regions. Extensive experiments across 17 benchmarks demonstrate that SurgMotion significantly outperforms state-of-the-art methods on surgical workflow recognition, achieving 14.6 percent improvement in F1 score on EgoSurgery and 10.3 percent on PitVis; on action triplet recognition with 39.54 percent mAP-IVT on CholecT50; as well as on skill assessment, polyp segmentation, and depth estimation. These results establish SurgMotion as a new standard for universal, motion-oriented surgical video understanding.

Abstract

Explore similar work

SurgMotion: A Video-Native Foundation Model for Universal Understanding of Surgical Videos

SurgAtlas: A Large-Scale Surgical Video-Language Dataset with 2,391 Hours of Open and Minimally Invasive Surgery

An Approach to Enriching Surgical Video Datasets for Fine-Grained Spatial-Temporal Understanding of Vision-Language Models