LapaFM: A Laparoscopic Foundation Model via Multi-Stage Continual Learning

Xiaobo Zhu; Shiwei Wu; Yu An; Yuechen Tao; He Sun; Song Zhang; Jianbo Wang; Xiaojing Wang

cs.CVOct 23, 2026

LapaFM: A Laparoscopic Foundation Model via Multi-Stage Continual Learning

Authors: Xiaobo Zhu, Shiwei Wu, Yu An, Yuechen Tao, He Sun, Song Zhang, Jianbo Wang, Xiaojing Wang,

Organizations: the CAS Key Laboratory of Molecular Imaging and Beijing Key Laboratory of Molecular Imaging, Institute of Automation, Chinese Academy of Sciences, Beijing 100190, China · the School of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing 100049, China · School of Computer Science and Engineering, Nanjing University of Science and Technology, Nanjing, Jiangsu Province, 210094, China · School of Engineering Medicine, Beihang University, Beijing, 100191, China · Key Laboratory of Big Data-Based Precision Medicine (Beihang University), Ministry of Industry and Information Technology of China, Beijing, 100191, China · Department of Gastroenterology, Lishui Central Hospital, The Fifth Affiliated Hospital of Wenzhou Medical University, Lishui, Zhejiang Province, 323020, China · Heping Hospital Affiliated to Changzhi Medical College, Changzhi, Shanxi Province, 425599, China · Shanghai Jiao Tong University School of Medicine, Ruijin Hospital, Department of General Surgery, Shanghai, 200025, China · Department of Gastrointestinal Surgery, Clinical Medical College and The First Affiliated Hospital of Chengdu Medical College, Chengdu, Sichuan Province, 610500, China · The Fourth Hospital of Hebei Medical University, Shijiazhuang, Hebei Province, 050011, China · Department of Gastrointestinal Surgery, the Third Affiliated Hospital of Sun Yat-sen University, Guangzhou 510630, China · the Department of Colorectal Surgery, National Cancer Center/National Clinical Research Center for Cancer/Cancer Hospital, Chinese Academy of Medical Sciences and Peking Union Medical College, Beijing, 100021, China · State Key Laboratory for Eye Health, Eye Hospital, Wenzhou Medical University, Wenzhou, Zhejiang Province, 325027, China

Abstract

Artificial intelligence-based analysis of intraoperative laparoscopic videos has the potential to improve surgical decision-making and precision. However, effective surgical assistance requires models that can generalize across diverse intraoperative tasks. Here we present LapaFM, a foundation model for laparoscopic surgery trained on more than 20,000 minutes of surgical videos using a multi-stage continual learning strategy that mitigates catastrophic forgetting and inter-task interference. LapaFM learns multimodal visual representations and supports multiple tasks, including smoke removal, surgical phase recognition, instrument and polyp segmentation, and lesion identification. The model was evaluated on 13 retrospective and two prospective cohorts collected from different imaging devices. LapaFM consistently outperformed task-specific models and existing foundation models across datasets, and also generalized to gastrointestinal endoscopy tasks, including polyp detection and classification. Notably, it recovered over 90% of smoke-obscured frames and achieved clinician-level or superior performance in lymph node metastasis prediction and prospective polyp classification.

Figures & tables

Fig. 1: A multi-stage continual learning strategy is employed during pretraining. In Stage I, the model learns white-light imaging (WLI) representations. In Stage II, pseudo-color imaging (PCI) features are introduced while reviewing white-light features via a frozen teacher network to mitigate modality conflict and catastrophic forgetting.

Fig. 2: Overview of the proposed smoke removal model. During training, the image reconstruction branch refines the output of the smoke removal branch by leveraging semantic priors extracted from the ground truth via LapaFM. A semantic embedding module is employed to guide high-quality reconstruction. During inference, only the smoke removal branch is retained, while the pretrained prior knowledge from LapaFM is transferred under the supervision of the distillation loss $\mathcal{L}_{\mathrm{distill}}$ .

Model	DS_Synthetic			DS_Real
Model	SSIM $\uparrow$	PSNR $\uparrow$	CIEDE-2000 $\downarrow$	SSIM $\uparrow$	PSNR $\uparrow$	CIEDE-2000 $\downarrow$
AECR-Net (CVPR’21) [ 15 ]	0.949	27.487	3.696	0.870	25.751	4.859
DS-CycleGAN (TCBB’22) [ 16 ]	0.792	20.412	9.495	0.766	21.927	7.688
Cyclic-DeGAN (CBM’20) [ 17 ]	0.865	21.280	7.525	0.833	22.397	7.173
DEA-Net (TIP’24) [ 18 ]	0.964	29.706	2.833	0.846	25.289	5.029
MB-Taylor-B V2 (TPAMI’25) [ 19 ]	0.969	29.551	2.658	0.871	25.883	4.853

TABLE I: Quantitative comparison of LapaFM with seven state-of-the-art desmoking/dehazing models or general image restoration models—AECR-Net, Ds-CycleGAN, Cyclic-DeGAN,DEA-Net, MB-Taylor-B V2, SGDN, and MambaIRv2-S—on both synthetic and real laparoscopic datasets. LapaFM achieves superior performance across all metrics

Fig. 3: Laparoscopic smoke removal visualization. Qualitative results demonstrate that LapaFM effectively removes surgical smoke while preserving anatomical details, outperforming other methods on both synthetic and real data.

Fig. 4: Performance validation for surgical stage recognition task. a. Performance comparison of models on the AutoLaparo dataset for surgical phase recognition task. b. Performance comparison of models on the Cholec80 dataset for surgical phase recognition task.

Model	IOU	DICE	IOU	DICE
	Endovis2017		Endovis2018
SSIS-Seg (TMI’22) [ 31 ]	0.867	0.926	0.613	0.661
Active_learning (MIA’24) [ 32 ]	0.736	0.846	0.705	0.816
MCF-SMSIS (CBM’24) [ 33 ]	0.722	0.831	0.663	0.794
SurgicalSAM (AAAI’24) [ 34 ]	0.756	0.857	0.699	0.820
MedSAM (NC’24) [ 35 ]	0.749	0.851	0.694	0.812

TABLE II: Quantitative results on surgical instrument segmentation show that LapaFM outperforms three task-specific models and two general-purpose segmentation models, achieving the highest IoU and Dice scores.

Fig. 5: Visualization of surgical instrument segmentation results. Qualitative results on instrument segmentation demonstrate that LapaFM produces more accurate and coherent edge predictions, supporting enhanced intraoperative navigation.

Fig. 6: Comparison of results for the task of predicting lymph node metastasis in colorectal cancer. a. Performance comparison of LapaFM on the validation set for the colorectal cancer lymph node metastasis prediction task. b. Performance comparison of LapaFM on the test set for the colorectal cancer lymph node metastasis prediction task.

Pretraining Strategy	Validation Set	Test Set
Pretraining Strategy	AUC	AUC
Joint Pretraining	0.783	0.722
Ours	0.797	0. 817

TABLE III: Comparison between multi-stage continuous learning and joint training for lymph node metastasis prediction, demonstrating improved multimodal modeling with reduced modality conflict and knowledge loss.

Model	IOU	DICE	IOU	DICE	IOU	DICE	IOU	DICE
	CVC-ClinicDB		CVC-ColonDB		ETIS		EndoScene
Polyper (AAAI’2024) [ 40 ]	0.889	0.942	0.668	0.756	0.785	0.865	0.867	0.924
CFANet (PR’2023) [ 41 ]	0.883	0.933	0.665	0.743	0.655	0.732	0.827	0.893
CoInNet (TMI’2023) [ 42 ]	0.887	0.930	0.729	0.797	0.690	0.759	0.863	0.909
BUNet (NN’2023) [ 43 ]	0.890	0.935	0.731	0.813	0.721	0.798	0.834	0.900
SurgicalSAM (AAAI’2024) [ 34 ]	0.505	0.644	0.330	0.460	0.238	0.342	0.472	0.623

TABLE IV: Quantitative results on surgical instrument segmentation show that LapaFM outperforms three task-specific models and two general-purpose segmentation models, achieving the highest IoU and Dice scores.

Model	Polyp Detection		Polyp Classification
	CH-CHILD		WCE-CCD validation set		WCE-CCD test set
	ACC	AUC	ACC	F1	ACC	F1
Swin-Transformer (ICCV’21) [ 36 ]	0.956	0.992	0.827	0.821	0.831	0.828
ConvNeXt (CVPR’22) [ 37 ]	0.979	0.992	0.761	0.741	0.795	0.780
Efficientnetv2 (ICML’21) [ 38 ]	0.961	0.994	0.772	0.766	0.792	0.782
GastroNet_5M (GAS’25) [ 30 ]	0.980	0.996	0.941	0.932	0.957	0.935

TABLE V: Comparison of LapaFM performance on polyp detection and classification across the CP-CHILD and WCE-CCD datasets.

Model	Propolyp-cls		Propolyp-detection
Model	ACC	AUC	ACC	AUC
Swin-Transformer(ICCV’21) [ 36 ]	0.646	0.535	0.554	0.523
ConvNeXt (CVPR’22) [ 37 ]	0.667	0.562	0.593	0.529
Efficientnetv2(ICML’21) [ 38 ]	0.671	0.576	0.619	0.645
GastroNet_5M (GAS’25) [ 30 ]	0.687	0.594	0.796	0.81
Dinov3 (Arxiv’25) [ 39 ]	0.675	0.582	0.787	0.779

TABLE VI: Performance comparison of LapaFM on prospective polyp classification and detection tasks across Propolyp-cls and Propolyp-detection dataset.

Fig. 7: Comparison between LapaFM and clinical experts. a. Lymph node metastasis prediction across five gastroenterologists with varying experience levels. b. Polyp classification across six endoscopy experts with varying experience levels.

Fig. 8: Robustness of LapaFM under limited supervision. Performance on instrument segmentation and polyp detection with 10%, 30%, and 50% training data demonstrates superior robustness compared to competing models.

Fig. 9: Effect of surgical smoke on downstream tasks and full-process surgical workflow support. a. Surgical smoke severely degrades segmentation and classification performance by obscuring the visual field. After applying LapaFM-based smoke removal, performance is significantly improved and approaches that of clear images, with no significant difference(ns, $P>0.05$ ; $***$ , $P<0.001$ ), indicating effective recovery of critical visual information. b. LapaFM supports the full laparoscopic workflow, from intraoperative smoke removal to downstream lesion classification.

Surgical Video Understanding

Surgical video understanding uses computer vision and multimodal AI to interpret procedures, instruments, actions, and events over time.

See all 118 papers

SurgMotion: A Video-Native Foundation Model for Universal Understanding of Surgical Videos
Feb 5, 2026Jinlin Wu, Felix Holm, Chuxi Chen +18Joint-Embedding Predictive Architecture Surgical Video Understanding
Center for Artificial Intelligence and Robotics, Hong Kong Institute of Science and Innovation, Chinese Academy of Sciences, Hong Kong, China · State Key Laboratory of Multimodal Artificial Intelligence Systems, Institute of Automation, Chinese Academy of Sciences, Beijing, China · Computer Aided Medical Procedures, Technical University of Munich, Munich, Germany +7
Scaling Video Pretraining for Surgical Foundation Models
Mar 31, 2026Sicheng Lu, Zikai Xiao, Jianhui Wei +7Surgical Video Understanding Video Representation Learning
The Johns Hopkins University, United States · Zhejiang University, China · Zhejiang University-University of Illinois Urbana-Champaign Institute, China +4
SurgiATM: A Physics-Guided Plug-and-Play Model for Deep Learning-Based Smoke Removal in Laparoscopic Surgery
Nov 7, 2025Mingyu Sheng, Jianan Fan, Dongnan Liu +3Physics-Guided Image Restoration Image Restoration
School of Computer Science, The University of Sydney, Sydney, NSW, Australia · Institute of Medical Robotics, School of Biomedical Engineering, Shanghai Jiao Tong University, Shanghai, China · Department of Radiology, Brigham and Women’s Hospital, and Harvard Medical School, Boston, MA, United States

Feb 5, 2026J/K move · Enter open · S save

SurgMotion: A Video-Native Foundation Model for Universal Understanding of Surgical Videos

Jinlin Wu, Felix Holm, Chuxi Chen, An Wang+17

Center for Artificial Intelligence and Robotics, Hong Kong Institute of Science and Innovation, Chinese Academy of Sciences, Hong Kong, China · State Key Laboratory of Multimodal Artificial Intelligence Systems, Institute of Automation, Chinese Academy of Sciences, Beijing, China · Computer Aided Medical Procedures, Technical University of Munich, Munich, Germany+7

Joint-Embedding Predictive Architecture Surgical Video Understanding Video Representation Learning

Open paper AskPDF arXiv

While foundation models have advanced surgical video analysis, current approaches rely predominantly on pixel-level reconstruction objectives that waste model capacity on low-level visual details, such as smoke, specular reflections, and fluid motion, rather than semantic structures essential for surgical understanding. We present SurgMotion, a video-native foundation model that shifts the learning paradigm from pixel-level reconstruction to latent motion prediction. Built on the Video Joint Embedding Predictive Architecture (V-JEPA), SurgMotion introduces three key technical innovations tailored to surgical videos: (1) motion-guided latent masked prediction to prioritize semantically meaningful regions, (2) spatiotemporal affinity self-distillation to enforce relational consistency, and (3) spatiotemporal feature diversity regularization (SFDR) to prevent representation collapse in texture-sparse surgical scenes. To enable large-scale pretraining, we curate SurgMotion-15M, the largest surgical video dataset to date, comprising 3,658 hours of video from 50 sources across 13 anatomical regions. Extensive experiments across 17 benchmarks demonstrate that SurgMotion significantly outperforms state-of-the-art methods on surgical workflow recognition, achieving 14.6 percent improvement in F1 score on EgoSurgery and 10.3 percent on PitVis; on action triplet recognition with 39.54 percent mAP-IVT on CholecT50; as well as on skill assessment, polyp segmentation, and depth estimation. These results establish SurgMotion as a new standard for universal, motion-oriented surgical video understanding.

Abstract

Surgical Video Understanding

Explore similar work

SurgMotion: A Video-Native Foundation Model for Universal Understanding of Surgical Videos

Scaling Video Pretraining for Surgical Foundation Models

SurgiATM: A Physics-Guided Plug-and-Play Model for Deep Learning-Based Smoke Removal in Laparoscopic Surgery