CliPPER: Contextual Video-Language Pretraining on Long-form Intraoperative Surgical Procedures for Event Recognition

Florian Stilz; Vinkle Srivastav; Nassir Navab; Nicolas Padoy

cs.CVMar 25, 2026

CliPPER: Contextual Video-Language Pretraining on Long-form Intraoperative Surgical Procedures for Event Recognition

Authors: Florian Stilz, Vinkle Srivastav, Nassir Navab, Nicolas Padoy

Organizations: University of Strasbourg, CNRS, INSERM, ICube, UMR7357, France · IHU Strasbourg, France · Technical University of Munich, Germany

Abstract

Video-language foundation models have proven to be highly effective in zero-shot applications across a wide range of tasks. A particularly challenging area is the intraoperative surgical procedure domain, where labeled data is scarce, and precise temporal understanding is often required for complex downstream tasks. To address this challenge, we introduce CliPPER (Contextual Video-Language Pretraining on Long-form Intraoperative Surgical Procedures for Event Recognition), a novel video-language pretraining framework trained on surgical lecture videos. Our method is designed for fine-grained temporal video-text recognition and introduces several novel pretraining strategies to improve multimodal alignment in long-form surgical videos. Specifically, we propose Contextual Video-Text Contrastive Learning (VTC_CTX) and Clip Order Prediction (COP) pretraining objectives, both of which leverage temporal and contextual dependencies to enhance local video understanding. In addition, we incorporate a Cycle-Consistency Alignment over video-text matches within the same surgical video to enforce bidirectional consistency and improve overall representation coherence. Moreover, we introduce a more refined alignment loss, Frame-Text Matching (FTM), to improve the alignment between video frames and text. As a result, our model establishes a new state-of-the-art across multiple public surgical benchmarks, including zero-shot recognition of phases, steps, instruments, and triplets. The source code and pretraining captions can be found at https://github.com/CAMMA-public/CliPPER.

Figures & tables

Figure 2: CliPPER overview: CliPPER processes multiple clips from the same video independently through modality-specific encoders with a Video Encoder for visual input and a Text Encoder for corresponding captions. The resulting pooled frame-level and text CLS embeddings are then passed through context encoders, which operate across clips from the same video to generate context-aware embeddings for each modality independently. On the Dual Encoder representations, we apply the standard video-text contrastive loss ( $\mathcal{L_{VTC}}$ ). The context-aware representations are optimized through $\mathbf{VTC_{CTX}}$ , a context-aware contrastive objective, together with a Cycle-Consistency Alignment loss that enforces bidirectional alignment consistency. In parallel, the model fuses all frame and text embeddings across clips using a Multi-Modal Encoder, enabling an additional objective: $\mathbf{FTM}$ , which guides the model toward fine-grained alignment by learning to localize which frames from multiple clips from the same video match a given text. Lastly, we fuse the visual and textual contextual embeddings in a separate step as well by applying a single Cross-Attention layer. On top of these fused embeddings, we independently predict the temporal order of the elements called COP , enabling the model to reason about sequence information.

Model	Cholec80	AutoLaparo	StrasBypass70	BernBypass70	GraSP	Average
VindLU [ 14 ]	$8.2$	$7.7$	$2.5$	$2.6$	$2.8$	$4.7$
SurgVLP [ 76 ]	$23.3$	$10.8$	$14.1$	$7.8$	$7.6$	$12.7$
HecVL [ 74 ]	$22.5$	$19.3$	$21.8$	$15.4$	$6.0$	$17.0$
PeskaVLP [ 75 ]	$30.5$	$25.4$	$26.5$	$19.2$	$7.7$	$21.8$
VindLU-SVL	$29.3$	$17.9$	$31.5$	$18.3$	$14.9$	$22.4$
Ours-SVL	$30.6$	$31.8$	$\underline{34.8}$	$\underline{21.7}$	$16.9$	$27.2$

Table 1: Zero-shot surgical phase recognition results: We report on F1-Score average across

4

different prompts. Our approach outperforms other methods significantly across all datasets. VindLU baseline uses the original pretrained weights from its source repository and VindLU-SVL is trained on the same narrations as Ours-SVL. Ours-YT is on the public YouTube narrations, and Ours-All specifies both SVL and YouTube videos combined. All other baselines are trained on the SVL as specified in the corresponding works.

Model	StrasBypass70	BernBypass70	GraSP	Average
VindLU [ 14 ]	$0.3$	$0.3$	$0.7$	$0.4$
SurgVLP [ 76 ]	$4.5$	$3.1$	$7.6$	$5.0$
HecVL [ 74 ]	$6.0$	$5.0$	$6.0$	$5.7$
PeskaVLP [ 75 ]	$5.2$	$4.1$	$7.7$	$5.7$
VindLU-SVL	$11.7$	$\underline{7.6}$	$6.3$	$8.7$
Ours-SVL	$\underline{13.3}$	$6.9$	$6.8$	$9.0$

Table 2: Zero-shot surgical step recognition results: We report on F1-score averaged across

4

different prompts. VindLU uses the original pretrained weights from the VindLU source repository and VindLU-SVL is trained on the same text and videos as Ours-SVL. Ours-YT is on the public YouTube narrations, and Ours-All specifies both SVL and YouTube videos combined. All other baselines are trained on the SVL dataset.

	CholecT50		ProstaTD		Average
Model	$ivt_{mAP}$	$i_{mAP}$	$ivt_{mAP}$	$i_{mAP}$	$ivt_{mAP}$	$i_{mAP}$
VindLU [ 14 ]	$3.0$	$26.4$	$3.4$	$36.7$	$3.2$	$31.5$
SurgVLP [ 76 ]	$3.7$	$27.5$	$3.9$	$39.1$	$3.8$	$33.3$
HecVL [ 74 ]	$3.5$	$27.9$	$4.2$	$41.1$	$3.9$	$34.5$
PeskaVLP [ 75 ]	$4.3$	$32.2$	$4.3$	$42.6$	$4.3$	$37.4$
VindLU-SVL	$5.0$	$36.3$	$4.4$	$40.4$	$4.7$	$38.4$

Table 3: Zero-shot surgical triplet ( ivt ) and instrument recognition results ( i ). We report the mean Average Precision (mAP) averaged across four different prompts. Our approach consistently outperforms prior methods on both datasets.

COP	CTX	FTM	CYC	VE	Cholec80	AutoLaparo	StrasBypass70	BernBypass70	GraSP	Average
$\times$	$\times$	$\times$	$\times$	$\times$	$29.4$	$18.0$	$30.4$	$17.4$	$14.9$	$22.4$
$\times$	$\checkmark$	$\checkmark$	$\checkmark$	$\checkmark$	$\underline{31.4}$	$26.5$	$33.9$	$\underline{21.6}$	$15.6$	$25.8$
$\checkmark$	$\checkmark$	$\times$	$\checkmark$	$\checkmark$	$27.9$	$\underline{30.5}$	$\underline{34.2}$	$20.1$	$\underline{16.2}$	$25.8$
$\checkmark$	$\checkmark$	$\checkmark$	$\times$	$\checkmark$	$\mathbf{32.3}$	$29.7$	$34.1$	$21.2$	$\underline{16.8}$	$\underline{26.8}$
$\checkmark$	$\checkmark$	$\checkmark$	$\checkmark$	$\checkmark$	$30.6$	$\mathbf{31.8}$	$\mathbf{34.8}$	$\mathbf{21.7}$	$\mathbf{16.9}$	$\mathbf{27.2}$
$\times$	$\times$	$\times$	$\times$	$\times$	$18.3$	$16.7$	$30.1$	$16.2$	$10.4$	$18.3$

Table 4: Ablation studies on phase recognition: We evaluate the performance of various model configurations using the F1-score. We compute the F1-score for each video and compute the average F1-score across all videos, then further average over four distinct text prompts. When the context-aware contrastive objective (

VTC_{CTX}

here in short CTX) is not used, the model defaults to using only the base frame (

v_{i,f}

) and text (

t_{i}

) embeddings for alignment. When the Video Encoder (VE) is not selected, then the ResNet50 image encoder is used.

Figure 3: Zero-shot Phase Recognition at varying temporal windows: The final plot shows the average performance across all datasets. The highlighted values indicate the improvement over the strongest baseline for both Ours-SVL and Ours (SVL + YouTube) .

Figure 4: Zero-shot Step Recognition at varying temporal windows: The final plot shows the average performance across all datasets. The highlighted values indicate the improvement over the strongest baseline for both Ours-SVL and Ours (SVL + YouTube) .

Model	Cholec80	AutoLaparo	StrasBypass70	BernBypass70	GraSP	Average
w/o progressive sampling	31.0	21.9	33.1	20.3	15.8	24.4
only Whisper	28.1	32.4	29.0	19.1	13.5	24.4
only GPT	32.5	30.8	31.9	19.8	14.6	25.9
Ours-SVL	30.6	31.8	34.8	21.7	16.9	27.2

Table 5: Sampling Ablations: This table highlights the importance of progressive sampling and the benefit of combining Whisper- and GPT-generated captions during pretraining. All pretraining experiments are conducted on the SVL dataset.

Surgical Video Understanding

Surgical video understanding uses computer vision and multimodal AI to interpret procedures, instruments, actions, and events over time.

See all 118 papers

Scaling Video Pretraining for Surgical Foundation Models
Mar 31, 2026Sicheng Lu, Zikai Xiao, Jianhui Wei +7Surgical Video Understanding Video Representation Learning
The Johns Hopkins University, United States · Zhejiang University, China · Zhejiang University-University of Illinois Urbana-Champaign Institute, China +4
HyperVLP: Enhancing Hierarchical Surgical Video-Language Pre-training in Hyperbolic Space
Jun 30, 2026Yaojun Hu, Kun Yuan, Nassir Navab +3Hierarchical Representation Learning Surgical Video Understanding
Zhejiang University, Hangzhou, China · University of Strasbourg, CNRS, INSERM, ICube, UMR7357, Strasbourg, France · IHU Strasbourg, Strasbourg, France +1
SurgNarrator: A Generative Retrieval Framework for Surgical Video Understanding
Aug 5, 2026Yuqing Feng, Jiawei Ma, Kevin Qinghong Lin +6Surgical Video Understanding Generative Retrieval
Department of Computer Science, University of Liverpool, Liverpool L69 3BX, U.K. · Department of Computer Science, City University of Hong Kong, Hong Kong SAR, China · Department of Engineering, University of Oxford, Oxford OX1 3PJ, U.K. +2

Mar 31, 2026J/K move · Enter open · S save

Scaling Video Pretraining for Surgical Foundation Models

Sicheng Lu, Zikai Xiao, Jianhui Wei, Danyu Sun+6

The Johns Hopkins University, United States · Zhejiang University, China · Zhejiang University-University of Illinois Urbana-Champaign Institute, China+4

Surgical Video Understanding Video Representation Learning

Open paper AskPDF arXiv

Surgical video understanding is essential for computer-assisted interventions, yet existing surgical foundation models remain constrained by limited data scale, procedural diversity, and inconsistent evaluation, often lacking a reproducible training pipeline. We propose SurgRec, a scalable and reproducible pretraining recipe for surgical video understanding, instantiated with two variants: SurgRec-MAE and SurgRec-JEPA. We curate a large multi-source corpus of 10,535 videos and 214.5M frames spanning endoscopy, laparoscopy, cataract, and robotic surgery. Building on this corpus, we develop a unified pretraining pipeline with balanced sampling and standardize a reproducible benchmark across 16 downstream datasets and four clinical domains with consistent data splits. Across extensive comparisons against SSL baselines and vision-language models, SurgRec consistently achieves superior performance across downstream datasets. In contrast, VLMs prove unreliable for fine-grained temporal recognition, exhibiting both performance gaps and sensitivity to prompt phrasing. Our work provides a reproducible, scalable foundation for the community to build more general surgical video models. All code, models, and data will be publicly released.

Abstract

Surgical Video Understanding

Explore similar work

Scaling Video Pretraining for Surgical Foundation Models

HyperVLP: Enhancing Hierarchical Surgical Video-Language Pre-training in Hyperbolic Space

SurgNarrator: A Generative Retrieval Framework for Surgical Video Understanding