On the Robustness of Temporal Vision-Language Models for Surgical Endoscopy Videos

Darakshan Rashid; Raza Imam; Ufaq Khan; Muhammad Bilal; Shazad Ashraf; Dwarikanath Mahapatra; Mohammad Yaqub; Muhammad Haris Khan

cs.CVAug 14, 2026

On the Robustness of Temporal Vision-Language Models for Surgical Endoscopy Videos

Authors: Darakshan Rashid, Raza Imam, Ufaq Khan, Muhammad Bilal, Shazad Ashraf, Dwarikanath Mahapatra, Mohammad Yaqub, Muhammad Haris Khan,

Organizations: Mohamed bin Zayed University of Artificial Intelligence (MBZUAI), UAE · Indian Institute of Technology Delhi, India · Birmingham City University, UK · University Hospitals Birmingham, UK ·

Abstract

Temporal vision-language models (TVLMs) offer a reusable, prompt-based interface for surgical video understanding, yet, their robustness under clinically realistic acquisition artifacts in endoscopy remains insufficiently characterized. In practice, degradations such as defocus, haze, motion blur, noise, cautery smoke, and packet loss introduce structured distribution shifts which may compromise video-text alignment. We study the robustness of temporal VLMs under such shifts caused by corruptions in clip frames. We introduce Endo-C6, a compact corruption benchmark of six endoscopy-realistic perturbations evaluated at a fixed high severity, and apply it to public Gastrointestinal (GI) endoscopy and laparoscopic cholecystectomy videos. Under a standardized prompt protocol, we benchmark 3 recent surgical TVLM baselines and analyze robustness in both mean and worst-case settings, spanning 294 dataset-level evaluations. Finally, we present RobustEndoCLIP, obtained by few-shot parameter-efficient tuning with VeRA, outperforming existing TVLM baselines. Our findings show that off-the-shelf TVLMs can exhibit severe worst-case collapse under endoscopy-specific corruptions, whereas lightweight few-shot adaptation can substantially improve corrupted performance and robustness without changing the prompt-based interface. We expect Endo-C6 to support standardized robustness reporting and promote more reliable clinical vision-language systems.

Figures & tables

Panel of Figure 1 — Figure 1: Endo-C6 benchmark overview. (a) Artifact taxonomy across endoscopic domains, organized by domain salience and temporal structure. (b) Dataset statistics across endoscopic domains covered in Endo-C6. Reported train-set counts follow the original dataset splits; unless otherwise stated, we train using only 16% of the listed training data. All evaluations are conducted on the full test splits reported in (b).

Panel of Figure 2 — Figure 2: Endo-C6 corrupted examples. Clean and corrupted frames (severity 5) for the six Endo-C6 corruptions on endoscopy clips of 3 datasets used in our benchmark.

Figure 3: A) Zero-shot inference maps an endoscopy image/frame and text class prompts into a shared embedding space. Prediction is the class whose prompt has the highest cosine similarity. B) Few-shot adaptation uses a small clean labeled subset from three datasets: CholecT50 (surgical phase labels), Kvasir (GI landmark/pathology class labels) and Temset24k (surgical action labels). The FC projection layer is adapted with Vector-based Random Matrix Adaptation (VeRA). Please also refer to Sec. 2.2 . C) t-SNE projections of video embeddings showing how each Endo-C6 corruption shifts the feature space relative to clean clips. D) On average across three corruption datasets of Endo-C6, RobustEndoCLIP outperforms other TVLM baselines by a large margin.

(a) CholecT50 / Laparoscopy (downstream)
Temporal MVLM $\downarrow$	Clean	Defocus	Fog	Shot	Motion	Packet	Smoke	Mean-C	Worst-C	$\boldsymbol{\Delta}$
SurgVLP	40.21	16.15	21.86	11.39	7.38	11.58	7.08	12.57	7.08	0.00
HecVLP	44.41	18.96	32.54	17.09	12.78	17.44	7.30	17.69	7.30	5.12
PeskaVLP	39.23	24.32	37.59	14.89	7.74	13.30	6.91	17.46	6.91	4.89
RobustEndoCLIP	47.06	42.09	42.92	42.66	16.83	46.30	38.91	38.29	16.83	25.72

Table 1: Robustness Analysis of Temporal VLMs on Endo-C6. Accuracy (%) on clean and corrupted test clips across three datasets. Bold denotes Best, while Underline denotes Second-best.

Figure 4: Label budget scaling. We compare the robustness of RobustEndoCLIP on different Endo-C6 datasets when tuning with different labeled fractions of the clean training set. We report Clean, Mean-C, and Worst-C across datasets.

Adaptation	# Params	Label Budget	CholecT50		Kvasir		TEMSET-24K		Average
			Clean	Mean	Clean	Mean	Clean	Mean	Clean	Mean
Zero-shot	0	0	40.21	12.57	13.75	12.38	2.82	3.86	18.93	9.60
LoRA	720896	16%	54.18	18.31	15.92	14.29	29.48	18.17	33.19	16.92
VeRA	1024	16%	47.06	38.29	36.50	22.20	26.67	26.89	36.74	29.13

Table 2: PEFT ablation. Comparison of LoRA vs VeRA (and zero-shot) under a fixed label budget, reported across datasets. For each dataset we report Clean and Mean-C, and we also report the Average across datasets. Best . Second-best .

Figure 5: Corruption stride analysis. (a) Mean-C for temporal VLMs under clean clips and random corruption applied to 50% vs. 100% of clip frames. (b) Per-corruption stride sensitivity, $\Delta\mathrm{Acc}=\mathrm{Acc}(50\%)-\mathrm{Acc}(100\%)$ , across TVLMs (lower is better). (c) Mean-C across tuning configurations (LoRA/VeRA, varying label budgets) under different stride settings. “Ours” and 16/VeRA are same representing RobustEndoCLIP.

VLM Robustness

Vision-language model robustness studies whether models remain reliable under visual, textual, and distribution shifts, adversarial inputs, and safety challenges.

See all 454 papers

VLM-RobustBench: A Comprehensive Benchmark for Robustness of Vision-Language Models
Mar 6, 2026Rohit Saxena, Alessandro Suglia, Pasquale MinerviniVLM Robustness VLM Evaluation
University of Edinburgh, UK
An Approach to Enriching Surgical Video Datasets for Fine-Grained Spatial-Temporal Understanding of Vision-Language Models
Apr 1, 2026Lennart Maack, Alexander SchlaeferMedical VQA Surgical Video Understanding
Hamburg University of Technology, Germany
SS-TPT: Stability and Suitability-Guided Test-Time Prompt Tuning for Adversarially Robust Vision-Language Models
Jun 5, 2026Sunoh Kim, Daeho UmVLM Robustness VLM Adaptation
Dankook University, Yongin, South Korea · University of Seoul, Seoul, South Korea

Mar 6, 2026J/K move · Enter open · S save

VLM-RobustBench: A Comprehensive Benchmark for Robustness of Vision-Language Models

Rohit Saxena, Alessandro Suglia, Pasquale Minervini

University of Edinburgh, UK

VLM Robustness VLM Evaluation

Open paper AskPDF arXiv

Vision-language models (VLMs) achieve strong performance on standard, high-quality datasets, but we still do not fully understand how they perform under real-world image distortions. We present VLM-RobustBench, a benchmark spanning 49 augmentation types across noise, blur, weather, digital, and geometric perturbations, evaluated under graded severities (low/mid/high) and binary transforms, yielding 133 corrupted settings. We evaluate VLMs from four families (Qwen, InternVL, Molmo, Gemma) on two complementary benchmarks: MMBench (visually grounded) and MMMU-Pro (reasoning-oriented). Our results reveal that visual severity is a weak predictor of difficulty: low-severity spatial perturbations often degrade performance more than visually severe photometric corruptions. In particular, low-severity glass_blur reduces MMBench accuracy by about 8 pp on average across models, while the largest drops arise from resampling and geometric distortions (e.g., upsample, elastic_transform), reaching up to 34 pp. Overall, our findings suggest current VLMs are semantically strong but spatially fragile, motivating the definition of novel robustness evaluation protocols and training regimes that emphasize resampling and geometric invariances.

Abstract

VLM Robustness

Explore similar work

VLM-RobustBench: A Comprehensive Benchmark for Robustness of Vision-Language Models

An Approach to Enriching Surgical Video Datasets for Fine-Grained Spatial-Temporal Understanding of Vision-Language Models

SS-TPT: Stability and Suitability-Guided Test-Time Prompt Tuning for Adversarially Robust Vision-Language Models