EndoComparer: Comparative Surgical Video Question Answering with Pseudo-Siamese Visual Adaptation and Perceive-then-Compare Reasoning
Authors: Shi Li, Lalith Sharan, Joël L. Lavanchy, Kun Yuan, Lorenzo Arboit, Nabani Banik, Elif Yazici, Maria Vannucci, Vinkle Srivastav, Pietro Mascagni, Nicolas Padoy
Organizations: ICube, UMR7357, University of Strasbourg, CNRS, INSERM, Strasbourg, 67000, France. · Institute of Image-Guided Surgery, IHU Strasbourg, Strasbourg, 67000, France. · University Digestive Health Care Center – Clarunis, Basel, 4002, Switzerland. · Department of Biomedical Engineering, University of Basel, Allschwil, 4123, Switzerland. · General Surgery Department, University of Turin, Turin, Italy. · Indian Institute of Technology (IIT) Madras, Chennai, India. · Fondazione Policlinico Universitario Agostino Gemelli IRCCS, Rome, Italy.
Purpose: Surgical training benefits from explicit comparison between cases: in patient condition, in technique and in skill proficiency. Surgical vision-language models (VLMs) offer a promising and flexible interface for this purpose. Yet current VLMs typically see one video at a time, and they often fall short when asked to compare videos. Methods: We propose EndoComparer, a VLM for comparative surgical video question answering with two components. (a) A Pseudo-Siamese Visual LoRA (PSVL) gives each video slot its own low-rank adapter over a shared frozen vision encoder, separating the two videos in the visual pathway at the cost of a single encoder pass. (b) A Perceive-then-Compare chain of thought (PtC-CoT) trains the model to first state what each video shows, then derive the answer from these statements. We further curate EndoDiffVQA, a benchmark of nearly 11K multiple-choice questions on video-pair over cholecystectomy and gastric bypass procedures, covering anatomy, tool, action and skill. Results: EndoComparer outperforms strong generalist and surgical VLMs on detecting and specifying difference between videos. Ablations show that two proposed components contribute complimentarily, and the model remains more robust under a video-swap control that exposes confusion between the two videos. Conclusion: Providing a dedicated visual reasoning path through PSVL and PtC-CoT makes cross-case comparison of surgical videos more robust for VLMs. EndoDiffVQA and EndoComparer provide a benchmark and a strong baseline for this task. Project page: https://camma-public.github.io/endocomparer
Figures & tables
Figure 1: An overview of EndoDiffVQA construction pipline.
Figure 2: An overview of EndoComparer model architecture.
4-subset macro
Strict per subset
Model
Detect
Specify
Strict
Chole- Cap
LRYGB- Cap
Chole- Skill
LRYGB- Skill
Zero-shot
Qwen3.5-9B ( Qwen Team, 2026a )
49.8 ±.5
57.3 ±.4
35.4 ±.9
41.3 ±.3
34.4 ±.7
31.6 ±.6
34.3 ±.3
Qwen3.8-27B ( Qwen Team, 2026b )
52.8 ±.7
58.9 † ±.2
35.2 ±.6
38.7 ±.3
33.8 ±.3
44.7 ±.6
23.6 ±1.0
InternVL3.5-8B ( Wang et al., 2025c )
49.7 ±.3
54.5 † ±.4
30.6 ±.8
44.7 ±1.1
25.5 ±.8
28.6 ±.6
23.6 ±1.2
InternVL3.5-30B ( Wang et al., 2025c )
56.6 ±.2
57.1 ±1.1
37.2 ±.5
44.5 ±.3
32.0 ±.3
36.3 ±.5
36.1 ±1.0
Table 1: Results on the EndoDiffVQA test split (2,300 items, %). Detect, Specify and Strict are 4-subset macros (see Metrics); the last four columns give Strict per subset. † : Specify macro over fewer than four subsets, because the model called fewer than 20 items Different in the others; not highlighted. Best per group in bold; the fine-tuned group includes ours.
Zero-shot
Fine-tuned
Qwen3.8 27B
InternVL 3.5-30B
Surg- VidLM
InternVL 3.5-8B
Surg- VidLM
Surg- TEMP
Surg- ViVQA
Qwen3.5 9B
Endo- Comparer
Full
35.2
37.2
32.1
47.0
45.1
46.8
32.6
48.3
54.0
Swap
33.1
34.5
27.8
41.6
38.3
39.2
27.4
42.5
53.6
Δ
− 2.1
− 2.7
− 4.3
− 5.4
− 6.8
− 7.6
− 5.2
− 5.8
−0.6
Table 2: Robustness to video order: Strict on the EndoDiffVQA test split (%, 4-subset macro). Full is the release order; Swap exchanges Video 1 and Video 2 and scores against the swapped answer. Δ is Swap − Full.
Table S1: Slot schema of the per-clip record: one anatomy entry per structure, one tools entry per instrument instance, one actions entry per instrument–tissue interaction, and one goal . Region: one of left, right, top, bottom, center, top-left, top-right, bottom-left, bottom-right, background, across the view . Slots that are extracted but feed no released template (texture, shape, exposure, movement direction) are omitted.
We can see a blue-green structure with thin red blood vessels on the top-left side of the camera, which is a porcelain gallbladder. The proximal side of the gallbladder and its peritoneum are dissected […] The grasper 2 is seen on the left side of the camera. […] (Not seen on this clip because of the camera’s view: The grasper 1 is grasping the distal side of the gallbladder.) The grasper 2 is retracting the proximal side of the gallbladder to the left side of the camera. […] The surgeon wants to dissect the gallbladder/cystic pedicle.
Table S2: Slot extraction on an abbreviated CholecT50 caption (one entry per record shown). The numbered graspers become two grasper entries, and the grasp narrated inside “(Not seen …)” is kept as an unseen action.
Step
Rule
Normalisation
Synonym folding
Surface variants map to one value: pinkish → pink , upper left → top-left , clip applier → clipper , gall bladder → gallbladder , bleeds → bleeding ; articles and punctuation are stripped.
Tool numbering
“grasper 2” → grasper ; any digit left in a tool name fails validation.
Object nouns
Specimen bag, trocar, compress, suture thread and needle filed under anatomy are moved to tools, and are never engaged and never an action’s actor.
Entity families
Names one structure carries across the two corpora ( omentum, mesentery, fat ) are kept but grouped, so absence (A1) must hold for the whole family.
Duplicates
Repeated entries are merged; slots of a structure marked not visible are cleared.
Table S3: Deterministic post-processing of the extracted slots.
ID
Fact per clip
Stem
Rule
Anatomy
A1
structure visible?
Is the {entity} visible in both videos?
same / only k / neither from the two flags; neither : a structure of the phase that neither caption mentions.
A2
colour of the structure
Is the {entity} the same colour in both videos?
same if equal, different if no colour word is shared, only k if absent from the other clip.
A3
region of the view the structure is in
Is the {entity} in the same part of the view in both videos?
As A2 over regions.
A4
state of the structure (bleeding, dissected, clipped, …)
Is the {entity} {state} in both videos?
As A1 per (structure, state); neither : a state the structure shows elsewhere in the source.
Tool
Table S4: The 13 question templates. Fact: what is read from each clip’s record; Rule: how the two facts give the relation of Eq. (1) of the main paper, a pair that fits no case being skipped. Throughout, only k requires the subject to be safely absent from the other clip (no same-organ alias such as omentum / mesentery / fat, no mention in its caption); overlapping values (“left” vs. “top-left”, “yellow” vs. “yellow-pink”) are never keyed as different ; specimen bag, trocar and other objects never count as instruments. C1 action kinds: manipulate (grasp, hold, retract, lift, push, pull, expose), divide (cut, divide), energy (coagulate, cauterize), clean (irrigate, suction, clean), and dissect, clip, staple, suture each on their own.
Rd.
Error
Example
Fix (hard check)
1
Object as actor
A specimen bag or a trocar counted as the instrument engaging tissue (T3, T4).
Object nouns never act; they are excluded as instruments.
1
Release read as engagement
“The grasper releases the gallbladder” marked the grasper as engaged with it (T4).
A tool that the caption says is releasing the target is not engaged.
2
True distractor
A distractor interaction or goal that one of the two clips actually shows (C1, C3).
No distractor may be true of either video.
2
Overlapping values as different
left vs. upper left keyed as a different position or entry side (A3, T2).
Overlapping values are never different , nor used as a distractor.
2
Value from a negation
“no unified tubular structure” turned into a tubular cystic duct (shape).
Values in negated clauses are dropped.
3
Absence from silence
“Is the cystic duct tubular in both videos?” answered “only in Video 1” because the other caption does not mention the shape.
Template disabled: shape does not change between clips.
Table S5: Representative errors found during the audit rounds and how each was fixed. Rd.: audit round in which the error was found. A hard check re-runs every fix on each build, and the build fails if any item is flagged.
Figure S1: Items per category and attribute in EndoDiffVQA.
Annotation type
Videos
Mean (s)
Variance (s 2 )
Median (s)
Captions (CholecT50, MultiBypass140)
1,442
9.2
1,224.8
2
Skill scores (CholeScore, MultiBypass140)
320
977.2
741,469.5
784
Table S6: Clip duration in EndoDiffVQA by annotation type, pooled over both procedures. Mean, variance and median are in seconds; variance is the population variance.
Figure S2: Example items of the Anatomy templates A1 (presence), A2 (colour), A3 (position) and A4 (state). Three frames per clip; the correct option is highlighted and each distractor is tagged with its error type.
Figure S3: Example items of the Tool templates T1 (presence), T2 (entry side), T3 (count) and T4 (engagement).
Figure S4: Example items of the Action templates C1 (interaction), C2 (target part) and C3 (goal).
Figure S5: Example items of the Skill templates S1 ( Chole-Skill-DQA , CholeScore criteria) and S2 ( LRYGB-Skill-DQA , GOALS domains). The clips are phase-level and last minutes, so the three frames span a whole surgical step.
Figure S6: Qualitative examples, templates A1 (Anatomy, presence) and A2 (Anatomy, colour) (legend in Section S3.2 ).
Figure S7: Qualitative examples, templates A4 (Anatomy, position) and A6 (Anatomy, state) (legend in Section S3.2 ).
Figure S8: Qualitative examples, templates T1 (Tool, presence) and T3 (Tool, entry side) (legend in Section S3.2 ).
Figure S9: Qualitative examples, templates T5 (Tool, count) and T6 (Tool, engagement) (legend in Section S3.2 ).
Figure S10: Qualitative examples, templates C1 (Action, interaction) and C2 (Action, target part) (legend in Section S3.2 ).
Figure S11: Qualitative examples, templates C4 (Action, goal) and S1 (Skill, CholeScore) (legend in Section S3.2 ).
Figure S12: Qualitative example, template S2 (Skill, MultiBypass140) (legend in Section S3.2 ).
Reasoning about surgical video -- which phase is underway, which instruments act, and when events occur -- underpins operative documentation, workflow analysis, and skill assessment. Yet surgical visual question answering is still largely built on single frames and closed labels that cannot express a duration, an ordering, or a count, and the video models that lift the frame limit typically compress every clip the same way regardless of the question, discarding the temporal evidence the answer needs. Worse, the answers are structurally heterogeneous, spanning five incompatible structures -- binary, classification, multi-label, counting, and sentence-temporal -- that no single training signal scores uniformly well. We propose SurgQ-VLM, a two-stage model for surgical video question answering. Stage 1 trains a question-conditioned dual-stream (slow/fast) compressor for a LoRA-adapted Qwen2.5-VL-7B: cross-attention conditions both streams on the question, and each clip's visual tokens shrink 49x from the raw ViT patches (11x vs. post-merge backbone tokens). Stage 2 post-trains with GRPO under an answer-type-aware reward that scores each question on the metric matching its answer structure (micro-F1 for label sets, normalised error for counts, a content-timing blend for temporal windows). On a sealed 21,840-question test split over Cholec80 and CholecT50, SurgQ-VLM reaches 0.779 keyword accuracy and a 10x BLEU-4 gain over zero-shot; GRPO adds 2.5 pp K-Acc (2.0 pp attributable to the answer-type-aware reward), concentrated in the structurally hard types, with answer format unchanged (ROUGE-L). Code is available at https://github.com/DhruvMakwana/SurgQ-VLM.
Purpose: Vision-language models (VLMs) have shown promising performance in surgical visual question answering (VQA). However, existing surgical VQA datasets often contain linguistic shortcuts, where question phrasing implicitly constrains the answer space. It remains unclear whether reported performance reflects visual understanding or reliance on such linguistic shortcuts. Methods: We introduce SurgCheck, a diagnostic benchmark for quantifying linguistic shortcut reliance in surgical VQA. SurgCheck employs a paired-question design in which each surgical frame is associated with an original question containing entity names and a less-biased counterpart that removes these names while preserving identical visual content and ground-truth answers. The resulting performance gap provides a diagnostic signal of shortcut reliance. To ensure that the less-biased question remains well-defined even without entity names, four grounding cues are incorporated: bounding box, arrow, spatial position, and periphrasis. We evaluate both general-purpose and surgical-specific VLMs under zero-shot and fine-tuned settings on SurgCheck. To evaluate open-ended zero-shot responses, we introduce an LLM-as-a-judge evaluation protocol. Results: Using SurgCheck, we observe consistent performance degradation on less-biased questions across five VLMs, despite identical visual inputs. Text-only ablation reveals minimal performance drops for action and target prediction, indicating that action and target prediction is largely driven by linguistic shortcuts rather than visual reasoning. Conclusion: SurgCheck provides a controlled diagnostic framework that exposes failure modes masked by linguistic bias in existing surgical VQA benchmarks. Our findings demonstrate that strong benchmark performance does not necessarily imply faithful visual understanding, underscoring the need for bias-aware evaluation in surgical VQA.
Jongmin Shin, Ka Young Kim, Eunki Cho +2
Department of Surgery, Samsung Medical Center, Seoul 06351, Republic of Korea · Kyung Hee University, Yongin 17104, Republic of Korea
A capable surgical AI model should reason from visual evidence; yet, many existing surgical benchmarks can be partially solved without visual input, as models may exploit procedural priors or common anatomy-instrument interactions to infer answers from language alone. To address this, we introduce SUREON, a video question-answering (QA) dataset and benchmark designed to evaluate visually grounded surgical reasoning. Our pipeline generates QA pairs from narrated surgical academic videos, refines incorrect options using multiple models to reduce linguistic biases, and uses trained adversarial guessers to filter out questions answerable without visual evidence. The resulting dataset contains 76K training pairs covering 168 procedures and 10 specialties, alongside a benchmark of 340 questions across 10 categories verified by expert surgeons. Benchmarking leading commercial and open-source models shows that surgical visual reasoning remains largely unsolved--no model exceeds 53% accuracy and even expert surgeons reach only 64.9%. We also introduce SUREON-VLM, a baseline trained on our curated data that avoids language shortcuts, outperforms existing open-source models on SUREON, and matches larger specialist models on multiple standard surgical benchmarks. Our training ablations show that distractor refinement and blind curation strengthen visual grounding, increasing the multimodal gain from 8.7 to 29.1 percentage points while reducing the training set from 176.6K to 76K filtered samples. SUREON thus offers both a verified testbed for surgical visual reasoning and training data that strengthens visual grounding.