EndoComparer: Comparative Surgical Video Question Answering with Pseudo-Siamese Visual Adaptation and Perceive-then-Compare Reasoning
Authors: Shi Li, Lalith Sharan, Joël L. Lavanchy, Kun Yuan, Lorenzo Arboit, Nabani Banik, Elif Yazici, Maria Vannucci, Vinkle Srivastav, Pietro Mascagni, Nicolas Padoy
Organizations: ICube, UMR7357, University of Strasbourg, CNRS, INSERM, Strasbourg, 67000, France. · Institute of Image-Guided Surgery, IHU Strasbourg, Strasbourg, 67000, France. · University Digestive Health Care Center – Clarunis, Basel, 4002, Switzerland. · Department of Biomedical Engineering, University of Basel, Allschwil, 4123, Switzerland. · General Surgery Department, University of Turin, Turin, Italy. · Indian Institute of Technology (IIT) Madras, Chennai, India. · Fondazione Policlinico Universitario Agostino Gemelli IRCCS, Rome, Italy.
Purpose: Surgical training benefits from explicit comparison between cases: in patient condition, in technique and in skill proficiency. Surgical vision-language models (VLMs) offer a promising and flexible interface for this purpose. Yet current VLMs typically see one video at a time, and they often fall short when asked to compare videos. Methods: We propose EndoComparer, a VLM for comparative surgical video question answering with two components. (a) A Pseudo-Siamese Visual LoRA (PSVL) gives each video slot its own low-rank adapter over a shared frozen vision encoder, separating the two videos in the visual pathway at the cost of a single encoder pass. (b) A Perceive-then-Compare chain of thought (PtC-CoT) trains the model to first state what each video shows, then derive the answer from these statements. We further curate EndoDiffVQA, a benchmark of nearly 11K multiple-choice questions on video-pair over cholecystectomy and gastric bypass procedures, covering anatomy, tool, action and skill. Results: EndoComparer outperforms strong generalist and surgical VLMs on detecting and specifying difference between videos. Ablations show that two proposed components contribute complimentarily, and the model remains more robust under a video-swap control that exposes confusion between the two videos. Conclusion: Providing a dedicated visual reasoning path through PSVL and PtC-CoT makes cross-case comparison of surgical videos more robust for VLMs. EndoDiffVQA and EndoComparer provide a benchmark and a strong baseline for this task. Project page: https://camma-public.github.io/endocomparer
Figures & tables
Figure 1: An overview of EndoDiffVQA construction pipline.
Figure 2: An overview of EndoComparer model architecture.
4-subset macro
Strict per subset
Model
Detect
Specify
Strict
Chole- Cap
LRYGB- Cap
Chole- Skill
LRYGB- Skill
Zero-shot
Qwen3.5-9B ( Qwen Team, 2026a )
49.8 ±.5
57.3 ±.4
35.4 ±.9
41.3 ±.3
34.4 ±.7
31.6 ±.6
34.3 ±.3
Qwen3.8-27B ( Qwen Team, 2026b )
52.8 ±.7
58.9 † ±.2
35.2 ±.6
38.7 ±.3
33.8 ±.3
44.7 ±.6
23.6 ±1.0
InternVL3.5-8B ( Wang et al., 2025c )
49.7 ±.3
54.5 † ±.4
30.6 ±.8
44.7 ±1.1
25.5 ±.8
28.6 ±.6
23.6 ±1.2
InternVL3.5-30B ( Wang et al., 2025c )
56.6 ±.2
57.1 ±1.1
37.2 ±.5
44.5 ±.3
32.0 ±.3
36.3 ±.5
36.1 ±1.0
Table 1: Results on the EndoDiffVQA test split (2,300 items, %). Detect, Specify and Strict are 4-subset macros (see Metrics); the last four columns give Strict per subset. † : Specify macro over fewer than four subsets, because the model called fewer than 20 items Different in the others; not highlighted. Best per group in bold; the fine-tuned group includes ours.
Zero-shot
Fine-tuned
Qwen3.8 27B
InternVL 3.5-30B
Surg- VidLM
InternVL 3.5-8B
Surg- VidLM
Surg- TEMP
Surg- ViVQA
Qwen3.5 9B
Endo- Comparer
Full
35.2
37.2
32.1
47.0
45.1
46.8
32.6
48.3
54.0
Swap
33.1
34.5
27.8
41.6
38.3
39.2
27.4
42.5
53.6
Δ
− 2.1
− 2.7
− 4.3
− 5.4
− 6.8
− 7.6
− 5.2
− 5.8
−0.6
Table 2: Robustness to video order: Strict on the EndoDiffVQA test split (%, 4-subset macro). Full is the release order; Swap exchanges Video 1 and Video 2 and scores against the swapped answer. Δ is Swap − Full.
Table S1: Slot schema of the per-clip record: one anatomy entry per structure, one tools entry per instrument instance, one actions entry per instrument–tissue interaction, and one goal . Region: one of left, right, top, bottom, center, top-left, top-right, bottom-left, bottom-right, background, across the view . Slots that are extracted but feed no released template (texture, shape, exposure, movement direction) are omitted.
We can see a blue-green structure with thin red blood vessels on the top-left side of the camera, which is a porcelain gallbladder. The proximal side of the gallbladder and its peritoneum are dissected […] The grasper 2 is seen on the left side of the camera. […] (Not seen on this clip because of the camera’s view: The grasper 1 is grasping the distal side of the gallbladder.) The grasper 2 is retracting the proximal side of the gallbladder to the left side of the camera. […] The surgeon wants to dissect the gallbladder/cystic pedicle.
Table S2: Slot extraction on an abbreviated CholecT50 caption (one entry per record shown). The numbered graspers become two grasper entries, and the grasp narrated inside “(Not seen …)” is kept as an unseen action.
Step
Rule
Normalisation
Synonym folding
Surface variants map to one value: pinkish → pink , upper left → top-left , clip applier → clipper , gall bladder → gallbladder , bleeds → bleeding ; articles and punctuation are stripped.
Tool numbering
“grasper 2” → grasper ; any digit left in a tool name fails validation.
Object nouns
Specimen bag, trocar, compress, suture thread and needle filed under anatomy are moved to tools, and are never engaged and never an action’s actor.
Entity families
Names one structure carries across the two corpora ( omentum, mesentery, fat ) are kept but grouped, so absence (A1) must hold for the whole family.
Duplicates
Repeated entries are merged; slots of a structure marked not visible are cleared.
Table S3: Deterministic post-processing of the extracted slots.
ID
Fact per clip
Stem
Rule
Anatomy
A1
structure visible?
Is the {entity} visible in both videos?
same / only k / neither from the two flags; neither : a structure of the phase that neither caption mentions.
A2
colour of the structure
Is the {entity} the same colour in both videos?
same if equal, different if no colour word is shared, only k if absent from the other clip.
A3
region of the view the structure is in
Is the {entity} in the same part of the view in both videos?
As A2 over regions.
A4
state of the structure (bleeding, dissected, clipped, …)
Is the {entity} {state} in both videos?
As A1 per (structure, state); neither : a state the structure shows elsewhere in the source.
Tool
Table S4: The 13 question templates. Fact: what is read from each clip’s record; Rule: how the two facts give the relation of Eq. (1) of the main paper, a pair that fits no case being skipped. Throughout, only k requires the subject to be safely absent from the other clip (no same-organ alias such as omentum / mesentery / fat, no mention in its caption); overlapping values (“left” vs. “top-left”, “yellow” vs. “yellow-pink”) are never keyed as different ; specimen bag, trocar and other objects never count as instruments. C1 action kinds: manipulate (grasp, hold, retract, lift, push, pull, expose), divide (cut, divide), energy (coagulate, cauterize), clean (irrigate, suction, clean), and dissect, clip, staple, suture each on their own.
Rd.
Error
Example
Fix (hard check)
1
Object as actor
A specimen bag or a trocar counted as the instrument engaging tissue (T3, T4).
Object nouns never act; they are excluded as instruments.
1
Release read as engagement
“The grasper releases the gallbladder” marked the grasper as engaged with it (T4).
A tool that the caption says is releasing the target is not engaged.
2
True distractor
A distractor interaction or goal that one of the two clips actually shows (C1, C3).
No distractor may be true of either video.
2
Overlapping values as different
left vs. upper left keyed as a different position or entry side (A3, T2).
Overlapping values are never different , nor used as a distractor.
2
Value from a negation
“no unified tubular structure” turned into a tubular cystic duct (shape).
Values in negated clauses are dropped.
3
Absence from silence
“Is the cystic duct tubular in both videos?” answered “only in Video 1” because the other caption does not mention the shape.
Template disabled: shape does not change between clips.
Table S5: Representative errors found during the audit rounds and how each was fixed. Rd.: audit round in which the error was found. A hard check re-runs every fix on each build, and the build fails if any item is flagged.
Figure S1: Items per category and attribute in EndoDiffVQA.
Annotation type
Videos
Mean (s)
Variance (s 2 )
Median (s)
Captions (CholecT50, MultiBypass140)
1,442
9.2
1,224.8
2
Skill scores (CholeScore, MultiBypass140)
320
977.2
741,469.5
784
Table S6: Clip duration in EndoDiffVQA by annotation type, pooled over both procedures. Mean, variance and median are in seconds; variance is the population variance.
Figure S2: Example items of the Anatomy templates A1 (presence), A2 (colour), A3 (position) and A4 (state). Three frames per clip; the correct option is highlighted and each distractor is tagged with its error type.
Figure S3: Example items of the Tool templates T1 (presence), T2 (entry side), T3 (count) and T4 (engagement).
Figure S4: Example items of the Action templates C1 (interaction), C2 (target part) and C3 (goal).
Figure S5: Example items of the Skill templates S1 ( Chole-Skill-DQA , CholeScore criteria) and S2 ( LRYGB-Skill-DQA , GOALS domains). The clips are phase-level and last minutes, so the three frames span a whole surgical step.
Figure S6: Qualitative examples, templates A1 (Anatomy, presence) and A2 (Anatomy, colour) (legend in Section S3.2 ).
Figure S7: Qualitative examples, templates A4 (Anatomy, position) and A6 (Anatomy, state) (legend in Section S3.2 ).
Figure S8: Qualitative examples, templates T1 (Tool, presence) and T3 (Tool, entry side) (legend in Section S3.2 ).
Figure S9: Qualitative examples, templates T5 (Tool, count) and T6 (Tool, engagement) (legend in Section S3.2 ).
Figure S10: Qualitative examples, templates C1 (Action, interaction) and C2 (Action, target part) (legend in Section S3.2 ).
Figure S11: Qualitative examples, templates C4 (Action, goal) and S1 (Skill, CholeScore) (legend in Section S3.2 ).
Figure S12: Qualitative example, template S2 (Skill, MultiBypass140) (legend in Section S3.2 ).