Organizations: Technical University of Munich, Germany · University of Strasbourg, France · University of Glasgow, United Kingdom · Nanyang Technological University, Singapore · University of Massachusetts Boston, USA
Surgical planning is commonly evaluated by trajectory similarity to a single observed execution, a paradigm borrowed from activity anticipation benchmarks such as Ego4D. This conflates two distinct concepts: task completion and protocol adherence and assumes similarity to a completed surgical trajectory implies a valid plan. In surgery, this assumption is flawed: the clinically relevant target is protocol adherence rather than mere completion, and the reference trajectory is only one observed execution that may itself deviate from the protocol. We therefore introduce an evaluation framework grounded in expert-defined protocol rules that directly measures protocol adherence. Under this framework, similarity-based metrics systematically fail: they penalize protocol-adherent plans that deviate from the reference trajectory, while rewarding protocol-violating plans that remain superficially similar to it. Evaluating Video-LLMs further reveals two separable failure modes: unreliable video-to-state grounding and under-constrained planning. Neither is resolved by model scaling alone, whereas structural knowledge injection provides consistent gains. Our results motivate a shift from trajectory similarity toward protocol adherence as the evaluation standard for surgical planning.
Figures & tables
Figure 1: Similarity = Completion = Protocol Adherence. Both candidates complete the gastric pouch creation phase (P2), yet differ in protocol adherence. The upper candidate closely resembles the reference SOP but violates a critical ordering constraint (S8 appears before S11, reversing the required order). The lower candidate deviates from the reference yet fully adheres to the protocol by respecting all procedural dependencies. Trajectory similarity cannot distinguish these cases. Step abbreviations are listed in Appendix A .
Traditional Metrics
LLM-based Judges
Subset
NED
JIS
ROA
Gemini 3 Pro
GPT 5.2
Claude 4.5 Sonnet
Hulu-Med 32B
Qwen3-32B
GPT-4o
Valid
18.8
40.3
93.2
99.5
97.4
61.8
99.0
81.2
83.8
OE
87.3
46.5
11.3
11.3
18.3
80.3
8.5
35.2
38.0
CE
86.8
85.3
17.6
85.3
97.1
97.1
58.8
85.3
91.2
BE
96.7
85.0
20.0
98.3
100.0
100.0
75.0
95.0
93.3
Invalid
89.9
71.4
17.1
63.8
69.8
92.0
45.7
70.4
72.9
Table 1: Accuracy (%) comparison of traditional similarity metrics and LLM-based judges across valid and erroneous subsets. NED: Normalized Edit Distance. JIS: Jaccard Index on Sequences. ROA: Relative Order Accuracy. OE: order error. CE: content error. BE: both error.
Figure 2: Evaluation for Surgical Planning. Left: Protocol adherence evaluation set construction via expert-defined protocol rules, separating valid and invalid step sequences. Right: Progressive evaluation of Video-LLMs, from end-to-end planning to planning with ground-truth steps and injected knowledge.
Task
Metric
Hulu-7B
Hulu-32B
Qwen 2.5-7B
Qwen 2.5-32B
DAMO
Lingshu-7B
Lingshu-32B
LLaVA
Qwen 3-7B
Qwen 3-7B Think
Real-world
StepAcc
23.5%
34.0%
21.7%
31.4%
14.4%
26.1%
39.4%
2.8%
28.3%
28.2%
Current
10.3%
23.8%
2.9%
20.9%
8.8%
7.4%
31.1%
1.3%
21.1%
20.9%
Next
2.9%
45.6%
2.4%
31.8%
1.4%
1.0%
46.5%
0.1%
22.1%
22.1%
Controlled
Current
27.9%
40.2%
13.7%
48.6%
22.5%
22.3%
51.1%
20.5%
47.3%
47.1%
Next
2.7%
31.0%
2.9%
29.4%
1.5%
4.5%
40.7%
0.1%
26.1%
25.8%
Table 2: Protocol adherence accuracy (%) across models. Task 1 (real-world) evaluates current step recognition (StepAcc) and downstream planning. Task 2 (controlled) focuses on phase-aware planning. Current/Next denote current-phase completion and next-phase planning. Best results in bold. Qwen: Qwen-VL model series.
Task
Metric
Hulu-7B
Hulu-32B
Qwen 2.5-7B
Qwen 2.5-32B
DAMO
Lingshu-7B
Lingshu-32B
LLaVA
Qwen 3-7B
Qwen 3-7B Think
Structural
Current
42.8%
66.0%
63.0%
73.0%
43.0%
46.1%
71.1%
23.4%
63.2%
63.1%
Next
51.2%
49.2%
64.3%
68.7%
38.0%
62.6%
70.5%
22.9%
64.5%
64.8%
Description
Current
26.8%
46.5%
14.0%
44.8%
26.9%
18.5%
49.7%
14.1%
53.2%
53.1%
Next
7.7%
26.7%
11.5%
56.7%
5.5%
0.9%
52.8%
0.1%
54.4%
54.3%
Combined
Current
36.2%
67.6%
55.0%
69.1%
38.6%
34.6%
70.3%
19.7%
61.6%
61.5%
Next
41.7%
76.5%
60.8%
89.2%
53.8%
56.5%
82.1%
21.3%
69.3%
69.0%
Table 3: Protocol adherence accuracy (%) of Task 3 under different knowledge injection settings. Current/Next denote current-phase completion and next-phase planning. Best results in bold. Qwen: Qwen-VL model series.
Appendix figures & tables8 assets
Supplementary material from the paper’s appendix.
Appendix
ID
Step name
ID
Step name
ID
Step name
S1
cavity_exploration
S16
biliary_limb_measurement
S31
alimentary_limb_opening
S2
trocar_placement
S17
jejunum_opening
S32
jejunojejunal_stapling
S3
retractor_placement
S18
gastrojejunal_stapling
S33
jejunojejunal_defect_closing
S4
crura_dissection
S19
gastrojejunal_defect_closing
S34
jejunojejunal_anastomosis_reinforcement
S5
his_angle_dissection
S20
mesenteric_opening
S35
staple_line_reinforcement
S6
horizontal_stapling
S21
jejunal_section
S36
mesenteric_defect_exposure
Appendix
Table 4: Surgical step abbreviations used in Figure 1 .
ID
Phase name
ID
Phase name
P1
Preparation
P7
Petersen space closure
P2
Gastric pouch creation
P8
Jejunojejunal anastomosis
P3
Omentum division
P9
Mesenteric defect closure
P4
Gastrojejunal anastomosis
P10
Cleaning & coagulation
P5
Anastomosis test
P11
Disassembling
P6
Jejunal separation
Appendix
Table 5: Surgical phase abbreviations used in the study.
Reliable surgical planning requires models that move beyond recognizing the current surgical step or imitating expert demonstrations, and instead anticipate how instrument motion reshapes subsequent operative states. Most surgical video understanding methods focus on recognizing phases, actions, or workflow states, while providing limited support for explicitly modeling instrument motion. Conversely, existing tool motion prediction methods can forecast instrument trajectories, but they generally do not capture the coupled evolution of future surgical video states. World models offer a natural framework for jointly modeling visual state transitions and instrument motion dynamics. However, existing surgical world model studies remain largely centered on visual generation quality, relying on generation-oriented metrics such as FVD and CD-FVD. These metrics are poorly aligned with instrument motion planning, as they do not directly measure whether predicted trajectories are geometrically accurate, temporally coherent, or actionable for downstream planning. This limitation is partly structural, since the field lacks public datasets and standardized evaluation protocols that provide the benchmarking infrastructure needed to assess motion-centric capabilities in surgical world models. In this paper, we introduce SurgWMBench, a vision-based benchmark for short-horizon surgical motion planning and dynamics prediction. Given intraoperative image sequences and historical instrument trajectory, SurgWMBench evaluates both near-future instrument motion prediction and stability under continuous rollout or input perturbations.
Huanrong Liu, Weiliang Huang, Bob Zhang +3
University of Macau · Xiamen University · National University of Singapore
Surgical procedures follow a phase-to-step hierarchy, yet the video-language models used to recognize them are evaluated with flat per-level metrics that ignore cross-level coherence and error structure. In this paper we make two contributions to address this problem, (i) we introduce SurgHiBench, the first hierarchy-aware evaluation suite for surgical video understanding, with three tasks measuring recognition, consistency, and severity across granularity levels. We evaluate a general-purpose CLIP model, a Euclidean surgical model, and, as second contribution: (ii) HyperSurg, a new hyperbolic model that enforces phase-step containment via entailment cones, across four (existing) datasets spanning three procedure types. The suite reveals that two models with the same accuracy can produce predictions of very different error severity, ranging from sibling confusions within the correct phase to unrelated cross-phase predictions. Hyperbolic geometry shifts predictions toward the correct procedural neighborhood, and these gains scale with the tree-likeness of each dataset's annotation hierarchy, providing a principled indicator when hierarchy-aware geometry helps.
Ana Manzano Rodríguez, Pascal Mettes, Marlies P. Schijven +1
Data Science Center HAVA-Lab, University of Amsterdam · Video & Image Sense Lab, Informatics Institute, University of Amsterdam · Amsterdam UMC Location University of Amsterdam, Surgery +2
Fine-grained spatiotemporal reasoning on surgical videos is critical, yet the capabilities of Multi-modal Large Language Models (MLLMs) in this domain remain largely unexplored. To bridge this gap, we introduce SurgCoT, a unified benchmark for evaluating chain-of-thought (CoT) reasoning in MLLMs across 7 surgical specialties and 35 diverse procedures. SurgCoT assesses five core reasoning dimensions: Causal Action Ordering, Cue-Action Alignment, Affordance Mapping, Micro-Transition Localization, and Anomaly Onset Tracking, through a structured CoT framework with an intensive annotation protocol (Question-Option-Knowledge-Clue-Answer), where the Knowledge field provides essential background context and Clue provides definitive spatiotemporal evidence. Evaluation of 10 leading MLLMs shows: 1) commercial models outperform open-source and medical-specialized variants; 2) significant gaps exist in surgical CoT reasoning; 3) SurgCoT enables effective evaluation and enhances progressive spatiotemporal reasoning. SurgCoT provides a reproducible testbed to narrow the gap between MLLM capabilities and clinical reasoning demands. Code: https://github.com/CVI-SZU/SurgCoT.
Gui Wang, YongSong Zhou, Kaijun Deng +4
School of Computer Science and Software Engineering, Shenzhen University · University of Nottingham · School of AI, Shenzhen University +1