Organizations: Technical University of Munich, Germany · University of Strasbourg, France · University of Glasgow, United Kingdom · Nanyang Technological University, Singapore · University of Massachusetts Boston, USA
Surgical planning is commonly evaluated by trajectory similarity to a single observed execution, a paradigm borrowed from activity anticipation benchmarks such as Ego4D. This conflates two distinct concepts: task completion and protocol adherence and assumes similarity to a completed surgical trajectory implies a valid plan. In surgery, this assumption is flawed: the clinically relevant target is protocol adherence rather than mere completion, and the reference trajectory is only one observed execution that may itself deviate from the protocol. We therefore introduce an evaluation framework grounded in expert-defined protocol rules that directly measures protocol adherence. Under this framework, similarity-based metrics systematically fail: they penalize protocol-adherent plans that deviate from the reference trajectory, while rewarding protocol-violating plans that remain superficially similar to it. Evaluating Video-LLMs further reveals two separable failure modes: unreliable video-to-state grounding and under-constrained planning. Neither is resolved by model scaling alone, whereas structural knowledge injection provides consistent gains. Our results motivate a shift from trajectory similarity toward protocol adherence as the evaluation standard for surgical planning.
Figures & tables
Figure 1: Similarity = Completion = Protocol Adherence. Both candidates complete the gastric pouch creation phase (P2), yet differ in protocol adherence. The upper candidate closely resembles the reference SOP but violates a critical ordering constraint (S8 appears before S11, reversing the required order). The lower candidate deviates from the reference yet fully adheres to the protocol by respecting all procedural dependencies. Trajectory similarity cannot distinguish these cases. Step abbreviations are listed in Appendix A .
Traditional Metrics
LLM-based Judges
Subset
NED
JIS
ROA
Gemini 3 Pro
GPT 5.2
Claude 4.5 Sonnet
Hulu-Med 32B
Qwen3-32B
GPT-4o
Valid
18.8
40.3
93.2
99.5
97.4
61.8
99.0
81.2
83.8
OE
87.3
46.5
11.3
11.3
18.3
80.3
8.5
35.2
38.0
CE
86.8
85.3
17.6
85.3
97.1
97.1
58.8
85.3
91.2
BE
96.7
85.0
20.0
98.3
100.0
100.0
75.0
95.0
93.3
Invalid
89.9
71.4
17.1
63.8
69.8
92.0
45.7
70.4
72.9
Table 1: Accuracy (%) comparison of traditional similarity metrics and LLM-based judges across valid and erroneous subsets. NED: Normalized Edit Distance. JIS: Jaccard Index on Sequences. ROA: Relative Order Accuracy. OE: order error. CE: content error. BE: both error.
Figure 2: Evaluation for Surgical Planning. Left: Protocol adherence evaluation set construction via expert-defined protocol rules, separating valid and invalid step sequences. Right: Progressive evaluation of Video-LLMs, from end-to-end planning to planning with ground-truth steps and injected knowledge.
Task
Metric
Hulu-7B
Hulu-32B
Qwen 2.5-7B
Qwen 2.5-32B
DAMO
Lingshu-7B
Lingshu-32B
LLaVA
Qwen 3-7B
Qwen 3-7B Think
Real-world
StepAcc
23.5%
34.0%
21.7%
31.4%
14.4%
26.1%
39.4%
2.8%
28.3%
28.2%
Current
10.3%
23.8%
2.9%
20.9%
8.8%
7.4%
31.1%
1.3%
21.1%
20.9%
Next
2.9%
45.6%
2.4%
31.8%
1.4%
1.0%
46.5%
0.1%
22.1%
22.1%
Controlled
Current
27.9%
40.2%
13.7%
48.6%
22.5%
22.3%
51.1%
20.5%
47.3%
47.1%
Next
2.7%
31.0%
2.9%
29.4%
1.5%
4.5%
40.7%
0.1%
26.1%
25.8%
Table 2: Protocol adherence accuracy (%) across models. Task 1 (real-world) evaluates current step recognition (StepAcc) and downstream planning. Task 2 (controlled) focuses on phase-aware planning. Current/Next denote current-phase completion and next-phase planning. Best results in bold. Qwen: Qwen-VL model series.
Task
Metric
Hulu-7B
Hulu-32B
Qwen 2.5-7B
Qwen 2.5-32B
DAMO
Lingshu-7B
Lingshu-32B
LLaVA
Qwen 3-7B
Qwen 3-7B Think
Structural
Current
42.8%
66.0%
63.0%
73.0%
43.0%
46.1%
71.1%
23.4%
63.2%
63.1%
Next
51.2%
49.2%
64.3%
68.7%
38.0%
62.6%
70.5%
22.9%
64.5%
64.8%
Description
Current
26.8%
46.5%
14.0%
44.8%
26.9%
18.5%
49.7%
14.1%
53.2%
53.1%
Next
7.7%
26.7%
11.5%
56.7%
5.5%
0.9%
52.8%
0.1%
54.4%
54.3%
Combined
Current
36.2%
67.6%
55.0%
69.1%
38.6%
34.6%
70.3%
19.7%
61.6%
61.5%
Next
41.7%
76.5%
60.8%
89.2%
53.8%
56.5%
82.1%
21.3%
69.3%
69.0%
Table 3: Protocol adherence accuracy (%) of Task 3 under different knowledge injection settings. Current/Next denote current-phase completion and next-phase planning. Best results in bold. Qwen: Qwen-VL model series.
Appendix figures & tables8 assets
Supplementary material from the paper’s appendix.
Appendix
ID
Step name
ID
Step name
ID
Step name
S1
cavity_exploration
S16
biliary_limb_measurement
S31
alimentary_limb_opening
S2
trocar_placement
S17
jejunum_opening
S32
jejunojejunal_stapling
S3
retractor_placement
S18
gastrojejunal_stapling
S33
jejunojejunal_defect_closing
S4
crura_dissection
S19
gastrojejunal_defect_closing
S34
jejunojejunal_anastomosis_reinforcement
S5
his_angle_dissection
S20
mesenteric_opening
S35
staple_line_reinforcement
S6
horizontal_stapling
S21
jejunal_section
S36
mesenteric_defect_exposure
Appendix
Table 4: Surgical step abbreviations used in Figure 1 .
ID
Phase name
ID
Phase name
P1
Preparation
P7
Petersen space closure
P2
Gastric pouch creation
P8
Jejunojejunal anastomosis
P3
Omentum division
P9
Mesenteric defect closure
P4
Gastrojejunal anastomosis
P10
Cleaning & coagulation
P5
Anastomosis test
P11
Disassembling
P6
Jejunal separation
Appendix
Table 5: Surgical phase abbreviations used in the study.
Data Science Center HAVA-Lab, University of Amsterdam · Video & Image Sense Lab, Informatics Institute, University of Amsterdam · Amsterdam UMC Location University of Amsterdam, Surgery +2