cs.CLJan 15, 2026

Beyond Trajectory Similarity: Evaluating Surgical Planning by Protocol Adherence

Authors: Ruochen Li, Kun Yuan, Yufei Xia, Yue Zhou, Qingyu Lu, Weihang Li, Youxiang Zhu, Nassir Navab

Organizations: Technical University of Munich, Germany · University of Strasbourg, France · University of Glasgow, United Kingdom · Nanyang Technological University, Singapore · 

Abstract

Surgical planning is commonly evaluated by trajectory similarity to a single observed execution, a paradigm borrowed from activity anticipation benchmarks such as Ego4D. This conflates two distinct concepts: task completion and protocol adherence and assumes similarity to a completed surgical trajectory implies a valid plan. In surgery, this assumption is flawed: the clinically relevant target is protocol adherence rather than mere completion, and the reference trajectory is only one observed execution that may itself deviate from the protocol. We therefore introduce an evaluation framework grounded in expert-defined protocol rules that directly measures protocol adherence. Under this framework, similarity-based metrics systematically fail: they penalize protocol-adherent plans that deviate from the reference trajectory, while rewarding protocol-violating plans that remain superficially similar to it. Evaluating Video-LLMs further reveals two separable failure modes: unreliable video-to-state grounding and under-constrained planning. Neither is resolved by model scaling alone, whereas structural knowledge injection provides consistent gains. Our results motivate a shift from trajectory similarity toward protocol adherence as the evaluation standard for surgical planning.

Figures & tables

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. SurgWMBench: A Vision-Based Benchmark for World-Modeling Surgical Instrument Motion Planning

    Aug 8, 2026Huanrong Liu, Weiliang Huang, Bob Zhang +3World Models for RoboticsWorld Model-Based Planning

  2. A Hierarchy-Aware Video-Language Model Evaluation and Hyperbolic Baseline for Surgery

    Sep 22, 2026Ana Manzano Rodríguez, Pascal Mettes, Marlies P. Schijven +1Video-Language Model EvaluationHierarchical Representation Learning

  3. SurgCoT: Advancing Spatiotemporal Reasoning in Surgical Videos through a Chain-of-Thought Benchmark

    Apr 22, 2026Gui Wang, YongSong Zhou, Kaijun Deng +4surgical video reasoning benchmarksSpatiotemporal Reasoning