cs.CVNov 8, 2026

How to "train" a surgeon? Benchmarking generalist agents in surgical scene understanding

Authors: Gengyuan Zhang, Xiao Han, Shijie Zhou, Abdullah Koukash, Jingpei Wu, Jindong Gu, Yuqicheng Zhu, Nassir Navab, 

Organizations: LMU Munich · MCML · TU Munich · relAI · 

Abstract

Evaluating general-purpose multimodal large language models (MLLMs) for surgical scene understanding remains limited due to scarce surgical data and fragmented evaluation protocols. We address these challenges through a two-stage evaluation framework for surgical scene understanding. In Stage I, we introduce SurgKnowBench, a diagnostic benchmark that harmonizes public surgical datasets across procedures and tasks and evaluates proprietary and open-source MLLMs to map the boundary of pretrained surgical competence. SurgKnowBench reveals that current models retain partial visual priors but remain weak on relational and procedural understanding and workflow assessment. Building on this failure frontier and motivated by the scaffolded nature of surgical training, we introduce SurgAgentGym, an agentic environment where general-purpose MLLM agents can use bounded knowledge search over SurgWiki and general-purpose visual grounding for surgical tasks. SurgAgentGym evaluates whether generalist agents can improve surgical understanding at inference time through tool-mediated evidence use, without additional surgical training. Our results show that agentic inference narrows the gap mainly for perception-heavy tasks, while fine-grained localization and expert-style assessment remain persistent bottlenecks. This opens a path toward future studies of agentic surgical understanding.

Figures & tables

Appendix figures & tables29 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. SurgRAW: Multi-Agent Workflow with Chain of Thought Reasoning for Robotic Surgical Video Analysis

    Mar 13, 2025Chang Han Low, Ziyue Wang, Tianyi Zhang +4Surgical Video UnderstandingMultimodal CoT Reasoning

  2. SurgXBench: Explainable Vision-Language Model Benchmark for Surgery

    May 16, 2025Jiajun Cheng, Xianwu Zhao, Sainan Liu +5VLM EvaluationSurgical Video Understanding

  3. SurgiQ: A Large-Scale Multi-Domain Benchmark for Evaluating Surgical Understanding in Large Language Models

    Jun 6, 2026Ayah Al-Naji, Edoardo Fazzari, Saif Alkindi +3LLM EvaluationSynthetic Benchmark Generation