How to "train" a surgeon? Benchmarking generalist agents in surgical scene understanding
Organizations: LMU Munich · MCML · TU Munich · relAI ·
Abstract
Evaluating general-purpose multimodal large language models (MLLMs) for surgical scene understanding remains limited due to scarce surgical data and fragmented evaluation protocols. We address these challenges through a two-stage evaluation framework for surgical scene understanding. In Stage I, we introduce SurgKnowBench, a diagnostic benchmark that harmonizes public surgical datasets across procedures and tasks and evaluates proprietary and open-source MLLMs to map the boundary of pretrained surgical competence. SurgKnowBench reveals that current models retain partial visual priors but remain weak on relational and procedural understanding and workflow assessment. Building on this failure frontier and motivated by the scaffolded nature of surgical training, we introduce SurgAgentGym, an agentic environment where general-purpose MLLM agents can use bounded knowledge search over SurgWiki and general-purpose visual grounding for surgical tasks. SurgAgentGym evaluates whether generalist agents can improve surgical understanding at inference time through tool-mediated evidence use, without additional surgical training. Our results show that agentic inference narrows the gap mainly for perception-heavy tasks, while fine-grained localization and expert-style assessment remain persistent bottlenecks. This opens a path toward future studies of agentic surgical understanding.
Figures & tables
| Data Source | Spatial | Temporal | Workflow | # samples | ||||||
|---|---|---|---|---|---|---|---|---|---|---|
| Dataset | Surgery | Tool | Trip. | Loc. | Phase | Action | CVS | Skill | Full | Hard |
| Cholec80 [ 5 ] | Lap. cholecystectomy | ✓ | 42,098 | 50 | ||||||
| CholecT50 [ 1 ] | Lap. cholecystectomy | ✓ | ✓ | ✓ | 16,722 | 220 | ||||
| Cholec80-CVS [ 8 ] | Lap. cholecystectomy | ✓ | 96 | 96 | ||||||
| HeiChole [ 2 ] | Lap. cholecystectomy | ✓ | ✓ | 36,137 | 86 | |||||
| MultiBypass140 [ 9 ] | Lap. Roux-en-Y bypass | ✓ | 171,633 | 50 | ||||||
| Model | Spatial Recognition | Temporal Reasoning | Workflow Assessment | Rank | ||||
|---|---|---|---|---|---|---|---|---|
| Tool Det. | Triplet Det. † | Tool Loc. † | Phase Rec. | Action Rec. | CVS Ass. | Skill Ass. | ||
| macro-F1 | macro-F1 | Acc. | macro-F1 | Acc./ | MAE / | |||
| Proprietary | ||||||||
| GPT-4o-mini [ 16 ] | 22.8 | 1.6 | 0.0 | 24.7 | 27.1 * | 15.6/0.9 | 0.84/0.2 | 7.7 [6.9,9.4] |
| Claude-Haiku-4.5 [ 17 ] | 18.2 | 2.9 | 1.6 | 18.4 | 21.8 | 17.0/-4.8 | 0.99/ 10.9 | 8.9 [8.4,9.9] |
| Open-source ( 10B) | ||||||||
| Model | Spatial Recognition | Temporal Reasoning | Workflow Assessment | ||||
|---|---|---|---|---|---|---|---|
| Tool Det. | Triplet Det. † | Tool Loc. † | Phase Rec. | Action Rec. | CVS Ass. | Skill Ass. | |
| F1 | F1 | Acc. | F1 | ||||
| A. Across-stage comparison I = direct answer, II = agent with tools | |||||||
| Proprietary | |||||||
| GPT-4o-mini [ 16 ] | — | ||||||
| Claude-Haiku-4.5 [ 17 ] | — | ||||||
Appendix figures & tables29 assets
Supplementary material from the paper’s appendix.
Appendix
| Dataset | Annotators | IAA reported? | Metric / value |
|---|---|---|---|
| Cholec80 [ 5 ] | 1 senior surgeon | No | Single annotator; none |
| CholecT50 [ 1 ] | 2 surgeons + 3rd-clinician mediation | No | Consensus only, no metric |
| Cholec80-CVS [ 8 ] | 4 (2 surgeons + 2 residents) | No | Discussion consensus, no metric |
| HeiChole [ 2 ] | Phase: 3 students / Skill: 2 students | Partial | Phase: Fleiss’ (train) / (test); Skill (GOALS): none |
| MultiBypass140 [ 9 ] | 2 board-certified surgeons | Yes | Cohen’s 96% (phase) / 81% (step) — cited from group’s prior ontology work |
| AutoLaparo [ 10 ] | 2 gynaecologists | No | Proofread / cross-checked, no metric |
| Model | Episode flip rate | Mean per-episode score variance |
|---|---|---|
| Qwen2.5-VL-7B | 0.32% | 0.0005 |
| Qwen3-VL-8B | 0.32% | 0.0003 |
| Model | Truncation (%) | Parameter mismatch (%) |
|---|---|---|
| Proprietary (API, unconstrained decoding) | ||
| GPT-4o-mini | 0.0 | |
| Claude-Haiku-4.5 | ||
| Open-source (constrained decoding) | ||
| InternVL2.5-8B | 0.0 | 0.0 |
| Qwen2.5-VL-7B | 0.0 | 0.0 |
| Model | R@1 | R@5 | R@10 | MRR | NDCG@10 | Time (s) | Dim |
| General-purpose dense encoders | |||||||
| BGE-base-en-v1.5 [ 67 ] | 0.448 | 0.696 | 0.771 | 0.553 | 0.605 | 363.6 | 768 |
| BGE-large-en-v1.5 [ 67 ] | 0.475 | 0.706 | 0.770 | 0.574 | 0.622 | 404.8 | 1024 |
| E5-large-v2 [ 68 ] | 0.448 | 0.689 | 0.757 | 0.551 | 0.601 | 947.4 | 1024 |
| GTE-large [ 69 ] | 0.460 | 0.695 | 0.761 | 0.558 | 0.607 | 1,292.9 | 1024 |
| Nomic-embed-text-v1.5 [ 70 ] | 0.432 | 0.668 | 0.741 | 0.530 | 0.581 | 370.8 | 768 |
| Model | CholecT50 | PitVis | Cataracts | AutoLaparo | GraSP |
|---|---|---|---|---|---|
| macro-F1 | macro-F1 | macro-F1 | macro-F1 | macro-F1 | |
| Proprietary | |||||
| GPT-4o-mini | 25.9 | 4.7 | 11.2 | 37.3 | 34.8 |
| Claude-Haiku-4.5 | 24.5 | 5.2 | 7.5 | 31.5 | 22.5 |
| Open-source ( 10B) | |||||
| InternVL2.5-8B | 24.4 | 5.0 | 6.0 | 14.7 | 2.0 |
| Model | Triplet | Instrument | Verb | Target |
|---|---|---|---|---|
| macro-F1 | macro-F1 | macro-F1 | macro-F1 | |
| Proprietary | ||||
| GPT-4o-mini | 1.6 | 13.5 | 9.4 | 8.7 |
| Claude-Haiku-4.5 | 2.9 | 24.7 | 18.1 | 8.4 |
| Open-source ( 10B) | ||||
| InternVL2.5-8B | 1.9 | 17.9 | 10.4 | 7.3 |
| Model | Cataract-101 | Cholec80 | AutoLaparo | GraSP | HeiCo | PitVis | MultiBypass140 |
|---|---|---|---|---|---|---|---|
| Acc. | Acc. | Acc. | Acc. | Acc. | Acc. | Acc. | |
| Proprietary | |||||||
| GPT-4o-mini | 30.0 | 40.1 | 24.1 | 12.5 | 27.0 | 7.3 | 31.9 |
| Claude-Haiku-4.5 | 19.2 | 20.6 | 17.8 | 14.9 | 25.7 | 7.9 | 22.7 |
| Open-source ( 10B) | |||||||
| InternVL2.5-8B | 30.5 | 38.9 | 20.3 | 10.1 | 17.2 | 34.5 | 14.8 |
| Model | Per-key Acc. | MAE | Dict-Acc. | |
|---|---|---|---|---|
| Acc. | MAE | Acc. | ||
| Proprietary | ||||
| GPT-4o-mini | 15.6 | 0.9 | 1.00 | 0.0 |
| Claude-Haiku-4.5 | 17.0 | -4.8 | 1.15 | 1.0 |
| Open-source ( 10B) | ||||
| InternVL2.5-8B | 71.2 | 0.0 | 0.32 | 45.8 |
| Model | MAE | |
|---|---|---|
| MAE | ||
| Proprietary | ||
| GPT-4o-mini | 0.84 | 0.2 |
| Claude-Haiku-4.5 | 0.99 | 10.9 |
| Open-source ( 10B) | ||
| InternVL2.5-8B | 0.86 | -7.2 |
| Model | [email protected] | mIoU | [email protected] | [email protected] |
|---|---|---|---|---|
| [email protected] | mIoU | [email protected] | [email protected] | |
| Proprietary | ||||
| GPT-4o-mini | 0.0 | 1.3 | 0.0 | 0.0 |
| Claude-Haiku-4.5 | 1.6 | 5.8 | 1.7 | 1.6 |
| Open-source ( 10B) | ||||
| InternVL2.5-8B | 11.3 | 13.1 | 11.7 | 11.0 |
| Model | macro-F1 | micro-F1 | Precision | Recall | Set-Acc. |
|---|---|---|---|---|---|
| macro-F1 | F1 | Precision | Recall | Acc. | |
| Proprietary | |||||
| GPT-4o-mini | 27.1 | 51.1 | 41.9 | 65.4 | 24.9 |
| Claude-Haiku-4.5 | 21.8 | 44.6 | 35.1 | 61.2 | 7.1 |
| Open-source ( 10B) | |||||
| InternVL2.5-8B | 11.7 | 10.9 | 10.1 | 11.7 | 7.7 |
| Model | CholecT50 | PitVis | Cataracts | AutoLaparo | GraSP |
|---|---|---|---|---|---|
| macro-F1 | macro-F1 | macro-F1 | macro-F1 | macro-F1 | |
| Proprietary | |||||
| GPT-4o-mini | 7.0 | 0.0 | 23.7 | 18.6 | 12.6 |
| Claude-Haiku-4.5 | 5.3 | 0.9 | 4.4 | 4.5 | 19.8 |
| Open-source ( 10B) | |||||
| InternVL2.5-8B | 8.5 | 4.6 | 11.7 | 9.8 | 0.0 |
| Model | Triplet | Instrument | Verb | Target |
|---|---|---|---|---|
| macro-F1 | macro-F1 | macro-F1 | macro-F1 | |
| Proprietary | ||||
| GPT-4o-mini | 0.0 | 16.5 | 11.4 | 0.0 |
| Claude-Haiku-4.5 | 0.0 | 21.2 | 14.0 | 5.7 |
| Open-source ( 10B) | ||||
| InternVL2.5-8B | 0.3 | 16.7 | 11.1 | 6.6 |
| Model | Cataract-101 | Cholec80 | AutoLaparo | GraSP | HeiCo | PitVis | MultiBypass140 |
|---|---|---|---|---|---|---|---|
| Acc. | Acc. | Acc. | Acc. | Acc. | Acc. | Acc. | |
| Proprietary | |||||||
| GPT-4o-mini | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 4.0 | 0.0 |
| Claude-Haiku-4.5 | 6.0 | 8.0 | 2.0 | 0.0 | 0.0 | 20.0 | 8.0 |
| Open-source ( 10B) | |||||||
| InternVL2.5-8B | 0.0 | 0.0 | 6.0 | 0.0 | 0.0 | 0.0 | 0.0 |
| Model | Per-key Acc. | MAE | Dict-Acc. | |
|---|---|---|---|---|
| Acc. | MAE | Acc. | ||
| Proprietary | ||||
| GPT-4o-mini | 18.8 | -1.0 | 0.96 | 2.1 |
| Claude-Haiku-4.5 | 23.3 | 2.2 | 0.87 | 2.1 |
| Open-source ( 10B) | ||||
| InternVL2.5-8B | 36.5 | -4.8 | 0.86 | 3.1 |
| Model | MAE | |
|---|---|---|
| MAE | ||
| Proprietary | ||
| GPT-4o-mini | 0.68 | 1.4 |
| Claude-Haiku-4.5 | 0.93 | 6.2 |
| Open-source ( 10B) | ||
| InternVL2.5-8B | 0.96 | 5.8 |
| Model | [email protected] | mIoU | [email protected] | [email protected] |
|---|---|---|---|---|
| mIoU | [email protected] | [email protected] | ||
| Proprietary | ||||
| GPT-4o-mini | 5.7 | 5.4 | 5.8 | 5.5 |
| Claude-Haiku-4.5 | 1.6 | 6.1 | 1.7 | 1.6 |
| Open-source ( 10B) | ||||
| InternVL2.5-8B | 0.7 | 4.3 | 0.7 | 0.8 |
| Model | macro-F1 | micro-F1 | Precision | Recall | Set-Acc. |
|---|---|---|---|---|---|
| macro-F1 | F1 | Precision | Recall | Acc. | |
| Proprietary | |||||
| GPT-4o-mini | 13.2 | 20.0 | 20.0 | 20.0 | 20.0 |
| Claude-Haiku-4.5 | 22.8 | 26.7 | 21.2 | 36.0 | 0.0 |
| Open-source ( 10B) | |||||
| InternVL2.5-8B | 18.0 | 30.9 | 28.3 | 34.0 | 32.0 |
| Model | Regime | Tool Det. | Triplet Det. | Tool Loc. | Phase Rec. | Action Rec. | CVS Ass. | Skill Ass. |
|---|---|---|---|---|---|---|---|---|
| F1 | F1 | Acc. | F1 | |||||
| InternVL2.5-8B | Tools off | |||||||
| Tools on | ||||||||
| Lingshu-32B | Tools off | |||||||
| Tools on |
| Spatial Recognition | Temporal Reasoning | Workflow Assessment | ||||||
| Model | Config | Tool Det. | Triplet Det. | Tool Loc. | Phase Rec. | Action Rec. | CVS Ass. | Skill Ass. |
| macro-F1 | macro-F1 | Acc. | macro-F1 | Acc./ | MAE / | |||
| Qwen3-VL-32B-Instruct [ 20 ] | Full | 7.0 | 0.0 | 17.8 | 5.1 | 26.5 | 40.6/7.7 | 0.69/-0.8 |
| w/o perceive() | 8.1 | 5.1 | 12.2 | 4.0 | 24.8 | 47.2/4.5 | 1.06/6.6 | |
| w/o retrieve() | 6.2 | 0.0 | 16.2 | 4.0 | 17.0 | 27.4/4.2 | 0.66/9.7 | |
| Qwen2.5-VL-72B-Instruct [ 19 ] | Full | 6.2 | 0.0 | 11.4 | 4.9 | 25.6 | 49.7/4.8 | 0.89/4.4 |
| Setting | Model | Tool Det. | Triplet Det. | Tool Loc. | Phase Rec. | Action Rec. | CVS Ass. | Skill Ass. |
|---|---|---|---|---|---|---|---|---|
| F1 | F1 | Acc. | F1 | |||||
| Main (budget , full KB, SAM2) | Qwen3-VL-8B | — | ||||||
| Qwen2.5-VL-7B | — | |||||||
| Main w/ SurgWiki -Open | Qwen3-VL-8B | — | ||||||
| Qwen2.5-VL-7B | — | |||||||
| Retrieve budget | Qwen3-VL-8B | — |
| Model | Knowledge base | Phase Acc. | Action F1 | CVS | Skill |
|---|---|---|---|---|---|
| Qwen3-VL-8B | Full KB | ||||
| Qwen3-VL-8B | SurgWiki -Open | ||||
| Qwen2.5-VL-7B | Full KB | ||||
| Qwen2.5-VL-7B | SurgWiki -Open |
| Task | Dataset | Reference Method | Reported Metric | Score |
| Spatial Recognition | ||||
| Tool Detection | CholecT50 | UniSurg [ 44 ] | AP-I | 91.55 |
| PitVis | DKFZ Ensemble [ 13 ] | Macro-F1 | 41.70 | |
| CATARACTS | OVFM [ 46 ] | Micro-AUC | 98.50 | |
| GraSP | TAPIS [ 12 ] | mAP | 94.33 | |
| Triplet Detection | CholecT50 | UniSurg [ 44 ] | AP-IVT | 39.54 |
| Task family | Outcome | Episodes | perceive() | retrieve() | Both | Warning | Calls/ep. |
|---|---|---|---|---|---|---|---|
| Spatial Recognition | Success | 71 | 66.2 | 69.0 | 60.6 | 25.4 | 2.62 |
| Spatial Recognition | Failure | 1129 | 81.9 | 60.2 | 59.5 | 22.8 | 2.28 |
| Temporal Reasoning | Success | 83 | 47.0 | 78.3 | 37.3 | 9.6 | 1.94 |
| Temporal Reasoning | Failure | 1517 | 63.4 | 69.1 | 47.9 | 20.8 | 2.12 |
| Workflow Assessment | Success | 41 | 58.5 | 63.4 | 48.8 | 9.8 | 2.59 |
| Workflow Assessment | Failure | 487 | 61.4 | 63.2 | 54.2 | 22.2 | 3.28 |
Surgical Video Understanding
Surgical video understanding uses computer vision and multimodal AI to interpret procedures, instruments, actions, and events over time.
See all 131 papers