Organizations: Department of Data Science, Michigan Technological University, Houghton, MI, USA · Department of Computer Science, Kennesaw State University, Marietta, GA, 30060, USA · Department of Computer Science, Michigan Technological University, Houghton, MI, USA · Department of Physics and Anatomy, Stony Brook University Hospital, Stony Brook, NY, USA · Department of Radiation and Cellular Oncology, University of Chicago, Chicago, IL, USA
Purpose: Surgical operative reports support procedural documentation, surgical training, performance assessment, and workflow review. However, extracting meaningful information often requires extensive manual review and clinical expertise. Automated surgical scene descriptions could reduce this burden by leveraging recurring surgical activities and prior knowledge. Methods: We propose GRAC (Graph-Based Retrieval-Augmented Captioning), a framework that represents surgical scenes as graphs capturing relationships among instruments, actions, and anatomical structures. Given an input scene graph, GRAC learns to retrieve relevant graph-caption pairs from training data. It compares instrument-tissue interactions between retrieved examples and the input scene, using comparison features to condition retrieved-caption representations during generation. Results: On the EndoVis-18 benchmark, GRAC achieved a CIDEr-D of 5.88, compared with 4.23 for SGT and 3.36 for the strongest conventional Transformer baseline. Exact instrument-action-target triplet-set accuracy reached 52.6%, compared with 32.7% for SGT and 26.2% for the strongest conventional baseline. These results demonstrate improvements in language quality and factual agreement. On TORS, GRAC achieved a CIDEr-D of 5.06, compared with 3.80 for the strongest baseline. Conclusion: GRAC demonstrates that combining structured surgical scene representations with retrieved examples improves surgical scene captioning and produces more contextually meaningful descriptions.
Figures & tables
Figure 1: Overview of GRAC. The input surgical scene graph is encoded to capture surgical interactions and retrieve similar prior scene graphs with their corresponding captions. GRAC then compares the retrieved graphs with the current scene and uses these comparisons to guide caption generation through a Transformer decoder.
Figure 2: Comparison of GRAC with interaction-feature-augmented baselines (†) and SGT on EndoVis-18 (392 frames). Bars show the mean over three seeds, with error bars for SD; in (a), circles are individual seeds. (a) CIDEr-D; *** p < 0.001 vs GRAC (paired frame-level bootstrap, B = 10,000, Holm-corrected). (b) Language metrics. (c) Instrument hallucination (lower is better). (d) Triplet-level factual correctness.
(a) Language-generation metrics
BLEU
Model
1
2
3
4
METEOR
ROUGE-L
CIDEr-D
Transformer †
0.653±0.021
0.604±0.019
0.556±0.017
0.514±0.016
0.458±0.009
0.697±0.018
3.356±0.144
M 2 Transformer †
0.645±0.048
0.591±0.047
0.541±0.051
0.501±0.057
0.444±0.012
0.688±0.004
3.116±0.356
AoA †
0.525±0.064
0.466±0.069
0.409±0.070
0.357±0.077
0.389±0.094
0.646±0.035
2.220±0.283
GET †
0.621±0.013
0.570±0.010
0.523±0.007
0.482±0.006
0.455±0.014
0.686±0.014
3.239±0.286
Table 1: Performance on EndoVis-18 (392 frames). (a) Language-generation metrics; values are mean (first line) and ± SD (second line) over three seeds, and higher is better for all metrics. (b) Paired frame-level bootstrap comparison of GRAC with each baseline in CIDEr-D (GRAC minus baseline; 10,000 resamples; two-sided, Holm-adjusted p-values).
Model
Exact set
Triplet prec.
Triplet recall
Triplet F1
Instr. halluc. ↓
Transformer †
0.253±0.023
0.473±0.020
0.569±0.023
0.516±0.021
0.056±0.019
M 2 Transformer †
0.221±0.059
0.437±0.034
0.530±0.005
0.479±0.022
0.041±0.010
AoA †
0.090±0.093
0.357±0.079
0.401±0.066
0.369±0.017
0.080±0.061
GET †
0.262±0.043
0.457±0.032
0.573±0.054
0.508±0.041
0.052±0.008
SGT
0.327±0.114
0.479±0.083
0.547±0.137
0.510±0.107
0.047±0.003
GRAC (ours)
0.526±0.008
0.616±0.010
0.657±0.013
0.636±0.012
0.037±0.015
Table 2: Triplet-based factual agreement on EndoVis-18 (392 frames). Values are mean ± SD over three seeds. Higher is better except instrument hallucination ( ↓ ).
Figure 3: Ablation of GRAC on EndoVis-18 (392 frames). Bars show CIDEr-D (mean ± SD over three seeds; dots are individual seeds; y-axis starts at 4.0). p-values compare each variant with GRAC using a paired frame-level bootstrap (B = 10,000), Holm-corrected across comparisons.
(a) Component configuration
Variant
Edge features
Retrieval
Graph comparison
Compat. loss
Query frames
No retrieval or edges
–
–
–
–
t
Graph only (with edges)
✓
–
–
–
t
Caption retrieval
✓
✓
–
–
t
Comparison, no compat. loss
✓
✓
✓
–
t
Temporal context
✓
✓
✓
✓
t−1,t,t+1
Table 3: GRAC ablations on EndoVis-18 (392 frames). (a) Component configuration of each variant; a check mark denotes an enabled component and a dash an excluded component; the default input is frame t . (b) Language-generation results; values are mean ± SD over three seeds, and higher is better.
Method
CIDEr-D
BLEU-4
Exact set
Triplet F1
Instr. halluc. ↓
Top-1 Copy
5.332±0.150
0.696±0.013
0.461±0.012
0.576±0.013
0.060±0.011
Caption Retrieval
5.805±0.142
0.710±0.013
0.520±0.015
0.630±0.013
0.049±0.010
GRAC (ours)
5.883±0.085
0.715±0.006
0.526±0.008
0.636±0.012
0.037±0.015
Table 4: Retrieval-only versus generated captions on EndoVis-18 (392 frames). Values are mean ± SD across three seeds; instrument hallucination is lower-is-better.
Figure 4: Qualitative examples from the EndoVis-18 benchmark (sequences 1, 5 and 16). One frame was selected per category, at the median position among the frames that met the category definition: (a) correct Top-1 retrieval, (b) Top-1 retrieval with a wrong action that GRAC corrects, (c) wrong Top-1 retrieval with all models failing, and (d) a baseline naming an instrument absent from the scene. Bold red marks an instrument, action or target that disagrees with the reference; when the instrument is wrong, the whole triplet is marked. ✓/✗ indicate whether the predicted triplet set matches the reference.
(a) Language generation
Model
BLEU-4
CIDEr-D
Δ CIDEr-D
95% CI
Adjusted p-value
Transformer †
0.396±0.011
3.80±1.35
+1.26
[0.31,2.25]
0.010
M 2 Transformer †
0.363±0.061
3.67±1.69
+1.39
[0.73,2.08]
<0.001
AoA †
0.359±0.045
3.61±1.19
+1.46
[0.86,2.11]
<0.001
GET †
0.307±0.019
1.97±0.24
+3.09
[1.51,4.70]
<0.001
SGT
0.387±0.038
3.73±1.46
+1.34
[0.59,2.11]
<0.001
Table 5: TORS test-set performance (36 frames). Values are mean ± SD over three seeds. (a) Language generation, including the CIDEr-D comparison with GRAC (GRAC minus baseline; paired frame-level bootstrap, 10,000 resamples; Holm-adjusted p-values). (b) Triplet-based factual agreement ( ↓ : lower is better).
Current surgical scene graph generation methods depend on dense multi-modal supervision and specialized hardware (synchronized RGB-D sensors, calibration rigs), making dataset construction expensive and restricting all existing benchmarks to simulated environments. We propose SAGE-OR, a feature-centric framework that replaces the traditional detect-then-reason paradigm with a decoupled representation-reasoning paradigm in which localization is derived from frozen foundation models, encoded implicitly in pre-computed features, and used without any localization supervision, while a lightweight graph transformer performs relational reasoning over cached features. We employ a semi-supervised formulation with general-purpose segmentation prompts to eliminate localization supervision while enabling unsupervised context augmentation through additional prompt-driven entities, such as hands, which are absent from annotations. General-purpose prompts are used to induce near-perfect recall, while precision is delegated to downstream attention-based reasoning, enabling simple adaptation to new entities via prompt-level modification. This design enables a lightweight 15M-parameter graph transformer that trains in 1.4 hours and runs relational inference at ∼1ms per frame with peak memory under 2GB, suitable for edge hardware used in the operating room; feature extraction runs offline as a separate caching stage (4.27s per frame). On the 4D-OR benchmark, the core model achieves 76% F1, matching the fully supervised 4D-OR baseline while eliminating all localization annotations, and unsupervised hand augmentation raises this to 86%, within 4 points of state-of-the-art (SOTA) methods requiring dense multi-modal supervision, providing a practical pathway for adaptation to new surgical settings without annotation other than relationship and class labels.
Brandon Leblanc, Charalambos Poullis
The Immersive and Creative Technologies Lab Concordia University Montreal, CA
Clinically reliable perception of surgical scenes is essential for advancing intelligent, context-aware intraoperative assistance such as instrument handoff guidance, collision avoidance, and workflow-aware robotic support. Existing surgical tool benchmarks primarily evaluate category-level segmentation, requiring models to detect all instances of predefined instrument classes. However, real-world clinical decisions often require resolving references to a specific instrument instance based on its functional role, spatial relation, or anatomical interaction capabilities not captured by current evaluation paradigms. We introduce GroundedSurg, the first language-conditioned, instance-level surgical grounding benchmark. Each instance pairs a surgical image with a natural-language description targeting a single instrument, accompanied by structured spatial grounding annotations including bounding boxes and point-level anchors. The dataset spans ophthalmic, laparoscopic, robotic, and open procedures, encompassing diverse instrument types, imaging conditions, and operative complexities. By jointly evaluating linguistic reference resolution and pixel-level localization, GroundedSurg enables a systematic and realistic evaluation of vision-language models in clinically realistic multi-instrument scenes. Extensive experiments demonstrate substantial performance gaps across modern segmentation and VLMs, highlighting the urgent need for clinically grounded vision-language reasoning in surgical AI systems. Code and data are publicly available at https://github.com/gaash-lab/GroundedSurg
Tajamul Ashraf, Abrar Ul Riyaz, Wasif Tak +4
King Abdullah University of Science and Technology (KAUST), Saudi Arabia · Gaash Research Lab, National Institute of Technology Srinagar, India · Thapar Institute of Engineering and Technology, India +1
Surgical scene understanding is a cornerstone of computer-assisted intervention. While recent advances, particularly in surgical image segmentation, have driven progress, real-world clinical applications require a more holistic understanding that jointly captures procedural context, semantic reasoning, and precise visual grounding. However, existing approaches typically address these components in isolation, leading to fragmented representations and limited semantic consistency. To address this limitation, we propose SurgMLLM, a unified surgical scene understanding framework that bridges high-level reasoning and low-level visual grounding within a single model. Given surgical videos, SurgMLLM fine-tunes a multimodal large language model (MLLM) to support structured interpretability reasoning, which is used to jointly model phases, instrument-verb-target (IVT) triplets, and triplet-entity segmentation tokens. These tokens are then temporally aggregated and serve as prompts for a segmentation network, enabling accurate pixel-wise grounding of triplet instruments and targets. The entire framework is trained end-to-end with a unified objective that couples language-based reasoning supervision with visual grounding losses, promoting coherent cross-task learning and clinically consistent scene representations. To facilitate unified evaluation, we introduce CholecT45-Scene, extending CholecT45 dataset with 64,299 frames of pixel-level mask annotations for instruments and targets, aligned with existing triplet labels. Extensive experiments show that SurgMLLM significantly advances surgical scene understanding, improving the primary triplet recognition metric AP_IVT from 40.7% to 46.0% and consistently outperforming prior methods in phase recognition and segmentation. These results highlight the effectiveness of unified reasoning-and-grounding for reliable, context-aware surgical assistance.
Jincai Huang, Shihao Zou, Yuchen Guo +5
Southern University of Science and Technology · Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences · Northwestern University +4