Organizations: Department of Data Science, Michigan Technological University, Houghton, MI, USA · Department of Computer Science, Kennesaw State University, Marietta, GA, 30060, USA · Department of Computer Science, Michigan Technological University, Houghton, MI, USA · Department of Physics and Anatomy, Stony Brook University Hospital, Stony Brook, NY, USA · Department of Radiation and Cellular Oncology, University of Chicago, Chicago, IL, USA
Purpose: Surgical operative reports support procedural documentation, surgical training, performance assessment, and workflow review. However, extracting meaningful information often requires extensive manual review and clinical expertise. Automated surgical scene descriptions could reduce this burden by leveraging recurring surgical activities and prior knowledge. Methods: We propose GRAC (Graph-Based Retrieval-Augmented Captioning), a framework that represents surgical scenes as graphs capturing relationships among instruments, actions, and anatomical structures. Given an input scene graph, GRAC learns to retrieve relevant graph-caption pairs from training data. It compares instrument-tissue interactions between retrieved examples and the input scene, using comparison features to condition retrieved-caption representations during generation. Results: On the EndoVis-18 benchmark, GRAC achieved a CIDEr-D of 5.88, compared with 4.23 for SGT and 3.36 for the strongest conventional Transformer baseline. Exact instrument-action-target triplet-set accuracy reached 52.6%, compared with 32.7% for SGT and 26.2% for the strongest conventional baseline. These results demonstrate improvements in language quality and factual agreement. On TORS, GRAC achieved a CIDEr-D of 5.06, compared with 3.80 for the strongest baseline. Conclusion: GRAC demonstrates that combining structured surgical scene representations with retrieved examples improves surgical scene captioning and produces more contextually meaningful descriptions.
Figures & tables
Figure 1: Overview of GRAC. The input surgical scene graph is encoded to capture surgical interactions and retrieve similar prior scene graphs with their corresponding captions. GRAC then compares the retrieved graphs with the current scene and uses these comparisons to guide caption generation through a Transformer decoder.
Figure 2: Comparison of GRAC with interaction-feature-augmented baselines (†) and SGT on EndoVis-18 (392 frames). Bars show the mean over three seeds, with error bars for SD; in (a), circles are individual seeds. (a) CIDEr-D; *** p < 0.001 vs GRAC (paired frame-level bootstrap, B = 10,000, Holm-corrected). (b) Language metrics. (c) Instrument hallucination (lower is better). (d) Triplet-level factual correctness.
(a) Language-generation metrics
BLEU
Model
1
2
3
4
METEOR
ROUGE-L
CIDEr-D
Transformer †
0.653±0.021
0.604±0.019
0.556±0.017
0.514±0.016
0.458±0.009
0.697±0.018
3.356±0.144
M 2 Transformer †
0.645±0.048
0.591±0.047
0.541±0.051
0.501±0.057
0.444±0.012
0.688±0.004
3.116±0.356
AoA †
0.525±0.064
0.466±0.069
0.409±0.070
0.357±0.077
0.389±0.094
0.646±0.035
2.220±0.283
GET †
0.621±0.013
0.570±0.010
0.523±0.007
0.482±0.006
0.455±0.014
0.686±0.014
3.239±0.286
Table 1: Performance on EndoVis-18 (392 frames). (a) Language-generation metrics; values are mean (first line) and ± SD (second line) over three seeds, and higher is better for all metrics. (b) Paired frame-level bootstrap comparison of GRAC with each baseline in CIDEr-D (GRAC minus baseline; 10,000 resamples; two-sided, Holm-adjusted p-values).
Model
Exact set
Triplet prec.
Triplet recall
Triplet F1
Instr. halluc. ↓
Transformer †
0.253±0.023
0.473±0.020
0.569±0.023
0.516±0.021
0.056±0.019
M 2 Transformer †
0.221±0.059
0.437±0.034
0.530±0.005
0.479±0.022
0.041±0.010
AoA †
0.090±0.093
0.357±0.079
0.401±0.066
0.369±0.017
0.080±0.061
GET †
0.262±0.043
0.457±0.032
0.573±0.054
0.508±0.041
0.052±0.008
SGT
0.327±0.114
0.479±0.083
0.547±0.137
0.510±0.107
0.047±0.003
GRAC (ours)
0.526±0.008
0.616±0.010
0.657±0.013
0.636±0.012
0.037±0.015
Table 2: Triplet-based factual agreement on EndoVis-18 (392 frames). Values are mean ± SD over three seeds. Higher is better except instrument hallucination ( ↓ ).
Figure 3: Ablation of GRAC on EndoVis-18 (392 frames). Bars show CIDEr-D (mean ± SD over three seeds; dots are individual seeds; y-axis starts at 4.0). p-values compare each variant with GRAC using a paired frame-level bootstrap (B = 10,000), Holm-corrected across comparisons.
(a) Component configuration
Variant
Edge features
Retrieval
Graph comparison
Compat. loss
Query frames
No retrieval or edges
–
–
–
–
t
Graph only (with edges)
✓
–
–
–
t
Caption retrieval
✓
✓
–
–
t
Comparison, no compat. loss
✓
✓
✓
–
t
Temporal context
✓
✓
✓
✓
t−1,t,t+1
Table 3: GRAC ablations on EndoVis-18 (392 frames). (a) Component configuration of each variant; a check mark denotes an enabled component and a dash an excluded component; the default input is frame t . (b) Language-generation results; values are mean ± SD over three seeds, and higher is better.
Method
CIDEr-D
BLEU-4
Exact set
Triplet F1
Instr. halluc. ↓
Top-1 Copy
5.332±0.150
0.696±0.013
0.461±0.012
0.576±0.013
0.060±0.011
Caption Retrieval
5.805±0.142
0.710±0.013
0.520±0.015
0.630±0.013
0.049±0.010
GRAC (ours)
5.883±0.085
0.715±0.006
0.526±0.008
0.636±0.012
0.037±0.015
Table 4: Retrieval-only versus generated captions on EndoVis-18 (392 frames). Values are mean ± SD across three seeds; instrument hallucination is lower-is-better.
Figure 4: Qualitative examples from the EndoVis-18 benchmark (sequences 1, 5 and 16). One frame was selected per category, at the median position among the frames that met the category definition: (a) correct Top-1 retrieval, (b) Top-1 retrieval with a wrong action that GRAC corrects, (c) wrong Top-1 retrieval with all models failing, and (d) a baseline naming an instrument absent from the scene. Bold red marks an instrument, action or target that disagrees with the reference; when the instrument is wrong, the whole triplet is marked. ✓/✗ indicate whether the predicted triplet set matches the reference.
(a) Language generation
Model
BLEU-4
CIDEr-D
Δ CIDEr-D
95% CI
Adjusted p-value
Transformer †
0.396±0.011
3.80±1.35
+1.26
[0.31,2.25]
0.010
M 2 Transformer †
0.363±0.061
3.67±1.69
+1.39
[0.73,2.08]
<0.001
AoA †
0.359±0.045
3.61±1.19
+1.46
[0.86,2.11]
<0.001
GET †
0.307±0.019
1.97±0.24
+3.09
[1.51,4.70]
<0.001
SGT
0.387±0.038
3.73±1.46
+1.34
[0.59,2.11]
<0.001
Table 5: TORS test-set performance (36 frames). Values are mean ± SD over three seeds. (a) Language generation, including the CIDEr-D comparison with GRAC (GRAC minus baseline; paired frame-level bootstrap, 10,000 resamples; Holm-adjusted p-values). (b) Triplet-based factual agreement ( ↓ : lower is better).
King Abdullah University of Science and Technology (KAUST), Saudi Arabia · Gaash Research Lab, National Institute of Technology Srinagar, India · Thapar Institute of Engineering and Technology, India +1