Organizations: Department of Electronic Engineering, The Chinese University of Hong Kong, Hong Kong SAR, China · Department of Mechanical Engineering, The University of Hong Kong, Hong Kong SAR, China · College of Mechatronics and Control Engineering, Shenzhen University, China · Division of Gastrointestinal Surgery, Shenzhen People’s Hospital, China · Department of Electronic and Electrical Engineering, Southern University of Science and Technology
Accurate and stable field-of-view (FoV) guidance is critical for safe and efficient minimally invasive surgery, yet existing approaches often conflate visual attention estimation with downstream camera control or rely on direct object-centric assumptions. In this work, we formulate surgical attention tracking as a spatio-temporal learning problem and model surgeon focus as a dense attention heatmap, enabling continuous and interpretable frame-wise FoV guidance. We propose SurgAtt-Tracker, a holistic framework that robustly tracks surgical attention by exploiting temporal coherence through proposal-level reranking and motion-aware refinement, rather than direct regression. To support systematic training and evaluation, we introduce SurgAtt-1.16M, a large-scale benchmark with a clinically grounded annotation protocol that enables comprehensive heatmap-based attention analysis across procedures and institutions. Extensive experiments on multiple surgical datasets demonstrate that SurgAtt-Tracker consistently achieves state-of-the-art performance and strong robustness under occlusion, multi-instrument interference, and cross-domain settings. Beyond attention tracking, our approach provides a frame-wise FoV guidance signal that can directly support downstream robotic FoV planning and automatic camera control.
Figures & tables
Figure 1: SurgAtt-Tracker enables AI-guided endoscope control by predicting an attention heatmap from raw endoscopic video.
Figure 2: Overview of the SurgAtt-1.16M dataset, illustrating its anatomical coverage, data sources, and unified organization across organs, procedures, and annotation types.
Figure 3: Overview of SurgAtt-Tracker. A frozen detector produces Top- K proposals and multi-scale pyramid features, which are converted into box-aligned embeddings by the Multi-Scale ROI Decoder (B); the AS-Rerank module performs temporal proposal reranking to select the Top-1 attention region (C), and MAA-Refine further refines it using motion-aware geometry (D) and visual evidence to yield the attention heatmap Ht .
Model
NSS ↑
CC ↑
SIM ↑
MSE ↓
MAE ↓
U-Net–based
Unet ( Ronneberger et al., 2015 )
1.051
0.393
0.490
0.079
0.191
TransUNet ( Chen et al., 2021 )
1.947
0.697
0.651
0.033
0.101
UNeXt ( Valanarasu and Patel, 2022 )
1.964
0.712
0.646
0.029
0.109
EMCAD ( Rahman et al., 2024 )
2.203
0.776
0.711
0.0261
0.089
Regression-based
Table 1: Quantitative comparison of attention heatmap prediction on SurgAtt-SZPH dataset using NSS, CC, SIM, MSE, and MAE.
Model
NSS ↑
CC ↑
SIM ↑
MSE ↓
MAE ↓
SurgAtt-AutoLaparo
Zero-shot
EMCAD
1.378
0.593
0.648
0.077
0.214
SalFoM
1.408
0.531
0.594
0.078
0.206
MCITrack
1.440
0.578
0.503
0.074
0.196
YOLOv12
1.551
0.658
0.520
0.059
0.146
Table 2: Zero-shot and fine-tuning performance on the SurgAtt-AutoLaparo and SurgAtt-Hamlyn datasets.
Figure 4: Qualitative comparison of attention heatmaps across diverse surgical scenarios: (A) single-instrument case; (B) multi-instrument without tissue interaction; (C) multi-instrument with tissue interaction; (D) multi-instrument with smoke interference.
Method
Rerank
Refine
Err ↓
IoU ↑
NSS ↑
CC ↑
MAE ↓
Detector-only baselines
YOLOv12 (Conf)
–
–
52.192
0.589
2.285
0.793
0.090
YOLOv12 (MinErr)
–
–
30.408
0.649
2.525
0.857
0.063
YOLOv12 (MaxIoU)
–
–
32.744
0.665
2.437
0.866
0.064
SurgAtt-Tracker
Ours (w/o Refine)
✓
×
36.975
0.627
2.313
0.823
0.079
Table 3: Reports detector-only best-case performance under three rules (Conf/MinErr/MaxIoU), serving as oracle baselines; SurgAtt-Tracker further incorporates AS-Rerank and MAA-Refine.
Top- M
NSS ↑
CC ↑
SIM ↑
MSE ↓
MAE ↓
0
1.973
0.729
0.621
0.047
0.144
3
2.379
0.859
0.792
0.024
0.086
5
2.580
0.871
0.829
0.015
0.051
7
2.195
0.812
0.753
0.033
0.107
Table 4: Ablation on the rerank loss weight (Top- M ).
Sample Gap
NSS ↑
CC ↑
SIM ↑
MSE ↓
MAE ↓
Avg Gap
2.592
0.863
0.788
0.045
0.090
Random Gap
2.339
0.862
0.798
0.034
0.065
Archer Gap
2.580
0.871
0.829
0.015
0.051
Table 5: Impact of temporal gap sampling during training.
Appendix figures & tables9 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 5: Construction pipeline of the SurgAtt-SZPH dataset. Raw laparoscopic videos are curated into high-quality surgical clips via optical-flow–based operation analysis and expert screening. Videos are sampled at 25 fps and grouped into five representative surgical scenes. During annotation, surgeons mark attention regions with bounding boxes, which are converted into continuous attention heatmaps for supervision. The resulting dataset provides dense, high-fidelity attention annotations across diverse surgical scenarios.
Dataset Name
Data Source
Surgical Organ
Procedure
Image Size
Train
Test
Total
SurgAtt-SZPH
Anonymous Hospital
Rectum, Stomach
Rectal and gastric cancer resections
1920x1080
800,000
200,000
1,000,000
SurgAtt-AutoLaparo
AutoLaparo
Uterus
Hysterectomy
1920x1080
60,000
15,000
75,000
SurgAtt-Hamlyn
Hamlyn Centre
Kidney
Nephrectomy
384x192
68,478
14,380
82,858
SurgAtt-1.16M
Aggregated
Multiple
Multiple laparoscopic procedures
Multiple
928,478
229,380
1,157,858
Appendix
Table 6: SurgAtt-1.16M forms a complete benchmark by aggregating three subsets under a unified annotation protocol, and summarizes the data source, organ, procedure, image resolution, and the number of frames in the train/test splits.
Category
Distal Gastric
Proximal Gastric
High Rectum
Middle Rectum
Low Rectum
Total
Split
Train
Test
Train
Test
Train
Test
Train
Test
Train
Test
Patients
4
1
4
1
4
1
4
1
4
1
25
Videos
20
5
20
5
20
5
20
5
20
5
125
Frames
180K
45K
180K
45K
180K
45K
180K
45K
180K
45K
1.12M
Appendix
Table 7: Two-stage curation and annotation pipeline for the SurgAtt-SZPH (SurgAtt-1M) subset. We first mine informative surgical segments via optical-flow–based motion filtering (A), then clinicians select high-quality clips under four clinical criteria (B), and finally surgeons annotate attention boxes that are converted into attention heatmaps at 25 fps across five surgical sites (C).
Figure 6: Visualization of heatmap generation.
Figure 7: Radar plots on five subsets (Distal/Proximal Gastric Cancer, Low/Middle/High Rectum) comparing representative SOTA methods over NSS/CC/SIM (higher better) and MSE/MAE (lower better).
Figure 8: Qualitative comparison of attention heatmap predictions on SurgAtt-SZPH across five surgical scenarios. SurgAtt-Tracker produces sharper and more stable attention aligned with clinically relevant regions compared with representative SOTA baselines.
Figure 9: Qualitative comparison on SurgAtt-Hamlyn under zero-shot and fine-tuning settings. SurgAtt-Tracker shows more stable and better-localized attention than RT-DETRv2 and YOLOv12 across both regimes.
Figure 10: Qualitative comparison on SurgAtt-AutoLaparo under zero-shot and fine-tuning settings. Our method maintains compact and consistent attention maps under domain shift and further improves after fine-tuning.
Figure 11: Top- K analysis of proposal ranking quality. AS-Rerank reorders the same YOLOv12 candidate pool using temporal consistency, increasing the likelihood that oracle-best proposals (Min-Err / Max-IoU) appear in the first few ranks.
Department of Electronic Engineering, The Chinese University of Hong Kong, Hong Kong SAR 999077, China · Shenzhen Loop Area Institute, Shen Zhen 518055, China · College of Mechatronics and Engineering, Shenzhen University, Shenzhen 518060, China +2