Organizations: UCL Hawkes Institute, University College London, London, UK · Visual Understanding Research Group, Department of Informatics, King’s College London, London, UK · School of Medicine, Nankai University, Tianjin, China · Division of Informatics, Imaging and Data Sciences, University of Manchester, Manchester, UK
Surgical spatio-temporal grounding (STG) requires locating, at each video time specified by a procedural question, the object that the question asks about. Existing approaches face a trade-off: vision language models understand the question context but produce imprecise coordinates, whereas open-set detectors provide localized candidate boxes whose confidence does not reflect which box answers the question. We introduce RefineRank, which closes this gap at the candidate-box level. A compact trainable module, RefineNet, combines the language and regional features of a frozen medical vision language model with the proposals of a frozen open-set detector: it predicts a bounded coordinate correction and a quality score for every candidate box, and a fixed decoding rule returns the original or refined box with the highest score. On the MedVidBench Official Rankings (Verified), RefineRank records 0.421 STG mIoU, the highest displayed STG score, while its global multi-metric rank is 11. In a controlled evaluation on separate training and evaluation videos, coordinate correction raises the candidate oracle upper bound from 0.6772 to 0.7302, and ranking the joint pool of original and refined candidates by their RefineNet scores improves STG mIoU from 0.2719 to 0.4534, whereas separately trained selectors over the same pool reach at most 0.4186. These results show that a small box-level module can reconcile question understanding with precise localization without retraining either backbone. Code is available at https://github.com/linzhe001/RefineRank.
Figures & tables
Figure 1: The complete RefineRank pipeline. The frozen MedVLM reads the complete timestamped question and sampled frames. It supplies the detector query, qlast , and two visual grids. GroundingDINO applies the query to the requested frames and provides original boxes and their metadata fdino,i . ROI pooling produces riinter and rifinal for the trainable RefineNet module. Its ranking head hrank outputs logits si and s^i , while the box head hbox outputs the correction δi . The final pool contains original boxes scored by si and refined boxes scored by s^i . At each requested time, the fixed decoding rule returns the candidate with the highest corresponding score; it has no learned parameters and no separate selector module is used.
Videos
Examples
Dataset
Train
Eval
Train
Eval
CholecTrack20
7
2
807
212
CoPESD
5
2
134
118
EgoSurgery
11
3
1405
624
Total
23
7
2,346
954
Table 1: Video-separated split of the controlled study. Each example is one requested timestamp with its sampled frame, question, and target box. The closed MedVidBench leaderboard test † split is not included.
MedVidBench Leaderboard
STG mIoU
uAI-NEXUS-MedVLM-1.0a-7B-RL-STG_final (ours)
0.421
uAI-NEXUS-MedVLM-1.0a-7B-RL [ 16 ]
0.202
uAI-NEXUS-MedVLM-1.0c-4B-SFT
0.190
uAI-NEXUS-MedVLM-1.0a-7B-SFT
0.177
uAI-NEXUS-MedVLM-1.0b-4B-RL
0.176
Table 2: MedVidBench Official Rankings (Verified), accessed 15 August 2026. Entries are ordered by STG mIoU, and the top five are shown.
Method
Dataset STG mIoU
Selected avg.
Oracle avg.
Cholec
CoPESD
Ego
MedVLM [ 16 ]
0.0929
0.0170
0.0190
0.0429
0.0429
MedVLM + GroundingDINO [ 16 , 11 ]
0.1350
0.3371
0.3435
0.2719
0.6772
RefineRank (ours)
0.3960
0.5972
0.3671
0.4534
0.7302
Table 3: Controlled STG mIoU on separate training and evaluation videos. Dataset averages weight the three datasets equally.
Candidate scoring rule
Dataset STG mIoU
Dataset avg.
Cholec
CoPESD
Ego
ExtraTrees [ 4 ]
0.2842
0.4243
0.2781
0.3289
MLP
0.3373
0.5767
0.3417
0.4186
Transformer encoder [ 17 ]
0.3447
0.5415
0.3261
0.4041
Native RefineNet scores (RefineRank)
0.3960
0.5972
0.3671
0.4534
Table 4: Selector study on the same joint pool of original and refined candidates generated after RefineNet training. RefineNet is frozen for every row. RefineRank trains no additional selector: its decoding rule, which has no learned parameters, uses RefineNet’s own scores. ExtraTrees, MLP, and the Transformer encoder are separately trained replacement scoring rules and are not RefineRank modules. None of them receives the RefineNet scores as an input feature. The MLP combines the metadata, query, and block-23 regional features; it is the same configuration as the block-23 row of Table 5 and therefore reports the same values.
Candidate MLP input
Dataset STG mIoU
Dataset avg.
Cholec
CoPESD
Ego
mean
fdino
0.1512
0.3578
0.3213
0.2767
fdino + qlast
0.2528
0.6069
0.3536
0.4044
fdino + qlast + visual block 7
0.3779
0.5488
0.3260
0.4176
fdino + qlast + visual block 15
0.3537
0.5411
0.3558
0.4169
fdino + qlast + visual block 23
0.3373
0.5767
0.3417
0.4186
Table 5: Input feature study with a separately trained box scoring MLP. The candidate pool is the joint pool of original and refined boxes generated by the trained and frozen RefineNet, and every row re-scores all of its candidates after 10 epochs of MLP training. The RefineNet scores are used neither as input features nor at selection time.
Figure 2: Spatial responses across frozen MedVLM visual blocks 7, 15, 23, and 31. For this visualization only, target region features are pooled after the multimodal forward and compared with every spatial token. Each map is normalized independently and is not an attention map or a causal explanation.
Figure 3: Six representative evaluation examples. Columns show direct MedVLM, MedVLM + GroundingDINO, RefineRank, and target boxes. The bottom two rows show shared failures. A displayed RefineRank prediction may therefore be either an original box or its refined counterpart, whichever received the higher score.
King Abdullah University of Science and Technology (KAUST), Saudi Arabia · Gaash Research Lab, National Institute of Technology Srinagar, India · Thapar Institute of Engineering and Technology, India +1