Organizations: UCL Hawkes Institute, University College London, London, UK · Visual Understanding Research Group, Department of Informatics, King’s College London, London, UK · School of Medicine, Nankai University, Tianjin, China · Division of Informatics, Imaging and Data Sciences, University of Manchester, Manchester, UK
Surgical spatio-temporal grounding (STG) requires locating, at each video time specified by a procedural question, the object that the question asks about. Existing approaches face a trade-off: vision language models understand the question context but produce imprecise coordinates, whereas open-set detectors provide localized candidate boxes whose confidence does not reflect which box answers the question. We introduce RefineRank, which closes this gap at the candidate-box level. A compact trainable module, RefineNet, combines the language and regional features of a frozen medical vision language model with the proposals of a frozen open-set detector: it predicts a bounded coordinate correction and a quality score for every candidate box, and a fixed decoding rule returns the original or refined box with the highest score. On the MedVidBench Official Rankings (Verified), RefineRank records 0.421 STG mIoU, the highest displayed STG score, while its global multi-metric rank is 11. In a controlled evaluation on separate training and evaluation videos, coordinate correction raises the candidate oracle upper bound from 0.6772 to 0.7302, and ranking the joint pool of original and refined candidates by their RefineNet scores improves STG mIoU from 0.2719 to 0.4534, whereas separately trained selectors over the same pool reach at most 0.4186. These results show that a small box-level module can reconcile question understanding with precise localization without retraining either backbone. Code is available at https://github.com/linzhe001/RefineRank.
Figures & tables
Figure 1: The complete RefineRank pipeline. The frozen MedVLM reads the complete timestamped question and sampled frames. It supplies the detector query, qlast , and two visual grids. GroundingDINO applies the query to the requested frames and provides original boxes and their metadata fdino,i . ROI pooling produces riinter and rifinal for the trainable RefineNet module. Its ranking head hrank outputs logits si and s^i , while the box head hbox outputs the correction δi . The final pool contains original boxes scored by si and refined boxes scored by s^i . At each requested time, the fixed decoding rule returns the candidate with the highest corresponding score; it has no learned parameters and no separate selector module is used.
Videos
Examples
Dataset
Train
Eval
Train
Eval
CholecTrack20
7
2
807
212
CoPESD
5
2
134
118
EgoSurgery
11
3
1405
624
Total
23
7
2,346
954
Table 1: Video-separated split of the controlled study. Each example is one requested timestamp with its sampled frame, question, and target box. The closed MedVidBench leaderboard test † split is not included.
MedVidBench Leaderboard
STG mIoU
uAI-NEXUS-MedVLM-1.0a-7B-RL-STG_final (ours)
0.421
uAI-NEXUS-MedVLM-1.0a-7B-RL [ 16 ]
0.202
uAI-NEXUS-MedVLM-1.0c-4B-SFT
0.190
uAI-NEXUS-MedVLM-1.0a-7B-SFT
0.177
uAI-NEXUS-MedVLM-1.0b-4B-RL
0.176
Table 2: MedVidBench Official Rankings (Verified), accessed 15 August 2026. Entries are ordered by STG mIoU, and the top five are shown.
Method
Dataset STG mIoU
Selected avg.
Oracle avg.
Cholec
CoPESD
Ego
MedVLM [ 16 ]
0.0929
0.0170
0.0190
0.0429
0.0429
MedVLM + GroundingDINO [ 16 , 11 ]
0.1350
0.3371
0.3435
0.2719
0.6772
RefineRank (ours)
0.3960
0.5972
0.3671
0.4534
0.7302
Table 3: Controlled STG mIoU on separate training and evaluation videos. Dataset averages weight the three datasets equally.
Candidate scoring rule
Dataset STG mIoU
Dataset avg.
Cholec
CoPESD
Ego
ExtraTrees [ 4 ]
0.2842
0.4243
0.2781
0.3289
MLP
0.3373
0.5767
0.3417
0.4186
Transformer encoder [ 17 ]
0.3447
0.5415
0.3261
0.4041
Native RefineNet scores (RefineRank)
0.3960
0.5972
0.3671
0.4534
Table 4: Selector study on the same joint pool of original and refined candidates generated after RefineNet training. RefineNet is frozen for every row. RefineRank trains no additional selector: its decoding rule, which has no learned parameters, uses RefineNet’s own scores. ExtraTrees, MLP, and the Transformer encoder are separately trained replacement scoring rules and are not RefineRank modules. None of them receives the RefineNet scores as an input feature. The MLP combines the metadata, query, and block-23 regional features; it is the same configuration as the block-23 row of Table 5 and therefore reports the same values.
Candidate MLP input
Dataset STG mIoU
Dataset avg.
Cholec
CoPESD
Ego
mean
fdino
0.1512
0.3578
0.3213
0.2767
fdino + qlast
0.2528
0.6069
0.3536
0.4044
fdino + qlast + visual block 7
0.3779
0.5488
0.3260
0.4176
fdino + qlast + visual block 15
0.3537
0.5411
0.3558
0.4169
fdino + qlast + visual block 23
0.3373
0.5767
0.3417
0.4186
Table 5: Input feature study with a separately trained box scoring MLP. The candidate pool is the joint pool of original and refined boxes generated by the trained and frozen RefineNet, and every row re-scores all of its candidates after 10 epochs of MLP training. The RefineNet scores are used neither as input features nor at selection time.
Figure 2: Spatial responses across frozen MedVLM visual blocks 7, 15, 23, and 31. For this visualization only, target region features are pooled after the multimodal forward and compared with every spatial token. Each map is normalized independently and is not an attention map or a causal explanation.
Figure 3: Six representative evaluation examples. Columns show direct MedVLM, MedVLM + GroundingDINO, RefineRank, and target boxes. The bottom two rows show shared failures. A displayed RefineRank prediction may therefore be either an original box or its refined counterpart, whichever received the higher score.
Referring surgical video segmentation requires segmenting a target instrument or tissue region across video frames according to a natural language expression. Recent Segment Anything Model 2 (SAM2) based two-stage methods (e.g., ReSurgSAM2) first ground the referred target in an initial or selected frame, then propagate the selected mask via tracking. Although effective, their performance is highly sensitive to the quality of the initial grounded mask: once an incorrect anchor is selected, subsequent tracking tends to propagate the error. This issue is especially challenging in surgical videos due to visually similar instruments, occlusion, and complex tissue-tool interactions. To address this issue, we propose ReGround-Surg, a lightweight reliability-guided anchor grounding framework to improve SAM2-based referring surgical video segmentation. It first predicts a text-conditioned spatial reliability map from the referring expression and current-frame visual features. The map is then reused in two complementary branches: a Gated Side Adapter enhances expression-relevant visual regions before text-to-vision fusion, while a Reliability-Weighted Vision-to-Text Attention module suppresses off-target visual evidence during prompt-token aggregation. Experiments on Ref-EndoVis17 and Ref-EndoVis18 show consistent improvements over state-of-the-art methods across three evaluation splits with negligible speed reduction. Code is publicly available at https://github.com/JiaxinWen1/ReGround-Surg.
Clinically reliable perception of surgical scenes is essential for advancing intelligent, context-aware intraoperative assistance such as instrument handoff guidance, collision avoidance, and workflow-aware robotic support. Existing surgical tool benchmarks primarily evaluate category-level segmentation, requiring models to detect all instances of predefined instrument classes. However, real-world clinical decisions often require resolving references to a specific instrument instance based on its functional role, spatial relation, or anatomical interaction capabilities not captured by current evaluation paradigms. We introduce GroundedSurg, the first language-conditioned, instance-level surgical grounding benchmark. Each instance pairs a surgical image with a natural-language description targeting a single instrument, accompanied by structured spatial grounding annotations including bounding boxes and point-level anchors. The dataset spans ophthalmic, laparoscopic, robotic, and open procedures, encompassing diverse instrument types, imaging conditions, and operative complexities. By jointly evaluating linguistic reference resolution and pixel-level localization, GroundedSurg enables a systematic and realistic evaluation of vision-language models in clinically realistic multi-instrument scenes. Extensive experiments demonstrate substantial performance gaps across modern segmentation and VLMs, highlighting the urgent need for clinically grounded vision-language reasoning in surgical AI systems. Code and data are publicly available at https://github.com/gaash-lab/GroundedSurg
Tajamul Ashraf, Abrar Ul Riyaz, Wasif Tak +4
King Abdullah University of Science and Technology (KAUST), Saudi Arabia · Gaash Research Lab, National Institute of Technology Srinagar, India · Thapar Institute of Engineering and Technology, India +1
Spatio-Temporal Video Grounding (STVG) aims to retrieve the visual trajectory of a specific object from a video stream as described by a natural language expression. However, most advanced methods struggle to balance global context modeling with precise boundary localization. Due to the prohibitive computational costs of processing long videos, these approaches typically resort to low-rate temporal downsampling and implicit motion modeling. This inevitably suppresses high-frequency boundary cues and neglects the explicit inter-frame dependencies required for precise boundary delineation. To address these limitations, we present ScanFocus, a novel coarse-to-fine framework that decouples the STVG task into a global spatio-temporal scan and a local boundary focus. Specifically, we utilize a unified vision-language fusion encoder combined with a lightweight Deformable Semantic-Motion Fusion module to efficiently align multimodal features and generate coarse proposals. To recover the suppressed fine-grained details, we introduce the Semantic-Guided Temporal Aggregator (SGTA) in the refinement stage. By densely sampling around coarse boundaries, SGTA explicitly models short-term temporal interactions under semantic guidance, capturing rapid motion changes for precise timestamp regression. Extensive experiments on three widely used benchmarks demonstrate the performance superiority of our proposed method over previous approaches. Code will be released at https://github.com/TenMinutes209/ScanFocus.