ABRR: Transforming Temporal Sentence Grounding with Wasserstein-Adaptive Refinement and Precision Ranking
Authors: Xiang Fang, Wanlong Fang, Junhao Dong, Daizong Liu
Organizations: College of Computing and Data Science, Nanyang Technological University, Singapore. · Institute for Math & AI, Wuhan University, Wuhan, 430072, China.
Temporal Sentence Grounding (TSG) aims to localize temporal segments in untrimmed videos that correspond to natural language queries, a critical task for video understanding with applications in video retrieval, summarization, and human-computer interaction. Despite significant progress driven by transformer-based models, existing TSG methods suffer from two key limitations: quantization errors due to temporal downsampling, which misalign segment boundaries, and suboptimal ranking of predictions based on semantic matching rather than localization precision. To address these challenges, we propose the Adaptive Boundary Refinement and Ranking (ABRR) framework, a novel, model-agnostic post-processing approach that enhances the performance of any TSG model without retraining. ABRR integrates three innovative components: (1) Temporal Distribution Smoothing, which mitigates noise in predicted score distributions using Gaussian kernel convolution; (2) Boundary Refinement via Optimal Transport, which leverages the 2-Wasserstein distance and Sinkhorn iterations to align coarse segment boundaries with true temporal moments, grounded in Monge-Kantorovich optimal transport theory; and (3) Quality-Driven Re-Ranking, which reorders predictions using a composite score balancing localization precision and semantic relevance. By correcting quantization errors and prioritizing accurately localized segments, ABRR achieves substantial improvements over state-of-the-art TSG models. Extensive experiments on benchmark datasets, including Charades-STA, ActivityNet Captions, and QVHighlights, demonstrate consistent gains at stringent IoU thresholds (e.g., IoU=0.7), alongside enhanced ranking performance. The framework's computational efficiency and compatibility with existing models make it a versatile and impactful solution for real-world TSG applications.
Figures & tables
Fig. 1: (a) Example of the Temporal Sentence Grounding (TSG) task. (b) Comparison between previous TSG models and our proposed ABRR.
Fig. 2: Overview of the ABRR framework. The pipeline includes temporal distribution smoothing, boundary refinement using optimal transport, and quality-driven re-ranking. Best viewed in color.
TABLE I: Performance comparison on Charades-STA and ActivityNet Captions, averaged over 3 runs with standard deviations. Bold indicates the best results, and ↑ denotes improvement with ABRR.
Aspect
MQVTG [ 95 ]
ABRR
Stage
Training/model design
Inference post-processing
Main target
Feature discriminability
Boundary/ranking precision
Retraining
Required
Not required
Core mechanism
Moment codebook
Wasserstein refinement
Compatibility
Integrated into models
Plug-and-play on outputs
TABLE II: Method-level comparison between MQVTG and ABRR. MQVTG addresses representation-level quantization during training, whereas ABRR addresses post-hoc temporal boundary quantization and ranking bias during inference.
TABLE III: Additional generalization results on QVHighlights validation set. We report moment retrieval metrics and do not modify highlight saliency scores.
TABLE V: Ablation study on Charades-STA, showing the impact of ABRR components on R@1 at IoU thresholds (0.3, 0.5, 0.7) and mIoU, averaged over 5 runs with standard deviations. Components: Smoothing (S), Boundary Refinement (BR), Quality-Driven Re-Ranking (QR).
TABLE VI: Ablation study on ActivityNet Captions, showing the impact of ABRR components on R@1 at IoU thresholds (0.3, 0.5, 0.7) and mIoU, averaged over 5 runs with standard deviations. Components: Smoothing (S), Boundary Refinement (BR), Quality-Driven Re-Ranking (QR).
TABLE VII: Comparison with simpler post-processing/calibration strategies on Charades-STA using UniVTG as the base model.
Configuration
σ=0.3
σ=0.5
σ=0.7
S + BR
31.8
32.1
31.9
S + BR + QR
33.9
34.2
33.8
S + shift-limited BR + QR
33.6
34.0
33.7
TABLE VIII: Interaction between smoothing variance σ and boundary refinement configurations on Charades-STA. Full ABRR uses S+BR+QR; performance is stable around σ=0.5 rather than being an artifact of a single isolated component.
TABLE IX: Sensitivity of the quality-score weight α across datasets using UniVTG + ABRR. A single α=0.7 is used in all main experiments.
Fig. 3: Main ablation study on Charades-STA with UniVTG. The corrected y-axis tick labels increase monotonically from bottom to top.
Fig. 4: Performance of the Temporal Distribution Smoothing module on Charades-STA.
Fig. 5: Sensitivity analysis of hyperparameters on Charades-STA with UniVTG + ABRR.
Fig. 6: Qualitative comparison of UniVTG vs. UniVTG + ABRR on different datasets (top: ActivityNet Captions; bottom: Charades-STA).
Fig. 7: Computational overhead of ABRR across representative base models. ABRR adds a lightweight post-processing cost compared with the base TSG inference time.
Fig. 8: Representative Sinkhorn convergence diagnostics. The relative marginal violation decreases rapidly and reaches the early-stopping tolerance within about 10 iterations on average.
Temporal sentence grounding in videos (TSGV) aims to localize a temporal segment that semantically corresponds to a sentence query from an untrimmed video. Most current methods adopt pre-trained query-agnostic visual encoders for offline feature extraction, and the video backbones are frozen and not optimized for TSGV. This leads to a task discrepancy issue for the video backbone trained for visual classification, but utilized for TSGV. To bridge this gap, we propose a fully end-to-end paradigm that jointly optimizes the video backbone and localization head. We first conduct an empirical study validating the effectiveness of end-to-end learning over frozen baselines across different model scales. Furthermore, we introduce a Sentence Conditioned Adapter (SCADA), which leverages sentence features to train a small portion of video backbone parameters adaptively. SCADA facilitates the deployment of deeper network backbones with reduced memory and significantly enhances visual representation by modulating feature maps through precise integration of linguistic embeddings. Experiments on two benchmarks show that our method outperforms state-of-the-art approaches. The code and models will be released.
Allen He, Qi Liu, Kun Liu +2
BASIS International School Park Lane Harbour · UCAS · JD Explore Academy +1
Spatio-Temporal Video Grounding (STVG) aims to retrieve the visual trajectory of a specific object from a video stream as described by a natural language expression. However, most advanced methods struggle to balance global context modeling with precise boundary localization. Due to the prohibitive computational costs of processing long videos, these approaches typically resort to low-rate temporal downsampling and implicit motion modeling. This inevitably suppresses high-frequency boundary cues and neglects the explicit inter-frame dependencies required for precise boundary delineation. To address these limitations, we present ScanFocus, a novel coarse-to-fine framework that decouples the STVG task into a global spatio-temporal scan and a local boundary focus. Specifically, we utilize a unified vision-language fusion encoder combined with a lightweight Deformable Semantic-Motion Fusion module to efficiently align multimodal features and generate coarse proposals. To recover the suppressed fine-grained details, we introduce the Semantic-Guided Temporal Aggregator (SGTA) in the refinement stage. By densely sampling around coarse boundaries, SGTA explicitly models short-term temporal interactions under semantic guidance, capturing rapid motion changes for precise timestamp regression. Extensive experiments on three widely used benchmarks demonstrate the performance superiority of our proposed method over previous approaches. Code will be released at https://github.com/TenMinutes209/ScanFocus.
Video Temporal Grounding (VTG) aims to localize the video segment that corresponds to a natural language query, which requires a comprehensive understanding of complex temporal dynamics. Existing Vision-LMMs typically perceive temporal dynamics via positional encoding, text-based timestamps, or visual frame numbering. However, these approaches exhibit notable limitations: assigning each frame a text-based timestamp token introduces additional computational overhead and leads to sparsity in visual attention, positional encoding struggles to capture absolute temporal information, and visual frame numbering often compromises spatial detail. To address these issues, we propose Temporal to Spatial Gridification (T2SGrid), a novel framework that reformulates video temporal understanding as a spatial understanding task. The core idea of T2SGrid is to process video content in clips rather than individual frames. we employ a overlapping sliding windows mechanism to segment the video into temporal clips. Within each window, frames are arranged chronologically in a row-major order into a composite grid image, effectively transforming temporal sequences into structured 2D layouts. The gridification not only encodes temporal information but also enhances local attention within each grid. Furthermore, T2SGrid enables the use of composite text timestamps to establish global temporal awareness. Experiments on standard VTG benchmarks demonstrate that T2SGrid achieves superior performance.
Chaohong Guo, Yihan He, Yongwei Nie +3
South China University of Technology. · Guangdong Laboratory of Artificial Intelligence and Digital Economy (SZ). · Bytedance Inc.