ABRR: Transforming Temporal Sentence Grounding with Wasserstein-Adaptive Refinement and Precision Ranking
Authors: Xiang Fang, Wanlong Fang, Junhao Dong, Daizong Liu
Organizations: College of Computing and Data Science, Nanyang Technological University, Singapore. · Institute for Math & AI, Wuhan University, Wuhan, 430072, China.
Temporal Sentence Grounding (TSG) aims to localize temporal segments in untrimmed videos that correspond to natural language queries, a critical task for video understanding with applications in video retrieval, summarization, and human-computer interaction. Despite significant progress driven by transformer-based models, existing TSG methods suffer from two key limitations: quantization errors due to temporal downsampling, which misalign segment boundaries, and suboptimal ranking of predictions based on semantic matching rather than localization precision. To address these challenges, we propose the Adaptive Boundary Refinement and Ranking (ABRR) framework, a novel, model-agnostic post-processing approach that enhances the performance of any TSG model without retraining. ABRR integrates three innovative components: (1) Temporal Distribution Smoothing, which mitigates noise in predicted score distributions using Gaussian kernel convolution; (2) Boundary Refinement via Optimal Transport, which leverages the 2-Wasserstein distance and Sinkhorn iterations to align coarse segment boundaries with true temporal moments, grounded in Monge-Kantorovich optimal transport theory; and (3) Quality-Driven Re-Ranking, which reorders predictions using a composite score balancing localization precision and semantic relevance. By correcting quantization errors and prioritizing accurately localized segments, ABRR achieves substantial improvements over state-of-the-art TSG models. Extensive experiments on benchmark datasets, including Charades-STA, ActivityNet Captions, and QVHighlights, demonstrate consistent gains at stringent IoU thresholds (e.g., IoU=0.7), alongside enhanced ranking performance. The framework's computational efficiency and compatibility with existing models make it a versatile and impactful solution for real-world TSG applications.
Figures & tables
Fig. 1: (a) Example of the Temporal Sentence Grounding (TSG) task. (b) Comparison between previous TSG models and our proposed ABRR.
Fig. 2: Overview of the ABRR framework. The pipeline includes temporal distribution smoothing, boundary refinement using optimal transport, and quality-driven re-ranking. Best viewed in color.
TABLE I: Performance comparison on Charades-STA and ActivityNet Captions, averaged over 3 runs with standard deviations. Bold indicates the best results, and ↑ denotes improvement with ABRR.
Aspect
MQVTG [ 95 ]
ABRR
Stage
Training/model design
Inference post-processing
Main target
Feature discriminability
Boundary/ranking precision
Retraining
Required
Not required
Core mechanism
Moment codebook
Wasserstein refinement
Compatibility
Integrated into models
Plug-and-play on outputs
TABLE II: Method-level comparison between MQVTG and ABRR. MQVTG addresses representation-level quantization during training, whereas ABRR addresses post-hoc temporal boundary quantization and ranking bias during inference.
TABLE III: Additional generalization results on QVHighlights validation set. We report moment retrieval metrics and do not modify highlight saliency scores.
TABLE V: Ablation study on Charades-STA, showing the impact of ABRR components on R@1 at IoU thresholds (0.3, 0.5, 0.7) and mIoU, averaged over 5 runs with standard deviations. Components: Smoothing (S), Boundary Refinement (BR), Quality-Driven Re-Ranking (QR).
TABLE VI: Ablation study on ActivityNet Captions, showing the impact of ABRR components on R@1 at IoU thresholds (0.3, 0.5, 0.7) and mIoU, averaged over 5 runs with standard deviations. Components: Smoothing (S), Boundary Refinement (BR), Quality-Driven Re-Ranking (QR).
TABLE VII: Comparison with simpler post-processing/calibration strategies on Charades-STA using UniVTG as the base model.
Configuration
σ=0.3
σ=0.5
σ=0.7
S + BR
31.8
32.1
31.9
S + BR + QR
33.9
34.2
33.8
S + shift-limited BR + QR
33.6
34.0
33.7
TABLE VIII: Interaction between smoothing variance σ and boundary refinement configurations on Charades-STA. Full ABRR uses S+BR+QR; performance is stable around σ=0.5 rather than being an artifact of a single isolated component.
TABLE IX: Sensitivity of the quality-score weight α across datasets using UniVTG + ABRR. A single α=0.7 is used in all main experiments.
Fig. 3: Main ablation study on Charades-STA with UniVTG. The corrected y-axis tick labels increase monotonically from bottom to top.
Fig. 4: Performance of the Temporal Distribution Smoothing module on Charades-STA.
Fig. 5: Sensitivity analysis of hyperparameters on Charades-STA with UniVTG + ABRR.
Fig. 6: Qualitative comparison of UniVTG vs. UniVTG + ABRR on different datasets (top: ActivityNet Captions; bottom: Charades-STA).
Fig. 7: Computational overhead of ABRR across representative base models. ABRR adds a lightweight post-processing cost compared with the base TSG inference time.
Fig. 8: Representative Sinkhorn convergence diagnostics. The relative marginal violation decreases rapidly and reaches the early-stopping tolerance within about 10 iterations on average.