Organizing Temporal Evidence in Hyperbolic Space for Video Moment Retrieval
Authors: Mengnan Liu, Sanping Zhou, Kun Xia, Jiaxuan Lin, Gang Hua, Le Wang
Organizations: State Key Laboratory of Human-Machine Hybrid Augmented Intelligence, Institute of Artificial Intelligence and Robotics, Xi’an Jiaotong University, Xi’an, Shaanxi 710049, China. · School of Computer Science and Technology, Xi’an Jiaotong University, Xi’an, Shaanxi 710049, China. · Amazon Alexa AI, Bellevue, WA 98004 USA.
Video moment retrieval aims to localize a query-relevant temporal segment in an untrimmed video. Existing methods typically perform video-text alignment in Euclidean space, where representations are embedded in a flat geometry. However, accurate moment localization requires organizing query-relevant evidence across multiple grounding states, where target regions, boundary transitions, and contextual distractors exhibit distinct temporal structures and relevance patterns. Such Euclidean representations lack an explicit geometry to jointly capture these relationships, motivating us to explore structured temporal evidence organization from a geometric perspective. Specifically, we propose Hyperbolic Temporal Evidence Organization (HTEO), a geometric framework that organizes query-aware temporal memory in hyperbolic space. HTEO maps temporal representations into a Poincaré ball and organizes temporal evidence through Radial Evidence Organization (REO) and Orthogonal Angular Refinement (OAR). REO adjusts temporal tokens' radial positions to encode the structural differences among temporal evidence, while OAR enhances query-conditioned semantic alignment through orthogonal rotation. The organized representation is then projected back to the Euclidean tangent space and injected into the decoder through a lightweight residual memory update, enabling hyperbolic geometric cues to be incorporated into standard Transformer decoders while preserving the original cross-modal memory. Furthermore, we introduce radial structure regularization and angular relevance supervision to explicitly guide temporal evidence organization and query-conditioned semantic alignment, respectively. Extensive experiments and analyses across multiple benchmarks demonstrate that HTEO improves temporal localization while providing an interpretable geometric organization of temporal evidence.
Figures & tables
Fig. 1: Motivation of hyperbolic temporal evidence organization for video moment retrieval. (a) In flat Euclidean memory, query-relevant tokens and contextual distractors may exhibit similar feature distances, making them difficult to distinguish based on flat similarity alone. (b) Such ambiguity can lead to false response peaks on distracting background regions, resulting in inaccurate temporal localization. (c) HTEO organizes query-aware temporal evidence in hyperbolic space through radius-angle geometry, where angular direction captures query-conditioned relevance and radial distance provides structural cues for temporal evidence organization. By introducing geometry-aware organization, HTEO enhances the discrimination between relevant evidence and semantically similar distractors, leading to more structured temporal representations for moment retrieval.
Fig. 2: Overview of the proposed Hyperbolic Temporal Evidence Organization (HTEO) framework. Given a video-query pair, HTEO first constructs query-aware temporal memory through multimodal feature fusion and maps it into a Poincaré ball via the exponential map. In hyperbolic space, temporal evidence is organized through Radial Evidence Organization (REO) and Orthogonal Angular Refinement (OAR), where REO regulates the radial structure to capture different temporal grounding characteristics, and OAR refines angular directions to enhance query-conditioned semantic relevance. The organized memory is then projected back to the Euclidean tangent space via the logarithmic map and incorporated into the decoder through a lightweight residual update for moment localization. The bottom panel illustrates the detailed geometric operations.
Fig. 3: Visualization of hyperbolic radius-angle behavior.
Method
MR
HD
R1
mAP
≥ Very Good
@ 0.5
@ 0.7
@ 0.5
@ 0.75
Avg.
mAP
HIT @ 1
M-DETR [ 23 ] (NeurIPS’2021)
52.89
33.02
54.84
29.40
30.73
35.69
55.60
QD-DETR [ 3 ] (CVPR’2023)
62.52
44.98
62.62
39.88
39.86
38.64
62.40
UniVTG [ 27 ] (ICCV’2023)
66.65
52.19
64.37
46.68
45.18
40.18
64.77
TaskWeave [ 28 ] (CVPR’2024)
64.26
50.06
65.39
46.47
45.38
-
-
TABLE I: Comparison of model performance on QVHighlights dataset. † indicates the model using LLM.
Method
TACoS
Charades-STA
R1 @ 0.3
R1 @ 0.5
R1 @ 0.7
mIoU
R1 @ 0.3
R1 @ 0.5
R1 @ 0.7
mIoU
M-DETR [ 23 ] (NeurIPS’2021)
38.0
24.7
12.0
25.5
65.8
52.1
30.6
45.5
MomentDiff [ 31 ] (NeurIPS’2023)
44.8
33.7
-
-
-
55.6
32.4
-
UniVTG [ 27 ] (ICCV’2023)
-
-
-
-
68.0
57.2
36.1
-
CG-DETR [ 32 ] (arXiv’2023)
52.2
39.6
22.2
36.5
70.4
58.4
36.3
50.1
TR-DETR [ 5 ] (AAAI’2024)
-
-
-
-
-
57.6
33.5
-
TABLE II: Comparison of model performance on TACoS and Charades-STA datasets.
Fig. 4: Visualization of hyperbolic radius-angle organization for temporal evidence modeling. (a) The radius-angle feature space shows the geometric separation among target, boundary, and background tokens. (b) The angular similarity curves demonstrate that angular direction provides query-conditioned relevance cues. (c) The temporal evolution of radius reveals structural variations around different temporal evidence regions and boundary transitions.
TABLE VI: Analysis of HTEO under different moment durations.
Fig. 5: Illustrative analysis of inside-outside response gaps. The left panel compares the maximum query-conditioned responses inside and outside the ground-truth interval, while the right panel shows the corresponding per-query inside-outside response gaps.
Fig. 6: Qualitative analysis of false peak correction using hyperbolic angular refinement. Four representative validation samples are shown where Euclidean similarity produces distractor-induced response peaks. Hyperbolic angular organization shifts dominant responses toward the ground-truth moments and suppresses irrelevant activations.
Region
Mean
Med.
Std.
Dist. (mean ± std)
Boundary
29.14
29.11
0.5012
0.5932 ± 0.0673
GT
28.75
28.77
0.3969
0.2490 ± 0.1442
Background
29.51
29.65
0.4879
2.8878 ± 2.2725
TABLE VII: Analysis of radial statistics in hyperbolic space. Mean, median, and standard deviation describe the radius distributions of different temporal regions, while Dist. denotes the geodesic distance to the GT moment center.
TABLE IX: Sensitivity to the curvature magnitude c .
Fig. 7: Visualization of temporal evidence organization in Euclidean and hyperbolic spaces. Compared with flat Euclidean representations, hyperbolic geometry yields more structured distributions of target, boundary, and background tokens, indicating improved evidence separability. The query texts are displayed as provided in the original dataset.
TABLE XI: Sensitivity to the residual scaling factor α .
Fig. 8: Qualitative comparison of HTEO with TR-DETR and LD-DETR on the QVHighlights dataset. HTEO achieves more accurate temporal localization with better alignment to GT. The queries are displayed as provided in the original dataset.
Joint video moment retrieval and highlight detection requires identifying query-relevant temporal segments while estimating clip-level saliency, yet DETR-style pipelines do not explicitly preserve query-relevant evidence throughout encoding, decoding, and cross-task prediction. We propose EviDETR, an evidence-preserving framework with three components. Semantic-aware Feature Reweighting (SFR) enhances query-relevant clip representations through saliency estimation and cross-modal interaction. A Temporal Top-2 Mixture-of-Experts (TTop2MoE) decoder performs query-adaptive refinement via sparse expert routing. MR-to-HD (MR2HD) fusion transfers span-level retrieval evidence to clip-level highlight prediction through confidence-weighted multi-scale aggregation. Using CLIP+SlowFast features, EviDETR achieves 69.29 [email protected], 54.77 [email protected], and 48.41 Avg. mAP for moment retrieval on QVHighlights, together with 41.83 HD-mAP and 68.33 HIT@1. Strong results on TACoS and Charades-STA further demonstrate cross-dataset transferability.
Haoran Sun, Yufan Li, Qichen Zhang +2
Beijing Normal-Hong Kong Baptist University, Zhuhai, China
Video MLLMs often struggle with fine-grained spatio-temporal reasoning, sometimes generating correct answers based on irrelevant frames or objects. Although outputting spatio-temporal evidence during reasoning is a promising direction, existing RL frameworks typically rely on geometry-only (IoU) rewards, which can be sensitive to boundary perturbations and overlook semantic alignment. To address this, we propose Semantic Evidence Reward (SER), which reformulates spatio-temporal evidence grounding as a constrained verification task. Instead of computing pixel-level overlap, SER uses a referee VLM as a local checker to evaluate model-generated evidence claims across two dimensions: relevance and localization quality, combined with a temporal penalty. This design reduces the reliance on dense box annotations and enables training directly on standard video QA data. On the V-STAR benchmark, SER achieves 49.6% mLGM, improving by 3.0 points over the strong evidence-grounded baseline Open-o3-Video, demonstrating its potential in enhancing both answer accuracy and evidence grounding.
Sheng Xia, Zhengqin Lai, Tianxiang Jiang +4
Nanjing University · Shanghai Innovation Institute · Harbin Institute of Technology, Shenzhen +2
Video Moment Retrieval (MR) and Highlight Detection (HD) are crucial tasks in video analysis that aim to localize specific moments and estimate clip-wise relevance based on a given text query. Recent approaches treat them as similar video grounding tasks and use the same architecture to solve them. These tasks require both fine-grained comprehension at the image level and high-level temporal understanding across the entire video. Existing approaches have primarily focused on temporal modeling using frame-level features, often neglecting the rich visual information related to the text query within individual frames. This oversight leads to inaccurate grounding results. To address this limitation, we propose a Comprehensive Spatial-Temporal Representation Learning Framework (CoSTL), which captures both fine-grained image-level information and temporal dynamics. Specifically, CoSTL incorporates a text-driven progressive fine-grained image encoder, performing a two-step text-driven knowledge extraction process to learn fine-grained spatial representations. Furthermore, a multi-scale temporal perception module captures comprehensive spatial-temporal representations, enhancing the model's ability to process temporal dynamics. We demonstrate state-of-the-art performance on four public benchmarks: QVHighlights, Charades-STA, TACoS, and TVSum.
Xin Dong, Wenjia Geng, Wenfeng Deng +1
Shenzhen International Graduate School, Tsinghua University · Pengcheng Laboratory