Organizing Temporal Evidence in Hyperbolic Space for Video Moment Retrieval
Authors: Mengnan Liu, Sanping Zhou, Kun Xia, Jiaxuan Lin, Gang Hua, Le Wang
Organizations: State Key Laboratory of Human-Machine Hybrid Augmented Intelligence, Institute of Artificial Intelligence and Robotics, Xi’an Jiaotong University, Xi’an, Shaanxi 710049, China. · School of Computer Science and Technology, Xi’an Jiaotong University, Xi’an, Shaanxi 710049, China. · Amazon Alexa AI, Bellevue, WA 98004 USA.
Video moment retrieval aims to localize a query-relevant temporal segment in an untrimmed video. Existing methods typically perform video-text alignment in Euclidean space, where representations are embedded in a flat geometry. However, accurate moment localization requires organizing query-relevant evidence across multiple grounding states, where target regions, boundary transitions, and contextual distractors exhibit distinct temporal structures and relevance patterns. Such Euclidean representations lack an explicit geometry to jointly capture these relationships, motivating us to explore structured temporal evidence organization from a geometric perspective. Specifically, we propose Hyperbolic Temporal Evidence Organization (HTEO), a geometric framework that organizes query-aware temporal memory in hyperbolic space. HTEO maps temporal representations into a Poincaré ball and organizes temporal evidence through Radial Evidence Organization (REO) and Orthogonal Angular Refinement (OAR). REO adjusts temporal tokens' radial positions to encode the structural differences among temporal evidence, while OAR enhances query-conditioned semantic alignment through orthogonal rotation. The organized representation is then projected back to the Euclidean tangent space and injected into the decoder through a lightweight residual memory update, enabling hyperbolic geometric cues to be incorporated into standard Transformer decoders while preserving the original cross-modal memory. Furthermore, we introduce radial structure regularization and angular relevance supervision to explicitly guide temporal evidence organization and query-conditioned semantic alignment, respectively. Extensive experiments and analyses across multiple benchmarks demonstrate that HTEO improves temporal localization while providing an interpretable geometric organization of temporal evidence.
Figures & tables
Fig. 1: Motivation of hyperbolic temporal evidence organization for video moment retrieval. (a) In flat Euclidean memory, query-relevant tokens and contextual distractors may exhibit similar feature distances, making them difficult to distinguish based on flat similarity alone. (b) Such ambiguity can lead to false response peaks on distracting background regions, resulting in inaccurate temporal localization. (c) HTEO organizes query-aware temporal evidence in hyperbolic space through radius-angle geometry, where angular direction captures query-conditioned relevance and radial distance provides structural cues for temporal evidence organization. By introducing geometry-aware organization, HTEO enhances the discrimination between relevant evidence and semantically similar distractors, leading to more structured temporal representations for moment retrieval.
Fig. 2: Overview of the proposed Hyperbolic Temporal Evidence Organization (HTEO) framework. Given a video-query pair, HTEO first constructs query-aware temporal memory through multimodal feature fusion and maps it into a Poincaré ball via the exponential map. In hyperbolic space, temporal evidence is organized through Radial Evidence Organization (REO) and Orthogonal Angular Refinement (OAR), where REO regulates the radial structure to capture different temporal grounding characteristics, and OAR refines angular directions to enhance query-conditioned semantic relevance. The organized memory is then projected back to the Euclidean tangent space via the logarithmic map and incorporated into the decoder through a lightweight residual update for moment localization. The bottom panel illustrates the detailed geometric operations.
Fig. 3: Visualization of hyperbolic radius-angle behavior.
Method
MR
HD
R1
mAP
≥ Very Good
@ 0.5
@ 0.7
@ 0.5
@ 0.75
Avg.
mAP
HIT @ 1
M-DETR [ 23 ] (NeurIPS’2021)
52.89
33.02
54.84
29.40
30.73
35.69
55.60
QD-DETR [ 3 ] (CVPR’2023)
62.52
44.98
62.62
39.88
39.86
38.64
62.40
UniVTG [ 27 ] (ICCV’2023)
66.65
52.19
64.37
46.68
45.18
40.18
64.77
TaskWeave [ 28 ] (CVPR’2024)
64.26
50.06
65.39
46.47
45.38
-
-
TABLE I: Comparison of model performance on QVHighlights dataset. † indicates the model using LLM.
Method
TACoS
Charades-STA
R1 @ 0.3
R1 @ 0.5
R1 @ 0.7
mIoU
R1 @ 0.3
R1 @ 0.5
R1 @ 0.7
mIoU
M-DETR [ 23 ] (NeurIPS’2021)
38.0
24.7
12.0
25.5
65.8
52.1
30.6
45.5
MomentDiff [ 31 ] (NeurIPS’2023)
44.8
33.7
-
-
-
55.6
32.4
-
UniVTG [ 27 ] (ICCV’2023)
-
-
-
-
68.0
57.2
36.1
-
CG-DETR [ 32 ] (arXiv’2023)
52.2
39.6
22.2
36.5
70.4
58.4
36.3
50.1
TR-DETR [ 5 ] (AAAI’2024)
-
-
-
-
-
57.6
33.5
-
TABLE II: Comparison of model performance on TACoS and Charades-STA datasets.
Fig. 4: Visualization of hyperbolic radius-angle organization for temporal evidence modeling. (a) The radius-angle feature space shows the geometric separation among target, boundary, and background tokens. (b) The angular similarity curves demonstrate that angular direction provides query-conditioned relevance cues. (c) The temporal evolution of radius reveals structural variations around different temporal evidence regions and boundary transitions.
TABLE VI: Analysis of HTEO under different moment durations.
Fig. 5: Illustrative analysis of inside-outside response gaps. The left panel compares the maximum query-conditioned responses inside and outside the ground-truth interval, while the right panel shows the corresponding per-query inside-outside response gaps.
Fig. 6: Qualitative analysis of false peak correction using hyperbolic angular refinement. Four representative validation samples are shown where Euclidean similarity produces distractor-induced response peaks. Hyperbolic angular organization shifts dominant responses toward the ground-truth moments and suppresses irrelevant activations.
Region
Mean
Med.
Std.
Dist. (mean ± std)
Boundary
29.14
29.11
0.5012
0.5932 ± 0.0673
GT
28.75
28.77
0.3969
0.2490 ± 0.1442
Background
29.51
29.65
0.4879
2.8878 ± 2.2725
TABLE VII: Analysis of radial statistics in hyperbolic space. Mean, median, and standard deviation describe the radius distributions of different temporal regions, while Dist. denotes the geodesic distance to the GT moment center.
TABLE IX: Sensitivity to the curvature magnitude c .
Fig. 7: Visualization of temporal evidence organization in Euclidean and hyperbolic spaces. Compared with flat Euclidean representations, hyperbolic geometry yields more structured distributions of target, boundary, and background tokens, indicating improved evidence separability. The query texts are displayed as provided in the original dataset.
TABLE XI: Sensitivity to the residual scaling factor α .
Fig. 8: Qualitative comparison of HTEO with TR-DETR and LD-DETR on the QVHighlights dataset. HTEO achieves more accurate temporal localization with better alignment to GT. The queries are displayed as provided in the original dataset.