Despite recent advances in Video Large Language Models (Vid-LLMs), Temporal Video Grounding (TVG) targeting precise localization of time segments of query events remains a grand challenge. Recent studies attempt to address this challenge by matching two separate tokens to all frame features and identifying the start and end frames of events based on the highest matching similarities. However, this approach relies heavily on the exact start and end timestamps, which are often ambiguous without capturing the semantic continuity and integrity of the event. To this end, we design E.M.Ground, a temporal grounding Vid-LLM that tackles the TVG challenge from a holistic and coherent event perception perspective. E.M.Ground introduces three novel designs: (i) It introduces a special <evt> token that aggregates information from all frames of query events with varying lengths, thereby preserving semantic continuity and integrity for accurate event matching. (ii) It introduces Savitzky-Golay smoothing to suppress the noise and abrupt changes in token-to-frame similarities across adjacent timestamps, thereby improving prediction accuracy. (iii) It aggregates multi-grained frame features to enhance the reliability of token-to-frame matching and improve temporal understanding, thereby compensating for the compression-induced information loss. Extensive experiments on widely adopted TVG benchmarks, along with in-depth analyses, demonstrate that E.M.Ground achieves strong overall performance, with the most pronounced improvements on single-event grounding and boundary-integrity tasks.
Figures & tables
Figure 1: Comparison between E.T.Chat and our proposed E.M.Ground. (a) E.T.Chat [ 34 ] uses two separate tokens to match the start and end frames, whereas E.M.Ground uses a unified <evt> token to match the complete event span. (b) Performance across different event-duration groups. The number in square brackets denotes the number of evaluated videos in each group, e.g. , 0-10s[n=379] , while the numerical annotations indicate the corresponding performance values. By preserving the semantic continuity and integrity of events, E.M.Ground achieves better and more stable performance across different event durations.
Figure 2: Overall architecture of our proposed E.M.Ground. Specifically, E.M.Ground perceives the query event from a holistic and coherent perspective. It introduces a special token <evt> to aggregate all frames within the ground-truth time spans, leverages multi-grained visual features to compensate for information loss, and refines the predictions with smoothing operations. Based on these designs, E.M.Ground effectively captures the semantic continuity and integrity of the query event.
Table 1: Zero-shot performance on the Temporal Video Grounding task is evaluated using the Charades-STA [ 18 ] ( Up ) and relevant E.T.Bench [ 34 ] subtasks ( Down ). The best results are highlighted in bold font.
Figure 3: Qualitative comparison between E.T.Chat [ 34 ] and E.M.Ground. Each example shows the query, video timeline, ground-truth segment (red bar), prediction of E.T.Chat (yellow bar), and the prediction of E.M.Ground (blue bar). The start and end timestamps beside each bar indicate the temporal boundaries of the corresponding segment. We present three representative failure modes of E.T.Chat, which are detailed in Sec. 4.3 . (a) No Overlap : E.T.Chat focuses on visually related objects, such as the person and the bag, but misses the queried action of taking a phone from the bag, resulting in a predicted segment that does not overlap with the ground-truth event. (b) Completely Contained : E.T.Chat only covers the middle part of the event and omits the beginning and ending phases, leading to an incomplete prediction fully contained within the ground-truth segment. (c) Partial Overlap : E.T.Chat partially overlaps with the ground-truth segment but includes irrelevant temporal regions and misses part of the target event. In contrast, our proposed E.M.Ground effectively mitigates all of these errors, providing more accurate temporal grounding.
Figure 4: Error analysis of Temporal Video Grounding. Left : Number of errors for each type made by E.T.Chat [ 34 ] and E.M.Ground. Right : mIoU corresponding to each error type. N.O. denotes predictions that have no overlap with the ground-truths; C.C. denotes cases where the ground-truths are completely contained within the predictions; P.O. denotes predictions partially overlap with the ground-truths.
Model
Original F1 ( ↑ )
F1 after relocation ( ↑ )
New-end selection ( ↑ )
Relocated-end selection ( ↓ )
E.T.Chat
38.6
30.7
22.6%
62.4%
Ours
52.0
49.9
68.7%
11.8%
Table 2: Performance after relocating the original event-ending frame on the E.T.Bench TVG subtask. “New-end selection” denotes selecting ve−1 , while “relocated-end selection” denotes incorrectly following the relocated ve .
E.T.Bench-Grounding
Setup
TVG
EPM
TAL
EVS
Baseline (E.T.Chat, dual token matching)
38.6
10.2
30.8
25.4
Baseline + Multi-grained feature
41.3
10.8
30.5
26.5
Single <evt> token matching
45.1
11.8
30.8
30.6
Baseline + Savitzky-Golay Smoothing
38.6
10.2
30.8
25.4
Baseline (E.T.Chat, dual token matching)
38.6
10.2
30.8
25.4
Table 3: Ablation studies for technical designs.
Figure 5: Visualization of attention patterns between the <evt> token and video frames across different LLM layers. Each block corresponds to the attention weight assigned to a specific frame. Darker colors indicate higher attention weights, while lighter colors indicate lower attention weights. The diverse attention patterns across layers show that different LLM layers capture complementary temporal cues, which motivates our multi-grained feature aggregation for event grounding. The quantitative effectiveness is validated by the controlled ablation in Tab. 3 .
Table 4: Performance comparison across different LLM-layer aggregation strategies.
E.T.Bench-Captioning
Methods
DVC
SLC
F1
Sim
F1
Sim
Video-ChatGPT(7B) [ 37 ]
8.8
11.3
5.7
10.2
PLLaVA (7B) [ 51 ]
13.3
10.6
9.7
11.8
VTimeLLM (7B) [ 23 ]
12.4
13.1
8.7
6.4
TimeChat (7B) [ 42 ]
16.6
12.5
5.6
9.2
Table 5: Zero-shot performance on the Dense Video Captioning subtasks of E.T.Bench [ 34 ] . The best results for each metric are highlighted in bold font.
YouCook2
Methods
Caption
Time
SODA_c
CIDEr
F1
TimeChat (7B) [ 42 ]
1.2
3.4
12.6
Seq2Time (7B) [ 11 ]
1.3
4.2
16.2
E.T.Chat(3.8B) [ 34 ]
1.5
4.7
15.1
VTG-LLM (7B) [ 19 ]
1.5
5.0
17.5
Table 6: Zero-shot performance on the YouCook2 dataset [ 55 ] , designed for the Dense Video Captioning task. The best results for each metric are highlighted in bold font.
QVHighlights
Methods
mAP
HIT@1
Momentor (7B) [ 40 ]
7.6
-
TimeChat (7B) [ 42 ]
14.5
23.9
NumPro (7B) [ 50 ]
15.3
24.3
VTG-LLM (7B) [ 19 ]
16.3
33.5
E.T.Chat (3.8B) [ 34 ]
20.0
44.8
Table 7: Zero-shot performance on the QVHighlights dataset [ 26 ] , designed for the Video Highlight Detection task. The best results for each metric are highlighted by bold font.
MVBench (Part 1)
Methods
Avg.
AA
AC
AL
AP
AS
CO
CI
EN
ER
VideoLLaMA
34.1
51.0
34.0
22.5
25.5
27.5
40.0
37.0
30.0
21.0
VideoChat
35.5
56.0
35.0
27.0
26.5
33.5
41.0
36.0
23.5
23.5
TimeChat
38.5
61.0
34.0
26.5
36.0
40.5
36.0
35.0
29.0
35.0
E.T.Chat
37.1
59.5
28.0
38.5
33.5
34.5
31.0
22.5
27.0
49.0
Ours
42.0
73.0
50.0
39.0
33.5
35.5
39.0
32.0
32.5
51.0
Table 8: Performance on the MVBench [ 28 ] . The best results are in bold , and Δ indicates improvement over E.T.Chat [ 34 ] .
Table 10: Comparison between fixed and adaptive thresholding strategies. The adaptive results are close to the fixed-threshold results, suggesting that E.M.Ground is not sensitive to the threshold value.
School of Computer Science and Technology, University of Chinese Academy of Sciences, Beijing, China · Beijing Key Laboratory of Embodied Intelligence Computing, Beijing, China · Faculty of Computing, Harbin Institute of Technology, Weihai, China +1
Media Analytics and Computing Lab, Xiamen University, Xiamen, China · Kling Team, Kuaishou Technology, China · The Chinese University of Hong Kong, HKSAR, China +1