Recent advancements in Video Large Language Models (VideoLLMs) have enabled strong performance across diverse multimodal video tasks. To reduce the high computational cost of processing dense video frames, efficiency-oriented methods such as frame selection have been widely adopted. While effective at minimizing redundancy, these methods often cause notable performance drops on tasks requiring temporal reasoning. Unlike humans, who can infer event progression from sparse visual cues, VideoLLMs frequently misinterpret temporal relations when intermediate frames are omitted. To address this limitation, we explore visual prompting (VP) as a lightweight yet effective way to enhance temporal understanding in VideoLLMs. Our analysis reveals that simply annotating each frame with explicit ordinal information helps the model perceive temporal continuity. This visual cue also supports frame-level referencing and mitigates positional ambiguity within a sparsely sampled sequence. Building on these insights, we introduce ViKey, a training-free framework that combines VP with a lightweight Keyword-Frame Mapping (KFM) module. KFM leverages frame indices as dictionary-like keys to link textual cues to the most relevant frames, providing explicit temporal anchors during inference. Despite its simplicity, our approach substantially improves temporal reasoning and, on some datasets, preserves dense-frame baseline performance with as few as 20% of frames.
Figures & tables
Figure 1: Illustrations of How Frame Selection Disrupts and Visual Prompting Restores Temporal Reasoning. Frame selection improves efficiency but breaks temporal continuity, causing VideoLLMs to misinterpret transitions. With frame-index visual prompts, the model regains temporal order and correctly identifies causal relations under sparse-frame conditions.
Figure 2: Probing VP for temporal understanding. (a) Positional Embedding Degradation: remove temporal order only, or collapse both temporal and spatial positions. (b) Frame-level Referencing in VideoLLMs: frame-index prompts enable lookup and reverse-lookup between indices and content. (c) Attention Analysis: VP increases attention to image tokens across layers.
Figure 3: Effect of VP under Position Degradation. Across both baselines, frame-index VP consistently improve accuracy under temporal-only and full-collapse settings, indicating added temporal cues and robustness to degraded positional signals.
(a) Look-up
Position
8 frames
16 frames
32 frames
64 frames
average
–
28.07
11.70
5.85
4.09
12.43
TL
67.25
60.23
55.56
39.18
55.56
TR
56.73
53.80
51.46
31.58
48.39
BL
77.19
64.91
63.16
53.22
64.62
BR
76.02
64.33
64.33
51.46
64.04
Table 1: Results on lookup and reverse-lookup tasks. The table compares performance across VP placements, with and without VP, for mapping frame numbers to scenes (lookup) and scenes back to frame numbers (reverse lookup).
Figure 4: Overall pipeline of ViKey . Given a sequence of video frames, we first apply visual prompting by overlaying frame-number VPs on each frame. For a user query, the system extracts key textual concepts and performs Keyword–Frame Mapping (KFM) to identify which frames best correspond to each keyword. The query is then rewritten to explicitly include the mapped frame indices, aligning textual cues with the numbered frames. Feeding this aligned query–frame pair into the VLM enables more accurate temporal understanding.
Models
TempCompass
MVBench
VideoMME
LongVideoBench
EO
AC
total
AS
ST
total
TR
E3E
O3O
SSS
total
64 Frames (1 FPS)
VideoLLM Baselines
GPT4.1 [ 31 ]
66.69
74.72
70.74
75.00
92.50
83.75
−
−
−
−
−
Qwen2.5-VL-7B [ 3 ]
75.00
80.26
77.65
79.50
92.00
85.75
38.42
57.45
48.48
40.21
48.63
LLaVA-Video-7B [ 53 ]
73.63
75.71
74.68
74.00
91.00
82.50
47.46
69.15
62.12
40.21
56.42
LLaVA-OneVision-7B [ 20 ]
71.24
71.66
71.45
73.00
92.00
82.50
43.50
67.02
56.06
36.08
52.53
Table 2: Main results. Results on the temporal understanding subsets of the TempCompass [ 26 ] , MVBench [ 21 ] , VideoMME [ 11 ] , and LongVideoBench [ 46 ] datasets. Our VP-based method consistently improves performance across all datasets, and even when the input is limited to 20% of the frames, it achieves accuracy comparable to the 1 FPS dense-frame setting in some cases.
Figure 5: Qualitative Results on Open-Ended Temporal Questions. Compared with the baseline LLaVA-Video-7B, ViKey uses explicit frame indices and keyword–frame mapping to better capture temporal order and thus produce more accurate answers to open-ended temporal queries. In particular, it often explicitly mentions the injected frame indices ( e.g . , “frame #01”, “frame #02”) in its free-form responses, indicating that its temporal reasoning is grounded on the visual prompts.
Figure 6: Performance Comparison under Varying Frame Sampling Ratios. Accuracy on the TempCompass [ 26 ] dataset under different frame sampling ratios for our method and the baseline [ 53 ] , and under varying token reduction levels for the token pruning method [ 12 ] . Our method with visual prompting consistently outperforms both baselines and maintains high accuracy even with fewer frames.
(a) VP and KFM
Exp.
VP
KFM
V-MME
LongVideoBench
#
TR
E3E
O3O
SSS
total
1
44.1
63.8
48.5
33.0
48.3
2
✓
49.2
62.8
54.6
32.0
49.0
3
✓
45.2
62.8
51.5
37.1
50.2
4
✓
✓
50.3
62.8
57.6
33.0
50.2
Table 3: Ablation Results. Part (a) shows effect of VP and KFM, and part (b) shows effect of different keyword extractors and mapping backbones. Results are reported on VideoMME (V-MME) [ 11 ] and LongVideoBench [ 46 ] .
Figure 7: Performance comparison across various VP styles. Styles 1 to 4 differ in how sequential cues are visually represented, and their impact is evaluated across four video understanding benchmarks.
Appendix figures & tables17 assets
Supplementary material from the paper’s appendix.
Appendix
Position
8 frames
16 frames
32 frames
64 frames
average
–
69.01
35.67
17.54
7.60
32.46
TL
100.0
99.42
100.0
99.42
99.71
TR
100.0
99.42
100.0
97.66
99.27
BL
100.0
100.0
100.0
100.0
100.0
BR
100.0
100.0
100.0
100.0
100.0
Appendix
Table 4: Results showing the accuracy of reverse lookup using VP, measured with a tolerance of ± 1 frame.
Dataset
Category
Questions
Videos
Min (s)
Max (s)
Mean (s)
Std (s)
TempCompass
EO
55
45
3.04
27.20
14.59
6.99
AC
45
39
3.00
24.68
10.46
5.19
Total
100
84
3.00
27.20
12.67
6.55
MVBench
AS
51
49
8.20
43.20
21.12
8.67
ST
49
49
20.00
20.00
20.00
0.00
Total
100
98
8.20
43.20
20.56
6.16
Appendix
Table 5: Statistics of the analysis subsets. For each dataset, we report the number of questions, number of videos, and video length statistics (in seconds) for the 100-example evaluation subsets used in our analysis.
Figure 8: Effect of VP Design Parameters. (a) Results with varying values of the font size control parameter s , and (b) results comparing the presence or absence of text outlines controlled by o .
Models
ARs
ARc
AP
CP
IS
OP
ORs
ORc
SP
SR
TP
mAcc
64 Frames (1 FPS)
Baseline [ 53 ]
56.84
62.30
76.92
48.13
75.23
64.03
57.27
71.19
64.81
78.57
70.91
66.02
+ Ours
55.79
65.81
76.02
46.64
75.23
63.31
58.37
70.34
64.81
80.36
70.91
66.15 ▲
20% Frames
Baseline [ 53 ]
55.09
57.51
67.42
41.79
73.99
58.99
55.73
61.02
57.41
83.93
60.00
59.44
+ Ours
54.74
58.15
68.78
41.04
73.68
54.68
55.95
62.71
59.26
82.14
63.64
61.34 ▲
Appendix
Table 6: Results on Non-Temporal Categories. Evaluation on the non-temporal categories of the VideoMME [ 11 ] dataset. The reported categories include: Action Reasoning (ARs), Action Recognition (ARc), Attribute Perception (AP), Counting Problem (CP), Information Synopsis (IS), OCR Problems (OP), Object Reasoning (ORs), Object Recognition (ORc), Spatial Perception (SP), Spatial Reasoning (SR), and Temporal Perception (TP).
Models
Sampling
VideoMME
LongVideoBench
TR
E3E
O3O
SSS
total
64 Frames (1 FPS)
Baseline [ 53 ]
AKS [ 41 ]
51.41
67.02
57.58
41.24
54.86
+ Ours
AKS [ 41 ]
53.67
72.34
57.58
39.18
56.03
20% Frames
Baseline [ 53 ]
AKS [ 41 ]
45.20
63.83
48.48
39.13
48.25
Appendix
Table 7: Integration with Keyframe Selection Methods. Results showing the effect of combining our method with the state-of-the-art keyframe selection model AKS [ 41 ] on long video datasets, VideoMME [ 11 ] and LongVideoBench [ 46 ] . This evaluates performance when applied to sampled frames from lengthy video inputs.
Models
TempCompass
MVBench
VideoMME
LongVideoBench
EO
AC
total
AS
ST
total
TR
E3E
O3O
SSS
total
30% Frames (Tokens)
Baseline [ 53 ]
71.24
69.32
70.27
67.00
90.00
78.50
44.63
63.83
45.45
36.08
48.64
+ Ours
73.63
71.95
72.78
67.50
93.00
80.25
50.28
64.89
46.97
38.14
50.19
+ Token Pruning [ 12 ]
71.97
63.14
67.51
65.50
91.00
78.25
45.20
64.89
45.45
32.99
47.86
20% Frames (Tokens)
Appendix
Table 8: Comparision with Token-Level Pruning Method. Results on the temporal understanding subsets of the TempCompass [ 26 ] , MVBench [ 21 ] , VideoMME [ 11 ] , and LongVideoBench [ 46 ] datasets.
Figure 9: Additional qualitative temporal results. Examples where our method provides temporally consistent answers, while the baseline VideoLLM fails to correctly capture the order of events.
Figure 10: Effect of Hyperparameter τ . Performance variation across four datasets with different values of the mapping threshold τ .
Figure 11: Effect of Hyperparameter s . Performance variation across four datasets as the font size control parameter s changes.
Figure 12: Effect of VP Position. Results on four datasets [ 26 , 21 , 11 , 46 ] when the visual prompt is inserted at different frame positions: bottom-left (BL), bottom-right (BR), top-left (TL), and top-right (TR).
Dataset
Category
Questions
Videos
Min (s)
Max (s)
Mean (s)
Std (s)
TempCompass
EO
1384
100
3.04
29.36
13.98
6.67
AC
1408
96
3.00
24.70
10.18
4.99
Total
2792
195
3.00
29.36
12.08
6.19
MVBench
AS
200
179
7.10
43.20
20.57
7.93
ST
200
200
20.00
20.00
20.00
0.00
Total
400
379
7.10
43.20
20.27
5.46
Appendix
Table 9: Statistics of the temporal evaluation splits. For each dataset, we report the number of questions, number of videos, and video length statistics (in seconds).
Models
TempCompass
MVBench
VideoMME
LongVideoBench
τ
s
o
τ
s
o
τ
s
o
τ
s
o
64 Frames (1 FPS)
Ours GPT4.1
1.00
12
✗
0.18
9
✗
-
-
-
-
-
-
Ours QwenVL
1.00
12
✗
0.20
9
✗
0.15
12
✗
0.15
15
✗
Ours LLaVA-Video
1.00
12
✗
0.18
9
✗
0.23
12
✗
0.22
12
✓
Ours LLaVA-OneVision
1.00
12
✗
0.18
9
✓
0.28
15
✓
0.18
12
✓
Appendix
Table 10: Hyperparameter Settings. Hyperparameters selected for four different VideoLLM baselines [ 31 , 3 , 53 , 20 ] across four datasets [ 26 , 21 , 11 , 46 ] , under two frame sampling conditions (64 frames and 20% frames), as reported in Table 2 of the main paper. Notation is provided Section D.2 .
Role / Dataset
Prompt
System (shared across all datasets)
system
You are a helpful assistant. Focus on the temporal relationships by referring to the number written in the bottom-left corner of each frame.
TempCompass
user
question{options} Task-specific post-prompt: - Multi-choice: ’Please directly give the best option.’ - Yes/No: ’Please answer yes or no.’ - Caption matching: ’Please directly give the best option.’ - Captioning: ’Please directly give the best option.’
Example
Which event happens first to the skillet? A. Burning in fire B. None of both C. Smoking Please directly give the best option.
MVBench
Appendix
Table 11: System prompt and dataset-specific user prompt templates with examples.
Role / Component
Prompt
System Prompt (Shared)
system
You are a helpful assistant that only extracts keywords and outputs them as a Python list.
User Prompt (Shared Core Instructions)
user
Follow these rules carefully: 1. Identify Key Phrases: Your goal is to extract key phrases from the question that refer to specific scenes, events, actions, or distinct items. 2. Exact Extraction: The extracted phrases must appear exactly as they do in the question. Do not modify or rephrase them. 3. Empty List Condition: If no relevant key phrases (as defined in Rule 1) are found in the question, you must return an empty list []. Now: Question: {$question} Your Answer:
Dataset-Specific User Prompt Examples
TempCompass
Example 1: Question: Which sentence better captures the essence of the video? Your Answer: [] Example 2: Question: Which description is a more suitable match for the video? Your Answer: []
Appendix
Table 12: System prompt, shared keyword-extraction instructions, and dataset-specific examples used for keyword extraction.
Table 25
Figure 13: Flexible zero-padding to reduce overlap with on-screen text. This helps reduce interference with subtitles and task-relevant content.
Figure 14: Example of ambiguous keyword–frame mapping. When a video contains many visually similar consecutive frames, CLIP-based similarity may not be sufficient for accurate mapping.
Temporal video understanding studies how models track actions, events, and changes over time to answer questions, locate moments, and predict what happens next.
Video Large Language Models (VideoLLMs) receive frames in sequential order and interpret how visual content evolves along the temporal axis, yet temporal reasoning remains a persistent weakness across architectures. Reversing the frame order of a video, a transformation that should invert temporal answers, often leaves the final prediction unchanged. We investigate where this failure originates by defining the temporal divergence vector τl, the layer-wise representational difference induced by reversing temporal order. Tracking its magnitude across layers reveals a consistent temporal divergence profile where the divergence peaks at intermediate layers and progressively diminishes toward the output. We confirm this peak is specific to temporal reasoning and functionally critical for predictions, establishing that VideoLLMs acquire temporal information at intermediate layers but fail to maintain it to the output. This progressive fading motivates our method, Temporal Activation Injection (TAI), which extracts τl at the peak of the profile for each input and reinjects it into subsequent layers following the measured decay. TAI requires no training and consistently improves temporal reasoning across three VideoLLMs and four benchmarks with negligible impact on non-temporal tasks. Code is available at https://github.com/Youngwoo-git/Before-It-Fades.
Youngwoo Shin, Yusung Ro, Minseo Kim +1
Korea Advanced Institute of Science and Technology (KAIST)
Long-form video understanding remains challenging for Video Large Language Models (VideoLLMs), as the dense frame sampling introduces massive visual tokens while sparse sampling risks missing critical temporal evidence and leading to LLM hallucination. Existing training-free token reduction methods either treat videos equally as static images or rely on segment-level merging heuristics, which weaken fine-grained spatiotemporal modeling and introduce additional overhead. In this paper, we propose EchoPrune, a lightweight and training-free token pruning method that improves temporal resolution under a fixed LLM-side visual token budget. Our core idea is to interpret redundant video tokens as temporal echoes: if a token is well reconstructed from the previous frame, it is merely a temporally redundant echo; otherwise, it may capture new events, motion, or query-relevant visual evidence. Based on this insight, EchoPrune scores visual tokens by (i) query-guided crossmodal relevance and (ii) temporal reconstruction error, measured by correspondence matching and echo matching across consecutive frames. The selected tokens preserve task-relevant cues and temporal novelty while suppressing predictable redundancy, allowing VideoLLMs to observe more frames without increasing the decoding budget. Extensive experiments on LLaVA-OV, Qwen2.5VL, and Qwen3VL across six video understanding benchmarks show that EchoPrune enables VideoLLMs to process up to 20x frames under the same token budget, yielding improved performance (+8.6%) and inference speedup (5.6x for prefilling) on Qwen2.5VL-7B.
Jiameng Li, Minye Wu, Jiezhang Cao +2
KU Leuven · Shanghai Jiaotong University · Weill Cornell Medicine
Video reasoning requires models to identify and verify temporally localized evidence within long video sequences. Recent Video Large Language Models (Video-LLMs) have shown promising reasoning abilities when aligned with reinforcement learning, yet existing approaches typically rely on outcome-based rewards that supervise only the final prediction. Such supervision provides limited guidance on how models should discover the relevant temporal evidence during intermediate reasoning. In this work, we propose TimeThink, a reinforcement learning framework that explicitly guides temporal evidence discovery in Video-LLMs. Our key idea is to treat temporal clue steps as the fundamental optimization primitive of video reasoning, where each reasoning step references a candidate time interval in the video. We introduce a step-wise temporal process reward that provides localized credit assignment for these clues and a joint process--outcome optimization objective that balances reasoning fidelity with task correctness. To enable scalable training, we construct TimeThink-RFT-20K, a dataset with automatically derived temporal evidence segments. Extensive experiments across video reasoning, temporal grounding, and general video understanding benchmarks show that TimeThink consistently improves both temporal localization and reasoning performance, achieving state-of-the-art results among open-source video RL models.
Handong Li, Longteng Guo, Zikang Liu +8
School of Artificial Intelligence, University of Chinese Academy of Sciences · Institute of Automation, Chinese Academy of Sciences · OPPO AI Center, OPPO Inc.