ViKey: Enhancing Temporal Understanding in Videos via Visual Prompting

Yeonkyung Lee; Dayun Ju; Youngmin Kim; Seil Kang; Seong Jae Hwang

cs.CVMar 24, 2026

ViKey: Enhancing Temporal Understanding in Videos via Visual Prompting

Authors: Yeonkyung Lee, Dayun Ju, Youngmin Kim, Seil Kang, Seong Jae Hwang

Organizations: Yonsei University

Abstract

Recent advancements in Video Large Language Models (VideoLLMs) have enabled strong performance across diverse multimodal video tasks. To reduce the high computational cost of processing dense video frames, efficiency-oriented methods such as frame selection have been widely adopted. While effective at minimizing redundancy, these methods often cause notable performance drops on tasks requiring temporal reasoning. Unlike humans, who can infer event progression from sparse visual cues, VideoLLMs frequently misinterpret temporal relations when intermediate frames are omitted. To address this limitation, we explore visual prompting (VP) as a lightweight yet effective way to enhance temporal understanding in VideoLLMs. Our analysis reveals that simply annotating each frame with explicit ordinal information helps the model perceive temporal continuity. This visual cue also supports frame-level referencing and mitigates positional ambiguity within a sparsely sampled sequence. Building on these insights, we introduce ViKey, a training-free framework that combines VP with a lightweight Keyword-Frame Mapping (KFM) module. KFM leverages frame indices as dictionary-like keys to link textual cues to the most relevant frames, providing explicit temporal anchors during inference. Despite its simplicity, our approach substantially improves temporal reasoning and, on some datasets, preserves dense-frame baseline performance with as few as 20% of frames.

Figures & tables

Figure 1: Illustrations of How Frame Selection Disrupts and Visual Prompting Restores Temporal Reasoning. Frame selection improves efficiency but breaks temporal continuity, causing VideoLLMs to misinterpret transitions. With frame-index visual prompts, the model regains temporal order and correctly identifies causal relations under sparse-frame conditions.

Figure 2: Probing VP for temporal understanding. (a) Positional Embedding Degradation: remove temporal order only, or collapse both temporal and spatial positions. (b) Frame-level Referencing in VideoLLMs: frame-index prompts enable lookup and reverse-lookup between indices and content. (c) Attention Analysis: VP increases attention to image tokens across layers.

Figure 3: Effect of VP under Position Degradation. Across both baselines, frame-index VP consistently improve accuracy under temporal-only and full-collapse settings, indicating added temporal cues and robustness to degraded positional signals.

(a) Look-up
Position	8 frames	16 frames	32 frames	64 frames	average
–	28.07	11.70	5.85	4.09	12.43
TL	67.25	60.23	55.56	39.18	55.56
TR	56.73	53.80	51.46	31.58	48.39
BL	77.19	64.91	63.16	53.22	64.62
BR	76.02	64.33	64.33	51.46	64.04

Table 1: Results on lookup and reverse-lookup tasks. The table compares performance across VP placements, with and without VP, for mapping frame numbers to scenes (lookup) and scenes back to frame numbers (reverse lookup).

Figure 4: Overall pipeline of ViKey . Given a sequence of video frames, we first apply visual prompting by overlaying frame-number VPs on each frame. For a user query, the system extracts key textual concepts and performs Keyword–Frame Mapping (KFM) to identify which frames best correspond to each keyword. The query is then rewritten to explicitly include the mapped frame indices, aligning textual cues with the numbered frames. Feeding this aligned query–frame pair into the VLM enables more accurate temporal understanding.

	Models	TempCompass			MVBench			VideoMME	LongVideoBench
	Models	EO	AC	total	AS	ST	total	TR	E3E	O3O	SSS	total
64 Frames (1 FPS)	VideoLLM Baselines
	GPT4.1 [ 31 ]	66.69	74.72	70.74	75.00	92.50	83.75	$-$	$-$	$-$	$-$	$-$
	Qwen2.5-VL-7B [ 3 ]	75.00	80.26	77.65	79.50	92.00	85.75	38.42	57.45	48.48	40.21	48.63
	LLaVA-Video-7B [ 53 ]	73.63	75.71	74.68	74.00	91.00	82.50	47.46	69.15	62.12	40.21	56.42
	LLaVA-OneVision-7B [ 20 ]	71.24	71.66	71.45	73.00	92.00	82.50	43.50	67.02	56.06	36.08	52.53

Table 2: Main results. Results on the temporal understanding subsets of the TempCompass [ 26 ] , MVBench [ 21 ] , VideoMME [ 11 ] , and LongVideoBench [ 46 ] datasets. Our VP-based method consistently improves performance across all datasets, and even when the input is limited to 20% of the frames, it achieves accuracy comparable to the 1 FPS dense-frame setting in some cases.

Figure 5: Qualitative Results on Open-Ended Temporal Questions. Compared with the baseline LLaVA-Video-7B, ViKey uses explicit frame indices and keyword–frame mapping to better capture temporal order and thus produce more accurate answers to open-ended temporal queries. In particular, it often explicitly mentions the injected frame indices ( e.g . , “frame #01”, “frame #02”) in its free-form responses, indicating that its temporal reasoning is grounded on the visual prompts.

Figure 6: Performance Comparison under Varying Frame Sampling Ratios. Accuracy on the TempCompass [ 26 ] dataset under different frame sampling ratios for our method and the baseline [ 53 ] , and under varying token reduction levels for the token pruning method [ 12 ] . Our method with visual prompting consistently outperforms both baselines and maintains high accuracy even with fewer frames.

(a) VP and KFM
Exp.	VP	KFM	V-MME	LongVideoBench
#	VP	KFM	TR	E3E	O3O	SSS	total
1			44.1	63.8	48.5	33.0	48.3
2	✓		49.2	62.8	54.6	32.0	49.0
3		✓	45.2	62.8	51.5	37.1	50.2
4	✓	✓	50.3	62.8	57.6	33.0	50.2

Table 3: Ablation Results. Part (a) shows effect of VP and KFM, and part (b) shows effect of different keyword extractors and mapping backbones. Results are reported on VideoMME (V-MME) [ 11 ] and LongVideoBench [ 46 ] .

Figure 7: Performance comparison across various VP styles. Styles 1 to 4 differ in how sequential cues are visually represented, and their impact is evaluated across four video understanding benchmarks.

Appendix figures & tables17 assets

Supplementary material from the paper’s appendix.

Appendix

Position	8 frames	16 frames	32 frames	64 frames	average
–	69.01	35.67	17.54	7.60	32.46
TL	100.0	99.42	100.0	99.42	99.71
TR	100.0	99.42	100.0	97.66	99.27
BL	100.0	100.0	100.0	100.0	100.0
BR	100.0	100.0	100.0	100.0	100.0

Appendix

Table 4: Results showing the accuracy of reverse lookup using VP, measured with a tolerance of

\pm

1 frame.

Dataset	Category	Questions	Videos	Min (s)	Max (s)	Mean (s)	Std (s)
TempCompass	EO	55	45	3.04	27.20	14.59	6.99
	AC	45	39	3.00	24.68	10.46	5.19
	Total	100	84	3.00	27.20	12.67	6.55
MVBench	AS	51	49	8.20	43.20	21.12	8.67
	ST	49	49	20.00	20.00	20.00	0.00
	Total	100	98	8.20	43.20	20.56	6.16

Appendix

Table 5: Statistics of the analysis subsets. For each dataset, we report the number of questions, number of videos, and video length statistics (in seconds) for the 100-example evaluation subsets used in our analysis.

Figure 8: Effect of VP Design Parameters. (a) Results with varying values of the font size control parameter $s$ , and (b) results comparing the presence or absence of text outlines controlled by $o$ .

Models	ARs	ARc	AP	CP	IS	OP	ORs	ORc	SP	SR	TP	mAcc
64 Frames (1 FPS)
Baseline [ 53 ]	56.84	62.30	76.92	48.13	75.23	64.03	57.27	71.19	64.81	78.57	70.91	66.02
+ Ours	55.79	65.81	76.02	46.64	75.23	63.31	58.37	70.34	64.81	80.36	70.91	66.15 $\blacktriangle$
20% Frames
Baseline [ 53 ]	55.09	57.51	67.42	41.79	73.99	58.99	55.73	61.02	57.41	83.93	60.00	59.44
+ Ours	54.74	58.15	68.78	41.04	73.68	54.68	55.95	62.71	59.26	82.14	63.64	61.34 $\blacktriangle$

Appendix

Table 6: Results on Non-Temporal Categories. Evaluation on the non-temporal categories of the VideoMME [ 11 ] dataset. The reported categories include: Action Reasoning (ARs), Action Recognition (ARc), Attribute Perception (AP), Counting Problem (CP), Information Synopsis (IS), OCR Problems (OP), Object Reasoning (ORs), Object Recognition (ORc), Spatial Perception (SP), Spatial Reasoning (SR), and Temporal Perception (TP).

Models	Sampling	VideoMME	LongVideoBench
Models	Sampling	TR	E3E	O3O	SSS	total
64 Frames (1 FPS)
Baseline [ 53 ]	AKS [ 41 ]	51.41	67.02	57.58	41.24	54.86
+ Ours	AKS [ 41 ]	53.67	72.34	57.58	39.18	56.03
20% Frames
Baseline [ 53 ]	AKS [ 41 ]	45.20	63.83	48.48	39.13	48.25

Appendix

Table 7: Integration with Keyframe Selection Methods. Results showing the effect of combining our method with the state-of-the-art keyframe selection model AKS [ 41 ] on long video datasets, VideoMME [ 11 ] and LongVideoBench [ 46 ] . This evaluates performance when applied to sampled frames from lengthy video inputs.

Models	TempCompass			MVBench			VideoMME	LongVideoBench
Models	EO	AC	total	AS	ST	total	TR	E3E	O3O	SSS	total
30% Frames (Tokens)
Baseline [ 53 ]	71.24	69.32	70.27	67.00	90.00	78.50	44.63	63.83	45.45	36.08	48.64
+ Ours	73.63	71.95	72.78	67.50	93.00	80.25	50.28	64.89	46.97	38.14	50.19
+ Token Pruning [ 12 ]	71.97	63.14	67.51	65.50	91.00	78.25	45.20	64.89	45.45	32.99	47.86
20% Frames (Tokens)

Appendix

Table 8: Comparision with Token-Level Pruning Method. Results on the temporal understanding subsets of the TempCompass [ 26 ] , MVBench [ 21 ] , VideoMME [ 11 ] , and LongVideoBench [ 46 ] datasets.

Figure 9: Additional qualitative temporal results. Examples where our method provides temporally consistent answers, while the baseline VideoLLM fails to correctly capture the order of events.

Figure 10: Effect of Hyperparameter $\tau$ . Performance variation across four datasets with different values of the mapping threshold $\tau$ .

Figure 11: Effect of Hyperparameter $s$ . Performance variation across four datasets as the font size control parameter $s$ changes.

Figure 12: Effect of VP Position. Results on four datasets [ 26 , 21 , 11 , 46 ] when the visual prompt is inserted at different frame positions: bottom-left (BL), bottom-right (BR), top-left (TL), and top-right (TR).

Dataset	Category	Questions	Videos	Min (s)	Max (s)	Mean (s)	Std (s)
TempCompass	EO	1384	100	3.04	29.36	13.98	6.67
	AC	1408	96	3.00	24.70	10.18	4.99
	Total	2792	195	3.00	29.36	12.08	6.19
MVBench	AS	200	179	7.10	43.20	20.57	7.93
	ST	200	200	20.00	20.00	20.00	0.00
	Total	400	379	7.10	43.20	20.27	5.46

Appendix

Table 9: Statistics of the temporal evaluation splits. For each dataset, we report the number of questions, number of videos, and video length statistics (in seconds).

Models	TempCompass			MVBench			VideoMME			LongVideoBench
Models	$\tau$	$s$	$o$	$\tau$	$s$	$o$	$\tau$	$s$	$o$	$\tau$	$s$	$o$
64 Frames (1 FPS)
Ours GPT4.1	1.00	12	✗	0.18	9	✗	-	-	-	-	-	-
Ours QwenVL	1.00	12	✗	0.20	9	✗	0.15	12	✗	0.15	15	✗
Ours LLaVA-Video	1.00	12	✗	0.18	9	✗	0.23	12	✗	0.22	12	✓
Ours LLaVA-OneVision	1.00	12	✗	0.18	9	✓	0.28	15	✓	0.18	12	✓

Appendix

Table 10: Hyperparameter Settings. Hyperparameters selected for four different VideoLLM baselines [ 31 , 3 , 53 , 20 ] across four datasets [ 26 , 21 , 11 , 46 ] , under two frame sampling conditions (64 frames and 20% frames), as reported in Table 2 of the main paper. Notation is provided Section D.2 .

Role / Dataset	Prompt
System (shared across all datasets)
system	You are a helpful assistant. Focus on the temporal relationships by referring to the number written in the bottom-left corner of each frame.
TempCompass
user	${question}$ {options} Task-specific post-prompt: - Multi-choice: ’Please directly give the best option.’ - Yes/No: ’Please answer yes or no.’ - Caption matching: ’Please directly give the best option.’ - Captioning: ’Please directly give the best option.’
Example	Which event happens first to the skillet? A. Burning in fire B. None of both C. Smoking Please directly give the best option.
MVBench

Appendix

Table 11: System prompt and dataset-specific user prompt templates with examples.

Role / Component	Prompt
System Prompt (Shared)
system	You are a helpful assistant that only extracts keywords and outputs them as a Python list.
User Prompt (Shared Core Instructions)
user	Follow these rules carefully: 1. Identify Key Phrases: Your goal is to extract key phrases from the question that refer to specific scenes, events, actions, or distinct items. 2. Exact Extraction: The extracted phrases must appear exactly as they do in the question. Do not modify or rephrase them. 3. Empty List Condition: If no relevant key phrases (as defined in Rule 1) are found in the question, you must return an empty list []. Now: Question: {$question} Your Answer:
Dataset-Specific User Prompt Examples
TempCompass	Example 1: Question: Which sentence better captures the essence of the video? Your Answer: [] Example 2: Question: Which description is a more suitable match for the video? Your Answer: []

Appendix

Table 12: System prompt, shared keyword-extraction instructions, and dataset-specific examples used for keyword extraction.

Table 25

Figure 13: Flexible zero-padding to reduce overlap with on-screen text. This helps reduce interference with subtitles and task-relevant content.

Figure 14: Example of ambiguous keyword–frame mapping. When a video contains many visually similar consecutive frames, CLIP-based similarity may not be sufficient for accurate mapping.

Temporal Video Understanding

Temporal video understanding studies how models track actions, events, and changes over time to answer questions, locate moments, and predict what happens next.

See all 371 papers

Before It Fades: Reinforcing Temporal Representations at Inference Time in VideoLLMs
Oct 1, 2026Youngwoo Shin, Yusung Ro, Minseo Kim +1Video-Language Models Activation Steering
Korea Advanced Institute of Science and Technology (KAIST)
EchoPrune: Interpreting Redundancy as Temporal Echoes for Efficient VideoLLMs
May 11, 2026Jiameng Li, Minye Wu, Jiezhang Cao +2Efficient VLM Inference Token Pruning
KU Leuven · Shanghai Jiaotong University · Weill Cornell Medicine
TimeThink: Reasoning with Time for Video LLMs
Jul 6, 2026Handong Li, Longteng Guo, Zikang Liu +8Temporal Video Grounding Process Reward Models
School of Artificial Intelligence, University of Chinese Academy of Sciences · Institute of Automation, Chinese Academy of Sciences · OPPO AI Center, OPPO Inc.

Oct 1, 2026J/K move · Enter open · S save

Before It Fades: Reinforcing Temporal Representations at Inference Time in VideoLLMs

Youngwoo Shin, Yusung Ro, Minseo Kim, Junmo Kim

Korea Advanced Institute of Science and Technology (KAIST)

Video-Language Models Activation Steering Temporal Representation Learning

Open paper AskPDF arXiv

Video Large Language Models (VideoLLMs) receive frames in sequential order and interpret how visual content evolves along the temporal axis, yet temporal reasoning remains a persistent weakness across architectures. Reversing the frame order of a video, a transformation that should invert temporal answers, often leaves the final prediction unchanged. We investigate where this failure originates by defining the temporal divergence vector

τ_l

, the layer-wise representational difference induced by reversing temporal order. Tracking its magnitude across layers reveals a consistent temporal divergence profile where the divergence peaks at intermediate layers and progressively diminishes toward the output. We confirm this peak is specific to temporal reasoning and functionally critical for predictions, establishing that VideoLLMs acquire temporal information at intermediate layers but fail to maintain it to the output. This progressive fading motivates our method, Temporal Activation Injection (TAI), which extracts

τ_l

at the peak of the profile for each input and reinjects it into subsequent layers following the measured decay. TAI requires no training and consistently improves temporal reasoning across three VideoLLMs and four benchmarks with negligible impact on non-temporal tasks. Code is available at https://github.com/Youngwoo-git/Before-It-Fades.

Abstract

Temporal Video Understanding

Explore similar work

Before It Fades: Reinforcing Temporal Representations at Inference Time in VideoLLMs

EchoPrune: Interpreting Redundancy as Temporal Echoes for Efficient VideoLLMs

TimeThink: Reasoning with Time for Video LLMs