Incentivizing Temporal-Awareness in Egocentric Video Understanding Models

Zhiyang Xu; Tian Qin; Bowen Jin; Zhengfeng Lai; Meng Cao; Lifu Huang; Peng Zhang

cs.CVMar 28, 2026

Incentivizing Temporal-Awareness in Egocentric Video Understanding Models

Authors: Zhiyang Xu, Tian Qin, Bowen Jin, Zhengfeng Lai, Meng Cao, Lifu Huang, Peng Zhang

Organizations: Virginia Tech · Harvard University · UIUC · UC Davis ·

Abstract

Multimodal large language models (MLLMs) have recently shown strong performance in visual understanding, yet they often lack temporal awareness, particularly in egocentric settings where reasoning depends on the correct ordering and evolution of events. This deficiency stems in part from training objectives that fail to explicitly reward temporal reasoning and instead rely on frame-level spatial shortcuts. To address this limitation, we propose Temporal Global Policy Optimization (TGPO), a reinforcement learning with verifiable rewards (RLVR) algorithm designed to incentivize temporal awareness in MLLMs. TGPO contrasts model outputs generated from temporally ordered versus shuffled video frames to derive calibrated, globally normalized reward signals that explicitly favor temporally coherent reasoning. Integrated with GRPO and GSPO, TGPO supports cold-start RL training and effectively suppresses spatial shortcut behaviors learned by existing MLLMs. Experiments across five egocentric video benchmarks demonstrate that TGPO consistently improves temporal grounding and causal coherence, outperforming prior RL-based video reasoning approaches. Our results suggest that TGPO offers a simple and scalable pathway toward temporally robust MLLMs for egocentric video understanding.

Figures & tables

Figure 1: Performance comparison of Temporal Global Policy Optimization ( TGPO ), GRPO, and GSPO on five egocentric benchmarks.

Figure 2: An overview of our proposed TGPO . In the upper half, the original video is passed into the policy model, and a group of responses are sampled. Rewards are assigned to each response in the group. In the bottom half, the policy model takes in a shuffled video and performs greedy decoding to generate responses. The reward model assigns scores to responses without temporal information.

Model	EgoSchema $\uparrow$	EgoPlan $\uparrow$	EgoPlan 2 $\uparrow$	VLM4D $\uparrow$	EgoTempo $\uparrow$
Qwen2.5-VL 3B
+ CoT	20.4	22.2	23.5	37.1	31.6
+ GRPO	46.5	36.5	37.1	46.8	40.0
+ GSPO	45.4	36.3	32.7	47.4	42.0
+ TGPO (GRPO)	49.6	36.8	42.3	49.6	45.2
+ TGPO (GSPO)	49.7	36.7	41.1	48.6	42.6

Table 1: Performance comparison of our method TGPO with two popular RL-based optimization methods and chain-of-thought (CoT) reasoning across egocentric benchmarks.

Method	VLM4D	EgoPlan	EgoPlan2	EgoSchema
Baseline
GRPO	1265.13	898.25	784.04	1159.03
GSPO	1305.99	693.21	552.01	1118.76
Ours
TGPO (GRPO)	1339.49	923.93	786.97	1306.32
TGPO (GSPO)	1333.75	905.82	544.95	1144.84

Table 2: Area Under the Curve (AUC) of reward over the first 3000 training steps for different optimization methods across datasets. A higher AUC reflects faster reward improvement and improved training stability.

Figure 3: Test reward over the 3000 training steps. The reward curves are reported on four benchmarks across GRPO, GSPO, and our two TGPO variants.

Table 6

Table 7

Model	EgoTempo $\uparrow$
Proprietary
Gemini-Flash	39.1
GPT-4o	40.1
Claude-3.5-Sonnet	13.1
Open-source MLLMs
Qwen2-VL (7B)	26.1

Table 7: Performance comparison on EgoTempo benchmark.

Figure 4: Qualitative comparison between TGPO and GRPO on the EgoSchema Benchmark.

Egocentric Video Understanding

Egocentric video understanding studies how AI interprets first-person video to recognize actions, track interactions, reason about space and time, and assist users.

See all 203 papers

Temporal-Aware Reasoning Optimization for Video Temporal Grounding
Jun 8, 2026Minghang Zheng, Zihao Yin, Yi Yang +2Temporal Video Grounding Temporal Reasoning
Wangxuan Institute of Computer Technology, Peking University · Central Media Technology Institute, Huawei Technologies Ltd. · State Key Laboratory of General Artificial Intelligence, Peking University.
TimeThink: Reasoning with Time for Video LLMs
Jul 6, 2026Handong Li, Longteng Guo, Zikang Liu +8Temporal Video Grounding Process Reward Models
School of Artificial Intelligence, University of Chinese Academy of Sciences · Institute of Automation, Chinese Academy of Sciences · OPPO AI Center, OPPO Inc.
EgoVITA: Learning to Plan and Verify for Egocentric Video Reasoning
Nov 23, 2025Yogesh Kulkarni, Pooyan FazliEgocentric Video Understanding Video Reasoning
Arizona State University

Jun 8, 2026J/K move · Enter open · S save

Temporal-Aware Reasoning Optimization for Video Temporal Grounding

Minghang Zheng, Zihao Yin, Yi Yang, Yuxin Peng+1

Wangxuan Institute of Computer Technology, Peking University · Central Media Technology Institute, Huawei Technologies Ltd. · State Key Laboratory of General Artificial Intelligence, Peking University.

Temporal Video Grounding Temporal Reasoning VLM Reasoning RL for VLMs

Open paper AskPDF arXiv

Multi-modal Large Language Models (MLLMs) have achieved remarkable progress in video temporal grounding with reinforcement learning for generating reasoning paths. However, existing models often produce superficial reasoning, which offers limited guidance for precise temporal localization. This limitation stems from (1) inefficient random exploration and (2) reward functions that focus solely on the answer correctness while ignoring reasoning quality. To address these issues, we propose TaRO (Temporal-Aware Reasoning Optimization), a framework that explicitly enhances the model's ability of thinking with time. First, we introduce a Constructive Reasoning Exploration that leverages pre-generated dense captions to construct reasoning paths grounded in explicit visual cues and timestamps, enabling efficient exploration of high-quality time-aware reasoning. Second, to evaluate reasoning quality, we design a Temporal-Sensitivity Reward. High-quality reasoning should be anchored to specific events and timestamps. If the event boundary under thinking is disrupted, such reasoning should become invalid, leading to a drop in the logit of the reasoning path. We utilize this drop as a critique of reasoning quality. Finally, TaRO follows a progressive curriculum, which starts by utilizing this reward to select better constructed reasoning paths, and evolves to a free exploration phase where the model autonomously generates effective reasoning. Experiments demonstrate that TaRO achieves state-of-the-art performance on VTG benchmarks. Code is available at https://github.com/oceanflowlab/TaRO.

Abstract

Egocentric Video Understanding

Explore similar work

Temporal-Aware Reasoning Optimization for Video Temporal Grounding

TimeThink: Reasoning with Time for Video LLMs

EgoVITA: Learning to Plan and Verify for Egocentric Video Reasoning