GraphThinker: Reinforcing Temporally Grounded Video Reasoning with Event Graph Thinking
Authors: Zixu Cheng, Da Li, Jian Hu, Yuhang Zang, Ziquan Liu, Shaogang Gong, Wei Li
Organizations: Queen Mary University of London · Samsung AI Centre Cambridge · Shanghai Artificial Intelligence Laboratory · Nanyang Technological University
Video reasoning requires a fine-grained understanding of the temporal dependencies and event-level relations between objects and events in videos. Current Multimodal Large Language Models (MLLMs) are prone to severe temporal hallucinations in video reasoning. An underlying cause of these hallucinations is weak visual-temporal grounding and the lack of explicit structure for modelling event relations. Models often rely on auxiliary text, such as dense captions, rather than explicitly anchoring their reasoning in actual visual evidence. However, these textual representations are inherently unstructured and fail to provide explicit causal constraints needed to guide the model's reasoning. In this work, we propose GraphThinker, a reinforcement finetuning method that constructs a structured event representation of a video and enforces visual grounding to jointly reduce reasoning hallucinations. Specifically, we employ an MLLM to construct an Event-based Video Scene Graph (EVSG) that captures both intra- and inter-event relations, guiding a structured video reasoning process. Moreover, we address the weak grounding issue by introducing a novel visual attention reward during reinforcement finetuning that encourages the model to actively attend to reliable visual cues. On the RexTime dataset, GraphThinker achieves an over 4% improvement in IoU=0.3 for moment localisation. On the VidHalluc dataset, GraphThinker achieves a 9.8% improvement in reducing temporal sequence hallucination and a 7.6% gain in Binary QA in reducing action hallucination, compared to the state-of-the-art methods.
Figures & tables
Figure 1: An overview of the GraphThinker . It first employs an MLLM to generate multi-grained dense captions for a video, and then prompts the same MLLM to select keywords as nodes to construct an event-based graph (EVSG). This EVSG is then used to serve as a fine-grained representation of structured event relations for reasoning. Given EVSGs, we further develop an Event Graph-based RL finetuning method with a visual attention reward to ground more attentive visual evidence. Together, GraphThinker achieves visually more grounded and temporally more consistent video reasoning.
Figure 2: An example of the proposed Event-based Video Scene Graph (EVSG). The EVSG comprises event subgraphs (derived from event-level captions), each subgraph corresponding to start–end timestamps and a set of triplets that represent object interactions and spatial relationships to capture intra-event semantics. Event subgraphs are sequentially linked by timestamp-based edges, forming a hierarchical structure that explicitly models both intra-event and inter-event relations.
Models
Param.
Type
Moment Localization
VQA
mIoU
R@1,IoU=0.3
R@1,IoU=0.5
Acc.
Acc.@IoU ≥ 0.5
GPT-4o OpenAI (2024)
–
ZS
36.28
45.33
34.00
73.67
28.67
Claude3-Opus Anthropic (2024)
–
ZS
23.61
30.67
17.67
68.67
13.67
Gemini-1.5-Pro Team et al. (2023)
–
ZS
28.43
35.67
25.00
68.00
18.33
GPT-4V OpenAI (2023)
–
ZS
26.74
33.33
22.00
63.33
16.67
Reka-Core Team et al. (2024)
–
ZS
27.95
36.33
24.00
59.67
17.00
Table 1: Comparison of SOTA methods on Rextime: “ZS”, “VLP”, “SFT”, and “RL” denote Zero-shot, Vision-Language Pretrained, Supervised Fine-Tuning, and Reinforcement Learning. † denotes results evaluated by us. “–” indicates not applicable. The shaded denote closed-source models.
Models
Params
Accuracy on ACH ↑
Accuracy on TSH ↑
Score on STH ↑
Binary QA ↑
MCQ ↑
Gemini-1.5-Pro Team et al. (2023)
∼ 200B
75.27
79.25
83.83
63.96
GPT-4o OpenAI (2024)
∼ 200B
81.15
90.95
82.00
71.58
Video-ChatGPT Maaz et al. (2024)
7B
9.50
24.58
30.17
7.70
Video-LLAVA Lin et al. (2024a)
7B
26.84
64.45
27.17
29.60
ShareGPT4Video Chen et al. (2024b)
8B
29.96
44.78
49.50
17.83
Table 2: Comparisons of methods on the three test sets on the VidHalluc benchmark, including Action Hallucination (ACH), Temporal Sequence Hallucination (TSH) and Scene Transition Hallucination (STH) test. † denotes results evaluated by us. *denotes finetuned on Rextime. The shaded denote closed-source models.
Figure 3: Ablation studies of Qwen2.5-VL-7B on Rextime validation set. DC: dense captions.
Figure 4: Parameter sensitivity on the Rextime validation set.
Figure 5: (a) Threshold sensitivity heatmap for rsim and rIoU , reporting [email protected] on the RexTime validation set with Qwen2.5-VL-3B. (b) Effect of the attention reward rattn on visual attention. (c) Inference cost analysis.
Figure 6: Illustration of GraphThinker reducing temporal hallucinations in reasoning. Qwen2.5-VL with dense captions after GRPO still produces hallucinated predictions, as redundant captions can obscure key evidence. In contrast, GraphThinker uses EVSG to organise key entities, relations, and their temporal order, guiding the model to focus on the relevant visual cues and produce more visually grounded and temporally consistent predictions. The visual attention reward further encourages attention to related video tokens during reasoning.
Appendix figures & tables11 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 7: The prompt template for multi-grained dense caption generation.
Figure 8: The prompt template for graph generation with middle-grained dense captioning.
Figure 9: The prompt template for graph refinement with coarse- and fine-grained dense captioning.
Figure 10: The prompt template for evaluation on the ReXTime [ 4 ] benchmark.
Table 4: Effect of different graph constructors on RexTime validation set. The baseline is fixed as Qwen2.5-VL-7B. “DC” denotes dense captions.
Aspect
Correct (%)
Partial (%)
Incorrect (%)
Event boundary correctness
76
18
6
Triplet faithfulness
72
16
12
Appendix
Table 5: Sample-level manual faithfulness audit of EVSG on 50 randomly sampled RexTime validation videos. Each aspect is rated as Correct, Partial, or Incorrect for the whole generated EVSG of each sample.
Figure 11: An illustration examples showing that GraphThinker improves causal reasoning in ReXTime [ 4 ] benchmark.
Figure 12: An illustration examples showing that GraphThinker mitigate Temporal Sequence Hallucination (TSH) in VidHalluc [ 30 ] benchmark.
Figure 13: An illustration examples showing that GraphThinker mitigate scene transition hallucination (STH) in VidHalluc [ 30 ] benchmark.
Figure 14: An illustration examples showing that GraphThinker mitigate action hallucination (ACH) in VidHalluc [ 30 ] benchmark.
School of Artificial Intelligence, University of Chinese Academy of Sciences · Institute of Automation, Chinese Academy of Sciences · OPPO AI Center, OPPO Inc.
Wangxuan Institute of Computer Technology, Peking University · Central Media Technology Institute, Huawei Technologies Ltd. · State Key Laboratory of General Artificial Intelligence, Peking University.