Reinforcing Consistency in Video MLLMs with Structured Rewards

Yihao Quan; Zeru Shi; Jinman Zhao; Ruixiang Tang

cs.CVApr 1, 2026

Reinforcing Consistency in Video MLLMs with Structured Rewards

Authors: Yihao Quan, Zeru Shi, Jinman Zhao, Ruixiang Tang

Organizations: Rutgers University · University of Toronto

Abstract

Multimodal large language models (MLLMs) have achieved remarkable progress in video understanding. However, seemingly plausible outputs often suffer from poor visual and temporal grounding: a model may fabricate object existence, assign incorrect attributes, or collapse repeated events while still producing a globally reasonable caption or answer. We study this failure mode through a compositional consistency audit that decomposes a caption into supporting factual and temporal claims, investigating whether a correct high-level prediction is actually backed by valid lower-level evidence. Our top-down audit reveals that even correct root relational claims often lack reliable attribute and existence support. This indicates that standard sentence-level supervision is a weak proxy for faithful video understanding. Furthermore, when turning to reinforcement learning (RL) for better alignment, standard sentence-level rewards often prove too coarse to accurately localize specific grounding failures. To address this, we replace generic sentence-level rewards with a structured reward built from factual and temporal units. Our training objective integrates three complementary components: (1) an instance-aware scene-graph reward for factual objects, attributes, and relations; (2) a temporal reward for event ordering and repetition; and (3) a video-grounded VQA reward for hierarchical self-verification. Across temporal, general video understanding, and hallucination-oriented benchmarks, this objective yields consistent gains on open-source backbones. These results suggest that structured reward shaping is a practical route to more faithful video understanding.

Figures & tables

Figure 1: Motivating observation. We evaluate a caption with a top-down compositional consistency audit. Starting from a root claim, we ask whether the supporting evidence beneath it is also correct. The audit set contains 200 human-created audit samples derived from AGQA-Decomp, with the root, attribute, and existence labels manually refined and verified.

Figure 2: Overview of our method. We compare the sampled caption and the reference caption in a shared structured space rather than at the whole-sentence level. After lightweight preprocessing, both captions are parsed into scene-graph elements, scored by revision-based factual, temporal, and VQA verification rewards, and then used to update the policy with REINFORCE plus KL regularization.

Metric	Base	Ours	$\Delta$
Root Relation Accuracy	36.6	47.8	+11.2
Attribute Conditional Accuracy	59.7	78.4	+18.7
Existence Conditional Accuracy	78.3	92.1	+13.8

Table 1: Post-training top-down audit under the same protocol as Figure 1 . Our method improves the full root-to-attribute-to-existence support chain, with especially large gains on the conditional attribute and existence accuracies.

Models	Temporal			Conventional			Hallucination
Models	TempComp.	TVBench	Avg.	Video-MME	MVBench	Avg.	VidHallucer	EventHall.	Avg.
Closed-source Models
GPT-4o	73.8	39.9	56.9	71.9	49.1	60.5	53.3	91.9	72.6
Gemini 1.5 Pro	67.1	47.6	57.4	75.0	60.5	67.8	37.8	80.4	59.1
Open-source Models
LLaVA-OV-7B	68.3	42.8	55.6	50.6	54.4	52.5	46.4	60.2	53.3

Table 2: Performance comparisons on benchmarks for temporal, conventional, and hallucination-oriented video understanding. Bold marks the best per group; highlights indicate the best benchmark results.

Figure 3: Reward-component and training-stage ablations. Results are averaged within temporal understanding, conventional video understanding, and hallucination-oriented benchmark groups.

Setting	$\mathbf{w_{\text{SG}}}$	$\mathbf{w_{\text{Temp}}}$	$\mathbf{w_{\text{VQA}}}$	Temporal	Conventional	Hallucination	Avg.
Ours	0.15	0.25	0.35	61.55	62.05	65.10	62.90
Rebalance to SG	0.20	0.25	0.35	61.43	62.10	65.22	62.92
Rebalance to Temp	0.15	0.30	0.35	61.70	61.96	65.00	62.89
Rebalance to VQA	0.15	0.25	0.40	61.31	62.14	65.21	62.89

Table 3: Ablation study on reward weights for Qwen3-VL-8B. SG refers to the scene graph reward. Results are averaged within temporal understanding, conventional video understanding, and hallucination-oriented benchmark groups.

Figure 4: Scaling behavior across model size and training data size. We study two axes of scaling for our post-training recipe: backbone size and training data size. Results are averaged within temporal understanding, conventional video understanding, and hallucination-oriented benchmark groups.

Category	Benchmark	Base			Ours
Category	Benchmark	8	32	64	8	32	64
Temporal	TempCompass	74.1	74.6	74.2	75.1 +1.0	75.5 +0.9	76.4 +2.2
Temporal	TVBench	47.0	46.8	47.9	48.0 +1.0	48.7 +1.9	49.2 +1.3
Conventional	Video-MME	56.7	64.2	67.0	57.4 +0.7	64.6 +0.4	67.3 +0.3
Conventional	MVBench	65.8	68.5	69.2	66.7 +0.9	69.1 +0.6	69.4 +0.2
Hallucination	VideoHallucer	55.7	60.3	62.2	56.6 +0.9	67.2 +6.9	63.1 +0.9

Table 4: Frame-budget ablations on Qwen3-VL-8B. Increasing sampled frames from 8 to 32 yields substantial gains across most benchmarks, while further scaling to 64 frames leads to performance saturation or slight degradation due to redundancy and noise.

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

Setting	SFT	RL
Backbone	Qwen3-VL-8B-Instruct	SFT-initialized Qwen3-VL-8B
Training data	TemporalBench short-caption	Aligned SFT version of the same pool
# training samples	1,891	1,891
Fine-tuning	LoRA ( $r{=}8$ , $\alpha{=}16$ , drop.=0.05)	LoRA ( $r{=}8$ , $\alpha{=}16$ , drop.=0.05)
Optimizer	AdamW	AdamW
Learning rate	$1\times 10^{-5}$	$5\times 10^{-6}$

Appendix

Table 5: Default 8B training recipe used in the main results.

Module / Parameter	Setting
Scene-graph parser	FACTUAL parser ( flan-t5-base-VG-factual-sg )
Beam size	5
Max input length	512
Max output length	128
Sentence preprocessing	spaCy ( en_core_web_sm ) sentence splitting

Appendix

Table 6: Parser and preprocessing settings used by the reward modules in the main experiments.

Root type	Template	Positive source	Negative source
Object existence	Is there a/an [OBJECT]?	reference support chain	none; we do not create open-world existence negatives
Attribute	Is the [OBJECT] [ATTRIBUTE]?	reference support chain	wrong attribute on a matched object slot
Relation	Does the [SUBJECT] [RELATION] the [OBJECT]?	reference support chain	wrong relation on matched endpoints
Event occurrence	Does [PARTICIPANTS] [EVENT]?	reference event occurrence	wrong participant binding on a matched event slot
Temporal order	Did [EVENT 1] happen before [EVENT 2]?	reference ordered pair	reversed supported order; event collapse/swap conflict

Appendix

Table 7: Question templates and supervision sources for the revised verification rewards. The local descriptor

d(o)

uses the reference phrase for the matched instance and may include a local modifier or index token when it is needed to distinguish repeated instances.

	Sentence-level quality audit (48 sentences)			Caption-level efficiency audit (100 captions)
Parser	Obj.	Attr.	Atom	Sec./cap.	Full pass (min)	API cost/pass ($)
FACTUAL parser	92.9	85.8	86.7	0.56	17.7	0.00
GPT-5.4 mini	89.2	91.3	87.1	3.46	109.1	3.52

Appendix

Table 8: Sentence-level parser-quality audit and caption-level efficiency audit for choosing the scene-graph parser used by our method. Quality is measured on 48 sampled training sentences from the TemporalBench short-caption pool with a GPT-5.4 judge. Atom pass asks whether a parse is usable for the object/attribute factual supervision used by our reward. Efficiency is measured on 100 training captions under the same sentence-by-sentence parsing pipeline and projected to the full 1,891-caption training pool. API cost is estimated from observed token usage and the public GPT-5.4 mini pricing in March 2026 ( OpenAI, 2026 ) .

Training pool	Caps.	Videos	TempComp.	TVBench	VMME	MVB	VidHall.	EventHall.	Total
Default 100%	1891	1891	0/410	0/2112	0/900	0/3851	0/1072	0/397	0/8193
Scaled 150%	2837	2757	0/410	0/2112	0/900	0/3851	0/1072	0/397	0/8193

Appendix

Table 9: Train–evaluation overlap audit for the TemporalBench-based training pools. We report exact normalized filename overlap against the official benchmark indices or dataset manifests. ‘Total’ is computed against the union of all six benchmark video sets, which contains 8,193 unique videos.

Caption Type	Score $\uparrow$		Acc. $\uparrow$
Caption Type	Base	+ Ours	Base	+ Ours
Short	2.274	2.348	44.04	45.49
Main object	2.345	2.404	45.50	46.51
Detailed	2.512	2.553	48.74	49.50
Camera	2.125	2.171	40.81	41.85
Background	2.116	2.117	40.74	40.73

Appendix

Table 10: VDC caption-generation results on the default Qwen3-VL-8B backbone. We report the GPT-4o-mini judged mean score (0–5) and binary accuracy for each caption type.

		Temporal Understanding			Conventional Video Understanding			Hallucination Examination
Scale	N	Temp Compass	TVBench	Avg.	Video- MME	MVBench	Avg.	Video Hallucer	Event Hallusion	Avg.	Avg.
Baseline	–	74.1	48.6	$61.35_{\scriptstyle\pm 0.00}$	56.7	65.8	$61.25_{\scriptstyle\pm 0.00}$	55.7	66.8	$61.25_{\scriptstyle\pm 0.00}$	$61.3_{\scriptstyle\pm 0.0}$
10%	189	74.2	48.7	$61.45_{\scriptstyle+0.10}$	56.9	65.9	$61.40_{\scriptstyle+0.15}$	55.8	72.1	$63.95_{\scriptstyle+2.70}$	$62.3_{\scriptstyle+1.0}$
25%	473	74.4	48.7	$61.55_{\scriptstyle+0.20}$	56.9	66.3	$61.60_{\scriptstyle+0.35}$	55.9	72.1	$64.00_{\scriptstyle+2.75}$	$62.4_{\scriptstyle+1.1}$
50%	946	74.5	49.1	$61.80_{\scriptstyle+0.45}$	56.9	66.6	$61.75_{\scriptstyle+0.50}$	56.2	72.8	$64.50_{\scriptstyle+3.25}$	$62.7_{\scriptstyle+1.4}$
100%	1891	75.1	49.2	$\mathbf{62.15}_{\scriptstyle+0.80}$	57.4	66.7	$62.05_{\scriptstyle+0.80}$	56.8	73.4	$\mathbf{65.90}_{\scriptstyle+4.65}$	$\mathbf{63.4}_{\scriptstyle+2.1}$

Appendix

Table 11: Ablation study on training data scale. Each group-wise Avg. is the arithmetic mean of the two benchmarks in that group, and the final Avg. is the arithmetic mean over all six benchmarks. Subscripts indicate the change relative to Baseline.

Category	Benchmark	Base			Ours
Category	Benchmark	4B	8B	32B	4B	8B	32B
Temporal	TempCompass	71.0	74.1	77.2	73.0 +2.0	75.1 +1.0	77.3 +0.1
Temporal	TVBench	41.7	47.0	51.3	47.1 +5.4	48.0 +1.0	51.4 +0.1
Conventional	Video-MME	53.6	56.7	60.1	54.4 +0.8	57.4 +0.7	60.3 +0.2
Conventional	MVBench	62.7	65.8	69.5	64.2 +1.5	66.7 +0.9	69.8 +0.3
Hallucination	VideoHallucer	51.4	55.7	62.0	55.8 +4.4	56.8 +1.1	62.5 +0.5

Appendix

Table 12: Model-size ablations for Qwen3-VL, grouped into temporal understanding, conventional video understanding, and hallucination-oriented evaluation.

	Temporal Understanding			Conventional Video Understanding			Hallucination Examination
Setting	TempCompass	TVBench	Avg	VideoMMe	MVBench	Avg	VideoHallucer	EventHallusion	Avg
Ours	75.1	48.0	61.55	57.4	66.7	62.05	56.8	73.4	65.10
Qwen3-VL-8B (Base)	74.1	47.0	60.55 -1.00	56.7	65.8	61.25 -0.80	55.7	66.8	61.25 -3.85
w/o SG Reward	74.6	47.4	61.00 -0.55	57.0	66.1	61.55 -0.50	56.2	71.3	63.75 -1.35
w/o Temp Reward	74.2	47.2	60.70 -0.85	57.1	66.5	61.80 -0.25	56.6	71.5	64.05 -1.05
w/o VQA Reward	74.8	47.7	61.25 -0.30	57.2	65.9	61.55 -0.50	56.0	69.9	62.95 -2.15

Appendix

Table 13: Ablation studies on our training strategy. SG refers to the scene graph reward. Benchmarks are grouped into temporal understanding, conventional video understanding, and hallucination.

Figure 5: Compact qualitative QA and verification comparison. The cases cover temporal QA, procedural MCQ, fine-grained MCQ, and open-ended scene-change reasoning. Across all four examples, the base model makes a locally plausible but grounded error, whereas our method recovers the correct answer with less ambiguity.

Figure 6: Compact qualitative captioning comparison. The selected cases are drawn from the stored TemporalBench short-caption outputs and highlight failures where the base model produces a plausible but factually wrong long caption, while our method restores the missing action or interaction details.

Video-Language Models

Video-language models connect visual content over time with natural language for understanding, describing, retrieving, and interacting with videos.

See all 161 papers

EventLens: Event-Structure Reinforcement Learning for Video Understanding
Oct 12, 2026Guowen Zhang, Boshen Xu, Zihao Yue +6Video Understanding Temporal Video Understanding
AIM3 Lab, Renmin University of China · Kling AI
Reinforcing Structured Chain-of-Thought for Video Understanding
Mar 26, 2026Peiyao Wang, Haotian Xu, Noranart Vesdapunt +6Structured CoT Reasoning RL for VLMs
Stony Brook University · Amazon
Reasoning as Intersection: Consensus-Frame Alignment for Visual Focus in Video-MLLMs
Jun 16, 2026Chengwen Liu, Zhe Huang, Jisheng Dang +3Multimodal Reward Modeling Video Reasoning
School of Information Science and Engineering, Lanzhou University, Lanzhou, China · Beijing University of Posts and Telecommunications, Beijing, China · Cloud and AI BU, Huawei, Shenzhen, China +1

Oct 12, 2026J/K move · Enter open · S save

EventLens: Event-Structure Reinforcement Learning for Video Understanding

Guowen Zhang, Boshen Xu, Zihao Yue, Ziheng Wang+5

AIM3 Lab, Renmin University of China · Kling AI

Video Understanding Temporal Video Understanding Temporal Video Grounding Reinforcement Learning Event-Based Vision

Open paper AskPDF arXiv

Recent multimodal large language models (MLLMs) achieve strong results on video question answering, yet often fail to recover the temporal structure that makes a video coherent. They may merge neighboring events, miss single-event progression, or infer implausible relations across events. We study structured temporal understanding as the ability to recover event structure from visual evidence, including event boundaries, single-event progression, and multi-event relations. We argue that this gap arises from a mismatch between current post-training objectives and temporal reasoning, where caption-style supervision allows models to rely on language shortcuts rather than visual temporal grounding. To address this, we propose EventLens, a vision-centric reinforcement learning framework that learns video understanding through event-structure recovery. Instead of reconstructing captions, EventLens trains models to recover temporal structure under controlled perturbations, such as altered temporal granularity, reversed progression, and shuffled event order. We instantiate this framework with a three-level temporal ontology and derive three verifiable task families: event segmentation, progression discrimination, and multi-event relation reasoning. These tasks admit deterministic rewards, enabling task-specialized GRPO training without human preference labels or learned reward models. We further introduce multi-teacher on-policy distillation (MT-OPD) to consolidate specialized policies into a unified model. Experiments on Qwen3-VL backbones show consistent improvements on temporal grounding and event-sensitive reasoning benchmarks, while preserving general video-QA performance. Learning curves further demonstrate that EventLens achieves comparable downstream transfer with significantly reduced training cost compared to direct mixed RL optimization.

Abstract

Video-Language Models

Explore similar work

EventLens: Event-Structure Reinforcement Learning for Video Understanding

Reinforcing Structured Chain-of-Thought for Video Understanding

Reasoning as Intersection: Consensus-Frame Alignment for Visual Focus in Video-MLLMs