When Rewards Disagree: Reward Weighting for Token-Efficient Video Reasoning

Zinuo Li; Yongxin Guo; Jun Liu; Jiawei Zhan; Xi Jiang; Mohammed Bennamoun; Farid Boussaid; Feng Zheng

cs.CLApr 6, 2026

When Rewards Disagree: Reward Weighting for Token-Efficient Video Reasoning

Authors: Zinuo Li, Yongxin Guo, Jun Liu, Jiawei Zhan, Xi Jiang, Mohammed Bennamoun, Farid Boussaid, Feng Zheng, +1 more

Organizations: University of Western Australia · Tencent Youtu Lab · CUHKSZ · SUSTech · Monash University

Abstract

Human perception interprets video dynamics by constructing structured mental representations of entities, actions, and temporal boundaries before engaging in high-level reasoning. In contrast, current Video-LLMs typically rely on unstructured Chain-of-Thought (CoT), where crucial visual cues are obscured by verbose narration. While recent efforts ground reasoning in intermediate visual evidence, they rely on task-specific schemas that fail to generalize across diverse video understanding tasks. To address this, we introduce Structured Event Evidence, a unified, temporally ordered schema designed for general video reasoning. However, post-training a model to extract structured evidence while reasoning introduces a severe multi-objective optimization bottleneck, as candidate rollouts must simultaneously satisfy schema compliance, task accuracy, and reasoning budget. Standard reinforcement learning (RL) algorithms attempt to balance these objectives via static scalarization, enforcing a rigid accuracy-length trade-off across training and obscuring fine-grained reward signals. We resolve this by proposing Group-Relative Advantage Balancing (GRAB), which dynamically computes Pareto-optimal objective weights within each rollout group via a minimum-norm criterion over centered rewards. Operating strictly in reward space, GRAB incurs virtually zero computational overhead (<0.002% relative runtime to GRPO) and eliminates brittle weight tuning. Supported by our new PARSE-60K dataset (60K annotations across 35K videos) and a three-stage progressive curriculum, we train PARSE-4B. Across temporal grounding and video question-answering benchmarks, GRAB generates 13% to 39% fewer tokens while maintaining competitive accuracy with exhaustively hand-tuned static weights -- without requiring hyperparameter weight sweeps.

Figures & tables

Figure 2: Comparison of unstructured video CoT and our structured reasoning approach. Our method first extracts structured event evidence and then performs evidence-grounded reasoning over temporal dependencies, yielding clearer cues and more interpretable predictions. Video from ActivityNet-Captions ( Krishna et al., 2017 ) .

Figure 3: GRPO vs. GRAB on a rollout group. GRPO averages the four rewards, assigning identical scores and advantages to $[1,1,0,0]$ and $[1,0,0,1]$ . GRAB applies dynamic minimum-norm weights (Algorithm 1 ) to centered rewards before $z$ -score normalization, correctly distinguishing response preferences.

Figure 4: How we build PARSE-60K. We first drop any video whose ID appears in an eval set. Stage 1 then cuts events, fills gaps, writes evidence, and judges quality. Stage 2 writes thinking traces and judges them again.

Model	Charades-TimeLens (R1@IoU)				ActivityNet-Captions (R1@IoU)				VideoMME		MLVU
Model	@0.3	@0.5	@0.7	Len	@0.3	@0.5	@0.7	Len	Acc	Len	TR	Ego	Len
Baselines 1fps
Qwen3-VL-4B-Inst.* Bai et al. (2025a)	56.5	37.8	18.4	-	50.2	35.8	21.6	-	63.4	-	80.1	60.3	-
Qwen3-VL-4B-Think.* Bai et al. (2025a)	56.3	37.4	17.8	1439	47.8	31.7	18.8	1895	63.1	4016	79.5	59.0	5192
Ablations 1fps
w/o Evidence = 16	54.0	37.0	16.5	553	50.7	34.3	23.2	1158	60.8	2961	76.2	56.8	3128

Table 1: Ablation study at the 4B scale on temporal grounding and general video understanding. We remove evidence, structured thinking, or RL, and compare GRAB against GRPO, DAPO, GDPO, fixed reward weighting, and a Lagrangian length baseline. @

\tau

is Recall@1 at IoU threshold

\tau

, VideoMME accuracy excludes subtitles, and TR and Ego are the Topic Reasoning and Ego Reasoning sub-tasks of MLVU. Len is the average number of generated tokens per response, so lower is better at comparable accuracy. A row ending in

=8

=16

gives the group size of its RL stage, and w/o RL has none. Entries marked “-” are not reported, since Qwen3-VL-4B-Instruct returns a bare answer whose length is not comparable with the reasoning rows.

Model	Charades-TimeLens			ActivityNet-TimeLens			ActivityNet-Captions
Model	[email protected]	[email protected]	[email protected]	[email protected]	[email protected]	[email protected]	[email protected]	[email protected]	[email protected]
Close-Source Models 2fps
GPT-4o ( Hurst et al., 2024 )	60.6	44.5	23.5	55.2	41.4	25.8	-	-	-
GPT-5 ( OpenAI, 2025 )	59.3	42.0	22.0	57.4	44.9	30.4	-	-	-
Gemini-2.0-Flash ( Comanici et al., 2025 )	66.4	53.5	27.1	62.9	54.0	37.7	50.4	33.2	19.9
Gemini-2.5-Flash ( Comanici et al., 2025 )	68.7	56.1	30.6	66.8	57.5	41.3	51.2	34.7	21.0

Table 2: Experiments on the public video temporal understanding benchmarks. * means this model is reproduced under the same video settings of our model, otherwise, the results are from official reports with more video frames.

	ETBench								VideoMME	MLVU		NExT-GQA
Metrics	RAR	ECA	RVQ	TVG	EPM	TAL	TEM	GVQ	Acc	TR (Acc)	Ego (Acc)	Acc
Metrics	Acc	Acc	Acc	F1	F1	F1	Rec	Acc	Acc	TR (Acc)	Ego (Acc)	Acc
Close-Source Models 2fps
GPT-4o ( Hurst et al., 2024 )	27.8	27.3	57.7	40.4	4.5	20.0	13.6	-	71.9	-	-	-
GPT-4v ( OpenAI, 2023 )	33.3	40.9	46.2	27.0	1.8	18.0	23.9	-	59.9	-	-	-
Open-Source Models

Table 3: Experiments on the public general video understanding benchmarks. TR and Ego mean Topic Reasoning and Ego Reasoning, respectively. Rows marked * are run by us at

1

fps, under the same video settings as PARSE-4B. The other rows are taken from official reports, which usually use more video frames.

Appendix figures & tables15 assets

Supplementary material from the paper’s appendix.

Appendix

Symbol	Meaning	Value
$L_{max}$	hard maximum response length (tokens)	$4096$
$L_{buffer}$	soft buffer below $L_{max}$ (tokens)	$2500$
$L_{target}$	$L_{max}-L_{buffer}$ , start of the length penalty	$1596$
$w^{\text{acc}}$	Fixed Weight (Accuracy), over (fmt, IoU, MC, len)	$(1,2,2,1)$
$w^{\text{def}}$	weight vector of the GRPO, DAPO and GDPO runs	$(1,1,1,1)$
$L_{\text{target}}^{\text{lag}}$	length target of the Lagrangian baseline	$2500$

Appendix

Table 4: Reward and solver hyperparameters of the Stage-3 run. Weights are given over the four implemented rewards; since a prompt activates only one of the two task metrics, a group sees

(w_{\text{fmt}},w_{\text{task}},w_{\text{len}})

, which is

(1,2,1)

for Fixed Weight (Accuracy) and

(1,1,1)

for the default runs.

Figure 5: Distribution of task types in the RL training data. Temporal Grounding constitutes the largest share (

53\%

), providing precise IoU-based supervision, while Spatial VQA (

21\%

), Reasoning VQA (

20\%

) and Temporal VQA (

6\%

) broaden the training signal toward spatial, semantic and temporal understanding.

Model	TempCompass			EventBench
Model	Overall	Event Order	Attr. Change	Overall	Temporal Reason.	Causal Reason.
Qwen3-VL-4B-Instruct	68.71	74.76	70.40	62.71	64.51	66.87
Qwen3-VL-4B-Thinking	68.60	73.20	69.00	60.44	62.57	65.22
PARSE-4B	70.64	76.96	73.85	63.48	65.42	69.47

Appendix

Table 5: Results on temporal and event reasoning benchmarks. TempCompass evaluates temporal perception; EventBench assesses event-level understanding including temporal and causal reasoning sub-tasks.

Model (8B)	VideoMME	CTL-mIoU	TempCompass	EventBench
8B-GRPO	64.81	43.73	65.79	64.87
8B-GRAB	66.98	46.81	66.23	66.10

Appendix

Table 6: GRAB vs. GRPO on the 8B model scale (LoRA fine-tuning). GRAB outperforms GRPO on all four benchmarks, so the margin is not confined to 4B. Both runs use half the RL data and a lower LoRA rank than the 4B experiments, so the absolute numbers sit below the 4B model and carry no scaling claim. CTL-mIoU is the mean IoU on Charades-TimeLens.

Group Size	GRAB (ms)	One Step (ms)
4	0.68	52,789
8	0.69	108,401
16	1.00	211,389

Appendix

Table 7: GRAB computational overhead. The weight computation is negligible (

<0.002\%

) relative to the total training step time.

Model	F1	BLEU-4	ROUGE-L	CIDEr
Qwen3-VL-4B-Instruct	44.2	10.6	27.1	4.1
PARSE-4B	54.7	16.6	35.1	17.3

Appendix

Table 8: Evidence quality evaluation. PARSE-4B generates substantially higher-quality structured evidence compared to the base model.

Run ( $G=16$ )	$\mathbf{w_{\text{len}}/w_{\text{task}}}$	[email protected]	[email protected]	[email protected]	Len
Fixed Weight (Accuracy), $w{=}(1,2,0)$	$0$	62.4	42.1	23.8	1180
Fixed Weight (Accuracy), $w{=}(1,2,1)$	$0.50$	63.2	42.9	24.5	931
GRPO, uniform weights $w{=}(1,1,1)$	$1.00$	59.8	40.6	21.8	846
Fixed Weight (Accuracy), $w{=}(1,2,4)$	$2.00$	57.5	37.8	19.6	468
GRAB	per group	63.7	43.4	24.6	579

Appendix

Table 9: Static reward weights on Charades-TimeLens at

G=16

, written as

(w_{\text{fmt}},w_{\text{task}},w_{\text{len}})

over the three objectives that can be live in a single group. Fixed Weight (Accuracy), GRPO and GRAB are the rows of the same name in Table 1 . The other two runs take the Fixed Weight (Accuracy) vector and set

w_{\text{len}}

to zero or to four times its tuned value, leaving

w_{\text{fmt}}

and

w_{\text{task}}

alone, so the four static runs trace a single ratio. Output length falls monotonically in

w_{\text{len}}/w_{\text{task}}

across them, so the column orders the family.

Run ( $G=16$ )	[email protected]	[email protected]	[email protected]	Len
GRPO	$59.8\pm 0.30$	$40.6\pm 0.30$	$21.8\pm 0.40$	$846\pm 12$
Fixed Weight (Accuracy)	$63.2\pm 0.30$	$42.9\pm 0.30$	$24.5\pm 0.40$	$931\pm 14$
GRAB	$\mathbf{63.7\pm 0.25}$	$\mathbf{43.4\pm 0.25}$	$\mathbf{24.6\pm 0.35}$	$\mathbf{579\pm 9}$

Appendix

Table 10: Spread across three random seeds on Charades-TimeLens. The means are the values reported in Table 1 , and the table adds the standard deviation over the seeds.

Quantity	Value
$\alpha^{*}_{\text{fmt}}$ , three live objectives	$0.15\pm 0.11$
$\alpha^{*}_{\text{task}}$ , three live objectives	$0.39\pm 0.06$
$\alpha^{*}_{\text{len}}$ , three live objectives	$0.46\pm 0.07$
$\alpha^{*}$ , two live objectives (self-check)	$0.50\pm 0.00$
[email protected] over GRPO, matched steps up to $100$	$+0.7$ to $+2.1$
Mean length against GRPO, same steps	$-8\%$ to $-19\%$

Appendix

Table 11: Weights the solver returns, measured over the PARSE-4B run. The first three rows give the mean and the standard deviation of

\boldsymbol{\alpha}^{*}_{q}

across the groups where all three objectives are live; the means sum to one, and uniform weighting would put

1/3

on each. The fourth row is the value Section G.3 gives for two live objectives, measured on real groups rather than assumed. The last two rows compare the two runs checkpoint by checkpoint at the same training step, on Charades-TimeLens, with the range covering the checkpoints through step

100

Model	Overall	Halluc.	Logic.	Interp.
Qwen3-VL-4B-Thinking	4.00	4.0	4.25	3.75
PARSE-4B	4.41	4.0	4.50	4.75

Appendix

Table 12: Reasoning quality evaluation by Gemini-2.5-Pro. PARSE-4B produces more logical and interpretable reasoning chains.

Figure 6: Evidence Generation prompt used in the data curation pipeline to produce structured event evidence from videos.

Figure 7: Quality Judge prompt used in the data curation pipeline to validate generated evidence.

Figure 8: Thinking Generation prompt used in the data curation pipeline to produce evidence-grounded reasoning traces.

Figure 9: Prompts for Stage 1 evidence training.

Figure 10: Prompts for Stage 2 thinking warm-start. Orange parts mean these parts are different from the previous parts.

Explore similar work

TimeThink: Reasoning with Time for Video LLMs
Jul 6, 2026Handong Li, Longteng Guo, Zikang Liu +8Temporal Video Grounding Process Reward Models
School of Artificial Intelligence, University of Chinese Academy of Sciences · Institute of Automation, Chinese Academy of Sciences · OPPO AI Center, OPPO Inc.
EventLens: Event-Structure Reinforcement Learning for Video Understanding
Oct 12, 2026Guowen Zhang, Boshen Xu, Zihao Yue +6Video Understanding Temporal Video Understanding
AIM3 Lab, Renmin University of China · Kling AI
Reinforce to Learn, Elect to Reason: A Dual Paradigm for Video Reasoning
Apr 6, 2026Songyuan Yang, Weijiang Yu, Jilin Ma +5Video Reasoning VLM Inference
National University of Defense Technology · Sun Yat-sen University

Jul 6, 2026J/K move · Enter open · S save

TimeThink: Reasoning with Time for Video LLMs

Handong Li, Longteng Guo, Zikang Liu, Dongze Hao+7

School of Artificial Intelligence, University of Chinese Academy of Sciences · Institute of Automation, Chinese Academy of Sciences · OPPO AI Center, OPPO Inc.

Temporal Video Grounding Process Reward Models Video Reasoning RL for VLMs

Open paper AskPDF arXiv

Video reasoning requires models to identify and verify temporally localized evidence within long video sequences. Recent Video Large Language Models (Video-LLMs) have shown promising reasoning abilities when aligned with reinforcement learning, yet existing approaches typically rely on outcome-based rewards that supervise only the final prediction. Such supervision provides limited guidance on how models should discover the relevant temporal evidence during intermediate reasoning. In this work, we propose TimeThink, a reinforcement learning framework that explicitly guides temporal evidence discovery in Video-LLMs. Our key idea is to treat temporal clue steps as the fundamental optimization primitive of video reasoning, where each reasoning step references a candidate time interval in the video. We introduce a step-wise temporal process reward that provides localized credit assignment for these clues and a joint process--outcome optimization objective that balances reasoning fidelity with task correctness. To enable scalable training, we construct TimeThink-RFT-20K, a dataset with automatically derived temporal evidence segments. Extensive experiments across video reasoning, temporal grounding, and general video understanding benchmarks show that TimeThink consistently improves both temporal localization and reasoning performance, achieving state-of-the-art results among open-source video RL models.

Abstract

Explore similar work

TimeThink: Reasoning with Time for Video LLMs

EventLens: Event-Structure Reinforcement Learning for Video Understanding

Reinforce to Learn, Elect to Reason: A Dual Paradigm for Video Reasoning