Behavior Pack Optimization for Video MLLM Post-Training

Zhaolu Kang; Shiyu Liu; Tailong Luo; Wei Zhang; Yingjie He; Lei Wei; Guansu Wang; Liang He

cs.CVOct 2, 2026

Behavior Pack Optimization for Video MLLM Post-Training

Authors: Zhaolu Kang, Shiyu Liu, Tailong Luo, Wei Zhang, Yingjie He, Lei Wei, Guansu Wang, Liang He,

Organizations: Peking University · Chengdu Minto Tech · The University of Melbourne · Stanford University

Abstract

Video multimodal large language models (MLLMs) keep climbing video question answering benchmarks, yet shuffling the frames, masking the segment that supports the answer, or occluding the target object barely changes their predictions. The accuracy rests on appearance and language priors, not on the temporal evidence the question asks for. We trace this to the unit of post-training: rewards are computed on a single response to the original clip, so the model is never asked to behave consistently across views. We propose Behavior Pack Optimization (BPO), which replaces the single response with a behavior pack of outputs across counterfactual views chosen by question type, scored jointly. The pack reward asks for stability when the intervention is irrelevant, sensitivity when key evidence is removed, and abstention when no evidence remains. To keep this objective stable at small pack sizes, BPO uses an anchor-relative advantage: the response on the original view serves as a per-prompt reference instead of a group mean over mixed views. On TempCompass, MVBench, and NExT-QA, BPO improves the macro accuracy of Qwen2.5-VL-7B-Instruct by 4.7 pp, the temporal-hard subset by 7.8 pp, and abstention F1 by 20.0 pp over a budget-matched vanilla GRPO baseline from the same SFT checkpoint. The gains transfer to Video-MME, LongVideoBench, and to LLaVA-Video-7B; ablations confirm they follow the view sets, not the rollout count. We hope this pack-level perspective offers a useful starting point for the video MLLM and multimodal post-training community as the field moves toward evidence-grounded video reasoning.

Figures & tables

Panel of Figure 1 — Figure 1: (a) GRPO samples $K$ rollouts from the original clip $v_{0}$ , centers rewards by the sampled-response mean $\bar{r}$ , and optimizes each response separately. (b) BPO samples one rollout per view from a question-type-conditioned set, scores the pack jointly with cross-view behavior contracts, and centers advantages relative to the $v_{0}$ anchor rollout.

Figure 2: Overview of Behavior Pack Optimization. Stage A installs the structured EST interface with SFT. Stage B routes each question to a question-typed view set, forms a behavior pack across counterfactual views, scores view-specific and cross-view contract rewards, and updates the policy with anchor-relative advantages.

Main Qwen2.5-VL-7B comparison
Setting	Macro $\uparrow$	Temp.-hard $\uparrow$	Key-sens $\uparrow$	Abstain-F1 $\uparrow$	Rollouts/ prompt	Stage-B FLOPs (rel.)
B0 Qwen2.5-VL-7B zero-shot	$55.8$	$48.2$	$12.4$	$0.31$	–	–
B1 + Stage A EST-SFT	$58.1$	$50.6$	$13.1$	$0.34$	–	–
B2 + vanilla single-video GRPO	$62.7\!\pm\!0.8$	$53.4\!\pm\!1.1$	$31.5\!\pm\!1.4$	$0.47\!\pm\!0.02$	$4$	$1.00$
B2.5 Video-R1 Feng et al. (2025) , reimpl.	$63.4\!\pm\!0.7$	$54.8\!\pm\!1.0$	$35.2\!\pm\!1.3$	$0.49\!\pm\!0.02$	$4$	$1.01$
B2.6 TPO Li et al. (2025a) , reimpl.	$63.1\!\pm\!0.9$	$54.2\!\pm\!1.2$	$33.8\!\pm\!1.5$	$0.51\!\pm\!0.02$	$4$	$1.02$

Table 1: Main results on TempCompass, MVBench, and NExT-QA. RL rows report mean

\pm

std over 5 seeds; FLOPs are normalized within Stage-B training. Bold and underline mark the best and second-best main 7B rows.

Setting	Video-MME		LongVideoBench
	Macro $\uparrow$	Temp.-hard $\uparrow$	Macro $\uparrow$	Temp.-hard $\uparrow$
B2 vanilla GRPO	$54.3$	$46.8$	$46.2$	$41.5$
B5 BPO	$\mathbf{58.1}$	$\mathbf{52.9}$	$\mathbf{49.8}$	$\mathbf{47.3}$
$\Delta$ (B5 $-$ B2)	$+3.8$	$+6.1$	$+3.6$	$+5.8$

Table 2: Extended benchmarks under the same Stage-B budget. Numbers are means over 3 seeds.

Panel of Figure 3 — Figure 3: Training dynamics: anchor-relative B4 is more stable than batch-normalized B3.

Trained on \ Eval	Q1 mask	Q1 temporal	Q2 mask	Q2 temporal
Run A: $Q1$ , $\mathcal{V}_{1}$ , $M$	$\mathbf{64.3}^{\dagger}$	$56.1$	$58.4$	$54.7$
Run B: $Q1$ , $\mathcal{V}_{2}$ , $T$	$57.8$	$55.9$	$56.2$	$55.1$
Run C: $Q2$ , $\mathcal{V}_{1}$ , $M$	$58.9$	$54.3$	$57.5$	$55.8$
Run D: $Q2$ , $\mathcal{V}_{2}$ , $T$	$57.4$	$55.2$	$56.9$	$\mathbf{62.1}^{\dagger}$

Table 3: Question-type

{\times}

view-set diagnostic matrix. Entries are temporal-hard accuracy; diagonal cells use 3 seeds and off-diagonal cells are single-seed checks.

Setting	Macro $\uparrow$	Temp.-hard $\uparrow$	Key-sens $\uparrow$	Reverse-sens $\uparrow$	Abstain-F1 $\uparrow$
Full BPO, B5	$\mathbf{67.4}$	$\mathbf{61.2}$	$\mathbf{54.6}$	$\mathbf{48.9}$	$\mathbf{0.67}$
$-\,R_{\text{ev}}$ , no evidence reward	$65.8$	$59.4$	$50.2$	$47.1$	$0.58$
$-\,$ stability contract	$66.1$	$59.7$	$52.8$	$47.5$	$0.65$
$-\,$ sensitivity contract	$65.4$	$57.3$	$38.4$	$35.2$	$0.55$
$-\,$ honest abstention	$66.2$	$60.1$	$53.1$	$47.8$	$0.43$
$-\,$ verifier, rule-only fallback	$64.1$	$56.8$	$46.7$	$43.5$	$0.52$

Table 4: Reward and view-set ablations. Each row removes one component from full BPO; numbers are means over 3 seeds.

Panel of Figure 4 — Figure 4: Budget frontier: BPO remains ahead across rollout budgets.

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

SFT data	Train loss $\downarrow$	Eval loss $\downarrow$	Parse rate $\uparrow$	EST validity $\uparrow$
v1 (original clips only)	$0.220$	$\mathbf{0.236}$	$\approx 1.00$	$\approx 1.00$
v2 (+ shuffle + reverse augmentation)	$0.228$	$0.402$	$\approx 1.00$	$\approx 1.00$

Appendix

Table 5: Stage A generalization on the same Qwen2.5-VL-3B backbone, two SFT data variants. Adding shuffled/reversed clips to the SFT training set raises validation loss while leaving structural parsing at the saturation level, motivating a pack-level (rather than data-level) treatment of cross-view behavior. Numbers are from the actual Stage A runs reused as the checkpoint base for BPO.

Setting	Rollouts/prompt	Pack size $\|\mathcal{V}\|$	FLOPs rel.	Time rel.
B2 vanilla single-video GRPO	$4$	$1$	$1.00$	$1.00$
B3 pack + batch-normalized pack baseline	$4$	$4$	$1.03$	$1.02$
B4 pack + anchor-relative	$4$	$4$	$1.03$	$1.02$
B5 BPO full	$4$	$4$	$1.04$	$1.03$

Appendix

Table 6: Per-step compute alignment for the RL rows in Table 1 .

Method	Benchmark (paper)	Paper $\uparrow$	Ours $\uparrow$	Gap	Notes
Video-R1 [ 4 ]	MVBench (temp.-hard)	$55.6$	$54.3$	$-1.3$	KL coef. matched to our B2; LR unchanged
TPO [ 5 ]	LongVideoBench (short)	$48.1$	$46.9$	$-1.2$	DPO $\beta\!=\!0.1$ (paper default); pair sampler = ours

Appendix

Table 7: Sanity replication of the two re-implemented baselines on their own benchmarks. Gaps to published numbers are reported to document any residual implementation drift; all gaps are within 1.5 pp, consistent with documented Qwen2.5-VL vs. original-backbone differences.

View set	$\|\mathcal{V}\|$	Macro	Q1 (mask)-hard	Q2 (temp.)-hard
V-min: $\{v_{0},\text{key-mask},\text{reverse}\}$	$3$	$65.4$	$57.2$	$55.8$
$\mathcal{V}_{1}$ (mask family)	$4$	$66.8$	$\mathbf{61.4}$	$56.9$
$\mathcal{V}_{2}$ (temporal family)	$4$	$66.5$	$57.8$	$\mathbf{60.7}$
$\mathcal{V}_{\text{full}}$ (oracle)	$6$	$\mathbf{67.6}$	$62.1$	$60.9$

Appendix

Table 8: View-set ablation. Each row trains BPO with the indicated view set under matched compute. Type-conditioned sets match the full oracle on their target type while using 4 views. Numbers are macro / subtype temporal-hard accuracy (%); the conditioned rows use 3 seeds, the min and full-oracle rows use 1 seed (resource constraint).

Panel of Figure 5 — Figure 5: Pack-size sweep on TempCompass. Panel a compares the measured advantage-variance ratio with the closed-form prediction from Proposition 2 . Panel b compares the measured cross-view reward correlation with the corresponding threshold. Per-view rollout counts are adjusted so that the total compute stays within 5% of the default pack-size setting.

Setting	Macro (oracle $\tau$ )	Macro (predicted $\tau$ )	$\Delta$
BPO (3B)	$61.9$	$60.2$	$1.7$
BPO (7B)	$68.8$	$67.4$	$1.4$

Appendix

Table 9: Oracle vs. predicted question-type performance gap. The gap stays below 2 pp on both backbones, ruling out a leakage explanation for the headline gains of Table 1 .

Setting	TempCompass	MVBench	NExT-QA	Macro
B2-Llava vanilla GRPO on LLaVA-Video-7B	$59.7$	$60.4$	$63.8$	$61.3$
B5-Llava BPO on LLaVA-Video-7B	$\mathbf{63.5}$	$\mathbf{65.2}$	$\mathbf{69.0}$	$\mathbf{65.9}$
$\Delta$ (B5-Llava $-$ B2-Llava)	$+3.8$	$+4.8$	$+5.2$	$+4.6$

Appendix

Table 10: Cross-backbone transfer: per-benchmark macro accuracy (%) for LLaVA-Video-7B under budget-aligned B2 and B5. 3 seeds per cell. Gaps transfer in direction and magnitude to the Qwen2.5-VL-7B gaps reported in Table 1 .

Indicator	Definition
key-sensitivity	$1-\mathrm{acc}(\text{key-mask})/\mathrm{acc}(\text{orig})$
non-key-stability	$\mathrm{acc}(\text{non-key-mask})/\mathrm{acc}(\text{orig})$
reverse-sensitivity	$\Pr(\text{answer flips on reverse})$ for type-2 questions
abstention F1	F1 over an insufficient-evidence subset, “insufficient” as positive

Appendix

Table 11: Definitions of the four behavior indicators reported throughout the paper. Here “acc” denotes per-view answer accuracy and “orig” denotes the original-view rollout.

Panel of Figure 6 — Figure 6: Verifier audit. Panel a reports verifier accuracy on human-labeled validation-pack samples stratified by structure category before annotation. Panel b reports the reward-hacking ratio, defined as the fraction of samples for which the verifier assigns a high score while human labels mark the answer as incorrect.

Method	Optim. unit	Intervention	Advantage baseline
LLaVA-Video / InternVideo (SFT) [ 9 , 10 ]	response	none	—
Video-R1 [ 4 ]	response group	global, fixed	sampled-response baseline
TPO [ 5 ]	response pair	global, manipulated input	implicit (DPO)
CounterVid / MixDPO [ 14 , 24 ]	response pair	global, synthesis-based	implicit (DPO)
BPO (ours)	behavior pack	question-type conditioned	anchor-relative

Appendix

Table 12: Positioning of BPO along three design axes. Prior work covers individual axes; the bottom row is the joint combination this paper studies.

Multimodal Large Language Models

Multimodal large language models combine language models with image, video, audio, and other inputs for grounded understanding and generation.

See all 1,092 papers

Reasoning as Intersection: Consensus-Frame Alignment for Visual Focus in Video-MLLMs
Jun 16, 2026Chengwen Liu, Zhe Huang, Jisheng Dang +3Multimodal Reward Modeling Video Reasoning
School of Information Science and Engineering, Lanzhou University, Lanzhou, China · Beijing University of Posts and Telecommunications, Beijing, China · Cloud and AI BU, Huawei, Shenzhen, China +1
VideoZeroBench: Probing the Limits of Video MLLMs with Spatio-Temporal Evidence Verification
Apr 2, 2026Jiahao Meng, Yue Tan, Qi Xu +9Video-Language Model Evaluation Long-Video QA
PKU · WHU · CASIA +2
Moment-Video: Diagnosing Temporal Fidelity of Video MLLMs on Momentary Visual Events
Jun 1, 2026Xiaolin Liu, Yilun Zhu, Xiangyu Zhao +9VLM Evaluation Video QA
Shandong University · Southeast University · Shanghai Jiao Tong University +1

Jun 16, 2026J/K move · Enter open · S save

Reasoning as Intersection: Consensus-Frame Alignment for Visual Focus in Video-MLLMs

Chengwen Liu, Zhe Huang, Jisheng Dang, Hong Peng+2

School of Information Science and Engineering, Lanzhou University, Lanzhou, China · Beijing University of Posts and Telecommunications, Beijing, China · Cloud and AI BU, Huawei, Shenzhen, China+1

Multimodal Reward Modeling Video Reasoning Visually Grounded Reasoning RL for VLMs RL for Video Reasoning

Open paper AskPDF arXiv

Reinforcement learning has improved the reasoning ability of large language models, but applying outcome-only rewards to video multimodal large language models (Video-MLLMs) provides limited guidance on which visual evidence should support the answer. Inspired by multisensory integration, where consistent cues can enhance the salience and reliability of perceptual estimates, we introduce Consensus Frame GRPO (CF-GRPO), a temporal-annotation-free process-level reward framework for evidence-aware video reasoning. CF-GRPO constructs a consensus frame prior from intrinsic video cues, including temporal coverage, scene-transition cues, and query-conditioned visual relevance. It then computes a model-side frame-use score from visual and response representations and optimizes their agreement through the Consensus Frame Reward (CFR). With salience-aware sparse aggregation and distribution sharpening, CFR provides a high-contrast reward signal without requiring human temporal annotations. Experiments show that VideoCFR achieves competitive performance across complex video reasoning benchmarks and improves several metrics over representative Video-MLLM and RL baselines, while the consensus prior provides an interpretable view of the evidence frames emphasized during training. The implementation is available at https://github.com/1Pansy/VideoCFR.

Abstract

Multimodal Large Language Models

Explore similar work

Reasoning as Intersection: Consensus-Frame Alignment for Visual Focus in Video-MLLMs

VideoZeroBench: Probing the Limits of Video MLLMs with Spatio-Temporal Evidence Verification

Moment-Video: Diagnosing Temporal Fidelity of Video MLLMs on Momentary Visual Events