Video multimodal large language models (MLLMs) keep climbing video question answering benchmarks, yet shuffling the frames, masking the segment that supports the answer, or occluding the target object barely changes their predictions. The accuracy rests on appearance and language priors, not on the temporal evidence the question asks for. We trace this to the unit of post-training: rewards are computed on a single response to the original clip, so the model is never asked to behave consistently across views. We propose Behavior Pack Optimization (BPO), which replaces the single response with a behavior pack of outputs across counterfactual views chosen by question type, scored jointly. The pack reward asks for stability when the intervention is irrelevant, sensitivity when key evidence is removed, and abstention when no evidence remains. To keep this objective stable at small pack sizes, BPO uses an anchor-relative advantage: the response on the original view serves as a per-prompt reference instead of a group mean over mixed views. On TempCompass, MVBench, and NExT-QA, BPO improves the macro accuracy of Qwen2.5-VL-7B-Instruct by 4.7 pp, the temporal-hard subset by 7.8 pp, and abstention F1 by 20.0 pp over a budget-matched vanilla GRPO baseline from the same SFT checkpoint. The gains transfer to Video-MME, LongVideoBench, and to LLaVA-Video-7B; ablations confirm they follow the view sets, not the rollout count. We hope this pack-level perspective offers a useful starting point for the video MLLM and multimodal post-training community as the field moves toward evidence-grounded video reasoning.
Figures & tables
Figure 1: (a) GRPO samples K rollouts from the original clip v0 , centers rewards by the sampled-response mean rˉ , and optimizes each response separately. (b) BPO samples one rollout per view from a question-type-conditioned set, scores the pack jointly with cross-view behavior contracts, and centers advantages relative to the v0 anchor rollout.
Figure 2: Overview of Behavior Pack Optimization. Stage A installs the structured EST interface with SFT. Stage B routes each question to a question-typed view set, forms a behavior pack across counterfactual views, scores view-specific and cross-view contract rewards, and updates the policy with anchor-relative advantages.
Setting
Macro ↑
Temp.-hard ↑
Key-sens ↑
Abstain-F1 ↑
Rollouts/ prompt
Stage-B FLOPs (rel.)
Main Qwen2.5-VL-7B comparison
B0 Qwen2.5-VL-7B zero-shot
55.8
48.2
12.4
0.31
–
–
B1 + Stage A EST-SFT
58.1
50.6
13.1
0.34
–
–
B2 + vanilla single-video GRPO
62.7±0.8
53.4±1.1
31.5±1.4
0.47±0.02
4
1.00
B2.5 Video-R1 Feng et al. (2025) , reimpl.
63.4±0.7
54.8±1.0
35.2±1.3
0.49±0.02
4
1.01
B2.6 TPO Li et al. (2025a) , reimpl.
63.1±0.9
54.2±1.2
33.8±1.5
0.51±0.02
4
1.02
Table 1: Main results on TempCompass, MVBench, and NExT-QA. RL rows report mean ± std over 5 seeds; FLOPs are normalized within Stage-B training. Bold and underline mark the best and second-best main 7B rows.
Setting
Video-MME
LongVideoBench
Macro ↑
Temp.-hard ↑
Macro ↑
Temp.-hard ↑
B2 vanilla GRPO
54.3
46.8
46.2
41.5
B5 BPO
58.1
52.9
49.8
47.3
Δ (B5 − B2)
+3.8
+6.1
+3.6
+5.8
Table 2: Extended benchmarks under the same Stage-B budget. Numbers are means over 3 seeds.
Figure 3: Training dynamics: anchor-relative B4 is more stable than batch-normalized B3.
Trained on \ Eval
Q1 mask
Q1 temporal
Q2 mask
Q2 temporal
Run A: Q1 , V1 , M
64.3†
56.1
58.4
54.7
Run B: Q1 , V2 , T
57.8
55.9
56.2
55.1
Run C: Q2 , V1 , M
58.9
54.3
57.5
55.8
Run D: Q2 , V2 , T
57.4
55.2
56.9
62.1†
Table 3: Question-type × view-set diagnostic matrix. Entries are temporal-hard accuracy; diagonal cells use 3 seeds and off-diagonal cells are single-seed checks.
Setting
Macro ↑
Temp.-hard ↑
Key-sens ↑
Reverse-sens ↑
Abstain-F1 ↑
Full BPO, B5
67.4
61.2
54.6
48.9
0.67
−Rev , no evidence reward
65.8
59.4
50.2
47.1
0.58
− stability contract
66.1
59.7
52.8
47.5
0.65
− sensitivity contract
65.4
57.3
38.4
35.2
0.55
− honest abstention
66.2
60.1
53.1
47.8
0.43
− verifier, rule-only fallback
64.1
56.8
46.7
43.5
0.52
Table 4: Reward and view-set ablations. Each row removes one component from full BPO; numbers are means over 3 seeds.
Figure 4: Budget frontier: BPO remains ahead across rollout budgets.
Appendix figures & tables10 assets
Supplementary material from the paper’s appendix.
Appendix
SFT data
Train loss ↓
Eval loss ↓
Parse rate ↑
EST validity ↑
v1 (original clips only)
0.220
0.236
≈1.00
≈1.00
v2 (+ shuffle + reverse augmentation)
0.228
0.402
≈1.00
≈1.00
Appendix
Table 5: Stage A generalization on the same Qwen2.5-VL-3B backbone, two SFT data variants. Adding shuffled/reversed clips to the SFT training set raises validation loss while leaving structural parsing at the saturation level, motivating a pack-level (rather than data-level) treatment of cross-view behavior. Numbers are from the actual Stage A runs reused as the checkpoint base for BPO.
Setting
Rollouts/prompt
Pack size ∣V∣
FLOPs rel.
Time rel.
B2 vanilla single-video GRPO
4
1
1.00
1.00
B3 pack + batch-normalized pack baseline
4
4
1.03
1.02
B4 pack + anchor-relative
4
4
1.03
1.02
B5 BPO full
4
4
1.04
1.03
Appendix
Table 6: Per-step compute alignment for the RL rows in Table 1 .
Method
Benchmark (paper)
Paper ↑
Ours ↑
Gap
Notes
Video-R1 [ 4 ]
MVBench (temp.-hard)
55.6
54.3
−1.3
KL coef. matched to our B2; LR unchanged
TPO [ 5 ]
LongVideoBench (short)
48.1
46.9
−1.2
DPO β=0.1 (paper default); pair sampler = ours
Appendix
Table 7: Sanity replication of the two re-implemented baselines on their own benchmarks. Gaps to published numbers are reported to document any residual implementation drift; all gaps are within 1.5 pp, consistent with documented Qwen2.5-VL vs. original-backbone differences.
View set
∣V∣
Macro
Q1 (mask)-hard
Q2 (temp.)-hard
V-min: {v0,key-mask,reverse}
3
65.4
57.2
55.8
V1 (mask family)
4
66.8
61.4
56.9
V2 (temporal family)
4
66.5
57.8
60.7
Vfull (oracle)
6
67.6
62.1
60.9
Appendix
Table 8: View-set ablation. Each row trains BPO with the indicated view set under matched compute. Type-conditioned sets match the full oracle on their target type while using 4 views. Numbers are macro / subtype temporal-hard accuracy (%); the conditioned rows use 3 seeds, the min and full-oracle rows use 1 seed (resource constraint).
Figure 5: Pack-size sweep on TempCompass. Panel a compares the measured advantage-variance ratio with the closed-form prediction from Proposition 2 . Panel b compares the measured cross-view reward correlation with the corresponding threshold. Per-view rollout counts are adjusted so that the total compute stays within 5% of the default pack-size setting.
Setting
Macro (oracle τ )
Macro (predicted τ )
Δ
BPO (3B)
61.9
60.2
1.7
BPO (7B)
68.8
67.4
1.4
Appendix
Table 9: Oracle vs. predicted question-type performance gap. The gap stays below 2 pp on both backbones, ruling out a leakage explanation for the headline gains of Table 1 .
Setting
TempCompass
MVBench
NExT-QA
Macro
B2-Llava vanilla GRPO on LLaVA-Video-7B
59.7
60.4
63.8
61.3
B5-Llava BPO on LLaVA-Video-7B
63.5
65.2
69.0
65.9
Δ (B5-Llava − B2-Llava)
+3.8
+4.8
+5.2
+4.6
Appendix
Table 10: Cross-backbone transfer: per-benchmark macro accuracy (%) for LLaVA-Video-7B under budget-aligned B2 and B5. 3 seeds per cell. Gaps transfer in direction and magnitude to the Qwen2.5-VL-7B gaps reported in Table 1 .
Indicator
Definition
key-sensitivity
1−acc(key-mask)/acc(orig)
non-key-stability
acc(non-key-mask)/acc(orig)
reverse-sensitivity
Pr(answer flips on reverse) for type-2 questions
abstention F1
F1 over an insufficient-evidence subset, “insufficient” as positive
Appendix
Table 11: Definitions of the four behavior indicators reported throughout the paper. Here “acc” denotes per-view answer accuracy and “orig” denotes the original-view rollout.
Figure 6: Verifier audit. Panel a reports verifier accuracy on human-labeled validation-pack samples stratified by structure category before annotation. Panel b reports the reward-hacking ratio, defined as the fraction of samples for which the verifier assigns a high score while human labels mark the answer as incorrect.
Method
Optim. unit
Intervention
Advantage baseline
LLaVA-Video / InternVideo (SFT) [ 9 , 10 ]
response
none
—
Video-R1 [ 4 ]
response group
global, fixed
sampled-response baseline
TPO [ 5 ]
response pair
global, manipulated input
implicit (DPO)
CounterVid / MixDPO [ 14 , 24 ]
response pair
global, synthesis-based
implicit (DPO)
BPO (ours)
behavior pack
question-type conditioned
anchor-relative
Appendix
Table 12: Positioning of BPO along three design axes. Prior work covers individual axes; the bottom row is the joint combination this paper studies.
Reinforcement learning has improved the reasoning ability of large language models, but applying outcome-only rewards to video multimodal large language models (Video-MLLMs) provides limited guidance on which visual evidence should support the answer. Inspired by multisensory integration, where consistent cues can enhance the salience and reliability of perceptual estimates, we introduce Consensus Frame GRPO (CF-GRPO), a temporal-annotation-free process-level reward framework for evidence-aware video reasoning. CF-GRPO constructs a consensus frame prior from intrinsic video cues, including temporal coverage, scene-transition cues, and query-conditioned visual relevance. It then computes a model-side frame-use score from visual and response representations and optimizes their agreement through the Consensus Frame Reward (CFR). With salience-aware sparse aggregation and distribution sharpening, CFR provides a high-contrast reward signal without requiring human temporal annotations. Experiments show that VideoCFR achieves competitive performance across complex video reasoning benchmarks and improves several metrics over representative Video-MLLM and RL baselines, while the consensus prior provides an interpretable view of the evidence frames emphasized during training. The implementation is available at https://github.com/1Pansy/VideoCFR.
Chengwen Liu, Zhe Huang, Jisheng Dang +3
School of Information Science and Engineering, Lanzhou University, Lanzhou, China · Beijing University of Posts and Telecommunications, Beijing, China · Cloud and AI BU, Huawei, Shenzhen, China +1
Video multimodal large language models achieve strong results on existing benchmarks, but answer accuracy alone does not establish whether they can locate the evidence needed to answer a question. We introduce VideoZeroBench, a challenging long-video benchmark with manually annotated question-answer pairs spanning 13 video domains. Questions target fine-grained cues, fleeting events, and evidence distributed across multiple segments. Temporal intervals and key-frame boxes are annotated where applicable. All questions undergo two rounds of cross-verification for answer validity and evidence quality. Our five-level diagnostic protocol compares answering with and without evidence hints, then combines answer correctness with independently evaluated temporal and spatial grounding. Across 19 evaluated models, the best standard QA accuracy is 24.8% (Level-3), achieved by Gemini-3.7-Flash. No model exceeds 1.8% when correct answers and accurate spatio-temporal localization are jointly required (Level-5). Analyses of atomic abilities, evidence spans, input modalities, and thinking-with-videos inference further characterize where the evaluated systems struggle. These findings motivate more precise evidence search and localization for long-video question answering. Our code and data are publicly released.
Video multimodal large language models (MLLMs) have made rapid progress in general and long-form video understanding, yet their ability to capture and use brief, answer-critical visual evidence remains underexplored. Momentary actions or state transitions may determine an answer while occupying only a small fraction of the video. Performance on broad video understanding tasks alone does not establish whether models can reliably identify and use this temporally localized evidence. We introduce Moment-Video, a benchmark for evaluating temporal fidelity through momentary visual event understanding. Questions are constructed around visually observable events or event sequences, requiring models to notice, count, describe, or reason about transient evidence. The benchmark contains 1,000 human-verified video-QA pairs across seven domains and 25 subcategories, covering four tasks: Temporal Occurrence, Temporal Counting, Action Description, and Temporal Reasoning. We evaluate 34 proprietary and open-source MLLMs. Under the main evaluation settings, the best-performing model, Seed-2.0-Pro, achieves 39.6% overall accuracy, while most open-source models remain below 25%. Diagnostic analyses show that denser frame sampling improves some models, but substantial errors persist even when sampled frames cover all annotated necessary event intervals in an audited subset. Performance also declines on longer videos, suggesting additional temporal-localization challenges. By exposing these limitations, Moment-Video provides a diagnostic benchmark to guide the evaluation and development of video MLLMs that more reliably understand brief, answer-critical visual events.
Xiaolin Liu, Yilun Zhu, Xiangyu Zhao +9
Shandong University · Southeast University · Shanghai Jiao Tong University +1