Watch Before You Answer: Learning from Visually Grounded Post-Training
Authors: Yuxuan Zhang, EunJeong Hwang, Huaisong Zhang, Penghui Du, Yiming Jia, Dongfu Jiang, Xuan He, Shenhui Zhang, +3 more
Organizations: University of British Columbia · Vector Institute · Etude AI · Kolors Team, Kuaishou Technology · University of Toronto · University of Waterloo · University of Illinois at Urbana-Champaign
It is critical for vision-language models (VLMs) to comprehensively understand visual, temporal, and textual cues. However, despite rapid progress in multimodal modeling, video understanding performance still lags behind text-based reasoning. In this work, we find that progress is even worse than previously assumed: commonly reported long video understanding benchmarks contain 40-60% of questions that can be answered using text cues alone. Furthermore, we find that these issues are also pervasive in widely used post-training datasets, potentially undercutting the ability of post-training to improve VLM video understanding performance. Guided by this observation, we introduce VidGround as a simple yet effective solution: using only the actual visually grounded questions without any linguistic biases for post-training. When used in tandem with RL-based post-training algorithms, this simple technique improves performance by up to 6.2 points relative to using the full dataset, while using only 69.1% of the original post-training data. Moreover, we show that data curation with a simple post-training algorithm outperforms several more complex post-training techniques, highlighting that data quality is a major bottleneck for improving video understanding in VLMs. These results underscore the importance of curating post-training data and evaluation benchmarks that truly require visual grounding to advance the development of more capable VLMs. Project page: http://vidground.etuagi.com.
Figures & tables
Figure 1: Performance decomposition on three video understanding benchmarks for four frontier VLMs: Qwen2.5-VL-7B and 32B (Q-7B, Q-32B) [ 7 ] , and Gemini-2.5-Pro and 3.1-Pro (G-2.5, G-3.1) [ 16 , 23 ] . Pink bars show text-only accuracy (no video input); blue bars show the additional visual gain from video access. The majority of benchmark performance comes from language priors rather than visual comprehension. Moreover, scaling up model size or version improves text-only reasoning but visual gain often remains flat or even decreases .
Model
VideoMME
VideoMMMU
MMVU
Random Choice
25.0
9.8
19.8
GPT-4o [ 1 ]
47.0 (+22.0)
38.6 (+28.8)
46.6 (+26.8)
GPT-5-mini [ 37 ]
45.2 (+20.2)
37.9 (+28.1)
53.3 (+33.5)
GPT-5 [ 37 ]
48.2 (+23.2)
41.0 (+31.2)
57.1 (+37.3)
Gemini-2.5-Pro [ 16 ]
53.3 (+28.3)
52.7 (+42.9)
60.6 (+40.8)
Gemini-3.1-Pro [ 23 ]
58.2 (+33.2)
61.1 (+51.3)
63.4 (+43.6)
Table 1: Text-only Answerability (TA) across video understanding benchmarks for frontier models. Each model receives only the question text and answer options—no video input—yet achieves accuracy far above random chance. (+ x ) denotes improvement relative to random choice. Results indicate that 40–60% of benchmark questions can be answered from text alone, revealing substantial linguistic bias in existing video understanding benchmarks.
Figure 2: Analysis of text-only answerable (TA) questions in video understanding benchmarks and post-training data. (a) We identify four common categories of linguistic shortcuts—textual cues, external knowledge, inferential strategies, and imagined content—that allow VLMs to answer correctly without watching the video. (b) These TA questions comprise 38–53% of popular benchmarks (classified using GPT-5-mini), with external knowledge being the dominant category in VideoMMMU and MMVU.
Frames
Method
VideoMME
VideoMMMU
MMVU
Avg.
Full
VG
16
Qwen2.5-VL-7B
58.2
45.0
60.5
54.6
42.9
TW-GRPO
58.2
48.6
61.8
56.2
( + 1.6)
44.1
( + 1.2)
LongVILA-R1-7B
55.5
38.8
59.1
51.1
( − 3.5)
39.7
( − 3.2)
Video-RTS
58.7
47.1
61.8
55.9
( + 1.3)
43.5
( + 0.6)
Qwen2.5-VL-7B-SFT
58.2
43.1
51.3
50.9
( − 3.7)
41.1
( − 1.8)
Table 2: Performance comparison of 7B-scale post-training methods on three video understanding benchmarks (VideoMME, VideoMMMU, MMVU). All methods except LongVILA-R1 are post-trained from Qwen2.5-VL-7B. Models are evaluated using 16, 32, and 64 frames per video. VidGround is post-trained with GRPO on visually grounded (VG) data only. Avg. columns report mean accuracy on the full benchmarks (Full) and on VG question subsets—questions that require video to answer (VG). Deltas show improvement over Qwen2.5-VL-7B: ( +x ) and ( −x ) . Bold indicates best; highlighted rows are ours.
Figure 3: Accuracy on visually grounded (VG) questions—questions that cannot be answered from text alone—across 16, 32, and 64 frames on VideoMME, VideoMMMU, and MMVU for four post-training methods. VG questions are identified by retaining only questions that cannot be answered from text alone, as classified by GPT-5-mini (see § 3 ). VidGround shows the strongest overall frame-scaling behavior, while baselines such as LongVILA-R1-7B and Video-R1 plateau or degrade on MMVU, and Video-RTS drops on VideoMMMU. This suggests that models post-trained on data containing linguistic shortcuts do not effectively leverage additional visual information.
Frames
Method
Data
VideoMME
VideoMMMU
MMVU
Avg.
Full
VG
16
Base
-
58.2
45.0
60.5
54.6
42.9
GRPO
Full
56.9
44.7
54.5
52.0
( − 2.6)
41.7
( − 1.2)
GRPO
VG
58.7
47.4
64.2
56.8
( + 2.2)
45.2
( + 2.3)
+clip-higher
VG
58.2
47.7
63.6
56.5
( + 1.9)
45.1
( + 2.2)
32
Base
-
60.7
45.4
62.3
56.1
44.4
Table 3: Ablation study on post-training data composition. We compare GRPO trained on two data variants of Video-R1-260K [ 20 ] : Full (all 263K samples) and VG (181K visually grounded samples—questions that require visual understanding to answer). +clip-higher denotes asymmetric clipping (see § 4.1 ). Avg. columns report mean accuracy on the full benchmarks (Full) and on VG question subsets—questions that require video to answer (VG). Deltas show improvement over Qwen2.5-VL-7B: ( +x ) and ( −x ) . Training on VG data consistently outperforms the Full variant despite using 31% less data. Highlighted rows are ours.
Figure 4: Qualitative comparison of reasoning paths on a VideoMMMU art analysis question. Given a video demonstrating visual art elements, the model must identify which elements are shown. VidGround (right) references specific visual elements observed in the video frames, such as lines, shapes, and colors (blue boxes), leading to the correct answer. In contrast, Video-R1 [ 20 ] (left) analyzes artistic concepts abstractly without grounding in the actual video content (red boxes), arriving at the wrong answer. This illustrates how post-training on visually grounded data encourages models to attend to visual evidence rather than relying on linguistic priors.
Appendix figures & tables18 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 5: Multi-model agreement analysis on Video-R1-260K [ 20 ] . Venn diagram showing the overlap of visually grounded (VG) questions—those that each model cannot answer correctly in text-only mode (no visual input)—across three frontier models: GPT (GPT-5-mini [ 37 ] ), Qwen (Qwen2.5-VL-7B [ 7 ] ), and Gemini (Gemini-3.1-Pro [ 23 ] ). GPT cannot answer 181,710 questions (69.1%) correctly without video, Qwen cannot answer 198,652 (75.5%), and their intersection (GPT ∩ Qwen) contains 154,860 questions (58.9%) that neither model can solve without visual input. All three models fail on the same 145,486 questions (55.3%), forming a robust core of visually grounded data. Inter-model agreement on VG questions is high (Jaccard index 68.7%), confirming that questions requiring visual understanding are consistently identified across diverse architectures.
Figure 6: Multi-model data curation pipelines for VidGround -M1 and VidGround -M2 variants (GPT: GPT-5-mini [ 37 ] , Qwen: Qwen2.5-VL-7B [ 7 ] , Gemini: Gemini-3.1-Pro [ 23 ] ). Both pipelines evaluate all ∼ 263K samples from Video-R1-260K [ 20 ] in text-only mode. GPT uses single-pass evaluation, Qwen uses circular evaluation with option permutation for MCQ + Pass@10 for open-ended, and Gemini uses 3-permutation circular evaluation for MCQ + direct text evaluation for open-ended. A question is retained as VG only if fewer than 2 models can answer it correctly without visual input. The key difference is the number of circular evaluation permutations for Qwen: (a) VidGround -M1 uses 2-permutation evaluation, retaining ∼ 161K samples (61.1%); (b) VidGround -M2 uses stricter 4-permutation evaluation, retaining ∼ 148K samples (56.2%). Highlighted boxes indicate the differing component.
Frames
Method
VideoMME
VideoMMMU
MMVU
Avg.
Full
VG
16
Qwen2.5-VL-7B
58.2
45.0
60.5
54.6
42.9
Video-R1 (Full, 263K)
56.9
44.7
54.5
52.0
( − 2.6)
41.7
( − 1.2)
VidGround (VG, 181K)
58.7
47.4
64.2
56.8
( + 2.2)
45.2
( + 2.3)
VidGround -M1 (161K)
58.5
48.0
64.4
57.0
( + 2.4)
45.9
( + 3.0)
VidGround -M2 (148K)
57.7
47.0
62.5
55.7
( + 1.1)
43.8
( + 0.9)
Appendix
Table 4: Comparison of data curation strategies on video understanding benchmarks. VidGround uses single-model curation (GPT-5-mini, 181K samples). VidGround -M1 and VidGround -M2 use progressively stricter multi-model consensus curation ( ≥ 2 models agree). All models post-trained with GRPO on Qwen2.5-VL-7B. Avg. columns report mean accuracy on the full benchmarks (Full) and on VG question subsets—questions that require video to answer. Deltas show improvement over Qwen2.5-VL-7B: ( +x ) and ( −x ) . Bold indicates best; highlighted rows are ours.
Method
MME
MMMU
MMBench
Qwen2.5-VL-7B
624.3
56.7
84.2
VidGround
648.9 ( + 24.6)
58.7 ( + 2.0)
84.5 ( + 0.3)
Appendix
Table 5: Performance on image QA benchmarks. VidGround maintains or improves performance on non-video tasks compared to the Qwen2.5-VL-7B base model, demonstrating that visually grounded post-training does not harm cross-task generalization. Deltas show improvement over Qwen2.5-VL-7B: ( +x ) . Bold indicates best; highlighted row is ours.
Model
VideoMME
VideoMMMU
MMVU
Avg.
Random Choice
25.0
9.8
19.8
18.2
Closed-source VLMs
GPT-4o [ 27 ]
47.0 (+22.0)
38.6 (+28.8)
46.6 (+26.8)
44.1 (+25.9)
GPT-5-mini [ 37 ]
45.2 (+20.2)
37.9 (+28.1)
53.3 (+33.5)
45.5 (+27.3)
GPT-5 [ 37 ]
48.2 (+23.2)
41.0 (+31.2)
57.1 (+37.3)
48.8 (+30.6)
Gemini-2.0-Flash [ 16 ]
43.1 (+18.1)
43.6 (+33.8)
53.4 (+33.6)
46.7 (+28.5)
Appendix
Table 6: Extended text-only answerability across video understanding benchmarks for 17 frontier models spanning closed-source VLMs, open-source VLMs, and text-only LLMs. Each model receives only the question text and answer options—no video input. ( +x ) denotes improvement over random chance. All model families achieve accuracy > 20 points above random, confirming pervasive linguistic bias. Bold indicates best.
Figure 7: Additional examples of text-only answerable (TA) questions from VideoMME [ 21 ] . VLMs can exploit the same four categories of linguistic biases identified in Figure 2(a) to answer benchmark questions without visual grounding. VLM responses from GPT-4o [ 27 ] .
Figure 14
Figure 10: Overall Pass@10 distribution for Video-R1-260K [ 20 ] . Same setup as Figure 9 . 51.6% of all questions achieve Pass@10 >0 by Qwen2.5-VL-7B [ 7 ] without visual information.
Hyperparameter
Value
Training Setup
model_name_or_path
Qwen/Qwen2.5-VL-7B-Instruct
max_prompt_length
16384
max_completion_length
768
per_device_train_batch_size
1
gradient_accumulation_steps
1
Appendix
Table 7: Training configuration for VidGround post-training. We apply GRPO [ 44 ] to Qwen2.5-VL-7B-Instruct [ 7 ] for one epoch on the VidGround filtered dataset (181K visually grounded samples from Video-R1-260K [ 20 ] ).
Figure 17
Figure 13: Art elements analysis question comparing VidGround (trained on visually grounded data) with Video-R1 [ 20 ] (trained on the full, unfiltered Video-R1-260K dataset). VidGround references the art elements most prominently featured in the video, while Video-R1 analyzes artistic concepts abstractly without considering the video content. Green highlights indicate correct reasoning grounded in video content; red highlights indicate text-based reasoning without visual grounding.
Figure 14: Psychology question. Same comparison setup as Figure 13 . VidGround grounds its reasoning in the neuron and its components introduced by the video, while Video-R1 [ 20 ] analyzes through general biology knowledge.
Figure 15: Medical science question. Same comparison setup as Figure 13 . VidGround locates the relevant content in the video about PaCO2 , while Video-R1 [ 20 ] evaluates options using linguistic priors.
Figure 16: Clinical medical imaging interpretation question. Same comparison setup as Figure 13 . This instance includes a reference image as part of the question prompt. VidGround refers to the video content about the pelvis and red dots pointing to specific bony landmarks, while Video-R1 [ 20 ] relies on prior knowledge to eliminate options without referencing the video.
Figure 17: Public health question. Same comparison setup as Figure 13 . This instance includes a reference image as part of the question prompt. VidGround grounds its analysis in video-presented concepts of OR (Odds Ratio), PAR (Population Attributable Risk), and AR (Attributable Risk), while Video-R1 [ 20 ] analyzes the question and options using prior knowledge.
Figure 18: Diagnostics and laboratory medicine question. Same comparison setup as Figure 13 . This instance includes a reference image as part of the question prompt. VidGround anchors its analysis in video content about eosinophils and their role in immunity, while Video-R1 [ 20 ] applies general knowledge without referencing the video.
Figure 19: Chemistry question. Same comparison setup as Figure 13 . VidGround references video explanations of intermolecular forces and applies them to each molecular pair, while Video-R1 [ 20 ] directly analyzes the options using prior knowledge.
Figure 20: Structural engineering question. Same comparison setup as Figure 13 . This instance includes a reference image as part of the question prompt. Both models reach the correct answer (A. unit moving load with unchanged direction) through different reasoning paths. VidGround references video content about influence lines in structural engineering and derives the answer from the demonstrated concept, while Video-R1 [ 20 ] eliminates options through prior knowledge without referencing the video. This example demonstrates that correct answers can emerge from either genuine video understanding or linguistic shortcuts, highlighting the importance of evaluating reasoning paths beyond accuracy metrics alone.
Multimodal large language models have made rapid progress in video understanding, yet existing benchmarks largely rely on simple prompts and provide limited evidence about whether models can satisfy explicit output constraints. We introduce VCIFBench, a benchmark for evaluating complex instruction following in video understanding. VCIFBench constructs constraint-rich instructions from both benchmark-adapted and directly video-grounded prompts, covering content, format, style, and structure requirements, and evaluates model outputs with a hybrid verification pipeline. The benchmark contains 306 satisfiable test instructions, a 540-pair DPO preference dataset, and a 30-item conflict diagnostic subset. Experiments on 10 MLLMs show that joint constraint satisfaction remains challenging. We further show that DPO training on VCIFBench data can improve instruction-following performance.
Huangchen Xu, Yuan Wu, Yi Chang
School of Artificial Intelligence, Jilin University · 2Engineering Research Center of Knowledge-Driven Human-Machine Intelligence, Jilin University · 3International Center of Future Science, Jilin University
Large video models have exhibited impressive performance on a wide range of visual question answering tasks, owing to the rise of powerful, pretrained text and vision encoders. The usefulness of such models have also been demonstrated on a wide range of benchmarks, with an important caveat - the dominant approach in these benchmarks evaluates multiple choice reasoning via text options. This is a natural way to test text-based reasoning in these models, and has led to significant insights regarding model behavior in the community. In this work, we ask a different question - what happens when the evaluation modality is visual, rather than text? We introduce three new vision-centric evaluation benchmarks in temporal frame retrieval, video future prediction, and causal memory distortion, all designed around evaluating visual understanding capabilities in large video models. Our approach complements the existing approaches to evaluate video understanding in frontier models. We show that current frontier models exhibit significant weakness when attempting to reason through visual queries, rather than text. We conclude with an extended analysis section that provides pointers for future improvements in visual understanding for large video models.
Rwiddhi Chakraborty, Yinong, Wang +7
Oliver · Carnegie Mellon University · Texas A&M University +3
The inherent complexity of video understanding makes it difficult to determine whether Video-LLM benchmark performance stems from visual perception, linguistic reasoning, or knowledge priors. While many benchmarks have emerged to assess high-level reasoning, shared criteria for evaluating video understanding remain largely overlooked. Instead of introducing yet another benchmark, we take a step back to re-examine the criteria for evaluating video understanding. In this work, we introduce Video-Oasis, a sustainable diagnostic suite for systematically auditing existing video understanding benchmarks. This audit reveals that 55% of existing benchmark samples are solvable without visual input or temporal context. After filtering these shortcuts, the remaining video-native challenges expose a substantial capability gap: state-of-the-art models perform only marginally above random guessing. Building on these findings, we use the distilled challenges as a testbed to investigate which algorithmic design choices contribute to robust video understanding. We hope our work provides a practical foundation for constructing rigorous video benchmarks and evaluating future Video-LLMs. Code is available at https://github.com/sejong-rcv/Video-Oasis.
Geuntaek Lim, Sungjune Park, Jaeyun Lee +5
Sejong University, South Korea · NAVER Cloud, South Korea