On-screen text can describe what a video shows, or it can frame what the video is supposed to mean. Current benchmarks treat both alike, testing whether a model can read the text rather than whether it should trust it. We study claim deference, multimodal large language models' tendency to let an on-screen claim override conflicting visual evidence. We introduce VisualTrust, a counterfactual benchmark with 5,063 cases across 14 claim-fact classes. Each case pairs a question with three versions of the same video: one with no claim, one with an accurate claim, and one with a misleading claim that alters a single grounded fact. The option the misleading claim supports is the trap answer. Across seven open-weight and four proprietary models, mean accuracy falls from 90.1% without a claim to 51.9% with a misleading claim. Among initially correct responses, 42.8% flip to the trap answer and only 0.6% become other errors. Strong no-claim accuracy does not ensure resistance. We also introduce DEAR (Decoupled Evidence Arbitration and Reading), a training-free agent for video question answering that separates evidence arbitration from reading. An independent check reports conflicting evidence, a selector weighs it without a fixed modality priority and blinds what it withholds, and the reader answers from the resulting video in a fresh context. On 500 videos across three proprietary models, DEAR raises accuracy under misleading claims from 47.2--64.0% to 81.8--85.2% and reduces trap-answer selection from 34.4--50.8% to 11.6--15.0%, while preserving no-claim accuracy.
Figures & tables
Figure 1: VisualTrust separates reading on-screen text from trusting it. (a,b) Existing benchmarks test whether models understand the video or read its on-screen text. (c) VisualTrust holds the video, question, and options fixed and varies only the rendered claim; in the example, the misleading claim redirects an otherwise correct response to the option it supports.
Figure 2: How one case is built. (a) The video, question, options, correct answer, and trap answer are fixed before either claim is written. (b) A single-point edit turns the accurate claim into its misleading counterpart, and both are rendered onto the same base, which itself supplies the no-claim condition. The case shown is Perception-Test:video_1879 .
Model
Accuracy (%) ↑
TFR (%) ↓
No claim
Accurate
Misleading
Open-weight
Qwen3.5-9B
89.1
98.7
47.1
47.5
Qwen3.5-27B
92.5
99.3
51.4
44.3
Qwen3.5-35B-A3B
91.3
99.0
47.8
47.7
Qwen3.5-122B-A10B
92.8
99.2
51.4
44.5
Table 1: Claim deference across eleven video models. Accuracy under each claim condition, and the trap-flip rate (TFR): the share of a model’s no-claim-correct responses that flip to the trap answer under the misleading claim.
Figure 3: Misleading claims redirect correct responses to the trap answer. (a) Accuracies under three claim conditions. (b) Destinations of no-claim-correct responses under the misleading claim.
Figure 4: DEAR separates evidence arbitration from reading. An empty selection leaves the video unchanged. The case shown is VideoMME:258-2 .
Method
Accuracy (%) ↑
Trap-answer rate (%) ↓
No claim
Accurate
Misleading
Qwen3.5-Plus
Direct (benchmark prompt)
89.4
96.6
49.4
47.6
Direct (reader prompt)
91.0
98.2
53.6
43.2
Self-Review
90.0
97.0
54.6
42.0
DEAR
90.4
98.0
85.2
11.6
Table 2: DEAR versus prompt-level baselines for mitigating claim deference. For each backbone, all methods answer the same 500 source videos under each claim condition. DEAR reduces the trap-answer rate more than either prompt-level baseline. Trap-answer rate uses all misleading-condition inputs as its denominator; execution failures count as incorrect but never as trap selections. Bold marks the best value per column within each backbone.
Appendix figures & tables16 assets
Supplementary material from the paper’s appendix.
Appendix
Class
Changed proposition
Example change
Cases
ENT
Identity of an entity, object, place, or scene
a cup → a bowl
532
ROL
Agent, patient, recipient, or instrument
boy pulls girl → girl pulls boy
438
EVT
Event occurrence, completion, or outcome
succeeds → fails
551
ACT
Action type or manner
walking → running
544
SET
Membership, count, or completeness
four people → three people
462
STA
Visible attribute, posture, or state
red clothing → blue clothing
484
Appendix
Table 3: Claim-fact classes. Examples illustrate a single-fact change. Counts describe the curated benchmark, not the prevalence of these facts in unconstrained video.
Class
Misleading
Accurate
ENT
29.19
0.11
ROL
49.28
0.29
EVT
72.05
1.08
ACT
46.72
0.32
SET
55.17
0.43
STA
27.49
0.08
Appendix
Table 4: Trap-flip rates by claim-fact class. Equal-model means (%). Within each model and class, both conditions use the same no-claim-correct cases and predesignated trap answers.
Figure 5: Trap-flip rates across claim-fact classes. Points weight the eleven models equally; intervals are pointwise 95% source-bootstrap intervals. The dashed line marks the overall misleading-claim mean, and parentheses give benchmark case counts. Within each model and class, both series use the same no-claim-correct cases and predesignated trap answers.
Model
Temperature
Top- p
Top- k
Presence penalty
Qwen3.5-9B/27B/35B/122B
1.0
0.95
20
1.5
Qwen3.5-397B
0.6
0.95
20
0.0
Gemma 4-26B/31B
1.0
0.95
64
1.5
Appendix
Table 5: Self-hosted sampling settings. Thinking is enabled and repetition penalty is 1 .
Model
Cases
Trap-answer rate (%)
Increase (pp)
No claim
Misleading
Open-weight models
Qwen3.5-9B
5063
7.1
51.9
44.8
Qwen3.5-27B
5063
5.2
47.9
42.7
Qwen3.5-35B-A3B
5063
5.6
51.4
45.8
Qwen3.5-122B-A10B
5063
4.5
47.8
43.3
Appendix
Table 6: Increase in trap-answer selection. Rates use all complete three-condition cases, including cases already incorrect without a claim. The increase is the difference between the two rates.
Model
Group
Cited as grounds
Cited and rejected
Not cited
Unclear
Qwen3.5-Plus
Flip
100
0
0
0
Still correct
4
86
10
0
Seed 2.1 Pro
Flip
75
0
23
2
Still correct
4
70
24
2
Gemini-3.5-Flash
Flip
99
0
1
0
Still correct
5
76
17
2
Appendix
Table 7: Claim use in reasoning traces, by model. Counts of flip and still-correct traces ( n=100 each) by claim-use label. The judge is not given the group.
Model
Acknowledge conflict
Explain away
Share (%)
Qwen3.5-Plus
61
60
98.4
Seed 2.1 Pro
46
38
82.6
Gemini-3.5-Flash
31
21
67.7
All
138
119
86.2
Appendix
Table 8: Acknowledged conflicts among flip traces. Traces that acknowledge evidence conflicting with the claim, and those among them that explain the conflict away while retaining the claim.
Selected content
Media transformation
Scope
Visual region
Black mask within a normalized rectangle and time interval
Can cover text or non-text content
Visual interval
Black mask over the whole frame for the selected interval
Preserves the video timeline
Audio track
Removes the entire audio track
No partial audio removal
None
Leaves the submitted media unchanged
No content removed
Appendix
Table 9: Evidence blinding operations. Each operation changes the media available to the reader; it does not prescribe a response.
Method
Accuracy (%) ↑
Trap-answer rate (%) ↓
No claim
Accurate
Misleading
DEAR
90.4
98.0
85.2
11.6
− Blinding
90.0
95.2
52.4
42.6
− Reader isolation
89.2
94.2
76.4
18.2
Appendix
Table 10: Effects of blinding and reader isolation. Qwen3.5-Plus on the 500 -video cohort in Table 2 . Each variant removes one DEAR component. Bold marks the best value per column.
Figure 6: Reassigning an action to match the claim. Frames show the misleading condition; responses compare the three claim conditions. Source: VidOR ( Shang et al., 2019 ) , “ At the shellfish restaurant ” by misskoco, CC BY-NC-SA 2.0 ; claim overlays added, and the adapted frames are shared under the same license. Case VidOR:vidor-new-v1-0037-3414991682-c01 .
Direct (benchmark prompt)
Reasoning
The user wants to identify who moves the large rough shellfish onto the grill using metal tongs.
1. Analyze the video: I need to watch the video and look for the specific action described: moving a “large rough shellfish” from the table to the grill using “metal tongs”.
2. Scan for the shellfish:
∙ 00:00 - 00:09: There are three smooth clams on the grill.
∙ 00:10 - 00:15: The camera pans to the right, showing a plate with large, rough-looking shellfish (likely sea snails or turban shells).
Appendix
Table 17
Figure 7: DEAR identifies the blocking object after blinding. The first pair precedes the claim and shows the blue-lid container being placed in the box. Source: Perception Test ( Patraucean et al., 2023 ) , copyright DeepMind Technologies Limited, CC BY 4.0 ; claim overlays and masks added. Case Perception-Test:pt-new-v1-valid-video_4608-q0 .
Direct (benchmark prompt)
Reasoning
The user wants me to identify the object preventing the box from closing based on the video.
1. Analyze the video:
∙ 00:00 - 00:02: The scene is set. There’s a cardboard box, a blue-lid container, a green plastic box, a small carton, and a black box on a book.
∙ 00:02 - 00:05: A person starts putting items into the box. First, the small carton, then the black box/book.
∙ 00:05 - 00:06: The person puts the blue-lid container into the box.
Appendix
Table 19
Direct (reader prompt)
Response
B. blue-lid container Justification: The video shows a person packing items into a cardboard box. After placing the blue-lidded container inside, followed by the green plastic box, the person attempts to fold the flaps down. A text overlay explicitly states, “That won’t close— the blue-lid container is sticking up past the rim,” directly identifying the blue-lidded container as the object preventing the box from closing.
Appendix
Table 20
Self-Review
Response
B. blue-lid container Justification: The video explicitly identifies the cause via a text overlay that appears as the person attempts to pack the box. The text states, “That won’t close— the blue-lid container is sticking up past the rim.” Although the green plastic box is placed on top, the video attributes the height issue preventing the flaps from closing to the blue-lidded container underneath.
Appendix
Table 21
DEAR
Reasoning
The user wants to identify the object preventing the box from closing.
1. Analyze the video:
∙ 00:00 - 00:01: An open cardboard box sits on a wooden pallet. Next to it are a blue-lidded container, a green-lidded container (white body), a small juice carton, a black box, and a book.
∙ 00:02 - 00:04: A person puts the juice carton, black box, and book into the cardboard box.
∙ 00:05 - 00:06: The person puts the blue-lidded container into the box.
Recent Vision-Language Models (VLMs) have made remarkable progress in multimodal understanding tasks, yet their evaluation on long video understanding remains unreliable. Due to limited frame inputs, key frames necessary for answering the question may be missing from the model's input. However, models that truthfully refuse to answer under such uncertainty are marked as incorrect, while those that guess may coincidentally produce the correct answer and thus obtain deceptively higher accuracy, leading to misleading evaluation results and encouraging models to guess rather than respond honestly. To address this issue, we introduce VirtueBench, a benchmark explicitly designed to assess model trustworthiness under uncertainty. VirtueBench constructs multiple frame-sampling levels for each video and provides ground truths that distinguish between answerable and unanswerable cases. Evaluations on 25 open-source and commercial VLMs reveal distinct refusal behaviors across different model families, with refusal accuracy ranging from over 70% in the best models to nearly 0% in the worst. Moreover, most models exhibit a substantial drop in refusal when the prompt does not explicitly require them to do so. These findings highlight the need for developing trustworthy VLMs for multimodal understanding, guided by benchmarks and leaderboards that emphasize reliability and trustworthiness.
Video question answering systems built on vision-language models often produce timestamped claims with high confidence even when unsupported by the cited frame. This deceptive hallucination arises because timestamps imply grounding without ensuring correctness, increasing user trust but not accuracy. We introduce a pipeline that closes this loop. A retrieval-augmented language model drafts answers with per-claim timestamp citations, and each cited frame is independently re-examined before being shown to the user. We compare against a plain baseline and ablate three verification designs, evaluated on both Apple Silicon (MLX) and Google Colab (HF Transformers, CUDA). Directly asking the vision model whether a frame supports a claim fails completely (0% catch rate on 40 claims) due to sycophancy. Blind re-captioning plus a general LLM judge improves results but is unstable, oscillating between 0% and 100% flagged depending on prompt phrasing. Replacing that judge with a small natural language inference model yields a stable, interpretable verifier that catches 79% of fabricated claims on adversarial false-premise questions while leaving true claims untouched. We release the full pipeline, evaluation harness, and implementations for both Apple Silicon and Colab. Code is available at https://github.com/yogesh-iitj/grounded-video-qa.
When asked which of two events came first, video large language models can fail in two opposite ways: cave to a false claim, or reject a true one. Prior video sycophancy work measures only the first and mitigates it by teaching the model to trust the user less, a fix known in text and image models to worsen the second. In video, both failures come from two causes the literature treats as one: availability, whether the sparse sampled frames contain the two events, and weighting, whether that evidence is trusted over the user. We separate them with two interventions that keep the claim fixed: a frame-preserving reorder that flips the claim's truth, and a sampling-offset shift that captures or misses both events at a fixed frame budget. When the events are missed, the two twins present identical frames, so each of the nine models we evaluate accepts a true and a false claim at the same rate, making Youden's J=0 by construction. Availability is necessary but not sufficient. Five of the nine read the order, yet four of those five still cave to the false claim, so their deference hits a weighting ceiling. Since trust cannot be calibrated over evidence that was never sampled, we propose a reversal test that cancels the model's order prior by scoring the sampled frames forward and reversed, then answers, resamples, or abstains without reading the claim. The test raises the order accuracy to 0.92-1.00 on the models that read the order and abstains rather than guesses on those that cannot.
Yuxin Cao, Wei Song, Jingling Xue +1
National University of Singapore, Singapore · University of New South Wales, Australia