VisualTrust: Video Reasoning under On-Screen Claim Framing

Huachao Zhu; Junjielong Xu; Sicheng Li; Liang Yang; Mingtao Chen; Zhichao Hu; Li Jiang; Pinjia He

cs.CVOct 15, 2026

VisualTrust: Video Reasoning under On-Screen Claim Framing

Authors: Huachao Zhu, Junjielong Xu, Sicheng Li, Liang Yang, Mingtao Chen, Zhichao Hu, Li Jiang, Pinjia He

Organizations: School of Data Science, The Chinese University of Hong Kong, Shenzhen · Tencent

Abstract

On-screen text can describe what a video shows, or it can frame what the video is supposed to mean. Current benchmarks treat both alike, testing whether a model can read the text rather than whether it should trust it. We study claim deference, multimodal large language models' tendency to let an on-screen claim override conflicting visual evidence. We introduce VisualTrust, a counterfactual benchmark with 5,063 cases across 14 claim-fact classes. Each case pairs a question with three versions of the same video: one with no claim, one with an accurate claim, and one with a misleading claim that alters a single grounded fact. The option the misleading claim supports is the trap answer. Across seven open-weight and four proprietary models, mean accuracy falls from 90.1% without a claim to 51.9% with a misleading claim. Among initially correct responses, 42.8% flip to the trap answer and only 0.6% become other errors. Strong no-claim accuracy does not ensure resistance. We also introduce DEAR (Decoupled Evidence Arbitration and Reading), a training-free agent for video question answering that separates evidence arbitration from reading. An independent check reports conflicting evidence, a selector weighs it without a fixed modality priority and blinds what it withholds, and the reader answers from the resulting video in a fresh context. On 500 videos across three proprietary models, DEAR raises accuracy under misleading claims from 47.2--64.0% to 81.8--85.2% and reduces trap-answer selection from 34.4--50.8% to 11.6--15.0%, while preserving no-claim accuracy.

Figures & tables

Figure 1: VisualTrust separates reading on-screen text from trusting it. (a,b) Existing benchmarks test whether models understand the video or read its on-screen text. (c) VisualTrust holds the video, question, and options fixed and varies only the rendered claim; in the example, the misleading claim redirects an otherwise correct response to the option it supports.

Figure 2: How one case is built. (a) The video, question, options, correct answer, and trap answer are fixed before either claim is written. (b) A single-point edit turns the accurate claim into its misleading counterpart, and both are rendered onto the same base, which itself supplies the no-claim condition. The case shown is Perception-Test:video_1879 .

Open-weight
Model	Accuracy (%) $\uparrow$			TFR (%) $\downarrow$
Model	No claim	Accurate	Misleading	TFR (%) $\downarrow$
Qwen3.5-9B	89.1	98.7	47.1	47.5
Qwen3.5-27B	92.5	99.3	51.4	44.3
Qwen3.5-35B-A3B	91.3	99.0	47.8	47.7
Qwen3.5-122B-A10B	92.8	99.2	51.4	44.5

Table 1: Claim deference across eleven video models. Accuracy under each claim condition, and the trap-flip rate (TFR): the share of a model’s no-claim-correct responses that flip to the trap answer under the misleading claim.

Figure 3: Misleading claims redirect correct responses to the trap answer. (a) Accuracies under three claim conditions. (b) Destinations of no-claim-correct responses under the misleading claim.

Figure 4: DEAR separates evidence arbitration from reading. An empty selection leaves the video unchanged. The case shown is VideoMME:258-2 .

Qwen3.5-Plus
Method	Accuracy (%) $\uparrow$			Trap-answer rate (%) $\downarrow$
Method	No claim	Accurate	Misleading	Trap-answer rate (%) $\downarrow$
Direct (benchmark prompt)	89.4	96.6	49.4	47.6
Direct (reader prompt)	91.0	98.2	53.6	43.2
Self-Review	90.0	97.0	54.6	42.0
DEAR	90.4	98.0	85.2	11.6

Table 2: DEAR versus prompt-level baselines for mitigating claim deference. For each backbone, all methods answer the same 500 source videos under each claim condition. DEAR reduces the trap-answer rate more than either prompt-level baseline. Trap-answer rate uses all misleading-condition inputs as its denominator; execution failures count as incorrect but never as trap selections. Bold marks the best value per column within each backbone.

Appendix figures & tables16 assets

Supplementary material from the paper’s appendix.

Appendix

Class	Changed proposition	Example change	Cases
ENT	Identity of an entity, object, place, or scene	a cup $\to$ a bowl	532
ROL	Agent, patient, recipient, or instrument	boy pulls girl $\to$ girl pulls boy	438
EVT	Event occurrence, completion, or outcome	succeeds $\to$ fails	551
ACT	Action type or manner	walking $\to$ running	544
SET	Membership, count, or completeness	four people $\to$ three people	462
STA	Visible attribute, posture, or state	red clothing $\to$ blue clothing	484

Appendix

Table 3: Claim-fact classes. Examples illustrate a single-fact change. Counts describe the curated benchmark, not the prevalence of these facts in unconstrained video.

Class	Misleading	Accurate
ENT	29.19	0.11
ROL	49.28	0.29
EVT	72.05	1.08
ACT	46.72	0.32
SET	55.17	0.43
STA	27.49	0.08

Appendix

Table 4: Trap-flip rates by claim-fact class. Equal-model means (%). Within each model and class, both conditions use the same no-claim-correct cases and predesignated trap answers.

Figure 5: Trap-flip rates across claim-fact classes. Points weight the eleven models equally; intervals are pointwise 95% source-bootstrap intervals. The dashed line marks the overall misleading-claim mean, and parentheses give benchmark case counts. Within each model and class, both series use the same no-claim-correct cases and predesignated trap answers.

Model	Temperature	Top- $p$	Top- $k$	Presence penalty
Qwen3.5-9B/27B/35B/122B	1.0	0.95	20	1.5
Qwen3.5-397B	0.6	0.95	20	0.0
Gemma 4-26B/31B	1.0	0.95	64	1.5

Appendix

Table 5: Self-hosted sampling settings. Thinking is enabled and repetition penalty is

1

Open-weight models
Model	Cases	Trap-answer rate (%)		Increase (pp)
Model	Cases	No claim	Misleading	Increase (pp)
Qwen3.5-9B	5063	7.1	51.9	44.8
Qwen3.5-27B	5063	5.2	47.9	42.7
Qwen3.5-35B-A3B	5063	5.6	51.4	45.8
Qwen3.5-122B-A10B	5063	4.5	47.8	43.3

Appendix

Table 6: Increase in trap-answer selection. Rates use all complete three-condition cases, including cases already incorrect without a claim. The increase is the difference between the two rates.

Model	Group	Cited as grounds	Cited and rejected	Not cited	Unclear
Qwen3.5-Plus	Flip	100	0	0	0
Qwen3.5-Plus	Still correct	4	86	10	0
Seed 2.1 Pro	Flip	75	0	23	2
Seed 2.1 Pro	Still correct	4	70	24	2
Gemini-3.5-Flash	Flip	99	0	1	0
Gemini-3.5-Flash	Still correct	5	76	17	2

Appendix

Table 7: Claim use in reasoning traces, by model. Counts of flip and still-correct traces (

n=100

each) by claim-use label. The judge is not given the group.

Model	Acknowledge conflict	Explain away	Share (%)
Qwen3.5-Plus	61	60	98.4
Seed 2.1 Pro	46	38	82.6
Gemini-3.5-Flash	31	21	67.7
All	138	119	86.2

Appendix

Table 8: Acknowledged conflicts among flip traces. Traces that acknowledge evidence conflicting with the claim, and those among them that explain the conflict away while retaining the claim.

Selected content	Media transformation	Scope
Visual region	Black mask within a normalized rectangle and time interval	Can cover text or non-text content
Visual interval	Black mask over the whole frame for the selected interval	Preserves the video timeline
Audio track	Removes the entire audio track	No partial audio removal
None	Leaves the submitted media unchanged	No content removed

Appendix

Table 9: Evidence blinding operations. Each operation changes the media available to the reader; it does not prescribe a response.

Method	Accuracy (%) $\uparrow$			Trap-answer rate (%) $\downarrow$
Method	No claim	Accurate	Misleading	Trap-answer rate (%) $\downarrow$
DEAR	90.4	98.0	85.2	11.6
$-$ Blinding	90.0	95.2	52.4	42.6
$-$ Reader isolation	89.2	94.2	76.4	18.2

Appendix

Table 10: Effects of blinding and reader isolation. Qwen3.5-Plus on the

500

-video cohort in Table 2 . Each variant removes one DEAR component. Bold marks the best value per column.

Figure 6: Reassigning an action to match the claim. Frames show the misleading condition; responses compare the three claim conditions. Source: VidOR ( Shang et al., 2019 ) , “ At the shellfish restaurant ” by misskoco, CC BY-NC-SA 2.0 ; claim overlays added, and the adapted frames are shared under the same license. Case VidOR:vidor-new-v1-0037-3414991682-c01 .

Direct (benchmark prompt)
Reasoning
The user wants to identify who moves the large rough shellfish onto the grill using metal tongs.
1. Analyze the video: I need to watch the video and look for the specific action described: moving a “large rough shellfish” from the table to the grill using “metal tongs”.
2. Scan for the shellfish:
$\bullet$ 00:00 - 00:09: There are three smooth clams on the grill.
$\bullet$ 00:10 - 00:15: The camera pans to the right, showing a plate with large, rough-looking shellfish (likely sea snails or turban shells).

Appendix

Table 17

Figure 7: DEAR identifies the blocking object after blinding. The first pair precedes the claim and shows the blue-lid container being placed in the box. Source: Perception Test ( Patraucean et al., 2023 ) , copyright DeepMind Technologies Limited, CC BY 4.0 ; claim overlays and masks added. Case Perception-Test:pt-new-v1-valid-video_4608-q0 .

Direct (benchmark prompt)
Reasoning
The user wants me to identify the object preventing the box from closing based on the video.
1. Analyze the video:
$\bullet$ 00:00 - 00:02: The scene is set. There’s a cardboard box, a blue-lid container, a green plastic box, a small carton, and a black box on a book.
$\bullet$ 00:02 - 00:05: A person starts putting items into the box. First, the small carton, then the black box/book.
$\bullet$ 00:05 - 00:06: The person puts the blue-lid container into the box.

Appendix

Table 19

Direct (reader prompt)

Response

B. blue-lid container Justification: The video shows a person packing items into a cardboard box. After placing the blue-lidded container inside, followed by the green plastic box, the person attempts to fold the flaps down. A text overlay explicitly states, “That won’t close— the blue-lid container is sticking up past the rim,” directly identifying the blue-lidded container as the object preventing the box from closing.

Appendix

Table 20

Self-Review

Response

B. blue-lid container Justification: The video explicitly identifies the cause via a text overlay that appears as the person attempts to pack the box. The text states, “That won’t close— the blue-lid container is sticking up past the rim.” Although the green plastic box is placed on top, the video attributes the height issue preventing the flaps from closing to the blue-lidded container underneath.

Appendix

Table 21

DEAR
Reasoning
The user wants to identify the object preventing the box from closing.
1. Analyze the video:
$\bullet$ 00:00 - 00:01: An open cardboard box sits on a wooden pallet. Next to it are a blue-lidded container, a green-lidded container (white body), a small juice carton, a black box, and a book.
$\bullet$ 00:02 - 00:04: A person puts the juice carton, black box, and book into the cardboard box.
$\bullet$ 00:05 - 00:06: The person puts the blue-lidded container into the box.

Appendix

Table 22

Video-Language Model Evaluation

Video-language model evaluation measures whether multimodal systems understand, ground, and reliably use visual and audio evidence over time.

See all 99 papers

VirtueBench: Evaluating Trustworthiness under Uncertainty in Long Video Understanding
Mar 7, 2026Xueqing Yu, Bohan Li, Yan Li +1Abstention in VLMs VLM Evaluation
Peking University · ByteDance
Catching Hallucinated Citations in Video-LLM Question Answering: A Self-Verification Pipeline and Verifier Ablation Study
Aug 16, 2026Yogesh KumarHallucination Detection in VLMs Video QA
Caved or Convinced: Temporal Sampling Gates Claim Deference in Video Large Language Models
Aug 4, 2026Yuxin Cao, Wei Song, Jingling Xue +1Selective Prediction Video-Language Model Evaluation
National University of Singapore, Singapore · University of New South Wales, Australia

Mar 7, 2026J/K move · Enter open · S save

VirtueBench: Evaluating Trustworthiness under Uncertainty in Long Video Understanding

Xueqing Yu, Bohan Li, Yan Li, Zhenheng Yang

Peking University · ByteDance

Abstention in VLMs VLM Evaluation Video-Language Model Evaluation Vision-Language Models Long-Video Understanding

Open paper AskPDF arXiv

Recent Vision-Language Models (VLMs) have made remarkable progress in multimodal understanding tasks, yet their evaluation on long video understanding remains unreliable. Due to limited frame inputs, key frames necessary for answering the question may be missing from the model's input. However, models that truthfully refuse to answer under such uncertainty are marked as incorrect, while those that guess may coincidentally produce the correct answer and thus obtain deceptively higher accuracy, leading to misleading evaluation results and encouraging models to guess rather than respond honestly. To address this issue, we introduce VirtueBench, a benchmark explicitly designed to assess model trustworthiness under uncertainty. VirtueBench constructs multiple frame-sampling levels for each video and provides ground truths that distinguish between answerable and unanswerable cases. Evaluations on 25 open-source and commercial VLMs reveal distinct refusal behaviors across different model families, with refusal accuracy ranging from over 70% in the best models to nearly 0% in the worst. Moreover, most models exhibit a substantial drop in refusal when the prompt does not explicitly require them to do so. These findings highlight the need for developing trustworthy VLMs for multimodal understanding, guided by benchmarks and leaderboards that emphasize reliability and trustworthiness.

Abstract

Video-Language Model Evaluation

Explore similar work

VirtueBench: Evaluating Trustworthiness under Uncertainty in Long Video Understanding

Catching Hallucinated Citations in Video-LLM Question Answering: A Self-Verification Pipeline and Verifier Ablation Study

Caved or Convinced: Temporal Sampling Gates Claim Deference in Video Large Language Models