Tinted Frames: Question Framing Blinds Vision-Language Models

Wan-Cyuan Fan; Jiayun Luo; Declan Kutscher; Leonid Sigal; Ritwik Gupta

cs.CVMar 19, 2026

Tinted Frames: Question Framing Blinds Vision-Language Models

Authors: Wan-Cyuan Fan, Jiayun Luo, Declan Kutscher, Leonid Sigal, Ritwik Gupta

Organizations: University of British Columbia · Vector Institute for AI · University of California, Berkeley

Abstract

Vision-Language Models (VLMs) have been shown to be blind, often underutilizing their visual inputs even on tasks that require visual reasoning. In this work, we demonstrate that VLMs are selectively blind. They modulate the amount of attention applied to visual inputs based on linguistic framing even when alternative framings demand identical visual reasoning. Using visual attention as a probe, we quantify how framing alters both the amount and distribution of attention over the image. Constrained framings, such as multiple choice and yes/no, induce substantially lower attention to image context compared to open-ended, reduce focus on task-relevant regions, and shift attention towards uninformative tokens. We further demonstrate that this attention misallocation is the principal cause of degraded accuracy and cross-framing inconsistency. Building on this mechanistic insight, we introduce a lightweight prompt-tuning method using learnable tokens that encourages the robust, visually grounded attention patterns observed in open-ended settings, improving visual grounding and improving performance across framings.

Figures & tables

Figure 1: VLM grounding changes as a function of question framing. Attention maps reveal that while the model actively attends to the target object (the chair) during open-ended generation, it exhibits disengagement and misallocation when the same question is posed as a Yes/No or MCQ task. Note that we employ attention rollout rather than averaging attention weights across layers and token. By recursively rolling out attention matrices, we trace information propagation from inputs of early layers to output embeddings. Qwen2.5-VL-7B is used for the visualization. The top 3 tokens with the highest attention are highlighted in red boxes, and the minimum and maximum values of the linear colormap are set to the same value for all images.

Figure 2: Illustration of impact of question framing. We hypothesize that question framing influences model predictions through visual attention. Framing alters attention allocation (F $\rightarrow$ A), which in turn degrades prediction quality (A $\rightarrow$ Y).

Panel of Figure 3 — Figure 3: Tested open-source VLMs have a significant inconsistency rates across framings and task types. (Left) Cross-framing inconsistency is evaluated by reframing questions with an LLM. (Right) With Qwen2.5-VL-7B, inconsistency is up to 26% on GQA and 38% on SeedBench.

Figure 4: Visual energy drops significantly on non-open-ended framings. (Top) There is a significant drop in attention applied to the portion of the image containing the object of interest, and a corresponding increase in attention to sink tokens, for yes/no and MCQ framing. (Bottom) Per-layer values of visual energy and box attention of Qwen2.5-VL-7B on GQA

{}^{\text{F}}

Figure 7: Learning to re-align attention . Given a VQA sample, we reframe the question/answer pairs into alternate framing with a frozen LLM. We append sets of learned tokens to constrained framings at the end of the input sequence. During training, we add an attention alignment objective on the attention rollout to align the attention from open-ended samples with constrained framings.

Figure 8: (Left) Performance comparison after appending ours learned soft tokens on input prompts on GQA

{}^{\text{F}}

and V

{}^{\text{*F}}

. (Right) Cross-framing inconsistency comparison between baseline (Qwen2.5-VL-7B) and ours.

Methods	General & Reasoning			Alignment		Fine-grained grounding
Methods	RealWorldQA †	MME	MMMU-Pro ‡	Hallusion ‡	POPE	HRBench8k	V ${}^{\text{*}}$
Gemma3-12B	61.87	2133.5	27.92	64.56	84.63	48.25	48.50
+ Ours	61.87	2180.0	28.09	66.14	84.98	49.50	49.79
GLM4.1V-9B-Base	69.41	2208.1	32.22	63.72	88.52	51.62	58.80
+ Ours	69.41	2237.8	31.41	67.51	87.24	52.25	61.80
LLaVA-OV1.5-8B	70.09	2276.2	30.52	66.04	88.82	58.00	62.23

Table 1: Performance impact of our method. † denotes the multiple-choice subset, and ‡ denotes the vision split.

Ablation	RWQA †	MMMU-Pro ‡	POPE	V ${}^{\text{*}}$	Avg
Baseline	63.70	31.45	87.68	66.95	62.45
Ours	66.44	31.50	88.65	69.53	64.03
w/o Attn Loss	64.84	32.89	87.05	65.67	62.61
w/o CE Loss	64.61	31.27	83.40	63.95	60.81

Table 2: Ablation on loss functions and learnable token count. RWQA denotes RealWorldQA. † denotes the multiple-choice subset, and ‡ denotes the vision split.

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 10 — Figure A1 : Human evaluation pipeline for open-ended and Yes/No question reframing. Note that for MCQ question reframing, we can use a rule-based approach (simply remove the options) to convert it into open-ended and then perform reframing to Yes/No.

From $\downarrow$ / To $\rightarrow$	Open-ended	Yes/No		MCQ
From $\downarrow$ / To $\rightarrow$	Correctness	Correctness	Spatial Cons.	Correctness	Exclusivity
Open-ended	—	96.5	96.9	94.8	95.7
Yes/No	92.2	—	—	94.8	93.7
MCQ	rule-based	92.6	95.4	—	—

Appendix

Table A1 : Human evaluation results for question reframing across different conversion types. MCQ

\rightarrow

Open-ended is handled via rule-based conversion and requires no human evaluation.

	Trainable Parameters	Time (min)	VRAM (peak)	VIRT (peak)
Qwen2.5VL-7B	57 K	187	33.8 GB	53.1 GB
Qwen3VL-8B	66 K	190	42.7 GB	65.9 GB
Gemma3-12B	61 K	380	39.6 GB	59.7 GB
GLM-4.1V-8B-Base	66 K	332	76.9 GB	116.1 GB
LLaVA-OneVision-1.5-8B	66 K	259	36.5 GB	54.6 GB

Appendix

Table A2 : Resource usage statistics for prompt tuning. Tested on 1 NVIDIA L40 with batch size 1 and gradient accumulation 16. VIRT stands for virtual memory size. GLM-4.1V-8B-Base is tested on 1 NVIDIA H100.

Position	General & Reasoning			Alignment		Fine-grained grounding
Position	RealWorldQA †	MME	MMMU-Pro ‡	Hallusion ‡	POPE	HRBench8k	V ${}^{\text{*}}$
Prefix	61.64	2272.2	28.15	65.51	88.28	51.62	66.52
Infix	66.44	2269.7	31.50	68.77	88.65	54.13	69.53
Postfix	63.47	2262.4	31.50	65.83	88.57	54.25	67.38

Appendix

Table A3 : Ablation study on learnable token positioning. † denotes the multiple-choice subset, and ‡ denotes the vision split.

Strategy	General & Reasoning			Alignment		Fine-grained grounding
Strategy	RealWorldQA †	MME	MMMU-Pro ‡	Hallusion ‡	POPE	HRBench8k	V ${}^{\text{*}}$
Baseline	63.70	2199.5	31.45	71.50	87.68	54.13	66.95
Confidence weighting	66.44	2269.7	31.50	68.77	88.65	54.13	69.53
Equal weighting	64.84	2205.6	30.25	65.51	89.09	52.50	68.67

Appendix

Table A4 : Ablation study on learnable token weighting strategies. † denotes the multiple-choice subset, and ‡ denotes the vision split.

Figure A2 : Visual energy drops significantly on non-open-ended framings.

Figure A3 : Visual energy drops significantly on non-open-ended framings.

Figure A4 : Qualitative examples of open-ended question reframing.

Figure A5 : Qualitative examples of Yes/No question reframing.

Figure A6 : Qualitative comparison of Baseline and Ours (Qwen2.5-VL-7B) .

Vision-Language Models

Vision-language models connect visual inputs with text for retrieval, question answering, generation, and grounded multimodal reasoning.

See all 721 papers

Arbitration Failure, Not Perceptual Blindness: How Vision-Language Models Resolve Visual-Linguistic Conflicts
Apr 10, 2026Farhad Nooralahzadeh, Omid Rohanian, Yi Zhang +2Vision-Language Grounding Visual Evidence Sensitivity in VLMs
Institute of Computer Science, Zurich University of Applied Sciences, Switzerland · University of Oxford, UK.
Listening makes Vision Clear for VLMs
Jun 22, 2026Yiyang Chen, Yixin Tan, Binrui ShenVLM Evaluation Visual Attention
Beijing Normal University
Look Where You Say You're Looking: Self-Grounded Attention for Visual Reasoning
Oct 4, 2026Uri Berger, Gal Chechik, Gal DalalVision-Language Grounding VLM Reasoning
NVIDIA Research · The Hebrew University of Jerusalem · University of Melbourne +1

Apr 10, 2026J/K move · Enter open · S save

Arbitration Failure, Not Perceptual Blindness: How Vision-Language Models Resolve Visual-Linguistic Conflicts

Farhad Nooralahzadeh, Omid Rohanian, Yi Zhang, Jonathan Fürst+1

Institute of Computer Science, Zurich University of Applied Sciences, Switzerland · University of Oxford, UK.

Vision-Language Grounding Visual Evidence Sensitivity in VLMs Faithfulness in VLMs VLM Interpretability

Open paper AskPDF arXiv

When a Vision-Language Model (VLM) sees a blue banana and answers "yellow", is the problem of perception or arbitration? We explore the question in ten VLMs with various sizes and reveal an Encoding-Grounding Dissociation: models that fail to report what they see (and thus provide a wrong answer) still encode the visual evidence as strongly as models that provide the correct answer. Using Multimodal Arbitration Crossover (MAC) analysis with layer-by-layer Logit Lens probing, we track the competition between visual and prior signals across every layer of each model. We show that visual attributes can be linearly decodable from early layers (AUC > 0.86). The accuracy remains nearly identical for both successful and failed samples. However, the gap in the final-layer logit - not the strength of encoding - better predicts grounding outcomes with a correlation of

ρ=

0.847. After having studied when VLMs base their answers on image clues rather than prior knowledge, we want to understand the causal relationships. We establish causality through full-sequence activation patching. The standard last-token interventions in LLM interpretability do not affect VLMs. In contrast, replacing the full token sequence at layers identified by MAC alters 60 to 84% of outputs. Partial-token decomposition shows that image tokens carry almost all of the causal impact, while text tokens have none. Scaling addresses the remaining architectural differences to achieve perfect retention. Moving from diagnosis to intervention, we show that training-free activation steering - both linear and sparse autoencoder-guided - in early layers can improve visual grounding by up to +3.8% with degrading performance in some setups. Overall, these findings lead to a clear conclusion: VLMs already see well, but the challenge is acting on what they see. Targeted interventions can help to bridge this gap.

Abstract

Vision-Language Models

Explore similar work

Arbitration Failure, Not Perceptual Blindness: How Vision-Language Models Resolve Visual-Linguistic Conflicts

Listening makes Vision Clear for VLMs

Look Where You Say You're Looking: Self-Grounded Attention for Visual Reasoning