When Helpful Text Hurts: Option-Redirecting Bias in Vision-Language Models

Tam Le Thi Thanh; Hoang Tran Van; Hong-Hanh Nguyen-Le; Thanh Duc Ngo

cs.AISep 26, 2026

When Helpful Text Hurts: Option-Redirecting Bias in Vision-Language Models

Authors: Tam Le Thi Thanh, Hoang Tran Van, Hong-Hanh Nguyen-Le, Thanh Duc Ngo

Organizations: University of Information Technology, VNU-HCM Ho Chi Minh City, Vietnam Vietnam National University Ho Chi Minh City, Vietnam · University College Dublin Dublin, Ireland

Abstract

In tri-modal visual question answering (VQA), auxiliary text is commonly used to complement visual and textual inputs, yet its reliability is often uncontrolled. While prior work studies modality conflicts in general, it remains unclear how different types of unreliable auxiliary text affect answer selection under fixed image-question-option contexts. In this work, we show that the most harmful auxiliary text is not necessarily the most factually incorrect, but the one that aligns with the question while contradicting the image and favoring a specific distractor, leading to systematic redirection of model predictions. To isolate this effect, we introduce the Textual Reliability Ladder, a controlled diagnostic protocol that decomposes auxiliary text along three axes: image consistency, question relevance, and option support. Across multiple datasets (ScienceQA, VCR, A-OKVQA, Causal-VidQA) and recent VLMs, we find that such distractor-supporting text induces the largest accuracy drops (up to 53.1%) and concentrates errors on specific incorrect options. To mitigate this failure mode, we propose a training-free inference-time intervention that explicitly counteracts this redirection effect via noise-stability steering and dynamic grounding, reducing redirected errors while largely preserving performance under faithful text. Our results highlight that auxiliary-text reliability must be understood at the decision level, rather than solely through factual correctness, and provide a practical pathway toward more robust tri-modal reasoning.

Figures & tables

Figure 1. Decision-level auxiliary-text bias in tri-modal multiple-choice VQA. We fix the image, question, and answer options, and vary only the auxiliary text to isolate its effect on answer selection. While faithful text supports correct reasoning and irrelevant text induces mild uncertainty, certain misleading text can systematically redirect predictions toward a specific incorrect option. We categorize auxiliary text into five regimes—Faithful, Off-Question, Contradictory, Option-Redirecting, and Irrelevant—along with a No-Aux baseline. Among these, Option-Redirecting text is particularly harmful: although aligned with the question, it is inconsistent with the image and implicitly favors a distractor, leading to concentrated errors rather than diffuse confusion. Schematic of tri-modal multiple-choice VQA. The image, question, and answer choices are fixed while the auxiliary text is varied across No-Aux, Faithful, Off-Question, Contradictory, Option-Redirecting, and Irrelevant conditions. Option-Redirecting text points toward a specific incorrect answer.

Textual type	Image	Question	Wrong ans.	Correct ans.
Faithful	$\checkmark$	$\checkmark$	$\times$	$\checkmark$
Off-Question	$\checkmark$	$\times$	$\times$	$\times$
Contradictory	$\times$	$\checkmark$	$\times$	$\times$
Option-Redirecting	$\times$	$\checkmark$	$\checkmark$	$\times$
Irrelevant	$\times$	$\times$	$\times$	$\times$

Table 1. Textual Reliability Ladder conditions and their diagnostic cues. Each condition is defined by three factors: whether the auxiliary text is grounded in the image , relevant to the question , and whether it implicitly supports a specific correct or incorrect answer option. The ladder spans from reliable (Faithful), to non-informative (Off-Question, Irrelevant), to decision-level misleading (Option-Redirecting).

Figure 2. Overview of our two-stage inference-time intervention. Stage 1 estimates global steering directions from contrastive residuals, while Stage 2 performs instance-level, question-aware grounding by aligning textual representations with question-relevant visual evidence. Together, the two stages reduce option-redirecting auxiliary-text bias while maintaining performance under non-harmful auxiliary text. Flow diagram of the two-stage inference-time intervention. Stage 1 computes global Noise-Stability Steering directions offline from contrastive residuals, and Stage 2 performs online Dynamic Grounding by aligning auxiliary-text representations with question-relevant visual evidence before answer prediction.

Figure 3. Accuracy of frozen VLMs under the Textual Reliability Ladder. Option-Redirecting text causes the largest and most consistent degradation across datasets and models Performance plots across A-OKVQA, ScienceQA, VCR, and Causal-VidQA for several frozen vision-language models under the Textual Reliability Ladder. Option-Redirecting text produces the largest and most consistent accuracy decrease.

Figure 4. Accuracy change after NSS+DG relative to the frozen baseline. Improvements concentrate under Option-Redirecting text, with small changes in other conditions. Heatmap of percentage-point accuracy changes from the combined NSS and DG intervention across models, datasets, and auxiliary-text conditions. The strongest positive changes are concentrated in the Option-Redirecting condition.

Metric	NSS only	DG only	NSS + DG
Avg. $\Delta$ Acc. (Faithful)	+0.11	-1.50	-1.30
Avg. $\Delta$ Acc. (Option-Redirecting)	+0.24	+6.72	+6.99
Avg. $\Delta$ Acc. (Non-OR)	+0.06	+0.07	+0.16

Table 2. Macro-averaged accuracy changes (

\Delta

, in percentage points) relative to frozen base models, aggregated across all model–dataset pairs. NSS introduces minimal shifts across conditions, while DG yields substantial improvements under Option-Redirecting text at the cost of slight degradation under Faithful text. The combined method ( NSS+DG ) achieves the best overall trade-off. Full regime-wise results are provided in Appendix D.3 . Non-OR includes No-Aux, Irrelevant, Off-Question, and Contradictory.

Appendix figures & tables25 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Table 3 — Table 3. Prompt templates used to generate Off-Question text.

Panel of Table 4 — Table 4. Prompt templates used to generate Contradictory text.

Panel of Table 5 — Table 5. Prompt templates used to generate Option-Redirecting text.

Model	Dataset	Noise-Stability Steering				Dynamic Grounding
Model	Dataset	$\alpha$ image	$\alpha$ text	Layer range	N_demos	$\beta$ DG	top $k$	Layer range
Qwen2.5VL-3B	Causal-VidQA	0.5	0.3	All	30	0.4	20	19–33
	ScienceQA	0.5	0.4	All	50	0.5	7	19–33
	A-OKVQA	0.4	0.4	All	50	0.4	7	19–33
	VCR	0.5	0.5	All	50	0.4	7	19–33
Qwen2.5VL-7B	Causal-VidQA	0.4	0.5	All	30	0.4	20	13–28

Appendix

Table 6. Best-performing hyperparameter configurations for our method (NSS + DG), used to produce the results in Figure 4 (percentage accuracy gain over the base model) across models and datasets. The reported settings are selected via brute-force search over predefined ranges, aiming to identify configurations that consistently provide strong performance gains while maintaining stability across different models and datasets.

Model	Causal-VidQA			ScienceQA			A-OKVQA			VCR
Model	Unsteered (Frozen)	Two-stage (Ours)	$\Delta(\%)$	Unsteered (Frozen)	Two-stage (Ours)	$\Delta(\%)$	Unsteered (Frozen)	Two-stage (Ours)	$\Delta(\%)$	Unsteered (Frozen)	Two-stage (Ours)	$\Delta(\%)$
Qwen2.5VL-3B	15.9	10.7	5.2	45.3	40.7	4.6	23.5	14.2	9.3	31.6	23.2	8.4
Qwen2.5VL-7B	15.8	12.6	3.2	59.0	43.8	15.2	26.1	17.0	9.1	38.5	29.2	9.3
Qwen3VL-4B	14.3	12.5	1.8	41.0	30.1	10.9	16.8	11.7	5.1	25.8	22.2	3.6
Qwen3VL-8B	16.4	13.8	2.6	39.3	29.5	9.8	18.2	12.1	6.1	30.8	20.8	10.0
InternVL3.5-4B	30.0	20.6	9.4	63.8	48.6	15.2	46.8	37.9	8.9	47.4	39.6	7.8

Appendix

Table 7. Redirect analysis across datasets. Redirect reports the percentage of samples (out of 1,000) whose predictions match the redirected option under option-targeted biased text (lower is better).

\Delta(\%)

denotes the absolute reduction in Redirect (in percentage points) relative to the no-intervention baseline, computed as

\Delta=\mathrm{Redirect}~{\text{Unsteer (Frozen)}}-\mathrm{Redirect}~{\text{Two-stages (Our)}}

(higher is better). Lower Redirect indicates stronger robustness to option-redirecting textual bias.

Panel of Table 8 — Table 8. Worked example of the Textual Reliability Ladder. We keep the image, question, and answer options fixed, and vary only the auxiliary text to instantiate the five ladder conditions (Faithful, Off-Question, Contradictory, Option-Redirecting, Irrelevant).

Figure 5. Image example Rocky beach scene with two boys sitting near the water in the lower left and a person under a red-and-white umbrella on rocks in the distance.

Text Type	A-OKVQA	ScienceQA	VCR	Causal-VidQA
Image	1000	1000	1000	3716
Faithful	$12.72\!\pm\!7.07$	$56.00\!\pm\!51.79$	$20.20\!\pm\!10.22$	$14.42\!\pm\!6.35$
Off-Question	$20.34\!\pm\!4.55$	$16.11\!\pm\!4.16$	$12.92\!\pm\!2.84$	$18.42\!\pm\!4.16$
Contradictory	$5.66\!\pm\!2.01$	$5.95\!\pm\!2.37$	$8.86\!\pm\!3.30$	$9.58\!\pm\!3.01$
Option-Redirecting	$12.86\!\pm\!6.86$	$53.33\!\pm\!49.00$	$18.86\!\pm\!8.71$	$14.22\!\pm\!6.74$
Irrelevant	$25.26\!\pm\!12.88$	$25.43\!\pm\!13.35$	$24.80\!\pm\!12.89$	$25.54\!\pm\!12.43$

Appendix

Table 9. Auxiliary-text length statistics for 1,000 evaluated instances per dataset. Cells report per-instance token lengths (mean

\pm

std). Strict instance-wise token-length matching is enforced only for Option-Redirecting relative to the corresponding Faithful text; other conditions are generated under fixed prompts with coarse length control.

Panel of Table 10 — Table 10. A concrete example used to construct NSS steering vectors.

Figure 6. Heatmap of percentage accuracy improvement over the base models using only Stage 1 (NSS) under different auxiliary-text conditions. Heatmap of percentage-point accuracy changes from Stage 1, Noise-Stability Steering, across models, datasets, and auxiliary-text conditions.

Panel of Table 11 — Table 11. Prompt templates used to evaluate.

Model	Dataset	Faithful	Contradictory	Off-Question	Option-Redirecting	Irrelevant	No-Aux
Qwen2.5VL-3B	Causal-VidQA	77.3	71.6	72.3	65.2	72.3	71.8
	ScienceQA	92.9	75.1	75.1	49.3	74.9	73.1
	A-OKVQA	90.9	83.2	82.9	67.4	83.1	82.7
	VCR	83.1	69.7	69.8	52.4	69.1	69.1
Qwen2.5VL-7B	Causal-VidQA	77.9	71.7	73.3	65.3	72.6	72.1
	ScienceQA	96.1	80.3	80.5	37.3	81.5	81.3

Appendix

Table 12. Ladder evaluation on frozen VLMs (no intervention). Accuracy (%) under different auxiliary-text conditions (Faithful, Contradictory, Off-Question, Option-Redirecting, Irrelevant, and No-Aux), measured on frozen VLMs without applying our method or any parameter updates. Each dataset uses 1,000 samples.

Model	Dataset	Faithful	Contradictory	Off-Question	Option-Redirecting	Irrelevant	No-Aux
Qwen2.5VL-3B	Causal-VidQA	77.4	71.5	72.1	65.8	72.0	71.7
	ScienceQA	93.0	75.2	75.4	49.3	75.1	73.2
	A-OKVQA	91.0	83.5	83.8	69.2	83.3	83.4
	VCR	83.6	69.6	69.0	52.4	69.4	69.1
Qwen2.5VL-7B	Causal-VidQA	77.8	71.4	73.5	65.6	72.8	72.3
	ScienceQA	95.8	80.9	80.4	37.6	81.8	81.4

Appendix

Table 13. Only NSS on frozen VLMs. Ladder results reported as accuracy (%) under different auxiliary-text conditions (Faithful, Contradictory, Off-Question, Option-Redirecting, Irrelevant, and No-Aux). Only NSS is applied at inference time (no training or parameter updates). Each dataset uses 1,000 samples.

Model	Dataset	Faithful	Contradictory	Off-Question	Option-Redirecting	Irrelevant	No-Aux
Qwen2.5VL-3B	Causal-VidQA	75.3	71.4	72.0	70.1	72.1	71.8
	ScienceQA	91.3	74.8	75.3	54.6	74.4	72.8
	A-OKVQA	89.1	83.1	83.8	75.1	83.5	83.0
	VCR	79.6	70.5	69.9	59.3	70.9	69.6
Qwen2.5VL-7B	Causal-VidQA	77.0	72.6	73.7	67.2	73.1	72.1
	ScienceQA	96.0	80.9	80.3	51.3	80.6	81.3

Appendix

Table 14. Only DG on frozen VLMs. Ladder results reported as accuracy (%) under different auxiliary-text conditions (Faithful, Contradictory, Off-Question, Option-Redirecting, Irrelevant, and No-Aux). Only DG is applied at inference time (no training or parameter updates). Each dataset uses 1,000 samples.

Model	Dataset	Faithful	Contradictory	Off-Question	Option-Redirecting	Irrelevant	No-Aux
Qwen2.5VL-3B	Causal-VidQA	75.7	72.1	71.9	70.2	72.1	72.1
	ScienceQA	91.5	75.1	75.3	54.5	75.3	73.4
	A-OKVQA	89.6	83.7	83.8	75.9	83.7	83.4
	VCR	79.6	70.6	70.2	59.6	70.1	69.8
Qwen2.5VL-7B	Causal-VidQA	77.1	72.9	73.9	68.0	73.3	72.3
	ScienceQA	95.9	81.0	80.6	51.6	80.8	81.4

Appendix

Table 15. Combined intervention on frozen VLMs. Ladder results reported as accuracy (%) under different auxiliary-text conditions (Faithful, Contradictory, Off-Question, Option-Redirecting, Irrelevant, and No-Aux). We apply the combined inference-time components (DG + NSS) without any training or parameter updates. Each dataset uses 1,000 samples.

Figure 7. Heatmap of percentage accuracy improvement over the base models using only Stage 2 (DG) under different auxiliary-text conditions. Gains are largest under Option-Redirecting textual descriptions, while remaining stable across other auxiliary-text settings. Heatmap of percentage-point accuracy changes from Stage 2, Dynamic Grounding, across models, datasets, and auxiliary-text conditions. The largest gains appear under Option-Redirecting text.

Figure 8. Layer-wise magnitude and cross-layer similarity of Stage 1 (NSS) and Stage 2 (DG) updates. Top: $\ell_{2}$ norms of NSS steering directions (vision/text) and DG adjustment vectors across layers, comparing stage-wise ablations ( Only NSS , Only DG ) with the Combined method. Bottom: cosine-similarity matrices across layers for the visual NSS directions and DG adjustments under the Combined setting. NSS directions remain stable and globally coherent across layers, while DG adjustments exhibit more localized, layer-dependent structure, supporting the role of NSS as a global static prior and DG as an instance-wise grounding correction. Layer-wise analysis of the intervention. The top panels plot the L2 magnitudes of NSS steering directions and DG adjustment vectors across layers. The bottom panels show cross-layer cosine-similarity matrices for the visual NSS directions and DG adjustments.

Figure 9. Input image for a successful correction case under Option-Redirecting auxiliary text. Beach umbrellas viewed from underneath, showing circular thatched canopies supported by wooden poles.

Figure 10. Input image for a failure case under Option-Redirecting auxiliary text, where the redirected distractor remains dominant after applying NSS+DG. An elephant standing beside a keeper in a green coat, with the elephant raising its trunk near another elephant and a crowd in the background.

Figure 11. Input image for a successful correction case under Option-Redirecting auxiliary text. A man in a kitchen drying a metal pot with a striped towel beside a sink.

Figure 12. Input image for a failure case under Option-Redirecting auxiliary text. Three giraffes reaching toward a suspended feeder, with one giraffe extending its neck upward.

Figure 13. Choice-level probability shift for the example in Figure 9 . NSS+DG reduces the redirected distractor and increases the correct answer probability. Grouped bar chart of answer-choice probabilities before and after NSS plus DG for the umbrella example. Probability decreases for the redirected answer nylon and increases for the correct answer thatched.

Figure 14. Choice-level probability shift for the example in Figure 11 . The intervention successfully rebalances probability mass toward the visually grounded answer. Grouped bar chart of answer-choice probabilities before and after NSS plus DG for the kitchen example, showing probability mass shifting toward the visually grounded correct answer.

Figure 15. Choice-level probability shift for the example in Figure 12 . Although NSS+DG changes the score distribution, the redirected distractor remains competitive, resulting in failure. Grouped bar chart of answer-choice probabilities before and after NSS plus DG for the giraffe example. The score distribution changes but the redirected distractor remains competitive.

Figure 16. Choice-level probability shift for the example in Figure 10 . The redirected distractor remains dominant after intervention, illustrating a failure case. Grouped bar chart of answer-choice probabilities before and after NSS plus DG for the elephant example. The redirected distractor remains dominant after the intervention.

Visual Question Answering

Visual Question Answering (VQA) develops systems that answer natural-language questions about images, videos, and visual documents.

See all 253 papers

Beyond OCR Accuracy: Text-Centric VQA Under Image Degradation with Modular and End-to-End
Sep 12, 2026Ritali Vatsi, Rachapudi Jagadeesh, Shruti Singh Baghel +3Visual Question Answering VLM Robustness
Indian Institute of Technology Mandi Mandi, India · DIT University Dehradun, India · Indian Institute of Technology Kharagpur Kharagpur, India
Do Vision-Language Models See or Guess? Measuring and Reducing Textual-Prior Reliance with a Phrasing-Controlled Benchmark
Jun 9, 2026Pratham Singla, Shivank Garg, Vihan Singh +1VLM Evaluation Vision-Language Models
Lossfunk · Indian Institute of Technology Roorkee · Raeth AI
Reasoning Dynamics and the Limits of Monitoring Modality Reliance in Vision-Language Models
Apr 16, 2026Danae Sánchez Villegas, Samuel Lewis-Lim, Nikolaos Aletras +1CoT Faithfulness Vision-Language Models
University of Copenhagen, Department of Computer Science · University of Sheffield, School of Computer Science

Sep 12, 2026J/K move · Enter open · S save

Beyond OCR Accuracy: Text-Centric VQA Under Image Degradation with Modular and End-to-End

Ritali Vatsi, Rachapudi Jagadeesh, Shruti Singh Baghel, Himani Sharma+2

Indian Institute of Technology Mandi Mandi, India · DIT University Dehradun, India · Indian Institute of Technology Kharagpur Kharagpur, India

Visual Question Answering VLM Robustness VLM Evaluation Optical Character Recognition

Open paper AskPDF arXiv

Text-centric Visual Question Answering (VQA) requires reading and reasoning over text embedded in images, a task made substantially harder when images suffer from real-world degradation such as motion blur, low resolution, or compression artifacts. While modular OCR-based pipelines and end-to-end vision-language models are both widely used for this task, their comparative robustness under degraded conditions remains underexplored. We present an empirical study comparing two modular pipelines with SA-DBNet, a custom detector architecture combining ResNet-18 with self-attention spatial modeling and deformable convolutions against an end-to-end vision-language baseline, evaluated on 4013 degraded images with 7000 question-answer pairs. Fine-tuned modular pipelines achieve up to 57.50% exact-match accuracy versus 38.00% for the end-to-end baseline, with domain-specific fine-tuning yielding a gain of up to 29.50 percentage points. Critically, we find that conventional OCR error metrics like Character Error Rate and Word Error Rate are unreliable predictors of downstream VQA performance, as semantic reasoning can compensate for recognition failures when contextual cues are present. These findings highlight the importance of task-aware evaluation for text-centric VQA systems under realistic visual conditions. Codes are available here

Abstract

Visual Question Answering

Explore similar work

Beyond OCR Accuracy: Text-Centric VQA Under Image Degradation with Modular and End-to-End

Do Vision-Language Models See or Guess? Measuring and Reducing Textual-Prior Reliance with a Phrasing-Controlled Benchmark

Reasoning Dynamics and the Limits of Monitoring Modality Reliance in Vision-Language Models