On the Faithfulness of Visual Thinking: Measurement and Enhancement

Zujing Liu; Junwen Pan; Qi She; Yuan Gao; Guisong Xia

cs.CVOct 27, 2025

On the Faithfulness of Visual Thinking: Measurement and Enhancement

Authors: Zujing Liu, Junwen Pan, Qi She, Yuan Gao, Guisong Xia

Organizations: Wuhan University · ByteDance

Abstract

Recent large vision-language models (LVLMs) can generate vision-text multimodal chain-of-thought (MCoT) traces after reinforcement fine-tuning (RFT). However, we observe that the visual information in MCoT is often inaccurate even when answers are correct, and task accuracy remains nearly unchanged despite the visual information being corrupted. This indicates a lack of faithfulness in the vision part of MCoT reasoning. We attribute this to the RL reward design in RFT, which solely incentivizes the format of interleaved vision-text cues, encouraging the model to incorporate visual information into its text reasoning steps without considering its correctness. In this paper, we first probe MCoT faithfulness by measuring how much the prediction changes when its visual and textual thoughts are intervened. Surprisingly, the model's predictions remain nearly unchanged under visual intervention but change significantly under textual intervention, indicating that the visual evidence is largely ignored. To further analyze the visual information, we introduce an automated LVLM-based evaluation metric that quantifies the faithfulness of visual cues from two perspectives: relevance and sufficiency. Our evaluation reveals that the visual information in current MCoT traces is simultaneously irrelevant and insufficient. To address this issue, we propose Sufficient-Component Cause Model (SCCM) learning. This approach encourages the MCoT to generate sufficient yet minimal visual components that are independently capable of leading to the correct answer. The proposed SCCM is annotation-free and plug-and-play, compatible with various RFT for MCoT. Empirical results demonstrate that SCCM substantially improves the visual faithfulness across a suite of fine-grained perception and reasoning benchmarks. Code is available at https://github.com/EugeneLiu01/Faithful_Thinking_with_Image.

Figures & tables

Panel of Figure 1 — Figure 1: Unfaithfulness problem in “Thinking with Images” diagrams. (a) Visual thinking failure modes, including irrelevant or insufficient visual information, where visual reasoning is misaligned with the final answer. (b) Quantitative analysis on V* Bench with current work [ 58 , 43 ] . We present the accuracy before and after interventions ( i.e. , mislocalized image region) on the Visual information within MCoT, showing that visual information intervention induced negligible change in accuracy.

Panel of Figure 2 — Figure 2: (a) The illustration of “thinking with images” paradigm. The visual information in MCoT is introduced by zoom-in tool call as observation tokens, and appended to the ongoing reasoning process to guide subsequent steps; (b) The Structural Causal Model (SCM) of MCoT and the answer. The directed arrow between two nodes indicates a causal dependency between them, which can be analyzed via intervention experiments.

Panel of Figure 3 — Figure 3: Textual vs. visual effect across benchmarks for DeepEyes and Pixel-Reasoner.

Figure 4: The overview of our proposed Sufficient-Component Cause Model (SCCM) learning to establish visual information as sufficient-component causes to correct answers. The SCCM framework requires that: 1) the visual information alone is sufficient to lead to the correct answer, enforced by the Visual Information Sufficiency reward $r_{s}$ ; and 2) the visual information involved is as minimal as possible, guided by the Visual Information Minimality reward $r_{m}$ .

Table 1: Intervention-based analysis on the textual and visual components of generated MCoT across different models on V* Bench and HR-Bench. The significance of the Textual/Visual Effect ( TE/VE ), measured as the difference in mean accuracy, is indicated in red . The corresponding

p

-values from McNemar’s test are shown in blue , where an asterisk (*) denotes statistical significance with

p\text{-value}<0.05

. Larger absolute effect values and lower

p\text{-values}

indicate stronger and more statistically reliable influence of the corresponding component (

\mathbf{T}

\mathbf{V}

) on the predicted answers.

Panel of Figure 5 — Figure 5: Training dynamics on V* Bench as the test dataset, with different ablation reward schemes. The visual information sufficiency is judged by Qwen2.5-VL-72B, as detailed in Appendix B .

Table 2: Relevance and sufficiency evaluation results of visual components on V* Bench and HR-Bench of different models. Bold and Underlined denote the best and second-best results.

Appendix figures & tables20 assets

Supplementary material from the paper’s appendix.

Appendix

Table A1 : Agreement between GPT-4o judgments and human annotations on the fully annotated V* Bench and sampled HR-Bench 4K/8K data.

\dagger

denotes sampled data.

Panel of Figure 9 — Figure A1 : Confusion matrices between GPT-4o judgments and human annotations for Relevance and Sufficiency on the fully annotated V* Bench and sampled ( $\dagger$ ) HR-Bench 4K/8K data.

Table A2 : Human-annotated relevance and sufficiency on the same data as in Table A1 . Bold and Underlined denote the best and second-best results.

\dagger

denotes sampled data.

Model	Relevance						Sufficiency
	HR-Bench 4K †			HR-Bench 8K †			HR-Bench 4K †			HR-Bench 8K †
	FSP	FCP	Avg.	FSP	FCP	Avg.	FSP	FCP	Avg.	FSP	FCP	Avg.
DeepEyes	51.00	25.00	38.00	30.30	17.17	23.74	60.00	26.00	43.00	33.33	16.16	24.75
Pixel-Reasoner	37.11	25.61	31.84	29.29	14.46	22.53	47.42	34.15	41.34	40.40	27.71	34.62
Ours	52.53	24.24	38.38	43.96	17.78	30.94	71.72	33.33	52.53	64.84	27.78	46.41

Appendix

Table A3 : GPT-4o-judged relevance and sufficiency on the sampled HR-Bench 4K/8K data. Bold and Underlined denote the best and second-best results.

\dagger

denotes sampled data.

Table A4 : Relevance and sufficiency on V* Bench under additional judge model, temperature, and prompts. Bold and Underlined denote the best and second-best results.

Figure A2 : A case from V* Bench showing interventions on MCoT generated by Pixel-Reasoner [ 43 ] . Specifically, the introduced mistake for intervention on textual components is underlined in red . The predicted answer is generated after the MCoT (whether intervened or not).

Model	V* Bench			HR-Bench 4K			HR-Bench 8K
Model	Attr.	Spat.	Avg.	FSP	FCP	Avg.	FSP	FCP	Avg.
DeepEyes	90.43	86.84	89.00	86.75	65.75	76.25	85.50	56.00	70.75
Pixel-Reasoner	90.09	83.33	87.43	83.94	66.67	76.06	86.48	58.80	73.72
Ours	93.91	86.84	91.10	85.86	58.84	72.35	89.81	54.44	72.20

Appendix

Table A5 : Base accuracy of the default reasoning process with No Intervention , conducted only on samples with generated MCoT, excluding non-MCoT cases.

Panel of Figure 15 — Figure A3 : The correctness flip rate under textual and visual intervention across different models and benchmarks, including Correct-to-Incorrect (C2I) and Incorrect-to-Correct (I2C).

Model	V* Bench			HR-Bench 4K			HR-Bench 8K
	Attr.	Spat.	Avg.	FSP	FCP	Avg.	FSP	FCP	Avg.
DeepEyes	-1.74	+1.32	-0.52	-0.25	-1.00	-0.63	-0.00	+1.00	+0.50
	0.5000	1.0000	1.0000	1.0000	0.3438	0.3323	1.0000	0.4240	0.5235
Pixel-Reasoner	+0.90	-0.00	+0.55	+0.26	-0.00	+0.14	-2.04	+0.90	-0.69
	1.0000	1.0000	1.0000	1.0000	1.0000	1.0000	${\text{{\color[rgb]{0,0,1}{0.0386}}}}^{*}$	0.6291	0.4583
Ours	-3.48	-2.63	-3.14	-0.51	+2.52	+1.01	-1.38	-1.39	-1.38

Appendix

Table A6 : Visual Effect ( VE ) and

p\text{-value}

from McNemar’s test of only MCoT cases for each model with intervention on visual components

\mathbf{V}

by random noise replacement. An asterisk (*) denotes statistical significance with

p\text{-value}<0.05

Figure A4 : Visual intervention via random noise replacement on correctness flip rates across different models and benchmarks, including Correct-to-Incorrect (C2I) and Incorrect-to-Correct (I2C).

Table A8 : Relevance and sufficiency evaluation results of visual components on V* Bench and HR-Bench of the ablation models under different reward schemes: (1) Naive, consisting only of accuracy and format rewards; (2) Curiosity, the curiosity-driven reward scheme proposed in [ 43 ] ; (3) SCCM, our proposed SCCM scheme with visual information sufficiency and minimality constraint; (4) SCCM w/o Minimality, an ablation variant of SCCM without the minimality constraint. We note that SCCM w/o Minimality (denoted *), is not comparable, as its cropped image is excessively large, being the same size as the original input image.

Reward Scheme	V* Bench		HR-Bench 4K		HR-Bench 8K
Reward Scheme	CRZ	TCC	CRZ	TCC	CRZ	TCC
Naive	0.1490	1.4345	0.2176	1.4587	0.1680	1.1862
Curiosity	0.0835	0.9895	0.1977	0.9850	0.2144	0.9750
SCCM	0.0429	1.0000	0.1429	0.9900	0.1273	0.9037
w/o * Minimality	1.9916	2.0000	1.9983	2.0000	1.9682	2.0025

Appendix

Table A9 : The Cropped Region Size (CRZ) and Tool Call Count (TCC) on V* Bench and HR-Bench of the ablation models under different reward schemes.

Table A10 : Relevance and sufficiency evaluation results of visual components on V* Bench and HR-Bench for models trained via RFT: (1) on the training dataset of Zheng et al. [58] ( i.e. , Ours ), and (2) on the training dataset of Su et al. [43] ( i.e. , Ablation ).

RFT Dataset	V* Bench		HR-Bench 4K		HR-Bench 8K
RFT Dataset	CRZ	TCC	CRZ	TCC	CRZ	TCC
Zheng et al. [58] Dataset ( Ours )	0.0429	1.0000	0.1429	0.9900	0.1273	0.9037
Su et al. [43] Dataset ( Ablation )	0.0442	0.9895	0.1584	0.9900	0.1553	0.9950

Appendix

Table A11 : The Cropped Region Size (CRZ), i.e. , the total aspect ratio of cropped regions relative to the original image, and Tool Call Count (TCC), i.e. , the average number of zoom-in tool calls, on V* Bench and HR-Bench of the ablation models under different RFT training datasets.

Model	V* Bench			HR-Bench 4K			HR-Bench 8K
Model	Attr.	Spat.	Avg.	FSP	FCP	Avg.	FSP	FCP	Avg.
SEAL	73.04	75.00	73.82	40.00	28.00	34.00	42.00	31.00	36.50
DeepEyes	90.43	86.84	89.00	86.75	65.75	76.25	85.50	56.00	70.75
Pixel-Reasoner	88.69	81.58	85.86	83.50	60.00	71.75	86.25	53.50	69.87
Ours	93.91	86.84	91.10	86.00	59.00	72.50	86.50	56.00	71.25

Appendix

Table A12 : The accuracy of different models on V* Bench and HR-Bench, covering all samples. Bold and Underlined denote the best and second-best results.

Model	V* Bench		HR-Bench 4K		HR-Bench 8K
Model	CRZ	TCC	CRZ	TCC	CRZ	TCC
DeepEyes	0.0074	1.0000	0.0371	1.0250	0.0256	1.0000
Pixel-Reasoner	0.0988	0.9581	0.1076	0.8875	0.0928	0.9087
Ours	0.0429	1.0000	0.1429	0.9900	0.1273	0.9037

Appendix

Table A13 : The Cropped Region Size (CRZ) and Tool Call Count (TCC) on V* Bench and HR-Bench of DeepEyes [ 58 ] , Pixel-Reasoner [ 43 ] and our model from SCCM-based RFT.

Panel of Figure 25 — Figure A6 : Relationship between MCoT faithfulness metrics (relevance and sufficiency) and task accuracy under different models. Each point denotes one model-task pair, with accuracy computed only on MCoT cases. Overall, no consistent monotonic relationship is observed.

Panel of Figure 26 — Figure A7 : Cases in V* Bench with MCoT responses generated by DeepEyes, Pixel-Reasoner and our SCCM-based RFT model.

Panel of Figure 27 — Figure A8 : Cases in V* Bench with MCoT responses generated by models in different reward schemes: (1) Curiosity Reward, the curiosity-driven reward scheme proposed in [ 43 ] ; (2) Naive Reward, consisting only of accuracy and format rewards; (3) SCCM w/o Minimality, an ablation variant of SCCM without the minimality constraint; (4) SCCM Reward, our proposed SCCM scheme with visual information sufficiency and minimality constraint.

Faithfulness in VLMs

Faithfulness in vision-language models (VLMs) concerns whether their answers and reasoning genuinely rely on visual evidence.

See all 87 papers

Attention-guided Fine-tuning of Multimodal Large Language Models Improves Chain-of-Thought Reasoning
Jun 1, 2026Sanchit Sinha, Guangzhi Xiong, Bohan Liu +2Multimodal CoT Reasoning Multimodal Large Language Models
University of Virginia
Counterfactual Tests for Measuring Chain-of-Thought Faithfulness in Visual Language Models
Sep 6, 2026Bayar Menzat, Maximilian Süss, Ruizhi Wang +3CoT Faithfulness Faithfulness in VLMs
Vienna University of Technology, Austria · University of Oxford, UK · Imperial College London, UK +1
Faithful-MR1: Faithful Multimodal Reasoning via Anchoring and Reinforcing Visual Attention
May 21, 2026Changyuan Tian, Zhicong Lu, Huaxing Liu +7Multimodal Large Language Models Multimodal Reasoning
AMAP, Alibaba Group · University of Chinese Academy of Sciences · Tsinghua University +1

Jun 1, 2026J/K move · Enter open · S save

Attention-guided Fine-tuning of Multimodal Large Language Models Improves Chain-of-Thought Reasoning

Sanchit Sinha, Guangzhi Xiong, Bohan Liu, Zhenghao He+1

University of Virginia

Multimodal CoT Reasoning Multimodal Large Language Models VLM Reasoning Fine-Tuning Visually Grounded Reasoning Multimodal Understanding+1

Open paper AskPDF arXiv

The effectiveness of Chain-of-Thought (CoT) prompting in Multimodal Large Language Models (MLLMs) remains uncertain: across several visual reasoning benchmarks, CoT prompting often degrades performance compared to direct prompting. In this paper, we provide a systematic analysis of CoT behavior in three modern MLLM families across model scales on datasets requiring step-wise visual evidence. Our analysis identifies two recurring failure modes: premature answer commitment and limited direct visual-token access during rationale generation. We further find that standard CoT-style Supervised Fine-Tuning (CoT-SFT) can mitigate these issues only partially, while often increasing reliance on textual priors and reducing counterfactual visual dependence. Motivated by these findings, we propose Attentive-CoT (Att-CoT), an attention-guided fine-tuning objective that encourages CoT trajectories to delay answer commitment while maintaining sustained visual-token access. Att-CoT can be plugged into any CoT-SFT training run without architectural changes. Experiments on three visual reasoning benchmarks across six MLLMs show that Att-CoT enhances CoT performance over standard fine-tuning.

Abstract

Faithfulness in VLMs

Explore similar work

Attention-guided Fine-tuning of Multimodal Large Language Models Improves Chain-of-Thought Reasoning

Counterfactual Tests for Measuring Chain-of-Thought Faithfulness in Visual Language Models

Faithful-MR1: Faithful Multimodal Reasoning via Anchoring and Reinforcing Visual Attention