Recent large vision-language models (LVLMs) can generate vision-text multimodal chain-of-thought (MCoT) traces after reinforcement fine-tuning (RFT). However, we observe that the visual information in MCoT is often inaccurate even when answers are correct, and task accuracy remains nearly unchanged despite the visual information being corrupted. This indicates a lack of faithfulness in the vision part of MCoT reasoning. We attribute this to the RL reward design in RFT, which solely incentivizes the format of interleaved vision-text cues, encouraging the model to incorporate visual information into its text reasoning steps without considering its correctness. In this paper, we first probe MCoT faithfulness by measuring how much the prediction changes when its visual and textual thoughts are intervened. Surprisingly, the model's predictions remain nearly unchanged under visual intervention but change significantly under textual intervention, indicating that the visual evidence is largely ignored. To further analyze the visual information, we introduce an automated LVLM-based evaluation metric that quantifies the faithfulness of visual cues from two perspectives: relevance and sufficiency. Our evaluation reveals that the visual information in current MCoT traces is simultaneously irrelevant and insufficient. To address this issue, we propose Sufficient-Component Cause Model (SCCM) learning. This approach encourages the MCoT to generate sufficient yet minimal visual components that are independently capable of leading to the correct answer. The proposed SCCM is annotation-free and plug-and-play, compatible with various RFT for MCoT. Empirical results demonstrate that SCCM substantially improves the visual faithfulness across a suite of fine-grained perception and reasoning benchmarks. Code is available at https://github.com/EugeneLiu01/Faithful_Thinking_with_Image.
Figures & tables
Figure 1: Unfaithfulness problem in “Thinking with Images” diagrams. (a) Visual thinking failure modes, including irrelevant or insufficient visual information, where visual reasoning is misaligned with the final answer. (b) Quantitative analysis on V* Bench with current work [ 58 , 43 ] . We present the accuracy before and after interventions ( i.e. , mislocalized image region) on the Visual information within MCoT, showing that visual information intervention induced negligible change in accuracy.
Figure 2: (a) The illustration of “thinking with images” paradigm. The visual information in MCoT is introduced by zoom-in tool call as observation tokens, and appended to the ongoing reasoning process to guide subsequent steps; (b) The Structural Causal Model (SCM) of MCoT and the answer. The directed arrow between two nodes indicates a causal dependency between them, which can be analyzed via intervention experiments.
Figure 3: Textual vs. visual effect across benchmarks for DeepEyes and Pixel-Reasoner.
Figure 4: The overview of our proposed Sufficient-Component Cause Model (SCCM) learning to establish visual information as sufficient-component causes to correct answers. The SCCM framework requires that: 1) the visual information alone is sufficient to lead to the correct answer, enforced by the Visual Information Sufficiency reward rs ; and 2) the visual information involved is as minimal as possible, guided by the Visual Information Minimality reward rm .
Table 1: Intervention-based analysis on the textual and visual components of generated MCoT across different models on V* Bench and HR-Bench. The significance of the Textual/Visual Effect ( TE/VE ), measured as the difference in mean accuracy, is indicated in red . The corresponding p -values from McNemar’s test are shown in blue , where an asterisk (*) denotes statistical significance with p-value<0.05 . Larger absolute effect values and lower p-values indicate stronger and more statistically reliable influence of the corresponding component ( T or V ) on the predicted answers.
Figure 5: Training dynamics on V* Bench as the test dataset, with different ablation reward schemes. The visual information sufficiency is judged by Qwen2.5-VL-72B, as detailed in Appendix B .
Table 2: Relevance and sufficiency evaluation results of visual components on V* Bench and HR-Bench of different models. Bold and Underlined denote the best and second-best results.
Appendix figures & tables20 assets
Supplementary material from the paper’s appendix.
Appendix
Table A1 : Agreement between GPT-4o judgments and human annotations on the fully annotated V* Bench and sampled HR-Bench 4K/8K data. † denotes sampled data.
Figure A1 : Confusion matrices between GPT-4o judgments and human annotations for Relevance and Sufficiency on the fully annotated V* Bench and sampled ( † ) HR-Bench 4K/8K data.
Table A2 : Human-annotated relevance and sufficiency on the same data as in Table A1 . Bold and Underlined denote the best and second-best results. † denotes sampled data.
Model
Relevance
Sufficiency
HR-Bench 4K †
HR-Bench 8K †
HR-Bench 4K †
HR-Bench 8K †
FSP
FCP
Avg.
FSP
FCP
Avg.
FSP
FCP
Avg.
FSP
FCP
Avg.
DeepEyes
51.00
25.00
38.00
30.30
17.17
23.74
60.00
26.00
43.00
33.33
16.16
24.75
Pixel-Reasoner
37.11
25.61
31.84
29.29
14.46
22.53
47.42
34.15
41.34
40.40
27.71
34.62
Ours
52.53
24.24
38.38
43.96
17.78
30.94
71.72
33.33
52.53
64.84
27.78
46.41
Appendix
Table A3 : GPT-4o-judged relevance and sufficiency on the sampled HR-Bench 4K/8K data. Bold and Underlined denote the best and second-best results. † denotes sampled data.
Table A4 : Relevance and sufficiency on V* Bench under additional judge model, temperature, and prompts. Bold and Underlined denote the best and second-best results.
Figure A2 : A case from V* Bench showing interventions on MCoT generated by Pixel-Reasoner [ 43 ] . Specifically, the introduced mistake for intervention on textual components is underlined in red . The predicted answer is generated after the MCoT (whether intervened or not).
Model
V* Bench
HR-Bench 4K
HR-Bench 8K
Attr.
Spat.
Avg.
FSP
FCP
Avg.
FSP
FCP
Avg.
DeepEyes
90.43
86.84
89.00
86.75
65.75
76.25
85.50
56.00
70.75
Pixel-Reasoner
90.09
83.33
87.43
83.94
66.67
76.06
86.48
58.80
73.72
Ours
93.91
86.84
91.10
85.86
58.84
72.35
89.81
54.44
72.20
Appendix
Table A5 : Base accuracy of the default reasoning process with No Intervention , conducted only on samples with generated MCoT, excluding non-MCoT cases.
Figure A3 : The correctness flip rate under textual and visual intervention across different models and benchmarks, including Correct-to-Incorrect (C2I) and Incorrect-to-Correct (I2C).
Model
V* Bench
HR-Bench 4K
HR-Bench 8K
Attr.
Spat.
Avg.
FSP
FCP
Avg.
FSP
FCP
Avg.
DeepEyes
-1.74
+1.32
-0.52
-0.25
-1.00
-0.63
-0.00
+1.00
+0.50
0.5000
1.0000
1.0000
1.0000
0.3438
0.3323
1.0000
0.4240
0.5235
Pixel-Reasoner
+0.90
-0.00
+0.55
+0.26
-0.00
+0.14
-2.04
+0.90
-0.69
1.0000
1.0000
1.0000
1.0000
1.0000
1.0000
{\text{{\color[rgb]{0,0,1}{0.0386}}}}^{*}
0.6291
0.4583
Ours
-3.48
-2.63
-3.14
-0.51
+2.52
+1.01
-1.38
-1.39
-1.38
Appendix
Table A6 : Visual Effect ( VE ) and p-value from McNemar’s test of only MCoT cases for each model with intervention on visual components V by random noise replacement. An asterisk (*) denotes statistical significance with p-value<0.05 .
Figure A4 : Visual intervention via random noise replacement on correctness flip rates across different models and benchmarks, including Correct-to-Incorrect (C2I) and Incorrect-to-Correct (I2C).
Figure 18
Table A8 : Relevance and sufficiency evaluation results of visual components on V* Bench and HR-Bench of the ablation models under different reward schemes: (1) Naive, consisting only of accuracy and format rewards; (2) Curiosity, the curiosity-driven reward scheme proposed in [ 43 ] ; (3) SCCM, our proposed SCCM scheme with visual information sufficiency and minimality constraint; (4) SCCM w/o Minimality, an ablation variant of SCCM without the minimality constraint. We note that SCCM w/o Minimality (denoted *), is not comparable, as its cropped image is excessively large, being the same size as the original input image.
Reward Scheme
V* Bench
HR-Bench 4K
HR-Bench 8K
CRZ
TCC
CRZ
TCC
CRZ
TCC
Naive
0.1490
1.4345
0.2176
1.4587
0.1680
1.1862
Curiosity
0.0835
0.9895
0.1977
0.9850
0.2144
0.9750
SCCM
0.0429
1.0000
0.1429
0.9900
0.1273
0.9037
w/o * Minimality
1.9916
2.0000
1.9983
2.0000
1.9682
2.0025
Appendix
Table A9 : The Cropped Region Size (CRZ) and Tool Call Count (TCC) on V* Bench and HR-Bench of the ablation models under different reward schemes.
Table A10 : Relevance and sufficiency evaluation results of visual components on V* Bench and HR-Bench for models trained via RFT: (1) on the training dataset of Zheng et al. [58] ( i.e. , Ours ), and (2) on the training dataset of Su et al. [43] ( i.e. , Ablation ).
RFT Dataset
V* Bench
HR-Bench 4K
HR-Bench 8K
CRZ
TCC
CRZ
TCC
CRZ
TCC
Zheng et al. [58] Dataset ( Ours )
0.0429
1.0000
0.1429
0.9900
0.1273
0.9037
Su et al. [43] Dataset ( Ablation )
0.0442
0.9895
0.1584
0.9900
0.1553
0.9950
Appendix
Table A11 : The Cropped Region Size (CRZ), i.e. , the total aspect ratio of cropped regions relative to the original image, and Tool Call Count (TCC), i.e. , the average number of zoom-in tool calls, on V* Bench and HR-Bench of the ablation models under different RFT training datasets.
Model
V* Bench
HR-Bench 4K
HR-Bench 8K
Attr.
Spat.
Avg.
FSP
FCP
Avg.
FSP
FCP
Avg.
SEAL
73.04
75.00
73.82
40.00
28.00
34.00
42.00
31.00
36.50
DeepEyes
90.43
86.84
89.00
86.75
65.75
76.25
85.50
56.00
70.75
Pixel-Reasoner
88.69
81.58
85.86
83.50
60.00
71.75
86.25
53.50
69.87
Ours
93.91
86.84
91.10
86.00
59.00
72.50
86.50
56.00
71.25
Appendix
Table A12 : The accuracy of different models on V* Bench and HR-Bench, covering all samples. Bold and Underlined denote the best and second-best results.
Model
V* Bench
HR-Bench 4K
HR-Bench 8K
CRZ
TCC
CRZ
TCC
CRZ
TCC
DeepEyes
0.0074
1.0000
0.0371
1.0250
0.0256
1.0000
Pixel-Reasoner
0.0988
0.9581
0.1076
0.8875
0.0928
0.9087
Ours
0.0429
1.0000
0.1429
0.9900
0.1273
0.9037
Appendix
Table A13 : The Cropped Region Size (CRZ) and Tool Call Count (TCC) on V* Bench and HR-Bench of DeepEyes [ 58 ] , Pixel-Reasoner [ 43 ] and our model from SCCM-based RFT.
Figure A6 : Relationship between MCoT faithfulness metrics (relevance and sufficiency) and task accuracy under different models. Each point denotes one model-task pair, with accuracy computed only on MCoT cases. Overall, no consistent monotonic relationship is observed.
Figure A7 : Cases in V* Bench with MCoT responses generated by DeepEyes, Pixel-Reasoner and our SCCM-based RFT model.
Figure A8 : Cases in V* Bench with MCoT responses generated by models in different reward schemes: (1) Curiosity Reward, the curiosity-driven reward scheme proposed in [ 43 ] ; (2) Naive Reward, consisting only of accuracy and format rewards; (3) SCCM w/o Minimality, an ablation variant of SCCM without the minimality constraint; (4) SCCM Reward, our proposed SCCM scheme with visual information sufficiency and minimality constraint.
The effectiveness of Chain-of-Thought (CoT) prompting in Multimodal Large Language Models (MLLMs) remains uncertain: across several visual reasoning benchmarks, CoT prompting often degrades performance compared to direct prompting. In this paper, we provide a systematic analysis of CoT behavior in three modern MLLM families across model scales on datasets requiring step-wise visual evidence. Our analysis identifies two recurring failure modes: premature answer commitment and limited direct visual-token access during rationale generation. We further find that standard CoT-style Supervised Fine-Tuning (CoT-SFT) can mitigate these issues only partially, while often increasing reliance on textual priors and reducing counterfactual visual dependence. Motivated by these findings, we propose Attentive-CoT (Att-CoT), an attention-guided fine-tuning objective that encourages CoT trajectories to delay answer commitment while maintaining sustained visual-token access. Att-CoT can be plugged into any CoT-SFT training run without architectural changes. Experiments on three visual reasoning benchmarks across six MLLMs show that Att-CoT enhances CoT performance over standard fine-tuning.
Chain-of-thought (CoT) may often look plausible, yet it may not faithfully reflect the model's decision-making process. While methods for measuring the faithfulness of CoTs for textual inputs have been increasingly introduced, using these methods for visual inputs is not straightforward. In this work, we adapt the family of counterfactual methods for measuring CoT faithfulness, namely the Counterfactual Test (CT) and Correlational Counterfactual Test (CCT), to visual inputs, and call them vCT and vCCT, respectively. Using vCT and vCCT, we benchmark eight recent open-source Vision Language Models (VLMs) on two datasets. Our analysis shows that CoTs do not reliably track visual evidence that influences model predictions: they may omit the removed object even when its removal causes a large prediction shift, yet mention it when the shift is small. We further find that Predict-then-Explain explanations align more strongly with perturbation-induced probability shifts than pre-answer CoTs, while binary vCT scores are often nearly saturated. We also include a reconstruction control, in which images pass through the same editing pipeline without object removal, and find that the main object-removal intervention induces larger shifts than reconstruction alone. We construct and release Counter-SNLI-VE and Counter-A-OKVQA, two datasets of image pairs that differ by a single object.
Bayar Menzat, Maximilian Süss, Ruizhi Wang +3
Vienna University of Technology, Austria · University of Oxford, UK · Imperial College London, UK +1
Reinforcement learning with verifiable rewards (RLVR) has emerged as a promising paradigm for advancing complex reasoning in large language models, and recent work extends RLVR to multimodal large language models (MLLMs). This transfer, however, surfaces a faithfulness challenge: faithful perception of task-relevant visual evidence and faithful use of that evidence during reasoning, leading to unsatisfactory gains on multimodal benchmarks. Specifically, existing perception supervision often operates on textual descriptions rather than natively on image regions, and faithful use is largely overlooked, exposing the perception-reasoning disconnect where correctly perceived evidence is dropped or contradicted during reasoning. To close these gaps, we propose Faithful-MR1, a training framework that anchors and reinforces visual attention to address both halves of faithful multimodal reasoning. The Anchoring stage turns perception into an explicit pre-reasoning subtask, supervising a dedicated <Focus> token's attention directly against image regions rather than through textual descriptions. The Reinforcing stage exposes faithful use through counterfactual image intervention, rewarding answer-correct trajectories that concentrate visual attention where vision causally matters. Extensive experiments demonstrate that Faithful-MR1 outperforms recent multimodal reasoning baselines on both Qwen2.5-VL-Instruct 3B and 7B backbones while using substantially less training data.
Changyuan Tian, Zhicong Lu, Huaxing Liu +7
AMAP, Alibaba Group · University of Chinese Academy of Sciences · Tsinghua University +1