The Cost of Reasoning: Chain-of-Thought Induces Overconfidence in Vision-Language Models

Robert Welch; Emir Konuk; Kevin Smith

cs.LGMar 17, 2026

The Cost of Reasoning: Chain-of-Thought Induces Overconfidence in Vision-Language Models

Authors: Robert Welch, Emir Konuk, Kevin Smith

Organizations: KTH Royal Institute of Technology, Stockholm, Sweden · Science for Life Laboratory, Stockholm, Sweden

Abstract

Vision-language models (VLMs) are increasingly deployed in high-stakes settings where reliable uncertainty quantification (UQ) is as important as predictive accuracy. Extended reasoning via chain-of-thought (CoT) prompting or reasoning-trained models has become ubiquitous in modern VLM pipelines, yet its effect on UQ reliability remains poorly understood. Our results show that reasoning tends to degrade the quality of many uncertainty estimates, even when it improves task accuracy. We identify implicit answer conditioning as the primary mechanism: as reasoning traces converge on a conclusion before the final answer is generated, token probabilities increasingly reflect consistency with the model's own reasoning trace rather than uncertainty about correctness. In effect, the model becomes overconfident in its answer. In contrast, agreement-based consistency remains robust and often improves under reasoning, making it a practical choice for uncertainty estimation in reasoning-enabled VLMs.

Figures & tables

Figure 1: Example of implicit answer conditioning: intermediate reasoning progressively commits the model to its eventual answer before that answer is explicitly generated. Colour indicates the model’s confidence in its final predicted answer $\hat{a}$ on the same input, without reasoning ( $P(\hat{a}\mid x,\theta)$ ) and at each reasoning token position $t$ with chain-of-thought reasoning ( $P(\hat{a}\mid x,r_{\leq t},\theta)$ ), where $x$ is the input and $r_{\leq t}$ is the reasoning trace up to token $t$ . Although the prediction is incorrect in both cases (3 inches; ground truth: 2), confidence rises sharply as the reasoning trace converges on the wrong answer, exceeding the no-CoT confidence.

		No CoT			CoT			$\Delta$ (CoT $-$ No CoT)
Type	UQ Estimate	AUGRC $\downarrow$	PRR $\uparrow$	Spear. $\uparrow$	AUGRC $\downarrow$	PRR $\uparrow$	Spear. $\uparrow$	AUGRC	PRR	Spear.
Oxford-IIIT Pet
Single-Generation (Acc: 79.3 / 76.2)	MSP	0.056	0.710	0.404	0.110	0.231	0.075	0.053	-0.478	-0.329
	PPL	0.057	0.711	0.399	0.111	0.211	0.061	0.054	-0.500	-0.338
	MTE	0.053	0.735	0.431	0.112	0.205	0.056	0.059	-0.530	-0.376
	SRC	0.060	0.709	0.377	0.100	0.314	0.151	0.041	-0.394	-0.226

Table 1: Selective generation results for Qwen3-VL-8B-IT with and without CoT prompting. Accuracy is reported as (Acc: No CoT / CoT ) for each setting.

		Instruct			Thinking			$\Delta$ (Thinking $-$ Instruct)
Type	UQ Estimate	AUGRC $\downarrow$	PRR $\uparrow$	Spear. $\uparrow$	AUGRC $\downarrow$	PRR $\uparrow$	Spear. $\uparrow$	AUGRC	PRR	Spear.
Oxford-IIIT Pet
Single-Generation (Acc: 79.3 / 77.2)	MSP	0.056	0.710	0.404	0.110	0.120	0.032	0.054	-0.590	-0.373
	PPL	0.057	0.711	0.399	0.112	0.089	0.021	0.055	-0.621	-0.378
	MTE	0.053	0.735	0.431	0.112	0.086	0.019	0.059	-0.649	-0.412
	SRC	0.060	0.709	0.377	0.075	0.538	0.320	0.016	-0.171	-0.057

Table 2: Selective generation results for Qwen3-VL-8B-IT (Instruct; no CoT) and Qwen3-VL-8B-Thinking (intrinsic reasoning, no CoT prompting). Accuracy is reported as (Acc: Instruct / Thinking) for each setting.

Figure 2: Confidence shifts induced by CoT reasoning. Bars show the percentage of samples whose confidence increases (solid colour), decreases (striped colour), or remains unchanged (gray) when moving from no-CoT to CoT inference, grouped by correctness. ATL-based estimates systematically increase confidence, including samples that become or remain incorrect, indicating confidence inflation unrelated to correctness. Self-reported confidence (SRC) and Consistency do not exhibit systematic inflation.

Panel of Figure 3 — Figure 3: Evidence for Implicit Answer Conditioning. (a) Aggregated spearman correlations between reasoning length and confidence under CoT (Fisher z-transformed; 95% CI). Longer reasoning traces correspond to lower confidence. (b) Confidence as a function of how often the final answer appears in the reasoning trace. Repeated answer mentions substantially increase answer-token likelihood. (c–d) Across datasets, answer frequency positively correlates with ATL-based uncertainty estimates. This correlation persists even for incorrect predictions, indicating correctness-agnostic confidence inflation. In contrast, Consistency and SRC show little sensitivity to answer frequency.

	No CoT (Acc 59.7)			CoT (Acc 57.7)			Masked-CoT (Acc 57.7)			Random Masked-CoT (Acc 57.7)
UQ Method	AUGRC $\downarrow$	Spear. $\uparrow$	PRR $\uparrow$	AUGRC $\downarrow$	Spear. $\uparrow$	PRR $\uparrow$	AUGRC $\downarrow$	Spear. $\uparrow$	PRR $\uparrow$	AUGRC $\downarrow$	Spear. $\uparrow$	PRR $\uparrow$
MSP	0.152	0.375	0.498	0.196	0.115	0.141	0.161	0.378	0.475	0.190	0.161	0.210
MTE	0.154	0.364	0.481	0.204	0.057	0.071	0.181	0.230	0.329	0.198	0.102	0.137
PPL	0.156	0.345	0.469	0.204	0.059	0.075	0.166	0.340	0.436	0.197	0.113	0.146

Table 3: Masking answer mentions in CoT improves uncertainty. Masked-CoT replaces occurrences of the predicted answer in the reasoning trace with [MASK] , while Random Masked-CoT masks an equal number of random reasoning tokens. Masking answer mentions substantially restores the ranking quality of ATL-based uncertainty estimates. Best values between CoT, Masked-CoT, and Random Masked-CoT are bold. Qwen3-VL-8B-IT on OK-VQA shown here, results for MathVista are given in the supp. material (Sec. L).

Appendix figures & tables13 assets

Supplementary material from the paper’s appendix.

Appendix

		Non-Reasoning			Reasoning
Model	Dataset	MSP	SRC	Cons.	MSP	SRC	Cons.
Gemma3-4B-IT	Oxford-IIIT Pet	0.301	0.383	0.313	0.161	0.287	0.161
	OK-VQA	0.349	0.416	0.450	0.410	0.557	0.352
	MMMU-Pro-Vision	0.581	0.497	0.752	0.682	0.366	0.542
	MathVista	0.397	0.585	0.521	0.513	0.440	0.268
Qwen3-VL-8B-IT	Oxford-IIIT Pet	0.061	0.198	0.143	0.170	0.260	0.103

Appendix

Table 4: Expected Calibration Error (ECE, 10 bins) across models and datasets. Lower is better; best per setting in bold .

Model	Temperature	Top- $p$	Top- $k$	Presence Penalty
Gemma3-4B-IT	1.0	0.95	64	1.0
InternVL3-9B	0.8	0.80	40	0.0
Qwen3-VL-8B-IT	1.0	1.0	40	2.0
Qwen3-VL-8B-Thinking	1.0	0.95	20	1.5
Qwen3-VL-32B-IT	0.7	0.80	20	1.5

Appendix

Table 5: Decoding hyperparameters used for each model family.

Panel of Figure 4 — Figure 4: Effect of sample size $k$ on uncertainty quality. Lower AUGRC is better, higher PRR is better.

		No CoT			CoT			$\Delta$ (CoT $-$ No CoT)
Type	UQ Estimate	AUGRC $\downarrow$	PRR $\uparrow$	Spear. $\uparrow$	AUGRC $\downarrow$	PRR $\uparrow$	Spear. $\uparrow$	AUGRC	PRR	Spear.
Gemma3-4B-IT
Single-Generation (Acc: 65.2 / 70.4)	MSP	0.122	0.498	0.380	0.139	0.200	0.068	0.018	-0.297	-0.312
	PPL	0.121	0.498	0.384	0.141	0.186	0.054	0.020	-0.312	-0.330
	MTE	0.118	0.521	0.404	0.136	0.219	0.090	0.018	-0.302	-0.314
	SRC	0.121	0.511	0.388	0.104	0.504	0.333	-0.016	-0.007	-0.055

Appendix

Table 6: Selective Generation Results on Oxford Pets

		No CoT			CoT			$\Delta$ (CoT $-$ No CoT)
Type	UQ Estimate	AUGRC $\downarrow$	PRR $\uparrow$	Spear. $\uparrow$	AUGRC $\downarrow$	PRR $\uparrow$	Spear. $\uparrow$	AUGRC	PRR	Spear.
Gemma3-4B-IT
Single-Generation (Acc: 38.8 / 35.4)	MSP	0.270	0.333	0.266	0.294	0.244	0.218	0.024	-0.090	-0.048
	PPL	0.278	0.291	0.207	0.305	0.188	0.135	0.027	-0.103	-0.072
	MTE	0.276	0.297	0.222	0.303	0.192	0.152	0.027	-0.105	-0.071
	SRC	0.270	0.328	0.268	0.289	0.304	0.261	0.018	-0.025	-0.007

Appendix

Table 7: Selective Generation Results on OK-VQA

		No CoT			CoT			$\Delta$ (CoT $-$ No CoT)
Type	UQ Estimate	AUGRC $\downarrow$	PRR $\uparrow$	Spear. $\uparrow$	AUGRC $\downarrow$	PRR $\uparrow$	Spear. $\uparrow$	AUGRC	PRR	Spear.
Gemma3-4B-IT
Single-Generation (Acc: 19.0 / 18.9)	MSP	0.401	0.118	0.037	0.398	0.036	0.069	-0.003	-0.082	0.032
	PPL	0.401	0.116	0.035	0.398	0.036	0.069	-0.003	-0.081	0.034
	MTE	0.398	0.129	0.059	0.397	0.039	0.075	-0.001	-0.090	0.016
	SRC	0.400	0.077	0.047	0.394	0.122	0.108	-0.006	0.045	0.060

Appendix

Table 8: Selective Generation Results on MMMU-Pro-Vision

		No CoT			CoT			$\Delta$ (CoT $-$ No CoT)
Type	UQ Estimate	AUGRC $\downarrow$	PRR $\uparrow$	Spear. $\uparrow$	AUGRC $\downarrow$	PRR $\uparrow$	Spear. $\uparrow$	AUGRC	PRR	Spear.
Gemma3-4B-IT
Single-Generation (Acc: 34.7 / 41.9)	MSP	0.292	0.317	0.256	0.269	0.202	0.150	-0.022	-0.115	-0.106
	PPL	0.300	0.239	0.192	0.275	0.152	0.110	-0.025	-0.087	-0.082
	MTE	0.302	0.229	0.182	0.276	0.152	0.107	-0.026	-0.077	-0.074
	SRC	0.340	0.022	-0.094	0.289	0.083	0.012	-0.051	0.060	0.106

Appendix

Table 9: Selective Generation Results on MathVista

Panel of Figure 5 — Figure 5: Sample-level confidence shifts under Chain-of-Thought prompting for Qwen3-VL-32B-Instruct across different datasets. Each subfigure corresponds to a dataset and reports confidence changes conditioned on correctness transitions.

		Answer Tokens	Answer Tokens	Reasoning Tokens
Model	Dataset	No Reasoning	Reasoning	Reasoning
Qwen3-VL-32B-IT	MMMU Pro Vision	1.0 $\pm$ 0.0	1.0 $\pm$ 0.0	521.9 $\pm$ 1742.1
	MathVista	1.8 $\pm$ 1.8	1.4 $\pm$ 1.1	206.8 $\pm$ 842.9
	OK-VQA	2.0 $\pm$ 1.0	2.2 $\pm$ 1.2	47.8 $\pm$ 24.7
	Oxford Pets	1.7 $\pm$ 0.4	1.7 $\pm$ 0.4	60.9 $\pm$ 11.3
Qwen3-VL-8B-IT	MMMU Pro Vision	1.0 $\pm$ 0.0	1.0 $\pm$ 0.0	186.0 $\pm$ 439.2

Appendix

Table 10: Average answer and reasoning lengths (number of tokens; mean

\pm

std).

Figure 6: Confidence shift in the final predicted answer’s MSP as a function of reasoning progress. Curves are split by whether the final answer was correct or incorrect. Solid lines show the mean, and shaded regions show the 95% confidence interval estimated via bootstrapping.

Panel of Figure 7 — Figure 7: Reasoning Trace Analysis on Incorrect Samples. On OK-VQA, reasoning traces that mention the predicted answer more frequently tend to yield higher final-answer MSP, even when the prediction is incorrect.

Figure 8: Sequential Confidence Trajectory. The log-sum-probability of the answer tokens across multiple rounds of reasoning and answering. Qwen3-VL-8B-IT on a random subset (n=200) of MathVista.

	No-CoT (Acc 62.9)			CoT (Acc 72.9)			Masked-CoT (Acc 72.9)			Random Mask (Acc 72.9)
UQ Method	AUGRC $\downarrow$	Spear. $\uparrow$	PRR $\uparrow$	AUGRC $\downarrow$	Spear. $\uparrow$	PRR $\uparrow$	AUGRC $\downarrow$	Spear. $\uparrow$	PRR $\uparrow$	AUGRC $\downarrow$	Spear. $\uparrow$	PRR $\uparrow$
MSP	0.142	0.313	0.522	0.130	0.043	0.115	0.129	0.055	0.200	0.147	-0.083	-0.032
MTE	0.143	0.304	0.526	0.134	0.013	0.075	0.119	0.133	0.264	0.144	-0.062	0.010
PPL	0.145	0.289	0.513	0.134	0.017	0.100	0.131	0.038	0.184	0.148	-0.094	-0.050

Appendix

Table 11: Masked-CoT Results on MathVista using Qwen3-VL-8B-IT. Masking answer mentions partially restores uncertainty ranking, while random masking of an equivalent number of reasoning tokens degrades ranking quality. These results are computed solely on the subset of image/question pairs where the reasoning traces include the predicted final answer.

Explore similar work

How Uncertainty Estimation Scales with Sampling in Reasoning Models
Mar 19, 2026Maksym Del, Markus Kängsepp, Marharyta Domnich +4LLM Uncertainty Estimation Uncertainty Quantification
Institute of Computer Science, University of Tartu
Reasoning Dynamics and the Limits of Monitoring Modality Reliance in Vision-Language Models
Apr 16, 2026Danae Sánchez Villegas, Samuel Lewis-Lim, Nikolaos Aletras +1CoT Faithfulness Vision-Language Models
University of Copenhagen, Department of Computer Science · University of Sheffield, School of Computer Science
From Perception to Integration: Revisiting the Internal Dynamics of Reasoning in Vision-Language Models
Sep 28, 2026Rong Yu Xu, Prayag Tiwari, Shaolei ZhangEfficient VLM Inference VLM Interpretability
Shenzhen College of International Education · Halmstad University, Sweden · Renmin University of China

Mar 19, 2026J/K move · Enter open · S save

How Uncertainty Estimation Scales with Sampling in Reasoning Models

Maksym Del, Markus Kängsepp, Marharyta Domnich, Ardi Tampuu+3

Institute of Computer Science, University of Tartu

LLM Uncertainty Estimation Uncertainty Quantification Confidence Estimation in Language Models Large Reasoning Models Self-Consistency in Language Models

Open paper AskPDF arXiv

Uncertainty estimation is critical for deploying reasoning language models, yet remains poorly understood under extended chain-of-thought reasoning. We study parallel sampling as a fully black-box approach using verbalized confidence and self-consistency. Across three reasoning models and 17 tasks spanning mathematics, STEM, and humanities, we characterize how these signals scale. Both self-consistency and verbalized confidence scale in reasoning models, but self-consistency exhibits lower initial discrimination and lags behind verbalized confidence under moderate sampling. Most uncertainty gains, however, arise from signal combination: with just two samples, a hybrid estimator improves AUROC by up to

+12

on average and already outperforms either signal alone even when scaled to much larger budgets, after which returns diminish. These effects are domain-dependent: in mathematics, the native domain of RLVR-style post-training, reasoning models achieve higher uncertainty quality and exhibit both stronger complementarity and faster scaling than in STEM or humanities.

Abstract

Explore similar work

How Uncertainty Estimation Scales with Sampling in Reasoning Models

Reasoning Dynamics and the Limits of Monitoring Modality Reliance in Vision-Language Models

From Perception to Integration: Revisiting the Internal Dynamics of Reasoning in Vision-Language Models