Linking Perception, Confidence and Accuracy in MLLMs

Yuetian Du; Yucheng Wang; Rongyu Zhang; Zhijie Xu; Boyu Yang; Ming Kong; Jie Liu; Qiang Zhu

cs.CVMar 12, 2026

Linking Perception, Confidence and Accuracy in MLLMs

Authors: Yuetian Du, Yucheng Wang, Rongyu Zhang, Zhijie Xu, Boyu Yang, Ming Kong, Jie Liu, Qiang Zhu

Organizations: Zhejiang University · University of Michigan · Alibaba Group · City University of Hong Kong

Abstract

Recent advances in Multi-modal Large Language Models (MLLMs) have predominantly focused on enhancing visual perception to improve accuracy. However, a critical question remains unexplored: Do models know when they do not know? Through a probing experiment, we reveal a severe confidence miscalibration problem in MLLMs. To address this, we propose Confidence-Driven Reinforcement Learning (CDRL), which uses original-noise image pairs and a novel confidence-based reward to enhance perceptual sensitivity and robustly calibrate the model's confidence. Beyond training benefits, calibrated confidence enables more effective test-time scaling as a free lunch. We further propose Confidence-Aware Test-Time Scaling (CA-TTS), which dynamically coordinates Self-Consistency, Self-Reflection, and Visual Self-Check modules guided by confidence signals. An Expert Model acts in multiple roles (e.g., Planner, Critic, Voter) to schedule these modules and provide external verification. Our integrated framework establishes new state-of-the-art results with consistent 8.8% gains across four benchmarks. More ablation studies demonstrate the effectiveness of each module and scaling superiority.

Figures & tables

Figure 1: Disconnection Between Model Confidence and Accuracy Under Perception Degradation. The X-axis (’Perception’) shows the input image with progressively increasing noise. The plot demonstrates that while the Mean Confidence remain highly stable (insensitive), the Model Accuracy descends sharply, revealing a significant gap between the model’s self-reported certainty and its actual performance as the visual input degrades.

Figure 2: Framework Overview. The upper panel (a) illustrates how original-noise image pairs are used to optimize the model via Reinforcement Learning, driven by a Confidence-based Calibration Reward and a Accuracy Reward. The bottom panel (b) shows the adaptive Confidence-Aware Test-Time Scaling (CA-TTS) system, where an Expert Model acts as a Planner, Voter, and Critic to coordinate the Self-Consistency, Self-Reflection, and Self-Check modules, which collaborate to produce the final answer.

Training Free Baselines (Qwen2.5-VL-7B)
	Math Reasoning						General VQA
Model/Dataset	Math-Vista testmini			Math-Vision test			MMStar test			MMMU val
	OE	MC	ALL	OE	MC	ALL	PE	RE	ALL	STEM	HASS	ALL
Pass@1	62.3	66.8	64.7	24.2	21.7	23.0	58.4	68.2	60.2	40.1	60.7	48.8
Majority Voting	68.4	73.7	69.8	26.2	33.2	30.1	63.6	77.0	69.0	53.2	62.5	57.5
Deepconf	69.3	74.7	70.7	26.4	32.3	29.6	58.2	71.0	61.1	51.5	62.5	56.1

Table 1: Evaluation Results on Visual Reasoning Benchmarks. Abbreviations: OE (Open-Ended), MC (Multi-Choice), PE (Perception), RE (Reasoning), STEM (Science, Technology, Engineering, and Mathematics), HASS (Humanities, Arts, and Social Sciences), and ALL (Overall). Best results are bold .

	Math-Vision test
Setting	OE	MC	ALL
Training-Free	24.24	21.71	22.96
CDRL	18.46	34.16	26.38
CA-TTS	37.99	42.95	37.99
CDRL+CA-TTS	38.44	45.60	42.35

Table 2: Ablation Study for CDRL and CA-TTS. Best results are bold , second-best are underlined .

	Math-Vision testmini
Expert Model	OE	MC	ALL
Majority Voting	22.09	31.30	27.65
Qwen-2.5-VL-7B	30.91	34.03	32.57
Qwen-2.5-VL-72B	28.52	37.70	34.21
Qwen-VL-Max	35.71	36.13	35.97
GPT-5	38.94	43.93	41.45

Table 3: Performance of Different Expert Models on Math-Vision. OE (Open-Ended), MC (Multi-Choice). The results on other datasets can be found in Appendix C .

Figure 3: Test-time scaling comparison on Math-Vision. Accuracy vs. number of samples for our CA-TTS (blue), Majority Voting (green), and DeepConf (yellow). The slope of our method (

\beta_{1}=3.65

) is 2.2-3.1

\times

steeper than baselines (

\beta_{2}=1.64

\beta_{3}=1.19

), demonstrating superior scaling potential with calibrated confidence.

Figure 4: A case study comparing the reasoning processes of ToT [ 57 ] and CA-TTS (Ours). ToT (upper) conducts a complex tree search that remains vulnerable to a single-point-of-failure in its final evaluation, leading it to the incorrect answer. In contrast, Our method (bottom) demonstrates a multi-stage, resilient process: an initial error from Self-Consistency (Answer: 4) is corrected by Self-Reflection (Answer: 6) and confirmed by Self-Check .

	Qwen2.5-VL-7B-Instruct			+CDRL (Ours)
Visual Uncertainty	CD $\downarrow$	ECE $\downarrow$	AUC $\uparrow$	CD $\downarrow$	ECE $\downarrow$	AUC $\uparrow$
Origin	0	64.57	54.81	0	62.24	59.42
Noised	-0.32	66.19	60.00	-1.39	63.75	60.19
Occlusion	-0.24	66.19	53.88	-1.13	65.05	56.34
Viewpoint	+0.09	64.99	55.19	-1.29	64.18	57.26
Mosaic	+0.11	65.41	60.01	-0.86	63.07	61.68

Table 4: Analysis of model confidence sensitivity under different visual uncertainty conditions on Math-Vision. We compare the base model with our CDRL -trained model across five visual conditions. CD (Confidence Drop) measures sensitivity to visual perturbations relative to the original image . ECE and AUC measure calibration quality. Best results are in bold .

Figure 5: The prompt template used for the Voter Expert. The model acts as a discriminator to assign probability scores to candidate choices, facilitating confidence-weighted voting.

Figure 6: The prompt template used for the Critic Expert. This prompt induces the expert model to generate a critique based on the original input and the model’s initial low-confidence response, aiding in the self-reflection process.

Methods	Math-Vista testmini			Math-Vision testmini			MMStar			MMMU			Overall
	Math Reasoning						General VQA
(Qwen3-VL-2B-Think)	OE	MC	ALL	OE	MC	ALL	PE	RE	ALL	STEM	HASS	ALL	Avg.
Pass@1	64.95	82.51	73.60	47.69	50.00	48.82	62.79	56.00	61.24	58.84	66.95	61.40	61.27
Majority Voting	72.30	89.86	80.95	60.87	62.50	61.70	67.10	66.49	66.24	72.90	81.01	75.46	71.09
DeepConf	72.77	87.92	80.24	52.17	62.50	57.45	67.36	68.09	66.24	73.83	82.28	75.93	69.97
Ours (CA-TTS)	77.46	90.34	83.81	60.87	66.67	63.83	69.71	75.53	70.36	78.50	83.54	79.63	74.41

Table 5: Performance on Qwen3-VL-2B-Thinking. Comparison of CA-TTS against other TTS baselines. Abbreviations follow previous tables. Best results are bold , and second-best results are underlined . The “Overall Avg” represents the mean of the “ALL” scores across the four datasets.

		Number of Samples ( $N$ )
Dataset	Method	1	2	4	8	16	32
	Majority Voting	26.38	25.00	29.93	30.26	30.92	34.41
Math-Vision testmini	DeepConf	26.38	26.64	28.95	28.95	30.26	32.15
	Ours (CDRL+CA-TTS)	30.60	34.21	41.78	43.75	46.38	48.44
	Majority Voting	64.70	67.78	68.56	68.39	73.11	75.60
Math-Vista testmini	DeepConf	64.70	67.89	67.69	69.28	71.32	75.50

Table 6: Detailed Scaling Results. Comparison of performance with varying number of samples (

N

). Best results are bold .

Models/Datasets	Math-Vista testmini			Math-Vision testmini			MMStar			MMMU			Overall
	Math Reasoning						General VQA
	OE	MC	ALL	OE	MC	ALL	PE	RE	ALL	STEM	HASS	ALL	Avg
Majority Voting	68.39	73.73	69.80	26.20	33.24	30.08	60.20	71.20	64.00	53.20	62.50	57.53	55.35
Qwen2.5-VL-7B	69.49	77.99	73.00	30.91	34.03	32.57	65.33	71.47	64.59	53.35	65.99	58.46	57.16
Qwen2.5-VL-72B	67.90	78.62	74.20	28.52	37.70	34.21	61.00	71.60	64.33	55.08	61.56	58.11	57.71
Qwen-VL-Max	69.35	79.14	74.47	35.71	36.13	35.97	61.20	73.40	64.80	56.25	67.05	61.45	59.17

Table 7: Performance of Different Expert Models. Abbreviations: OE (Open-Ended), MC (Multi-Choice), PE (Perception), RE (Reasoning), STEM (Science, Technology, Engineering, and Mathematics), HASS (Humanities, Arts, and Social Sciences), and ALL (Overall). Best results are bold , and second-best results are underlined . The “Overall Avg” column is calculated as the average of the “ALL” scores from the four datasets.

Framework + CA-TTS	Math-Vista	Math-Vision	MMStar	MMMU	Average
Qwen2.5-VL-7B	75.8	39.1	65.6	59.2	59.9
R1-OneVision	70.9	38.2	60.9	59.8	57.5
VL-Rethinker	74.3	38.1	64.3	57.1	58.5
We-Think	77.1	39.8	66.8	60.2	61.0
Ours (CDRL)	79.5	42.4	71.3	66.3	64.9

Table 8: Baseline Models Results with Gemini 2.5 Pro. Comparison of various frameworks when equipped with our CA-TTS plug-in.

Settings	Math-Vista	Math-Vision	MMStar	MMMU	Avg.	Time Cost (s)
Ours (CA-TTS)	79.5	42.4	71.3	66.3	64.9	11.03
w/o Self-Consistency	70.5	35.0	67.4	58.4	57.8	4.66
w/o Self-Reflection	74.6	37.9	69.1	65.5	61.8	8.55
w/o Self-Check	74.2	39.1	70.5	65.9	62.4	8.85
Majority Voting	69.8	30.1	69.0	57.5	56.6	5.76

Table 9: Ablation Study and Inference Cost. Evaluation of individual modules and their corresponding time consumption.

Figure 7: Scaling Superiority on Math-Vista. This plot compares the accuracy of Our method (blue dots) against the Majority Voting (green squares) and DeepConf (yellow triangles) baselines as the number of samples increases from 1 to 32. The results show that the slope of our trendline (

\beta_{1}=2.85

) is substantially steeper than those of the baseline methods (

\beta_{2}=2.01

and

\beta_{3}=1.88

), indicating that the performance advantage and potential of our method widens as more samples are provided.

Figure 8: Scaling Superiority on MMStar. This plot compares the accuracy of Our method (blue dots) against the Majority Voting (green squares) and DeepConf (yellow triangles) baselines as the number of samples increases from 1 to 32. The results show that the slope of our trendline (

\beta_{1}=2.48

) is substantially steeper than those of the baseline methods (

\beta_{2}=1.89

and

\beta_{3}=1.72

), indicating that the performance advantage and potential of our method widens as more samples are provided.

Figure 9: Scaling Superiority on MMMU. This plot compares the accuracy of Our method (blue dots) against the Majority Voting (green squares) and DeepConf (yellow triangles) baselines as the number of samples increases from 1 to 32. The results show that the slope of our trendline (

\beta_{1}=3.02

) is substantially steeper than those of the baseline methods (

\beta_{2}=1.85

and

\beta_{3}=1.63

), indicating that the performance advantage and potential of our method widens as more samples are provided.

Figure 10: Generality of Confidence Miscalibration under Visual Degradation. We tested the impact of various visual degradation types on two representative MLLMs. The overlapping confidence distributions indicate that MLLMs struggle to calibrate their confidence correctly when handling degraded visual inputs, highlighting the necessity of our Confidence-Aware framework.

Figure 11: A case study of CA-TTS on MMStar. Our method (bottom) demonstrates a multi-stage, resilient process: an initial error from Self-Consistency (Answer: B) is corrected by Self-Reflection (Answer: C) and confirmed by Self-Check .

Figure 12: A case study of CA-TTS on Math-Vista. Our method (bottom) demonstrates a multi-stage, resilient process: a error from Self-Reflection and Self-Consistency (Answer: B) is corrected by Self-Check (Answer: C).

Multimodal Large Language Models

Multimodal large language models combine language models with image, video, audio, and other inputs for grounded understanding and generation.

See all 1,091 papers

Instinct vs. Reflection: Unifying Token and Verbalized Confidence in Multimodal Large Models
Apr 19, 2026Yunkai Dang, Yifan Jiang, Yizhu Jiang +3Selective Prediction Confidence Estimation in Language Models
School of Artificial Intelligence, Nanjing University.
Scaling with Confidence: Calibrating Confidence of LLMs for Adaptive Test Time Scaling
Jul 2, 2026Xuqing Yang, Yi Yuan, Shanzhe Lei +1LLM Inference Efficiency RL for Language Models
Shanghai Jiao Tong University · Southeast University · Shanghai AI Laboratory
Process Supervision of Confidence Margin for Calibrated LLM Reasoning
Apr 25, 2026Liaoyaqi Wang, Chunsheng Zuo, William Jurayj +2Confidence Estimation in Language Models RL for Language Model Reasoning
Johns Hopkins University

Apr 19, 2026J/K move · Enter open · S save

Instinct vs. Reflection: Unifying Token and Verbalized Confidence in Multimodal Large Models

Yunkai Dang, Yifan Jiang, Yizhu Jiang, Anqi Chen+2

School of Artificial Intelligence, Nanjing University.

Selective Prediction Confidence Estimation in Language Models Multimodal Large Language Models Language Model Calibration LLM Reliability LLM Uncertainty Estimation+1

Open paper AskPDF arXiv

Multimodal Large Language Models (MLLMs) have demonstrated exceptional capabilities in various perception and reasoning tasks. Despite this success, ensuring their reliability in practical deployment necessitates robust confidence estimation. Prior works have predominantly focused on text-only LLMs, often relying on computationally expensive self-consistency sampling. In this paper, we extend this to multimodal settings and conduct a comprehensive evaluation of MLLMs' response confidence estimation. Our analysis reveals a significant instinct-reflection misalignment: the model's implicit token-level support frequently diverges from its verbal self-assessment confidence. To address this misalignment, we propose a monotone confidence fusion framework to merge dual-channel signals and cross-channel consistency to estimate correctness. Subsequently, an order-preserving mean alignment step is applied to correct global bias, which improves calibration while preserving the risk-coverage trade-off for selective prediction. Experiments on diverse open-source and closed-source MLLMs show that our method consistently yields more reliable confidence estimates and improves both calibration and failure prediction. Code will be available at https://github.com/Yunkaidang/Instinct-vs.-Reflection.

Abstract

Multimodal Large Language Models

Explore similar work

Instinct vs. Reflection: Unifying Token and Verbalized Confidence in Multimodal Large Models

Scaling with Confidence: Calibrating Confidence of LLMs for Adaptive Test Time Scaling

Process Supervision of Confidence Margin for Calibrated LLM Reasoning