VisConf: Quality-Aware Consensus for Best-of-N in Vision-Language Model Reasoning

Muhammad Usman; Muhammad Qasim Ayub; Usman Ahad; Muhammad Tahir

cs.CVOct 17, 2026

VisConf: Quality-Aware Consensus for Best-of-N in Vision-Language Model Reasoning

Authors: Muhammad Usman, Muhammad Qasim Ayub, Usman Ahad, Muhammad Tahir

Organizations: Department of Computer Science, Lahore University of Management Sciences · Department of Electrical Engineering, Lahore University of Management Sciences

Abstract

Best-of-N methods improve text-only LLM reasoning by sampling multiple trajectories before selecting an answer. For Vision-Language Models (VLMs), answer frequency and model confidence alone can be insufficient, as sampled generations also vary in their engagement with the visual input. We find that distributional confidence and visual attention are complementary indicators of rollout quality. Each is limited on its own, while trajectories that show strong signals in both are more likely to be correct. Motivated by this, we introduce VisConf, a rollout-quality score that multiplicatively combines visual engagement with candidate-calibrated Self-Certainty, penalizing rollouts that are weak in either signal. We incorporate Rank-Weighted Consensus (RWC), a tuning-free aggregation rule that converts within-question VisConf ranks into answer-level vote weights. Across three VLMs on MathVista, MMMU-Pro, and MMStar, VisConf with RWC consistently outperforms strong BoN baselines. Our analysis shows particularly large gains when correct trajectories are present but underrepresented in the candidate pool.

Figures & tables

Panel of Figure 1 — Figure 1: Analysis on Qwen2.5-VL-3B using MathVista, MMMU-Pro, and MMStar. (a) Accuracy as a function of confidence and visual engagement. (b) Discriminative strength measured by the accuracy gap between the top and bottom score quartiles. Combining confidence and visual engagement is more informative, but how the signals are combined also matters.

Figure 2: Overview of the VisConf and RWC framework. Given multiple sampled VLM rollouts, VisConf aggregates calibrated confidence and visual engagement independently over each trajectory, and combines them into a rollout-quality score, rewarding trajectories that are both confident and visually engaged. RWC uses VisConf scores to effectively select high-quality answer groups.

Method	Qwen2.5-VL-3B			InternVL3-8B			Gemma4-E2B			Avg.
	$N=8$	$N=16$	$N=32$	$N=8$	$N=16$	$N=32$	$N=8$	$N=16$	$N=32$	Avg.
FirstAns	48.51	48.51	48.51	56.55	56.55	56.55	47.34	47.34	47.34	50.80
Self-Consistency	53.70	54.75	55.07	60.96	61.71	61.80	49.85	50.23	50.35	55.38
Self-Certainty	49.78	49.95	49.94	58.83	59.03	59.07	49.43	49.62	49.81	52.83
+ Borda ( $p=0.3$ )	53.83	54.87	55.14	61.21	61.97	62.25	50.12	50.50	50.71	55.62
+ Borda ( $p=1.2$ )	53.53	54.72	55.33	61.21	62.03	62.78	50.35	50.82	50.91	55.74

Table 1: Averaged accuracy (%) across MathVista, MMMU-Pro, and MMStar at varying rollout budgets

N

. Per-dataset results are presented in Appendix B.1 . VisConf + RWC achieves the highest average for every model and sampling budget.

Panel of Figure 3 — Figure 3: Effect of scaling $N$ on accuracy across three benchmarks. (a) VisConf top-1 selection scales strongly but underperforms Self-Consistency. (b-c) VisConf + RWC consistently outperforms top-1 and Self-Consistency, with its advantage persisting or increasing with scaling.

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

$\boldsymbol{\alpha}$	$\boldsymbol{\lambda}$	Aggregation	Qwen2.5-VL-3B			InternVL3-8B			Gemma4-E2B			Avg.
			$N=8$	$N=16$	$N=32$	$N=8$	$N=16$	$N=32$	$N=8$	$N=16$	$N=32$
0.25	1.25	BoN	49.88	50.17	50.45	56.31	55.00	52.99	48.92	49.38	49.95	51.45
0.25	1.25	RWC	53.40	54.61	55.51	60.78	61.68	62.07	50.16	50.70	50.84	55.53
0.50	1.25	BoN	50.09	50.34	50.57	58.22	58.11	57.84	49.16	49.74	50.27	52.70
0.50	1.25	RWC	53.68	54.93	55.72	61.27	62.16	62.65	50.30	50.86	51.19	55.86

Appendix

Table 3: LookBack Best-of-N and Rank-Weighted Consensus (RWC) performance under two values of

\alpha

, with

\lambda=1.25

Method	MathVista			MMMU-Pro			MMStar			Avg.
	$N=8$	$N=16$	$N=32$	$N=8$	$N=16$	$N=32$	$N=8$	$N=16$	$N=32$	Avg.
Qwen2.5-VL-3B
Oracle	82.76	87.01	89.40	80.31	88.10	92.95	85.50	91.57	95.67	88.14
FirstAns	54.01	54.01	54.01	41.20	41.20	41.20	50.31	50.31	50.31	48.51
Self-Consistency	61.04	62.48	63.20	45.30	46.38	46.88	54.77	55.38	55.13	54.51
Self-Certainty	55.65	55.70	55.40	42.97	43.25	43.76	50.72	50.89	50.67	49.89

Appendix

Table 4: Full accuracy results (%) on MathVista, MMMU-Pro, and MMStar for

N\in\{8,16,32\}

. Avg. is the unweighted mean across the nine dataset-budget combinations within each model.

Comparator	$\boldsymbol{N=8}$	$\boldsymbol{N=16}$	$\boldsymbol{N=32}$	Overall
LookBack + RWC	$+0.47\;[0.35,\,0.59]$	$+0.45\;[0.31,\,0.61]$	$+0.56\;[0.24,\,0.87]$	$+0.49\;[0.32,\,0.66]$
Self-Consistency	$+0.72\;[0.56,\,0.86]$	$+0.87\;[0.68,\,1.07]$	$+1.34\;[0.97,\,1.72]$	$+0.98\;[0.76,\,1.18]$
Self-Cert. + Borda ( $p=1.2$ )	$+0.52\;[0.40,\,0.66]$	$+0.58\;[0.40,\,0.74]$	$+0.74\;[0.39,\,1.09]$	$+0.61\;[0.43,\,0.80]$
VisConf (top- $1$ )	$+2.03\;[1.79,\,2.28]$	$+2.61\;[2.21,\,3.02]$	$+3.08\;[2.42,\,3.77]$	$+2.57\;[2.15,\,3.00]$

Appendix

Table 5: Paired accuracy improvements of VisConf + RWC over each comparator in percentage points with

95\%

question-cluster bootstrap confidence intervals. Overall averages across sampling budgets, models, and benchmarks.

Panel of Figure 6 — Figure 6: Joint enrichment of rollout correctness by raw confidence and visual engagement. The dashed line denotes overall rollout accuracy, and shaded regions show 95% confidence intervals.

Correct rollouts	Questions	Rescues	Regressions	Net
$1$ – $4$	521	6	1	$+5$
$5$ – $8$	355	18	1	$+17$
$9$ – $12$	340	52	17	$+35$
$13$ – $15$	197	34	10	$+24$
$1$ – $15$	1,413	110	29	$+81$
$16$ – $32$	2,397	0	24	$-24$

Appendix

Table 7: Rescues and regressions of VisConf + RWC relative to Self-Consistency on InternVL3-8B at

N=32

, pooled across MathVista, MMMU-Pro, and MMStar.

Aggregation	MathVista			MMMU-Pro			MMStar			Avg.
	$N=8$	$N=16$	$N=32$	$N=8$	$N=16$	$N=32$	$N=8$	$N=16$	$N=32$	Avg.
RWC	61.08	62.97	64.50	46.12	47.24	48.38	55.06	55.75	56.00	55.23
Borda ( $p=0.3$ )	61.46	63.02	64.10	45.79	46.80	47.28	55.00	55.59	55.73	54.97
Borda ( $p=0.7$ )	61.46	63.11	64.40	46.04	47.09	48.15	55.10	55.69	55.87	55.21
Borda ( $p=1.2$ )	61.14	62.91	64.10	46.10	47.20	48.38	55.07	55.74	56.07	55.19
Borda ( $p=2.0$ )	60.41	62.43	63.60	45.92	47.12	48.32	54.85	55.73	56.00	54.93

Appendix

Table 8: Accuracy (%) on Qwen2.5-VL-3B, obtained by applying RWC or parameterized Borda aggregation to the same VisConf rollout scores. Avg is the unweighted mean across MathVista, MMMU-Pro, and MMStar for

N\in\{8,16,32\}

Reasoning for VLMs

See all 39 papers

The Cost of Reasoning: Chain-of-Thought Induces Overconfidence in Vision-Language Models
Mar 17, 2026Robert Welch, Emir Konuk, Kevin SmithConfidence Estimation for VLMs VLM Reasoning
KTH Royal Institute of Technology, Stockholm, Sweden · Science for Life Laboratory, Stockholm, Sweden
Be Faithful When Response: Returning Fluent and Grounded Answers for Vision-Language Models Reinforcement Learning
Jun 29, 2026Peng, Lee, Yin Zhang +9RL for VLMs Vision-Language Grounding
GMLab · Hong Kong Polytechnic University · XPENG Robotics
The Mirage of Calibrated Confidence: Trajectory-Independence of Verbalized Confidence in Vision-Language Models
Sep 16, 2026Jisoo Yang, Jaeho Han, Trung X. Pham +1VLM Evaluation Confidence Estimation in Language Models
Department of Artificial Intelligence, Chung-Ang University, Republic of Korea · Van Lang University, Ho Chi Minh City, Vietnam

Mar 17, 2026J/K move · Enter open · S save

The Cost of Reasoning: Chain-of-Thought Induces Overconfidence in Vision-Language Models

Robert Welch, Emir Konuk, Kevin Smith

KTH Royal Institute of Technology, Stockholm, Sweden · Science for Life Laboratory, Stockholm, Sweden

Confidence Estimation for VLMs VLM Reasoning Reasoning for VLMs

Open paper AskPDF arXiv

Vision-language models (VLMs) are increasingly deployed in high-stakes settings where reliable uncertainty quantification (UQ) is as important as predictive accuracy. Extended reasoning via chain-of-thought (CoT) prompting or reasoning-trained models has become ubiquitous in modern VLM pipelines, yet its effect on UQ reliability remains poorly understood. Our results show that reasoning tends to degrade the quality of many uncertainty estimates, even when it improves task accuracy. We identify implicit answer conditioning as the primary mechanism: as reasoning traces converge on a conclusion before the final answer is generated, token probabilities increasingly reflect consistency with the model's own reasoning trace rather than uncertainty about correctness. In effect, the model becomes overconfident in its answer. In contrast, agreement-based consistency remains robust and often improves under reasoning, making it a practical choice for uncertainty estimation in reasoning-enabled VLMs.

Abstract

Reasoning for VLMs

Explore similar work

The Cost of Reasoning: Chain-of-Thought Induces Overconfidence in Vision-Language Models

Be Faithful When Response: Returning Fluent and Grounded Answers for Vision-Language Models Reinforcement Learning

The Mirage of Calibrated Confidence: Trajectory-Independence of Verbalized Confidence in Vision-Language Models