Ensembles of diverse Large Vision Language Models have demonstrated performance beyond that of single models across multiple datasets and tasks. However, the necessary aggregation process usually relies on training, parameter fitting, or complex task-specific pipelines. Furthermore, evaluation is limited, making a fair comparison difficult. We create a unified pipeline where we perform candidate response selection: we build an ensemble by selecting the best answer from a set provided by different models. We comprehensively evaluate across 34 different means of confidence estimation, 24 ensemble configurations, 20 datasets, and two prompt specifications. We show that simple training and parameter-free approaches can be sufficient, even for ensembles of different architectures. These simple self-assessment approaches rely solely on each constituent model's confidence, without explicit calibration, yet are able to perform better than said constituent models. They are also competitive with more involved alternatives on standard metrics such as accuracy, including on open-ended problems. However, we also discover that dataset, ensemble composition, and prompt wording are key confounding variables insufficiently explored by prior work that significantly affect ensemble performance, such that the ensemble can perform worse than its best constituent model. These must be optimized for to get the most out of ensembles, at the cost of limiting their generalizability. We recommend that practitioners consider training-free ensembles only if deployment conditions can be precisely characterized, with Best-of-N configurations of strong models being a good starting point.
Figures & tables
Figure 1: An overview of the testbed, showing the variables V1 , V2 , V3 and V4 that can affect how one example question from the MMVet dataset is processed through to evaluation. See Sec. 4.1 for details and Sec. 4.2 for a description of the variables.
Figure 2: Comparison of different ensembles, presented as the Relative Gain (%) over the best constituent model, chosen optimally for each dataset, prompt and seed combination. Results show the mean and 95% confidence interval, across all seeds and datasets compatible with the particular ensemble. The confidence interval is calculated using bootstrapping with 2000 iterations. Left: The P(True) Search Approach. Right: Using the optimal approach for each ensemble, dataset and prompt.is calculated using bootstrapping with 2000 iterations.
Figure 3: Relative Gain (%) over the best constituent model, chosen optimally for each dataset, prompt and seed combination. The best possible approach for each ensemble configuration, dataset, and prompt is shown. This shows the best performance that can be obtained compared to an individual model, if we are able to fully characterize the deployment environment. This also serves as a breakdown of the right side of Fig. 2 . Results are averaged across both prompt types (except CMMMU and MMSafetyBench which are only Normal ) and all seeds.
Figure 4: Mean performance of different approaches by ensemble configuration, across all datasets, prompt types and seeds. Green and red shading represents the Relative Gain (%) over the best constituent model (whose mean performance is shown in blue), chosen optimally for each dataset, prompt and seed combination.
Figure A5 : Comparison of different model selection approaches against ensemble configurations. The average relaxed win rate (within 0.5% of the winner) is shown across prompt types, datasets, and seeds. Unlike Figs. A1 and A2 we include the full set of approaches.
Appendix figures & tables9 assets
Supplementary material from the paper’s appendix.
Appendix
Dataset Name
VLMEvalKit Identifier
Type
Questions
OCRBench [ 33 ]
OCRBench
OCR
1000
MMVet [ 66 ]
MMVet
Open-ended VQA
218
MMStar [ 6 ]
MMStar
Multiple choice VQA
1500
MMMU [ 67 ]
MMMU_DEV_VAL
Mixed VQA
900
MMBench EN [ 32 ]
MMBench_DEV_EN_V11
Multiple choice VQA
1164
MathVista [ 34 ]
MathVista_MINI
Mixed Math VQA
1000
Appendix
Table A1 : Datasets used in this work, their identifiers in VLMEvalKit [ 9 ] , their question types, and the number of questions evaluated. The number of questions may be different from total published dataset size as we use only splits with public ground truth data. Mixed VQA indicates a combination of multiple choice and open-ended VQA. Safety is unique; see [ 31 ] . Metrics are discussed in Appendix A1 .
Approach
All Plots
Model Score
✓
Model Score (L. Norm.)
✓
Mean Log Confidence
✓
Mean Confidence
✓
Mean Entropy
✓
Sequence Prob.
✓
Appendix
Table A2 : List of all the approaches tested in this work. The All Plots column indicates if the approach is relevant enough to be shown on all plots, as due to space constraints we elected to only use a subset in most plots with some exceptions such as Fig. A5 . Abbreviations: L. - Length, Norm. - Normalized, Prob. - Probability, Tok. - Tokens, Srch. - Search, Conf. - Confidence, Min. - Minimum, Max. - Maximum, Bott. - Bottom, W. - Width.
Description
Models
Best-of- N Approaches
Qwen 3.5 Best-of- N Instruct
Qwen 3.5 9B Instruct
Qwen 3.5 Best-of- N Thinking
Qwen 3.5 9B Thinking
GLM 4.6V Best-of- N Instruct
GLM 4.6V Flash Instruct
GLM 4.6V Best-of- N Thinking
GLM 4.6V Flash Thinking
Diverse Approaches
Appendix
Table A3 : Complete list of ensemble configurations used in our experiments and their constituent models. See Tabs. A6 and A5 for the performance of each model, relevant citations, and sampling parameters.
Figure A1 : Comparison of different model selection approaches, presented as the relaxed average win rate (within 0.5% of the winner) across all datasets, ensemble configurations, prompt types and seeds. Results show the mean and 95% confidence interval. The confidence interval is calculated using bootstrapping with 2000 iterations.
Figure A2 : Comparison of different model selection approaches, presented as the average win rate across all datasets, ensemble configurations, prompt types and seeds. Results show the mean and 95% confidence interval. The confidence interval is calculated using bootstrapping with 2000 iterations. This is the same plot as Fig. A1 except we use a strict win rate definition.
Figure A3 : Comparison of different ensembles by dataset, presented as the Relative Gain (%) over the best possible performance using one of the models that make up the ensemble, chosen optimally for each dataset, prompt and seed combination. This is the same as Fig. 3 but with the approach fixed to P(True) Search instead of the best possible.
Figure A4 : Comparison of different ensembles by dataset, presented as the Relative Gain (%) over the best possible performance using one of the models that make up the ensemble, chosen optimally for each dataset, prompt and seed combination. This is the same as Fig. 3 but with the approach fixed to P(True) Confidence instead of the best possible.
Correct
P(True) Response
P(True) Conf.
P(True) Search
Quest. #
Model
5
GLM 4.6V
✓
<|begin_of_box|>True<|end_of_box|><|user|>
0.4
94.6
Qwen 3
✓
True<|im_end|>
100.0
100.0
Qwen 3.5
✓
True<|im_end|>
99.4
99.4
89
GLM 4.6V
✗
<|begin_of_box|>True<|end_of_box|><|user|>
2.6
98.9
Qwen 3
✗
True<|im_end|>
100.0
100.0
Appendix
Table A4: Detailed comparison of P(True) Confidence and P(True) Search for selected questions and one run on the MMBench EN dataset. Quest. # denotes the index of the question in the dataset. Correct indicates that the model response for that question matched ground truth. P(True) Response is the response to the P(True) query as a string (see Appendix A2 ), while the last two P(True) columns show the confidence in the answer as reported by each approach.
Model
Hugging Face Repository
Temperature
Max Tokens
GLM-4.6V-Flash Inst. [ 52 ]
zai-org/GLM-4.6V-Flash
0.8
16384
GLM-4.6V-Flash Thnk. [ 52 ]
zai-org/GLM-4.6V-Flash
0.8
32768
Gemma-3-4B Inst. [ 51 ]
google/gemma-3-4b-it
−
16384
InternVL2-8B Inst. [ 8 ]
OpenGVLab/InternVL2-8B
0.8
16384
InternVL3.5-2B Inst. [ 58 ]
OpenGVLab/InternVL3_5-2B
0.8
16384
InternVL3.5-8B Inst. [ 58 ]
OpenGVLab/InternVL3_5-8B-Instruct
0.8
16384
Appendix
Table A5: Sampling parameters for each model variant. Thnk. : Thinking, Inst. : Instruct. Prompt type ( Appendix A14 ) has no effect on sampling parameters. A temperature of − indicates VLLM [ 24 ] defaults are used i.e . read from the Hugging Face model repository if possible. Abbreviations: Inst. - Instruct, Thnk. - Thinking.
Ensembling Vision-Language Models (VLMs) from different providers maximizes benchmark accuracy, yet models from the same architectural family share correlated errors that standard voting ignores. We study this structure across 17 VLMs from 8 families on VQAv2, TextVQA, and GQA. Family-correlated errors reduce effective ensemble dimensionality to 2.5-3.6 independent voters and create a Misleading tier (1.5-6.5% of questions) where correlated majority errors destroy accuracy to 0% despite the best model being correct. We propose three family-aware methods. Hierarchical Family Voting (HFV) aggregates within families before voting across them, recovering +18-26 pp on the Misleading tier. QualRCCV, a training-free method weighting models by calibration, family quality, and inverse family size, is the first to beat calibrated voting on all three benchmarks (p<0.05). Learned Candidate Scoring (LCS) trains a cross-validated classifier to re-rank candidate answers using support breadth, family diversity, and model quality, achieving the largest gains: +0.68% VQAv2, +0.61% TextVQA, +2.45% GQA -- all significant -- and is the only learned method that never degrades any benchmark. On VQAv2 test-standard (EvalAI), LCS reaches 87.83% with 12 models, confirming generalization.
Test-time compute (TTC) strategies have emerged as a lightweight approach to boost reasoning in large language models (LLMs). However, their application and benefits for vision-language models (VLMs) remain underexplored. We present a systematic study of TTC across seven VLMs and six benchmarks, specifically analyzing feature-based scoring and majority voting methods. We find that feature heuristics fail and voting yields only modest gains in single-model settings. We theoretically show that this limitation stems from a lack of prediction diversity: when outputs are highly correlated, voting provides little benefit. In contrast, multi-model ensembles offer richer diversity, yet standard majority voting fails to account for varying model capabilities. To address this, we propose Entropy-based TTC (ETTC), which selects the most confident prediction based on predictive entropy. Our method reduces to majority voting in the single-model case, but in model ensembles, it leverages confidence disparities to prioritize stronger models. We prove that ETTC outperforms majority voting under mild assumptions and empirically demonstrate that it consistently surpasses both voting and the best individual model. Crucially, our results show that smaller models can synergistically enhance larger ones, unlocking ensembling gains not achievable with standard strategies.
Heterogeneous large language models (LLMs) exhibit complementary strengths, motivating routing and multi-model inference methods that seek to identify reliable predictions for each input. However, exploiting this complementarity remains challenging even when a correct prediction is available within the ensemble. We investigate this challenge across heterogeneous LLM ensembles and find that a simple single-round aggregation rule is highly competitive with learned routing methods, yet substantial oracle headroom remains. Further analysis shows that the evaluated single-round response signals provide limited discriminative power for identifying reliable predictions. In contrast, controlled re-presentations of the same question elicit complementary evidence about prediction reliability. Motivated by these findings, we introduce RE-PRESENT, a training-free framework that selectively re-presents disputed questions and directly aggregates candidate-level evidence across alternative views. Across seven multiple-choice benchmarks and two heterogeneous model ensembles, RE-PRESENT improves over strong single-round aggregation while offering a favorable accuracy--cost trade-off compared with self-consistency and multi-model debate.
Juming Xiong, Weixin Liu, Kevin Guo +7
Vanderbilt University · Vanderbilt University Medical Center