Ensembles of diverse Large Vision Language Models have demonstrated performance beyond that of single models across multiple datasets and tasks. However, the necessary aggregation process usually relies on training, parameter fitting, or complex task-specific pipelines. Furthermore, evaluation is limited, making a fair comparison difficult. We create a unified pipeline where we perform candidate response selection: we build an ensemble by selecting the best answer from a set provided by different models. We comprehensively evaluate across 34 different means of confidence estimation, 24 ensemble configurations, 20 datasets, and two prompt specifications. We show that simple training and parameter-free approaches can be sufficient, even for ensembles of different architectures. These simple self-assessment approaches rely solely on each constituent model's confidence, without explicit calibration, yet are able to perform better than said constituent models. They are also competitive with more involved alternatives on standard metrics such as accuracy, including on open-ended problems. However, we also discover that dataset, ensemble composition, and prompt wording are key confounding variables insufficiently explored by prior work that significantly affect ensemble performance, such that the ensemble can perform worse than its best constituent model. These must be optimized for to get the most out of ensembles, at the cost of limiting their generalizability. We recommend that practitioners consider training-free ensembles only if deployment conditions can be precisely characterized, with Best-of-N configurations of strong models being a good starting point.
Figures & tables
Figure 1: An overview of the testbed, showing the variables V1 , V2 , V3 and V4 that can affect how one example question from the MMVet dataset is processed through to evaluation. See Sec. 4.1 for details and Sec. 4.2 for a description of the variables.
Figure 2: Comparison of different ensembles, presented as the Relative Gain (%) over the best constituent model, chosen optimally for each dataset, prompt and seed combination. Results show the mean and 95% confidence interval, across all seeds and datasets compatible with the particular ensemble. The confidence interval is calculated using bootstrapping with 2000 iterations. Left: The P(True) Search Approach. Right: Using the optimal approach for each ensemble, dataset and prompt.is calculated using bootstrapping with 2000 iterations.
Figure 3: Relative Gain (%) over the best constituent model, chosen optimally for each dataset, prompt and seed combination. The best possible approach for each ensemble configuration, dataset, and prompt is shown. This shows the best performance that can be obtained compared to an individual model, if we are able to fully characterize the deployment environment. This also serves as a breakdown of the right side of Fig. 2 . Results are averaged across both prompt types (except CMMMU and MMSafetyBench which are only Normal ) and all seeds.
Figure 4: Mean performance of different approaches by ensemble configuration, across all datasets, prompt types and seeds. Green and red shading represents the Relative Gain (%) over the best constituent model (whose mean performance is shown in blue), chosen optimally for each dataset, prompt and seed combination.
Figure A5 : Comparison of different model selection approaches against ensemble configurations. The average relaxed win rate (within 0.5% of the winner) is shown across prompt types, datasets, and seeds. Unlike Figs. A1 and A2 we include the full set of approaches.
Appendix figures & tables9 assets
Supplementary material from the paper’s appendix.
Appendix
Dataset Name
VLMEvalKit Identifier
Type
Questions
OCRBench [ 33 ]
OCRBench
OCR
1000
MMVet [ 66 ]
MMVet
Open-ended VQA
218
MMStar [ 6 ]
MMStar
Multiple choice VQA
1500
MMMU [ 67 ]
MMMU_DEV_VAL
Mixed VQA
900
MMBench EN [ 32 ]
MMBench_DEV_EN_V11
Multiple choice VQA
1164
MathVista [ 34 ]
MathVista_MINI
Mixed Math VQA
1000
Appendix
Table A1 : Datasets used in this work, their identifiers in VLMEvalKit [ 9 ] , their question types, and the number of questions evaluated. The number of questions may be different from total published dataset size as we use only splits with public ground truth data. Mixed VQA indicates a combination of multiple choice and open-ended VQA. Safety is unique; see [ 31 ] . Metrics are discussed in Appendix A1 .
Approach
All Plots
Model Score
✓
Model Score (L. Norm.)
✓
Mean Log Confidence
✓
Mean Confidence
✓
Mean Entropy
✓
Sequence Prob.
✓
Appendix
Table A2 : List of all the approaches tested in this work. The All Plots column indicates if the approach is relevant enough to be shown on all plots, as due to space constraints we elected to only use a subset in most plots with some exceptions such as Fig. A5 . Abbreviations: L. - Length, Norm. - Normalized, Prob. - Probability, Tok. - Tokens, Srch. - Search, Conf. - Confidence, Min. - Minimum, Max. - Maximum, Bott. - Bottom, W. - Width.
Description
Models
Best-of- N Approaches
Qwen 3.5 Best-of- N Instruct
Qwen 3.5 9B Instruct
Qwen 3.5 Best-of- N Thinking
Qwen 3.5 9B Thinking
GLM 4.6V Best-of- N Instruct
GLM 4.6V Flash Instruct
GLM 4.6V Best-of- N Thinking
GLM 4.6V Flash Thinking
Diverse Approaches
Appendix
Table A3 : Complete list of ensemble configurations used in our experiments and their constituent models. See Tabs. A6 and A5 for the performance of each model, relevant citations, and sampling parameters.
Figure A1 : Comparison of different model selection approaches, presented as the relaxed average win rate (within 0.5% of the winner) across all datasets, ensemble configurations, prompt types and seeds. Results show the mean and 95% confidence interval. The confidence interval is calculated using bootstrapping with 2000 iterations.
Figure A2 : Comparison of different model selection approaches, presented as the average win rate across all datasets, ensemble configurations, prompt types and seeds. Results show the mean and 95% confidence interval. The confidence interval is calculated using bootstrapping with 2000 iterations. This is the same plot as Fig. A1 except we use a strict win rate definition.
Figure A3 : Comparison of different ensembles by dataset, presented as the Relative Gain (%) over the best possible performance using one of the models that make up the ensemble, chosen optimally for each dataset, prompt and seed combination. This is the same as Fig. 3 but with the approach fixed to P(True) Search instead of the best possible.
Figure A4 : Comparison of different ensembles by dataset, presented as the Relative Gain (%) over the best possible performance using one of the models that make up the ensemble, chosen optimally for each dataset, prompt and seed combination. This is the same as Fig. 3 but with the approach fixed to P(True) Confidence instead of the best possible.
Correct
P(True) Response
P(True) Conf.
P(True) Search
Quest. #
Model
5
GLM 4.6V
✓
<|begin_of_box|>True<|end_of_box|><|user|>
0.4
94.6
Qwen 3
✓
True<|im_end|>
100.0
100.0
Qwen 3.5
✓
True<|im_end|>
99.4
99.4
89
GLM 4.6V
✗
<|begin_of_box|>True<|end_of_box|><|user|>
2.6
98.9
Qwen 3
✗
True<|im_end|>
100.0
100.0
Appendix
Table A4: Detailed comparison of P(True) Confidence and P(True) Search for selected questions and one run on the MMBench EN dataset. Quest. # denotes the index of the question in the dataset. Correct indicates that the model response for that question matched ground truth. P(True) Response is the response to the P(True) query as a string (see Appendix A2 ), while the last two P(True) columns show the confidence in the answer as reported by each approach.
Model
Hugging Face Repository
Temperature
Max Tokens
GLM-4.6V-Flash Inst. [ 52 ]
zai-org/GLM-4.6V-Flash
0.8
16384
GLM-4.6V-Flash Thnk. [ 52 ]
zai-org/GLM-4.6V-Flash
0.8
32768
Gemma-3-4B Inst. [ 51 ]
google/gemma-3-4b-it
−
16384
InternVL2-8B Inst. [ 8 ]
OpenGVLab/InternVL2-8B
0.8
16384
InternVL3.5-2B Inst. [ 58 ]
OpenGVLab/InternVL3_5-2B
0.8
16384
InternVL3.5-8B Inst. [ 58 ]
OpenGVLab/InternVL3_5-8B-Instruct
0.8
16384
Appendix
Table A5: Sampling parameters for each model variant. Thnk. : Thinking, Inst. : Instruct. Prompt type ( Appendix A14 ) has no effect on sampling parameters. A temperature of − indicates VLLM [ 24 ] defaults are used i.e . read from the Hugging Face model repository if possible. Abbreviations: Inst. - Instruct, Thnk. - Thinking.