Hidden Clones: Exposing and Fixing Family Bias in Vision-Language Model Ensembles

Zacharie Bugaud

cs.CVMar 17, 2026

Hidden Clones: Exposing and Fixing Family Bias in Vision-Language Model Ensembles

Authors: Zacharie Bugaud

Organizations: Astera Institute

Abstract

Ensembling Vision-Language Models (VLMs) from different providers maximizes benchmark accuracy, yet models from the same architectural family share correlated errors that standard voting ignores. We study this structure across 17 VLMs from 8 families on VQAv2, TextVQA, and GQA. Family-correlated errors reduce effective ensemble dimensionality to 2.5-3.6 independent voters and create a Misleading tier (1.5-6.5% of questions) where correlated majority errors destroy accuracy to 0% despite the best model being correct. We propose three family-aware methods. Hierarchical Family Voting (HFV) aggregates within families before voting across them, recovering +18-26 pp on the Misleading tier. QualRCCV, a training-free method weighting models by calibration, family quality, and inverse family size, is the first to beat calibrated voting on all three benchmarks (p<0.05). Learned Candidate Scoring (LCS) trains a cross-validated classifier to re-rank candidate answers using support breadth, family diversity, and model quality, achieving the largest gains: +0.68% VQAv2, +0.61% TextVQA, +2.45% GQA -- all significant -- and is the only learned method that never degrades any benchmark. On VQAv2 test-standard (EvalAI), LCS reaches 87.83% with 12 models, confirming generalization.

Figures & tables

Model	Family	Size	VQAv2	TextVQA	GQA
molmo2raw	Molmo	8B	86.3	77.9	59.0
molmo2	Molmo	8B	85.0	77.9	59.0
fullft	Qwen2.5	7B	84.8	67.2	60.6
7b_lora	Qwen2.5	7B	83.6	82.9	61.2
7b_lora_full	Qwen2.5	7B	83.6	82.4	61.3
72b_zs	Qwen2.5	72B	82.8	81.2	59.7

Table 1: Model inventory and individual accuracy across three benchmarks. Family dominance varies: Molmo leads on VQAv2, while Qwen2.5-VL LoRA variants lead on TextVQA; LLaVA-NeXT leads on GQA. Models fine-tuned on VQAv2 (fullft) transfer poorly to TextVQA (67%). InternVL3 and Phi-4 collapse below 50% on TextVQA. All 17 models evaluated on all benchmarks.

Figure 1: Gap decomposition across benchmarks. Calibrated voting captures only a small fraction of the gap between single-best and oracle accuracy, especially on VQAv2 and GQA.

Tier	% Q’s	Single	Cal	HFV	$\Delta$
T0: Trivial	41.6%	97.8	97.9	98.0	$+$ 0.1
T1: Easy	47.7%	91.5	91.7	90.2	$-$ 1.5
T2: Misleading	2.5%	78.9	0.0	26.0	+26.0
T3: Hard	7.1%	0.0	30.6	29.5	$-$ 1.1
T4: Impossible	1.1%	0.0	0.0	0.0	0

Table 2: Difficulty taxonomy on VQAv2 (17 models). The Misleading tier (T2, 2.5%) shows catastrophic failure: the best model achieves 79% but calibrated voting collapses to 0% . HFV recovers +26.0 pp of this tier.

Figure 2: Hierarchical clustering (Ward linkage) on error correlation distance. Family-colored leaves reveal that architecture families cluster together, confirming correlated within-family errors.

Split	Overall	Yes/No	Number	Other
test-dev	87.66	97.35	80.77	80.88
test-standard	87.83	97.33	81.00	81.12

Table 3: VQAv2 test-set results (EvalAI). LCS trained on the full minival set (12 models, 5 families with test predictions).

Training-free methods
Method	VQAv2	TextVQA	GQA
Single best	86.29 [85.9, 86.7]	82.88 [81.9, 83.9]	64.25 [63.4, 65.1]
Majority vote	86.25	85.67	63.72
Calibrated vote	86.70 [86.3, 87.1]	85.87 [85.0, 86.8]	64.02 [63.2, 64.9]
RCCV ( $\rho{=}0.4$ )	86.80	85.97	64.30
QualRCCV ( $\rho{=}0.4,\gamma{=}1$ )	86.87 [86.4, 87.3]	86.07 [85.2, 87.0]	64.33 [63.6, 65.2]

Table 4: Main results across three benchmarks (95% bootstrap CIs, 17 models, 8 families). QualRCCV is the first training-free method to beat calibrated voting on all three benchmarks (all

p{<}0.05

). LCS achieves the largest overall gains (

+0.68\%

VQAv2,

+0.61\%

TextVQA,

+2.45\%

GQA)—significant on all three benchmarks and the only learned method with this property. † 5-fold cross-validated.

Benchmark	T2 %	Cal	HFV	$\Delta$
VQAv2	2.5%	0%	26.0%	+26.0
TextVQA	1.5%	0%	18.3%	+18.3
GQA	6.5%	0%	23.7%	+23.7

Table 5: Misleading tier (T2) recovery across benchmarks (17 models). In every case, calibrated voting achieves 0% on T2 questions where the best model is correct. HFV consistently recovers a large fraction.

Figure 3: Per-family accuracy across benchmarks. HFV helps when family quality is relatively balanced (VQAv2, GQA) but hurts when one family is dramatically weaker (InternVL3 at 49% on TextVQA).

Ensemble	VQAv2	TextVQA	GQA
Balanced (best per family)	86.63	85.13	64.02
Full 17-model (calibrated)	86.70	85.87	64.02
$\Delta$	$-$ 0.07	$-$ 0.74	+0.00

Table 6: Balanced ensemble (one best model per family) vs. full 17-model calibrated ensemble. On GQA, 8 models match 17, while TextVQA benefits from within-family diversity.

Variant	Accuracy	$\Delta$ vs Cal
Calibrated vote	86.70%	—
QualRCCV	86.87%	$+0.17\%$
LCS (full)	87.25%	$+0.55\%$
Feature ablation
w/o quality	87.12%	$+0.42\%$
w/o consensus	87.23%	$+0.53\%$

Table 7: LCS ablation on VQAv2 (17 models, simplified 17-feature variant with GradientBoosting for interpretability; Table 4 reports the enhanced LCS with 80+ features and LightGBM). All variants use 5-fold cross-validation.

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Metric	VQAv2	TextVQA	GQA
Models ( $M$ )	17	17	17
Families ( $F$ )	8	8	8
$\lambda_{1}$ variance	58.0%	50.9%	62.5%
Top-5 variance	75.2%	—	—
Eff. dimensionality	2.86	3.59	2.49
Within-family $r$	0.67	0.59	0.73

Appendix

Table 8: Effective ensemble dimensionality across benchmarks (17 models from 8 families; Section 4.3 ). Only 2.5–3.6 effective independent voters exist among 17 models. The first eigenvalue captures 51–63% of error variance, confirming strong shared failure modes.

Partition	$F$	Accuracy
Per-model	11	86.65
Merged (Qwen)	3	86.36
Original families	5	86.86
Split Qwen2.5	7	86.96
Calibrated (flat)	–	86.73

Appendix

Table 9: Effect of family granularity on VQAv2 HFV accuracy (11-model subset; overall-accuracy weights for comparability across partitions). Splitting Qwen2.5 into training-paradigm sub-families (6 families) yields the best result. Merging families hurts.

$k$	4	5	6	7	8	9	10	11	12
HFV acc. (%)	86.91	86.84	86.75	86.63	86.58	86.61	86.60	86.56	86.42
ARI	0.44	0.29	0.30	0.36	0.42	0.43	0.45	0.51	0.54

Appendix

Table 10: Data-driven family discovery via spectral clustering on the error correlation affinity matrix (

k=4,\ldots,12

clusters) on VQAv2 with 17 models. HFV accuracy for each discovered grouping is compared to true architecture families (86.57%) and calibrated voting (86.70%).

Type	$n$	Cal (%)	HFV (%)	$\Delta$ ( $p$ )
number	6,667	80.08	80.28	$+$ 0.19 (—)
yes/no	6,667	97.19	97.26	$+$ 0.06 (—)
other	6,667	82.83	82.18	$-$ 0.65 (—)

Appendix

Table 11: Per-question-type HFV improvement over calibrated voting on VQAv2. HFV strongly benefits number questions but hurts on free-form other questions.

$k$	Mean gap	% positive	Corr(gap, imbal.)
3	$-$ 0.11	17%	$-$ 0.34
5	$-$ 0.09	36%	$-$ 0.09
7	$+$ 0.01	47%	$+$ 0.09
9	$+$ 0.13	73%	$+$ 0.42
11	$+$ 0.13	100%	—

Appendix

Table 12: HFV

-

calibrated accuracy gap as a function of ensemble size

k

(200 random multi-family subsets per

k

, VQAv2). HFV reliably improves over calibrated voting only when

k\geq 9

models.

Figure 4: Error PCA of model accuracy vectors on VQAv2. Architecture families (color-coded) cluster together in the error landscape, confirming family structure is a real property—not an assumption.

Figure 5: LCS vs. calibrated voting accuracy as a function of ensemble size on VQAv2 and GQA. LCS gains grow with pool size while calibrated voting degrades from within-family redundancy.

Figure 6: Per-question-type LCS improvement over calibrated voting. On VQAv2, number questions benefit most (

+2.01\%

p{<}0.001

) while yes/no and other types gain

<0.1\%

. On GQA, query (

+3.48\%

) and logical (

+1.39\%

) question types show the largest gains.

Vision-Language Models

Vision-language models connect visual inputs with text for retrieval, question answering, generation, and grounded multimodal reasoning.

See all 736 papers

Scaling Vision-Language Models Is Not Enough to Mitigate Bias
Jul 30, 2026Ioannis Sarridis, Ioannis Kompatsiaris, Symeon PapadopoulosVLM Evaluation VLM Robustness
Information Technologies Institute, CERTH Thessaloniki, Greece
Vision Verification Enhanced Fusion of VLMs for Efficient Visual Reasoning
Mar 13, 2026Selim Furkan Tekin, Yichang Xu, Gaowen Liu +3VLM Hallucination Mitigation VLM Reasoning
Georgia Institute of Technology · Cisco Systems
When Vision-Language Models Judge Without Seeing: Exposing Informativeness Bias
Apr 20, 2026Xiaohan Zou, Roshan Sridhar, Mohammadtaher Safarzadeh +1VLM Evaluation Vision-Language Alignment
The Pennsylvania State University · Oracle AI

Jul 30, 2026J/K move · Enter open · S save

Scaling Vision-Language Models Is Not Enough to Mitigate Bias

Ioannis Sarridis, Ioannis Kompatsiaris, Symeon Papadopoulos

Information Technologies Institute, CERTH Thessaloniki, Greece

VLM Evaluation VLM Robustness Vision-Language Models

Open paper AskPDF arXiv

Vision-Language Models (VLMs) such as CLIP are now foundational to multimodal systems, yet their robustness to spurious correlations remains poorly understood at scale. We present the first large-scale empirical study of 194 publicly available VLMs, including 16 model families, covering a wide range of model sizes, 24 training datasets, and three evaluation benchmarks, namely ImageNet (overall performance), CelebA (typical single-attribute bias), and UrbanCars (complex multi-attribute biases). Across these settings, the Spearman correlation between model scale and performance weakens as evaluation shifts from ImageNet (

ρ{=}0.68

) to single-attribute (

ρ{=}0.48

) and further to multi-attribute (

ρ{=}0.05

) bias benchmarks. In contrast, properties of the training data (size and quality) show more consistent relationships with worst-group accuracy across both bias benchmarks. Notably, curated datasets yield improvements of up to 25% over uncurated alternatives at a comparable scale. Finally, the effect of architectural choices (e.g., patch size, image resolution) is highly context-dependent, varying with the nature of the benchmark, including the type of bias and its spatial distribution within images.

Abstract

Vision-Language Models

Explore similar work

Scaling Vision-Language Models Is Not Enough to Mitigate Bias

Vision Verification Enhanced Fusion of VLMs for Efficient Visual Reasoning

When Vision-Language Models Judge Without Seeing: Exposing Informativeness Bias