When More Foundation Models Means Less: Diagnosing and Addressing Multi-View Fusion Failure

Yibo Liu; Bowen Jiang

cs.CVAug 18, 2026

When More Foundation Models Means Less: Diagnosing and Addressing Multi-View Fusion Failure

Authors: Yibo Liu, Bowen Jiang

Organizations: Hainan Campus, Beijing University of Posts and Telecommunications Lingshui, Hainan, China

Abstract

Foundation-model hubs turn multi-view fusion into a selection problem: from a large heterogeneous encoder pool, which views should be fused, and how many? We show that downstream performance is non-monotonic in the number of fused encoders; later views can be redundant or task-misaligned, causing accuracy to saturate or decline. We formalise this setting as view-set composition and propose KAGES (Kernel-Alignment Greedy Encoder Selector), a label-aware method that orders frozen encoders by their marginal gain in centred kernel-target alignment. KAGES requires no downstream classifier training during selection, evaluates each candidate in $\mathcal{O}(n^2)$ time independent of encoder dimension, and admits a conditional $(1-e^{-γ})$ prefix-wise guarantee under monotonicity and a positive submodularity ratio. Across five recognition regimes and low-shot, larger-pool, and full-data protocols, KAGES improves average AULC over full fusion by 3.9, 5.8, and 3.3 points, respectively, and exceeds DPP and facility-location selection in average AULC. Image retrieval exhibits later, task-dependent saturation along the KAGES ordering, while peak-then-decline reproduces in frozen-LLM fusion. These results show that effective large-pool fusion depends on selecting a compact, task-aligned set of views rather than indiscriminately fusing more encoders.

Figures & tables

Figure 1: View-set composition in frozen foundation-model fusion. Left: a heterogeneous pool of frozen vision backbones. Middle: downstream accuracy is non-monotonic in the number of fused views, with most datasets peaking at $3$ to $5$ encoders and declining, while a single task-aligned encoder beats every fusion on Country211. Right: two failure modes behind the decline. Type I ( representation redundancy ): a new encoder repeats information already captured. Type II ( task-model misalignment ): a representationally distinct encoder injects task-irrelevant cues, so diversity alone is not effectiveness. The marginal-utility theory of Section 2.2 explains the shape from a single gain-cost crossing (Theorem 1 ); the two failure modes are formalised in supplementary App. A.3.6 .

Regime	generic	fine-grained	texture	geo-spatial	symbol
Dataset	CIFAR-100	Pets	DTD	Country211	GTSRB
1 (CLIP)	60.29	85.04	64.41	$\mathbf{13.50}$	$\mathbf{61.15}$
2 (+DINOv2)	80.11	92.86	72.67	9.85	51.54
3 (+ConvNeXt)	$\mathbf{80.37}$	$\mathbf{93.75}$	73.52	9.91	52.73
4 (+SigLIP)	80.18	93.46	74.05	9.82	52.34
5 (+MAE)	80.20	93.65	74.17	9.71	51.97

Table 1: Peak-then-decline at 10-shot across all five recognition regimes. Encoders are added one at a time along the literature-tracked path of Section 2.3 . Each cell is gated-fusion accuracy (%) after the indicated set is in place. Bold : per-column peak.

Dataset (regime)	peak $k$	Acc	peak $k$	Acc	peak $k$	Acc	$\Delta$
	$n=10$ -shot		$n=100$ -shot		$n=$ full data
CIFAR-100 (generic)	3	80.37	3	84.26	5	87.36	+2
Pets (fine-grained)	3	93.75	4	95.86	5	95.87	+2
DTD (texture)	6	74.18	3	80.78	5	82.53	-1
Country211 (geo-spatial)	1	13.50	1	24.01	1	25.65	0
GTSRB (symbol / digit)	1	61.15	1	82.35	5	86.55	+4

Table 2:

n

-scaling across all five recognition regimes: peak position along the literature-tracked path at three sample sizes. For each dataset and each of three sample sizes (10 shots per class, 100 shots per class, full training set), we report the peak position along the path and the gated-fusion accuracy (%) at that peak. The right-most column gives the shift

\Delta=

(peak at full data)

-

(peak at 10-shot). The peak moves right with

n

3

5

datasets (CIFAR-100, Pets, GTSRB); Country211 is single-encoder saturated and stays at

k=1

; DTD shows a non-monotonic dependence on

n

(further discussed in the text). Full per-

k

curves are in supplementary App. A.3.5 .

Protocol B (16-shot, Concat+MLP)
Method	C100	Pets	DTD	C211	GTSRB	Avg
Fuse-All	78.54	94.06	75.35	10.16	57.18	63.06
Random	77.06	92.97	74.87	11.10	59.16	63.03
Diversity	80.61	90.91	74.38	11.27	59.12	63.26
KAGES (ours)	83.84	94.70	77.82	12.16	66.12	66.93
$\Delta$ vs Fuse-All	+5.3	+0.6	+2.5	+2.0	+8.9	+3.9

Table 3: Per-dataset AULC (%) on all three protocols: KAGES beats the multi-view defaults on every dataset, by a wide margin where heterogeneous fusion matters. Against full fusion (Fuse-All), random and diversity selection, KAGES is best in every cell. The

\Delta

rows give KAGES minus Fuse-All: the gain is largest on the structured-symbol task GTSRB (

+8.9

+15.5

points) and on the geo regime at full data. Stronger subset-selection baselines (DPP, submodular) are reported in the text; KAGES matches or exceeds them too. Means over 5 seeds

\{17,23,42,71,137\}

; bold : best per column within a protocol.

Figure 2: KAGES front-loads high-utility encoders and stays at the top of the multi-view baseline envelope (Protocol F, full data). Per-dataset prefix accuracy along each method’s ordering. KAGES (navy, star at its peak) is plotted over the gray envelope spanning the standard multi-view defaults (full fusion, random, and diversity selection), with their per-

k

median in gray. KAGES reaches a strong representation at small

k

on every regime while the defaults climb slowly, with the clearest separation on the texture and symbol tasks. Country211 is the anti-fusion regime in which CLIP alone saturates geolocation, so all methods converge.

Figure 3: Accuracy curves over fused LLM count for MMLU and ARC-Challenge. The gray band spans diversity-greedy and fixed-order baselines, the gray line shows their median, and KAGES is shown in navy with stars marking its peak. The peak-then-decline pattern reproduces at the LLM layer.

Dataset	1	2	3	4	5	6	Peak
CIFAR-100	60.29	80.11	80.37	80.18	80.20	79.92	3
Pets	85.04	92.86	93.75	93.46	93.65	93.49	3
DTD	64.41	72.67	73.52	74.05	74.17	74.18	6
Country211 $\star$	13.50	9.85	9.91	9.82	9.71	9.58	1
GTSRB $\star$	61.15	51.54	52.73	52.34	51.97	52.58	1

Table A.1: 10-shot accuracy (%) as encoders are incrementally added along the literature-tracked path (CLIP

\to

DINOv2

\to

ConvNeXt

\to

SigLIP

\to

MAE

\to

ResNet-50; gated fusion). These are the per-dataset curves underlying Figure 1 and Table 1 . Bold : peak.

\star

: CLIP alone is optimal.

Dataset	$k=1$	3	5	8	10	Peak $k$	Best R@1
DTD	47.4	66.6	70.1	70.5	74.6	10	74.6
GTSRB	50.4	56.8	55.7	59.2	60.2	10	60.2
Pets	89.8	93.3	93.5	94.3	94.0	8	94.3

Table A.2: Image retrieval Recall@1 (%) vs.

k

. Retrieval gains are task-dependent and saturate at larger

k

than classification.

	Method	DTD	Flowers	GTSRB	Avg
Single	CLIP zero-shot $\dagger$	44.3	67.4	–	–
	CoOp $\dagger$ ( Zhou et al., 2022 )	68.7	95.7	–	–
	Tip-Adapter-F $\dagger$ ( Zhang et al., 2022 )	73.7	97.2	–	–
	TaskRes $\dagger$ ( Yu et al., 2023 )	73.4	97.0	–	–
	CaFo $\dagger$ ( Zhang et al., 2023 )	74.6	97.5	–	–
Multi	All Models	77.9	99.6	61.6	79.7

Table A.3: Fair 16-shot comparison: multi-model fusion vs. single-model CLIP adaptation (%). All methods use 16 shots per class.

\dagger

: from original papers.

Method	C100	Pets	DTD	C211	GTSRB	Avg
Fuse-All	78.54	94.06	75.35	10.16	57.18	63.06
Random	77.06	92.97	74.87	11.10	59.16	63.03
Diversity	80.61	90.91	74.38	11.27	59.12	63.26
DPP	83.62	93.21	72.48	11.87	64.88	65.21
Submodular	82.82	93.31	77.64	11.34	65.53	66.13
KAGES (ours)	83.84	94.70	77.82	12.16	66.12	66.93

Table A.4: Full per-dataset AULC (%) on Protocol B (16-shot, Concat+MLP), means over 5 seeds

\{17,23,42,71,137\}

. Bold : best per column; underlined : second-best.

Dataset	Method	$k=1$	2	3	4	5	6	Peak
MMLU-500	KAGES	67.0	75.0	69.0	72.0	71.0	67.0	75.0
	diversity-greedy	67.0	66.0	70.0	69.0	71.0	68.0	71.0
	fixed order	67.0	68.0	72.0	70.0	69.0	67.0	72.0
ARC-Ch-500	KAGES	88.0	86.0	89.0	91.0	88.0	87.0	91.0
	diversity-greedy	88.0	86.0	89.0	90.0	89.0	87.0	90.0
	fixed order	85.0	88.0	89.0	85.0	91.0	89.0	91.0

Table A.5: Linear-probe accuracy (%) across

k

on frozen LLMs. Peak-then-decline reproduces on both tasks. KAGES peaks earlier and higher on MMLU.

Figure A.1: CKA matrices across 6 LLMs. Mistral-Llama CKA reaches $0.82$ on MMLU (both English-heavy pretraining), while Gemma (Google, distilled) stays below $0.30$ with every other model. This echoes our vision finding: training corpus/objective, not architecture, drives representation similarity.

Multi-View Feature Fusion

See all 23 papers

Beyond Argmax: A Mechanistic Study of Semantic Retention in Frozen Foundation-Model Composition for Generalized Few-Shot 3D Segmentation
Sep 10, 2026Silas Kwabla Gah, Ebenezer Owusu3D Foundation Models Ensemble Learning
Department of Computer Science University of Ghana, Legon
Feature Alignment Determines Fusion Strategy: A Comparative Study of Cross-Attention and Concatenation in Multimodal Learning
May 31, 2026Zhiqiang Zhou, Xuezhen XieCross-Attention Multimodal Fusion
Hunan Chemical Industry Vocational and Technical College
Multi-View Foundation Models
Dec 17, 2025Leo Segre, Or Hirschorn, Shai AvidanMulti-View Consistency Vision Foundation Model Adaptation
Tel Aviv University

Sep 10, 2026J/K move · Enter open · S save

Beyond Argmax: A Mechanistic Study of Semantic Retention in Frozen Foundation-Model Composition for Generalized Few-Shot 3D Segmentation

Silas Kwabla Gah, Ebenezer Owusu

Department of Computer Science University of Ghana, Legon

3D Foundation Models Ensemble Learning Information Bottleneck 3D Semantic Segmentation Few-Shot Semantic Segmentation

Open paper AskPDF arXiv

Classical classifier-combination work distinguishes score-level fusion from hard decision-level voting. We revisit this distinction where independently pretrained, frozen foundation models are composed at inference time for generalized few-shot 3D segmentation. We ask: how much useful semantic information is lost when heterogeneous sources are collapsed to a single class before they can interact? We answer with a same-input semantic-retention intervention. Dense RegionPLC and sparse cross-view SAM3 evidence, model weights, masks, geometry, vocabularies, and fusion rules are frozen; only the number of semantic alternatives retained before interaction is varied via a matched top-k ladder. On 156 held-out ScanNet200 scenes, top-1 reaches 28.47 harmonic-mean (HM) IoU while full distribution fusion reaches 34.87 HM (+6.40, 95% CI [+5.24,+7.64]). The pattern replicates on 50 ScanNet++ scenes: 23.02 vs. 26.50 HM (+3.48, 95% CI [+1.64,+5.93]). The conclusion is robust: full-distribution HM is stable across sparse-source weights 0.3--0.7; alternative operators (max, geometric pooling) also outperform top-1; and a GroundingDINO--SAM2.1 source-replacement diagnostic shows monotonic HM increase from 14.77 to 18.75 with full retention. Calibration diagnostics reveal opposite miscalibration of the two sources, yet correcting calibration does not eliminate the retention advantage. Across datasets and source stacks, most information is recovered by retaining a compact set of plausible alternatives. The contribution is a controlled diagnosis of premature semantic collapse as a repeatable information bottleneck in heterogeneous frozen-model composition.

Abstract

Multi-View Feature Fusion

Explore similar work

Beyond Argmax: A Mechanistic Study of Semantic Retention in Frozen Foundation-Model Composition for Generalized Few-Shot 3D Segmentation

Feature Alignment Determines Fusion Strategy: A Comparative Study of Cross-Attention and Concatenation in Multimodal Learning

Multi-View Foundation Models