Foundation-model hubs turn multi-view fusion into a selection problem: from a large heterogeneous encoder pool, which views should be fused, and how many? We show that downstream performance is non-monotonic in the number of fused encoders; later views can be redundant or task-misaligned, causing accuracy to saturate or decline. We formalise this setting as view-set composition and propose KAGES (Kernel-Alignment Greedy Encoder Selector), a label-aware method that orders frozen encoders by their marginal gain in centred kernel-target alignment. KAGES requires no downstream classifier training during selection, evaluates each candidate in O(n2) time independent of encoder dimension, and admits a conditional (1−e−γ) prefix-wise guarantee under monotonicity and a positive submodularity ratio. Across five recognition regimes and low-shot, larger-pool, and full-data protocols, KAGES improves average AULC over full fusion by 3.9, 5.8, and 3.3 points, respectively, and exceeds DPP and facility-location selection in average AULC. Image retrieval exhibits later, task-dependent saturation along the KAGES ordering, while peak-then-decline reproduces in frozen-LLM fusion. These results show that effective large-pool fusion depends on selecting a compact, task-aligned set of views rather than indiscriminately fusing more encoders.
Figures & tables
Figure 1: View-set composition in frozen foundation-model fusion. Left: a heterogeneous pool of frozen vision backbones. Middle: downstream accuracy is non-monotonic in the number of fused views, with most datasets peaking at 3 to 5 encoders and declining, while a single task-aligned encoder beats every fusion on Country211. Right: two failure modes behind the decline. Type I ( representation redundancy ): a new encoder repeats information already captured. Type II ( task-model misalignment ): a representationally distinct encoder injects task-irrelevant cues, so diversity alone is not effectiveness. The marginal-utility theory of Section 2.2 explains the shape from a single gain-cost crossing (Theorem 1 ); the two failure modes are formalised in supplementary App. A.3.6 .
Regime
generic
fine-grained
texture
geo-spatial
symbol
Dataset
CIFAR-100
Pets
DTD
Country211
GTSRB
1 (CLIP)
60.29
85.04
64.41
13.50
61.15
2 (+DINOv2)
80.11
92.86
72.67
9.85
51.54
3 (+ConvNeXt)
80.37
93.75
73.52
9.91
52.73
4 (+SigLIP)
80.18
93.46
74.05
9.82
52.34
5 (+MAE)
80.20
93.65
74.17
9.71
51.97
Table 1: Peak-then-decline at 10-shot across all five recognition regimes. Encoders are added one at a time along the literature-tracked path of Section 2.3 . Each cell is gated-fusion accuracy (%) after the indicated set is in place. Bold : per-column peak.
n=10 -shot
n=100 -shot
n= full data
Dataset (regime)
peak k
Acc
peak k
Acc
peak k
Acc
Δ
CIFAR-100 (generic)
3
80.37
3
84.26
5
87.36
+2
Pets (fine-grained)
3
93.75
4
95.86
5
95.87
+2
DTD (texture)
6
74.18
3
80.78
5
82.53
-1
Country211 (geo-spatial)
1
13.50
1
24.01
1
25.65
0
GTSRB (symbol / digit)
1
61.15
1
82.35
5
86.55
+4
Table 2: n -scaling across all five recognition regimes: peak position along the literature-tracked path at three sample sizes. For each dataset and each of three sample sizes (10 shots per class, 100 shots per class, full training set), we report the peak position along the path and the gated-fusion accuracy (%) at that peak. The right-most column gives the shift Δ= (peak at full data) − (peak at 10-shot). The peak moves right with n on 3 of 5 datasets (CIFAR-100, Pets, GTSRB); Country211 is single-encoder saturated and stays at k=1 ; DTD shows a non-monotonic dependence on n (further discussed in the text). Full per- k curves are in supplementary App. A.3.5 .
Method
C100
Pets
DTD
C211
GTSRB
Avg
Protocol B (16-shot, Concat+MLP)
Fuse-All
78.54
94.06
75.35
10.16
57.18
63.06
Random
77.06
92.97
74.87
11.10
59.16
63.03
Diversity
80.61
90.91
74.38
11.27
59.12
63.26
KAGES (ours)
83.84
94.70
77.82
12.16
66.12
66.93
Δ vs Fuse-All
+5.3
+0.6
+2.5
+2.0
+8.9
+3.9
Table 3: Per-dataset AULC (%) on all three protocols: KAGES beats the multi-view defaults on every dataset, by a wide margin where heterogeneous fusion matters. Against full fusion (Fuse-All), random and diversity selection, KAGES is best in every cell. The Δ rows give KAGES minus Fuse-All: the gain is largest on the structured-symbol task GTSRB ( +8.9 to +15.5 points) and on the geo regime at full data. Stronger subset-selection baselines (DPP, submodular) are reported in the text; KAGES matches or exceeds them too. Means over 5 seeds {17,23,42,71,137} ; bold : best per column within a protocol.
Figure 2: KAGES front-loads high-utility encoders and stays at the top of the multi-view baseline envelope (Protocol F, full data). Per-dataset prefix accuracy along each method’s ordering. KAGES (navy, star at its peak) is plotted over the gray envelope spanning the standard multi-view defaults (full fusion, random, and diversity selection), with their per- k median in gray. KAGES reaches a strong representation at small k on every regime while the defaults climb slowly, with the clearest separation on the texture and symbol tasks. Country211 is the anti-fusion regime in which CLIP alone saturates geolocation, so all methods converge.
Figure 3: Accuracy curves over fused LLM count for MMLU and ARC-Challenge. The gray band spans diversity-greedy and fixed-order baselines, the gray line shows their median, and KAGES is shown in navy with stars marking its peak. The peak-then-decline pattern reproduces at the LLM layer.
Dataset
1
2
3
4
5
6
Peak
CIFAR-100
60.29
80.11
80.37
80.18
80.20
79.92
3
Pets
85.04
92.86
93.75
93.46
93.65
93.49
3
DTD
64.41
72.67
73.52
74.05
74.17
74.18
6
Country211 ⋆
13.50
9.85
9.91
9.82
9.71
9.58
1
GTSRB ⋆
61.15
51.54
52.73
52.34
51.97
52.58
1
Table A.1: 10-shot accuracy (%) as encoders are incrementally added along the literature-tracked path (CLIP → DINOv2 → ConvNeXt → SigLIP → MAE → ResNet-50; gated fusion). These are the per-dataset curves underlying Figure 1 and Table 1 . Bold : peak. ⋆ : CLIP alone is optimal.
Dataset
k=1
3
5
8
10
Peak k
Best R@1
DTD
47.4
66.6
70.1
70.5
74.6
10
74.6
GTSRB
50.4
56.8
55.7
59.2
60.2
10
60.2
Pets
89.8
93.3
93.5
94.3
94.0
8
94.3
Table A.2: Image retrieval Recall@1 (%) vs. k . Retrieval gains are task-dependent and saturate at larger k than classification.
Method
DTD
Flowers
GTSRB
Avg
Single
CLIP zero-shot †
44.3
67.4
–
–
CoOp † ( Zhou et al., 2022 )
68.7
95.7
–
–
Tip-Adapter-F † ( Zhang et al., 2022 )
73.7
97.2
–
–
TaskRes † ( Yu et al., 2023 )
73.4
97.0
–
–
CaFo † ( Zhang et al., 2023 )
74.6
97.5
–
–
Multi
All Models
77.9
99.6
61.6
79.7
Table A.3: Fair 16-shot comparison: multi-model fusion vs. single-model CLIP adaptation (%). All methods use 16 shots per class. † : from original papers.
Method
C100
Pets
DTD
C211
GTSRB
Avg
Fuse-All
78.54
94.06
75.35
10.16
57.18
63.06
Random
77.06
92.97
74.87
11.10
59.16
63.03
Diversity
80.61
90.91
74.38
11.27
59.12
63.26
DPP
83.62
93.21
72.48
11.87
64.88
65.21
Submodular
82.82
93.31
77.64
11.34
65.53
66.13
KAGES (ours)
83.84
94.70
77.82
12.16
66.12
66.93
Table A.4: Full per-dataset AULC (%) on Protocol B (16-shot, Concat+MLP), means over 5 seeds {17,23,42,71,137} . Bold : best per column; underlined : second-best.
Dataset
Method
k=1
2
3
4
5
6
Peak
MMLU-500
KAGES
67.0
75.0
69.0
72.0
71.0
67.0
75.0
diversity-greedy
67.0
66.0
70.0
69.0
71.0
68.0
71.0
fixed order
67.0
68.0
72.0
70.0
69.0
67.0
72.0
ARC-Ch-500
KAGES
88.0
86.0
89.0
91.0
88.0
87.0
91.0
diversity-greedy
88.0
86.0
89.0
90.0
89.0
87.0
90.0
fixed order
85.0
88.0
89.0
85.0
91.0
89.0
91.0
Table A.5: Linear-probe accuracy (%) across k on frozen LLMs. Peak-then-decline reproduces on both tasks. KAGES peaks earlier and higher on MMLU.
Figure A.1: CKA matrices across 6 LLMs. Mistral-Llama CKA reaches 0.82 on MMLU (both English-heavy pretraining), while Gemma (Google, distilled) stays below 0.30 with every other model. This echoes our vision finding: training corpus/objective, not architecture, drives representation similarity.
Classical classifier-combination work distinguishes score-level fusion from hard decision-level voting. We revisit this distinction where independently pretrained, frozen foundation models are composed at inference time for generalized few-shot 3D segmentation. We ask: how much useful semantic information is lost when heterogeneous sources are collapsed to a single class before they can interact? We answer with a same-input semantic-retention intervention. Dense RegionPLC and sparse cross-view SAM3 evidence, model weights, masks, geometry, vocabularies, and fusion rules are frozen; only the number of semantic alternatives retained before interaction is varied via a matched top-k ladder. On 156 held-out ScanNet200 scenes, top-1 reaches 28.47 harmonic-mean (HM) IoU while full distribution fusion reaches 34.87 HM (+6.40, 95% CI [+5.24,+7.64]). The pattern replicates on 50 ScanNet++ scenes: 23.02 vs. 26.50 HM (+3.48, 95% CI [+1.64,+5.93]). The conclusion is robust: full-distribution HM is stable across sparse-source weights 0.3--0.7; alternative operators (max, geometric pooling) also outperform top-1; and a GroundingDINO--SAM2.1 source-replacement diagnostic shows monotonic HM increase from 14.77 to 18.75 with full retention. Calibration diagnostics reveal opposite miscalibration of the two sources, yet correcting calibration does not eliminate the retention advantage. Across datasets and source stacks, most information is recovered by retaining a compact set of plausible alternatives. The contribution is a controlled diagnosis of premature semantic collapse as a repeatable information bottleneck in heterogeneous frozen-model composition.
Silas Kwabla Gah, Ebenezer Owusu
Department of Computer Science University of Ghana, Legon
The choice between cross-attention and concatenation for multimodal fusion remains governed by practitioner intuition rather than principled understanding. In this paper, we demonstrate that feature alignment quality, not data scale alone, is the primary determinant of which fusion strategy excels. Through controlled experiments on Flickr8k using two feature extraction backbones (ResNet18 and CLIP ViT-B/32), we show that concatenation outperforms cross-attention by 4.1-5.1 percentage points across all tested scales (2048-16384 samples) when features are pre-aligned by a vision-language pretraining objective. We provide a theoretical explanation grounded in sample complexity analysis: concatenation requires O(d_v + d_t) samples to learn its fusion projection, while cross-attention requires O(d_v * d_t) samples to learn bilinear attention weights, over 256 times as many for 512-dimensional CLIP features. When features are already aligned, the approximation error gap between the two methods vanishes, and concatenation's sample efficiency dominates at all practical dataset sizes. An alignment degradation study confirms a monotonic trend: as feature alignment degrades, concatenation's advantage grows from 1.3% to 2.8%. These findings provide a principled decision framework for fusion method selection in multimodal systems, with direct implications for the design of Multimodal Large Language Models.
Zhiqiang Zhou, Xuezhen Xie
Hunan Chemical Industry Vocational and Technical College
Foundation models are vital tools in various Computer Vision applications. They take as input a single RGB image and output a deep feature representation that is useful for various applications. However, in case we have multiple views of the same 3D scene, they operate on each image independently and do not always produce consistent features for the same 3D point. We propose a way to convert a Foundation Model into a Multi-View Foundation Model. Such a model takes as input a set of images and outputs a feature map for each image such that the features of corresponding points are as consistent as possible. This approach bypasses the need to build a consistent 3D model of the features and allows direct manipulation in the image space. Specifically, we show how to augment Transformers-based foundation models (i.e., DINO, SAM, CLIP) with intermediate 3D-aware attention layers that help match features across different views. As leading examples, we show surface normal estimation and multi-view segmentation tasks. Quantitative experiments show that our method improves feature matching considerably compared to current foundation models.