Multimodal LLMs are typically trained by first aligning a vision encoder with a language model via projector training on large image-caption datasets, followed by joint training of the full multimodal model. Despite its widespread use, the role of the projector training stage remains poorly understood. We find that (i) using only 10 to 20% of the data for training the projector already recovers most downstream performance gains, and (ii) projectors trained on different datasets can be linearly interpolated while largely preserving performance. These results suggest that projector training mainly performs a coarse alignment, which many different projectors achieve equally well. Motivated by these findings, we propose PORTAL, a training-free projector initialization that requires neither paired image-caption data nor gradient-based optimization, but only per-modality summary statistics. PORTAL computes an optimal-transport-based map between Gaussian approximations of the vision and language feature distributions on a shared principal subspace, so that the projector training stage can be skipped entirely. Without any paired data, PORTAL matches the standard pipeline trained on 558k image-caption pairs across six LLM backbones, two vision encoders, and 16 benchmarks. It outperforms that pipeline in 3 of 7 (vision encoder, LLM) settings and reaches an average 98% of its score across the other four.
Figures & tables
Figure 1: Projector alignment as a coarse alignment with a broad solution space. ( Left ) Standard pipeline: during projector alignment, the vision encoder and language model are frozen, and only the projector is trained to map visual tokens into a representation space compatible with the LLM using image-caption pairs. ( Middle ) We show that projectors trained on different image-caption datasets can be linearly interpolated while largely preserving downstream performance, indicating that there is a large solution space for the weights of the projector. ( Right ) This observation motivates a closed-form solution of projector alignment. We propose a training-free projector PORTAL constructed by optimal transport between the two modalities’ marginal statistics.
Figure 2: Stage-I gain saturates early. Relative gain in downstream performance over the no-pretraining baseline by Cambrian-1 category, as a function of Stage-I data fraction on LCS-558k (left) and ShareGPT-1.2M (right). OCR and Chart benefit the most, while Knowledge and Vision-Centric scores saturate early.
Figure 3: Cross-data projectors are linearly interpolable. Cross-data interpolation Wα=(1−α)WLCS+αWShareGPT , evaluated per Cambrian-1 category. The optimal α varies by category, but every interior blend outperforms both endpoints on the aggregate score, with the α=0.5 mixture peaking above either trained projector.
Knowledge
General
OCR
Vision
LLM
Scale
Vision
Init
AI2D
SciQA
MathV
MMMU
GQA
MME
SEED L
MMB
TxtVQA
OCR-Bv2
DocVQA
ChartQA
CV-2D
CV-3D
BLINK
RealWQ
Avg
%BL
Vicuna 1.5
7B
CLIP
Baseline †
54.5
67.9
26.1
34.3
66.6
74.5
67.7
63.2
57.3
12.5
22.0
16.5
54.7
58.3
37.4
54.9
48.0
100.0
Random
51.5
69.1
27.2
32.8
64.2
69.2
57.2
58.1
52.8
11.0
17.1
15.2
52.5
53.8
34.7
51.9
44.9 ±0.60
93.5 ±1.6
PORTAL
53.4
68.2
27.4
34.4
66.4
69.9
67.9
60.4
55.1
12.7
20.5
16.3
56.2
60.7
36.0
53.5
47.4 ±0.56
98.8 ±1.5
SigLIP
Baseline †
55.8
68.0
27.2
34.0
68.7
73.6
69.1
66.0
61.5
13.3
25.3
19.1
57.5
60.8
28.1
56.6
49.0
100.0
Random
51.8
67.9
26.7
35.1
56.8
60.6
46.5
52.3
43.8
7.9
8.9
12.8
47.1
57.5
25.6
45.9
40.5 ±0.58
82.5 ±1.1
Table 1: PORTAL generalizes across LLM backbones and vision encoders. All configurations use identical Stage-II instruction tuning (LLaVA-665k [ 2 ] ). PORTAL requires zero Stage-I paired captions and zero Stage-I gradient steps. Baseline † : uses 558k paired captions for Stage-I pretraining. Bold : best among zero training data methods. Avg : unweighted mean of the 16 per-benchmark scores. %BL: this average as a percentage of the trained baseline (BL), which is therefore 100% . Subscripts on the Avg and %BL columns give the 95% bootstrap confidence-interval half-width over per-sample outcomes, from 1000 resamples paired across methods. LLM backbones: Vicuna 1.5 7B [ 36 ] , LLaMA 3.1 8B [ 37 ] , Qwen 2.5 7B [ 38 ] , LLaMA 3.2 1B [ 37 ] , Qwen 2.5 0.5B [ 38 ] , OLMo 2 1B [ 39 ] . Vision encoders: CLIP ViT-L/14-336 [ 40 ] , SigLIP SO400M [ 41 ] . L : 500-sample Lite subset from lmms-eval [ 42 ] .
Figure 4: Alignment compute efficiency on Vicuna 1.5 7B/CLIP. Average over the 16 benchmarks as a function of the projector alignment budget, with identical Stage-II tuning on LLaVA-665k. PORTAL stays above Random at every budget. The marked point indicates the crossover at roughly 4 k captions, where PORTAL matches the 558 k-trained baseline at ∼135× fewer alignment FLOPs. The horizontal dashed line shows the 558 k baseline.
Projector
Init
Avg
%BL
MLP
Trained †
41.6
100.0
Random
41.5
99.9
PORTAL
44.2
106.2
Linear
Trained †
39.9
100.0
Random
37.6
94.2
PORTAL
44.7
111.8
Table 2: Linear vs. MLP-2x-GELU projector on LLaMA 3.2 1B/CLIP. Full per-benchmark breakdown in Appendix J .
Target
Avg
%BL
Trained †
51.3
100.0
Random
51.4
100.2
ℓ=0 (embed)
52.5
102.3
ℓ=4
52.3
101.9
ℓ=8
52.0
101.4
Table 3: Layer ablation for PORTAL-ctx on LLaMA 3.1 8B/CLIP. Avg over the 16-benchmark suite, %BL relative to trained Stage-I.
Appendix figures & tables17 assets
Supplementary material from the paper’s appendix.
Appendix
Stage I (Pretrain)
Stage II (Finetune)
Learning rate
1×10−3
2×10−5
Schedule
Cosine
Cosine
Warmup
200 steps
3% ratio
Weight decay
5×10−2
0
Per-device batch size
32
4
Gradient accumulation
1
4
Appendix
Table 4: Training hyperparameters. Identical to LLaVA-1.5.
Dataset
Used for
License
Source
Training data
LCS-558k
Stage-I pretrain
CC-BY-4.0
[ 2 ]
ShareGPT-1.2M
Stage-I pretrain
CC-BY-NC-4.0
[ 44 ]
LLaVA-665k
Stage-II finetune
CC-BY-4.0
[ 2 ]
Cambrian-737k
PORTAL image pool
Apache-2.0
[ 7 ]
COCO val2017
PORTAL image pool
CC-BY-4.0
[ 45 ]
Appendix
Table 5: Datasets used in our experiments.
Category
Benchmark
Samples
Metric
Knowledge
AI2D [ 46 ]
3,088
Exact match
ScienceQA [ 47 ]
2,017
Exact match
MathVista [ 48 ]
1,000
LLM-as-judge
MMMU [ 49 ]
900
Accuracy
General
GQA [ 50 ]
12,578
Exact match
MME [ 51 ]
2,374
Perception / 20
Appendix
Table 6: Evaluation benchmarks grouped by Cambrian-1 category. L = Lite subset (500 samples).
Knowledge
General
OCR
Vision
Pretrain
AI2D
SciQA
MathV
MMMU
GQA
MME
SEED L
MMB
TxtVQA
OCR-Bv2
DocVQA
ChartQA
CV-2D
CV-3D
BLINK
RealWQ
Avg
%BL
0%
34.5
47.4
25.2
26.6
51.2
49.7
41.6
10.1
17.0
7.0
8.9
10.3
36.6
51.7
35.3
41.6
30.9
80.9
10%
37.5
53.0
24.3
28.0
56.5
56.4
58.6
39.3
32.1
7.8
13.1
10.6
43.7
53.2
35.7
44.4
37.1
97.1
20%
39.1
55.4
24.4
27.6
56.8
58.7
59.0
39.0
33.6
8.3
14.2
11.3
44.6
51.2
35.5
44.2
37.7
98.5
30%
39.1
53.7
25.8
29.3
56.6
57.6
58.6
37.7
34.8
7.6
14.9
10.6
44.6
51.8
35.7
44.3
37.7
98.5
40%
39.2
55.2
25.9
29.1
56.7
58.2
59.2
41.8
35.2
8.1
15.0
10.9
45.6
52.9
34.8
44.6
38.3
100.1
Appendix
Table 7: Per-benchmark results for each LCS-558k data fraction. Architecture: CLIP ViT-L/14 (336 px) and MobileLLaMA 1.4B [ 61 ] , with a two-layer MLP projector ( 1024→2048→2048 ). Rows: fraction of the 558 k-sample LCS corpus used for Stage-I pretraining ( 0% corresponds to no Stage-I, random projector init). Columns: 16 benchmarks grouped into four categories adapted from the Cambrian-1 taxonomy (Appendix C ), followed by the overall average and its ratio to the 100% reference. All scores are normalized to 0 – 100 (MME is divided by 20 , others are already percentages). L : 500 -sample Lite subset from lmms-eval [ 42 ] .
Knowledge
General
OCR
Vision
Pretrain
AI2D
SciQA
MathV
MMMU
GQA
MME
SEED L
MMB
TxtVQA
OCR-Bv2
DocVQA
ChartQA
CV-2D
CV-3D
BLINK
RealWQ
Avg
%BL
0%
34.5
47.4
25.2
26.6
51.2
49.7
41.6
10.1
17.0
7.0
8.9
10.3
36.6
51.7
35.3
41.6
30.9
81.4
10%
38.4
53.7
24.3
28.1
55.1
54.5
53.5
50.0
29.6
6.9
12.5
11.2
40.8
52.4
35.2
43.1
36.8
97.0
20%
37.9
54.2
24.1
28.1
55.7
55.1
55.4
54.5
30.7
6.5
12.7
11.1
41.7
53.8
32.3
44.1
37.4
98.4
30%
38.7
53.9
24.4
28.3
56.4
55.7
56.4
51.5
31.1
6.8
13.1
11.2
40.5
52.9
35.6
45.2
37.6
99.0
40%
38.0
53.9
24.8
28.4
56.4
57.5
57.2
50.8
31.4
6.8
13.2
11.6
44.4
52.8
35.3
46.4
38.1
100.2
Appendix
Table 8: Per-benchmark results for each ShareGPT-1.2M data fraction. Layout matches Table 7 but with the ShareGPT-1.2M corpus used for Stage-I pretraining.
Initialization
Avg16
% of Baseline
Stage-I trained baseline
48.03
100.0
Random
44.90
93.5
CORAL
46.36
96.5
Gromov-Wasserstein
47.06
98.0
PORTAL
47.43
98.8
Appendix
Table 9: Comparison with alternative alignment methods. All methods except the conventional Stage-I baseline require no paired image-caption samples.
Method
Paired Samples
Projector Training
Construction
Random
No
No
Random initialization
CORAL
No
No
Closed form
GW
No
No
Iterative optimization
PORTAL
No
No
Closed form
Appendix
Table 10: Characteristics of the unpaired initialization methods considered in Table 9 .
LLM
Vision Encoder
Knowledge
General
OCR
Vision
Overall
Vicuna 1.5 7B
CLIP
100.3
97.3
96.6
100.5
98.8
Vicuna 1.5 7B
SigLIP
100.6
98.1
97.6
101.0
99.3
LLaMA 3.1 8B
CLIP
103.6
100.7
103.9
103.8
102.7
LLaMA 3.2 1B
CLIP
102.5
109.8
112.1
103.1
106.2
Qwen 2.5 7B
CLIP
98.5
99.4
92.6
99.3
98.3
Qwen 2.5 0.5B
CLIP
100.9
113.2
121.1
97.9
106.1
Appendix
Table 11: PORTAL performance as a percentage of the corresponding Stage-I-trained baseline, broken down by task category. The last row averages the retention rate across the seven (vision encoder, LLM) configurations.
Category
Baseline
PORTAL
Δ
95% CI
Knowledge
45.95
46.68
+0.72
[−0.40,+1.88]
General
59.60
59.88
+0.27
[−0.71,+1.27]
OCR
30.28
31.45
+1.18
[+0.67,+1.68]
Vision
50.12
51.28
+1.16
[−0.13,+2.46]
Overall
46.49
47.32
+0.83
[+0.36,+1.35]
Appendix
Table 12: Performance under large-scale Stage-II multimodal instruction tuning with 3.2M samples. The confidence interval reports the 95% CI for the difference between PORTAL and the Stage-I-trained baseline.
LLM
Vision Encoder
Baseline
Random
PORTAL
Vicuna 1.5 7B
CLIP
85.5
85.3
85.7 (+0.2)
Qwen 2.5 7B
CLIP
85.7
85.5
86.4 (+0.7)
LLaMA 3.1 8B
CLIP
84.0
84.2
84.9 (+0.9)
Vicuna 1.5 7B
SigLIP
84.6
79.1
85.1 (+0.5)
OLMo 2 1B
CLIP
85.3
84.8
85.5 (+0.2)
LLaMA 3.2 1B
CLIP
84.1
84.0
86.2 (+2.1)
Appendix
Table 13: POPE F1 scores across all seven (vision encoder, LLM) configurations. Numbers in parentheses indicate the improvement of PORTAL over the conventionally Stage-I-trained baseline.
Task
Baseline
Random
PORTAL
AI2D
49.3
46.7
48.0
ScienceQA
68.2
65.2
66.2
MathVista
26.9
25.9
26.2
MMMU
32.3
34.6
33.9
GQA
60.6
59.7
60.7
MME-P
60.8
59.6
65.4
Appendix
Table 14: Performance with the higher-capacity C-Abstractor connector. Despite substantially increasing connector capacity and introducing explicit token mixing and sequence compression, PORTAL remains essentially indistinguishable from conventional Stage-I training.
Type
Distribution
n
d
deff
Mardia b1
Exc. kurt.
AD rej.
Source
CLIP ViT-L/14 patches
5,000
1,024
4.5
0.0022
0.487
100%
Source
SigLIP SO400M patches
5,000
1,152
50.9
0.0026
0.278
100%
Target
Vicuna 7B embedding table
5,000
4,096
1,429.6
0.0005
0.071
84%
Target
LLaMA 3.1 8B embedding table
5,000
4,096
1,713.2
0.0005
0.131
98%
Target
Qwen 2.5 7B embedding table
5,000
3,584
830.1
0.0004
0.114
100%
Appendix
Table 15: Distributional audit on top k=128 principal components. The columns most relevant to the argument in Section 4 are deff and Mardia b1 . The Anderson-Darling rejection rate (AD rej.) and the excess kurtosis identify where the deviation from Gaussianity is concentrated.
Encoder
LLM
Scale
Source
Random
PORTAL
Target
PORTAL deff
Random Avg
PORTAL Avg
Trained Avg
CLIP
Vicuna 7B
pop
0.234
0.234
0.100
0.056
266.7
44.9
47.4
48.0
intra
0.252
0.251
0.235
0.056
266.7
44.9
47.4
48.0
CLIP
LLaMA 3.1 8B
pop
0.234
0.234
0.101
0.057
257.4
50.4
52.2
50.8
intra
0.252
0.251
0.234
0.057
257.4
50.4
52.2
50.8
CLIP
Qwen 2.5 7B
pop
0.234
0.234
0.094
0.054
289.3
48.6
51.7
52.6
intra
0.252
0.251
0.245
0.054
289.3
48.6
51.7
52.6
Appendix
Table 16: Cosine dispersion ω of projector outputs and downstream average. For each (encoder, LLM) pair, Source is ω of the raw source patches, Random and PORTAL are ω after applying the corresponding projector, and Target is ω of the LLM’s input embedding table restricted to the top- 512 principal subspace that PORTAL matches. Random and PORTAL denote the first layer W1x+b1 of each projector. PORTAL deff reports the effective dimension of the PORTAL output cloud. Downstream Avg columns from Table 1 .
LLM
Method
∥Δμ∥
Rel.
Spec. err.
deff
cv
ω (pop)
Vicuna 7B
target
0.000
0.000
0.000
319.7
0.123
0.056
Random+GELU
7.144
47.33
24.25
3.6
0.682
0.212
PORTAL (preact)
0.021
0.142
0.066
266.7
0.281
0.100
PORTAL+GELU
0.077
0.508
0.470
266.9
0.281
0.100
Trained Stage-I
33.18
219.8
142.4
2.4
1.047
0.232
LLaMA 3.1 8B
target
0.000
0.000
0.000
308.2
0.135
0.057
Appendix
Table 17: Post-projection statistics on CLIP patch tokens. Distributional fit to the target distribution for four projector constructions, plus a target row for reference. Smaller ∥Δμ∥ and spectrum error indicate a closer fit to the target. The deff and ω values should be read against the target row of the same LLM.
Knowledge
General
OCR
Vision
Pool
Src
#Img
AI2D
SciQA
MathV
MMMU
GQA
MME
SEED L
MMB
OCR-Bv2
DocVQA
ChartQA
CV-2D
CV-3D
BLINK
RealWQ
Avg
%BL
Baseline †
LCS
558k
46.5
66.2
26.9
33.6
59.7
58.4
56.2
46.1
10.1
12.8
10.8
53.5
56.4
37.2
47.3
41.4
100.0
Random
–
0
46.6
66.9
26.3
34.2
59.8
58.1
56.0
45.2
10.4
12.8
11.0
54.2
54.9
37.4
46.9
41.4
99.8
PORTAL
Cambrian
4096
48.2
66.7
26.9
34.3
62.0
64.3
59.6
53.6
10.8
16.8
12.0
53.5
55.8
37.7
49.4
43.4
104.8
PORTAL
COCO
1024
47.6
67.1
29.2
33.7
62.1
66.0
59.6
54.2
11.4
16.3
12.0
54.6
57.3
38.9
49.7
44.0
106.1
Appendix
Table 18: Image-pool ablation for the PORTAL construction, per-benchmark breakdown. Setup: LLaMA 3.2 1B/CLIP, Stage-II on LLaVA-665k. Both PORTAL variants use rank k=512 and require no paired captions. Bold : best between the two PORTAL rows per benchmark. Avg and %BL are computed over the 15 benchmarks shown.
Knowledge
General
OCR
Vision
Projector
Init
AI2D
SciQA
MathV
MMMU
GQA
MME
SEED L
MMB
TxtVQA
OCR-Bv2
DocVQA
ChartQA
CV-2D
CV-3D
BLINK
RealWQ
Avg
%BL
MLP-2x-GELU
Baseline †
46.5
66.2
26.9
33.6
59.7
58.4
56.2
46.1
43.7
10.1
12.8
10.8
53.5
56.4
37.2
47.3
41.6
100.0
Random
46.6
66.9
26.3
34.2
59.8
58.1
56.0
45.2
43.8
10.4
12.8
11.0
54.2
54.9
37.4
46.9
41.5
99.9
PORTAL
47.6
67.1
29.2
33.7
62.1
66.0
59.6
54.2
47.1
11.4
16.3
12.0
54.6
57.3
38.9
49.7
44.2
106.2
Linear
Baseline †
46.5
65.5
27.1
33.3
54.7
53.1
47.3
49.8
42.6
10.1
11.0
10.0
50.1
54.4
36.8
46.7
39.9
100.0
Random
46.0
63.2
24.6
30.9
53.0
52.8
46.1
43.8
36.8
7.9
7.3
8.8
45.6
52.8
37.7
44.4
37.6
94.2
Appendix
Table 19: Linear versus MLP-2x-GELU projector, per-benchmark breakdown. Setup: LLaMA 3.2 1B/CLIP, Stage-II on LLaVA-665k. Each architecture is anchored to its own trained baseline for %BL. Bold : best between Random and PORTAL within an architecture. Underline : second best.
Figure 5: PCA rank sensitivity on LLaMA 3.2 1B/CLIP. Average over the 16 benchmarks as a function of the rank k used in the PORTAL construction, with identical Stage-II tuning. All ranks outperform the trained baseline ( 41.6 ). k=512 is optimal ( 44.2 ).
The typical training process of a multimodal large language model (MLLM) involves adapting both the language model backbone and the projector between the backbone and a modality-specific encoder. We ask whether fine-tuning the backbone of an MLLM is necessary to adapt it to a new modality. Through experiments on 3D MLLMs, we find that training only the projector is sufficient to achieve strong multimodal performance relative to existing baseline models and our jointly trained MLLMs with the same encoder and backbone. We also show that joint training leads to undesirable drift in existing capabilities of the language model, which projector-only training avoids by definition. Furthermore, projector-only training has approximately twice the training sample throughput of joint training. We validate our findings across different language model backbones via 3D classification and captioning benchmarks as well as standard benchmarks evaluating language, vision, and spatial reasoning capabilities.
Training a small projector between a frozen vision encoder and language model is an established approach to multimodal learning. As the parameter count of language models scales dramatically, we revisit which vision capabilities this approach can add while keeping their pretrained weights fixed. Here we train a 50M parameter projector from the vision encoder of Kimi K2.6 to GLM 5.2 and 5.3, both models without native vision capabilities, and further present a reproducible recipe for training these adapters at scale. We study the following: (a) how vision capabilities of multimodal models scale as purely the language model side scales, and (b) what specific vision capabilities are able to be imbued into a pure language model at scale, and which ones remain limited. We evaluate on MMMU-Pro and BLINK, examining both overall performance and results on individual visual tasks.
Jaehoon Lee, Harry Partridge, Mudith Jayasekara +3
Despite the impressive results achieved by multimodal large language models (MLLMs), their training typically relies on jointly curated multimodal data, requiring substantial human effort to construct multi-way aligned datasets and thereby limiting scalability across domains. In this work, we explore training MLLMs by only leveraging multiple paired modalities as a surrogate for the full joint multimodal distribution. Specifically, we first provide a theoretical analysis of the conditions under which the representations are identifiable with only observing pairwise modalities. Building on this analysis, we propose a representation learning framework for aligning latent representations across modalities using only pairwise data. The framework consists of two stages: latent representation alignment and cross-modal recomposition. Specifically, in the first stage, we learn the shared latent space across modalities by both self-modal reconstruction and pair-wise contrastive learning. We also incorporate an inductive bias in the contrastive learning process by partially aligning and minimal latent specification. In stage two, we integrate the encoder of newly introduced modalities with the decoders of the pre-trained modalities to facilitate cross-modal transfer and generation. We evaluate our method by newly adding 3D point clouds and tactile modalities into pre-trained MLLMs with three modality pairs and show that, by learning an aligned latent representation space, our model achieves strong cross-modal performance.
Yan Li, Yunlong Deng, Yuewen Sun +3
Mohamed bin Zayed University of Artificial Intelligence · Carnegie Mellon University