Rethinking Projector Training in Multimodal LLMs

Fabian Gröger; Naël Ouerghemi; Shuo Wen; Maria Brbić

cs.CVOct 13, 2026

Rethinking Projector Training in Multimodal LLMs

Authors: Fabian Gröger, Naël Ouerghemi, Shuo Wen, Maria Brbić

Organizations: EPFL · University of Basel · HSLU

Abstract

Multimodal LLMs are typically trained by first aligning a vision encoder with a language model via projector training on large image-caption datasets, followed by joint training of the full multimodal model. Despite its widespread use, the role of the projector training stage remains poorly understood. We find that (i) using only $10$ to $20\%$ of the data for training the projector already recovers most downstream performance gains, and (ii) projectors trained on different datasets can be linearly interpolated while largely preserving performance. These results suggest that projector training mainly performs a coarse alignment, which many different projectors achieve equally well. Motivated by these findings, we propose PORTAL, a training-free projector initialization that requires neither paired image-caption data nor gradient-based optimization, but only per-modality summary statistics. PORTAL computes an optimal-transport-based map between Gaussian approximations of the vision and language feature distributions on a shared principal subspace, so that the projector training stage can be skipped entirely. Without any paired data, PORTAL matches the standard pipeline trained on 558k image-caption pairs across six LLM backbones, two vision encoders, and 16 benchmarks. It outperforms that pipeline in $3$ of $7$ (vision encoder, LLM) settings and reaches an average $98\%$ of its score across the other four.

Figures & tables

Figure 1: Projector alignment as a coarse alignment with a broad solution space. ( Left ) Standard pipeline: during projector alignment, the vision encoder and language model are frozen, and only the projector is trained to map visual tokens into a representation space compatible with the LLM using image-caption pairs. ( Middle ) We show that projectors trained on different image-caption datasets can be linearly interpolated while largely preserving downstream performance, indicating that there is a large solution space for the weights of the projector. ( Right ) This observation motivates a closed-form solution of projector alignment. We propose a training-free projector PORTAL constructed by optimal transport between the two modalities’ marginal statistics.

Panel of Figure 2 — Figure 2: Stage-I gain saturates early. Relative gain in downstream performance over the no-pretraining baseline by Cambrian-1 category, as a function of Stage-I data fraction on LCS-558k (left) and ShareGPT-1.2M (right). OCR and Chart benefit the most, while Knowledge and Vision-Centric scores saturate early.

Figure 3: Cross-data projectors are linearly interpolable. Cross-data interpolation

W_{\alpha}=(1{-}\alpha)W_{\text{LCS}}+\alpha W_{\text{ShareGPT}}

, evaluated per Cambrian-1 category. The optimal

\alpha

varies by category, but every interior blend outperforms both endpoints on the aggregate score, with the

\alpha{=}0.5

mixture peaking above either trained projector.

LLM	Scale	Vision	Init	AI2D	SciQA	MathV	MMMU	GQA	MME	SEED ${}^{\text{L}}$	MMB	TxtVQA	OCR-Bv2	DocVQA	ChartQA	CV-2D	CV-3D	BLINK	RealWQ	Avg	%BL
				Knowledge				General				OCR				Vision
Vicuna 1.5	7B	CLIP	Baseline †	54.5	67.9	26.1	34.3	66.6	74.5	67.7	63.2	57.3	12.5	22.0	16.5	54.7	58.3	37.4	54.9	48.0	100.0
			Random	51.5	69.1	27.2	32.8	64.2	69.2	57.2	58.1	52.8	11.0	17.1	15.2	52.5	53.8	34.7	51.9	44.9 ±0.60	93.5 ±1.6
			PORTAL	53.4	68.2	27.4	34.4	66.4	69.9	67.9	60.4	55.1	12.7	20.5	16.3	56.2	60.7	36.0	53.5	47.4 ±0.56	98.8 ±1.5
		SigLIP	Baseline †	55.8	68.0	27.2	34.0	68.7	73.6	69.1	66.0	61.5	13.3	25.3	19.1	57.5	60.8	28.1	56.6	49.0	100.0
			Random	51.8	67.9	26.7	35.1	56.8	60.6	46.5	52.3	43.8	7.9	8.9	12.8	47.1	57.5	25.6	45.9	40.5 ±0.58	82.5 ±1.1

Table 1: PORTAL generalizes across LLM backbones and vision encoders. All configurations use identical Stage-II instruction tuning (LLaVA-665k [ 2 ] ). PORTAL requires zero Stage-I paired captions and zero Stage-I gradient steps. Baseline

{\dagger}

: uses 558k paired captions for Stage-I pretraining. Bold : best among zero training data methods. Avg : unweighted mean of the 16 per-benchmark scores. %BL: this average as a percentage of the trained baseline (BL), which is therefore

100\%

. Subscripts on the Avg and %BL columns give the 95% bootstrap confidence-interval half-width over per-sample outcomes, from 1000 resamples paired across methods. LLM backbones: Vicuna 1.5 7B [ 36 ] , LLaMA 3.1 8B [ 37 ] , Qwen 2.5 7B [ 38 ] , LLaMA 3.2 1B [ 37 ] , Qwen 2.5 0.5B [ 38 ] , OLMo 2 1B [ 39 ] . Vision encoders: CLIP ViT-L/14-336 [ 40 ] , SigLIP SO400M [ 41 ] .

{}^{\text{L}}

: 500-sample Lite subset from lmms-eval [ 42 ] .

Figure 4: Alignment compute efficiency on Vicuna 1.5 7B/CLIP. Average over the 16 benchmarks as a function of the projector alignment budget, with identical Stage-II tuning on LLaVA-665k. PORTAL stays above Random at every budget. The marked point indicates the crossover at roughly

4

k captions, where PORTAL matches the

558

k-trained baseline at

\sim

135{\times}

fewer alignment FLOPs. The horizontal dashed line shows the

558

k baseline.

Projector	Init	Avg	%BL
MLP	Trained †	$41.6$	$100.0$
	Random	$41.5$	$99.9$
	PORTAL	$\mathbf{44.2}$	$\mathbf{106.2}$
Linear	Trained †	$39.9$	$100.0$
	Random	$37.6$	$94.2$
	PORTAL	$\mathbf{44.7}$	$\mathbf{111.8}$

Table 2: Linear vs. MLP-2x-GELU projector on LLaMA 3.2 1B/CLIP. Full per-benchmark breakdown in Appendix J .

Target	Avg	%BL
Trained †	$51.3$	$100.0$
Random	$51.4$	$100.2$
$\ell{=}0$ (embed)	$\mathbf{52.5}$	$\mathbf{102.3}$
$\ell{=}4$	$52.3$	$101.9$
$\ell{=}8$	$52.0$	$101.4$

Table 3: Layer ablation for PORTAL-ctx on LLaMA 3.1 8B/CLIP. Avg over the 16-benchmark suite, %BL relative to trained Stage-I.

Appendix figures & tables17 assets

Supplementary material from the paper’s appendix.

Appendix

	Stage I (Pretrain)	Stage II (Finetune)
Learning rate	$1\times 10^{-3}$	$2\times 10^{-5}$
Schedule	Cosine	Cosine
Warmup	200 steps	3% ratio
Weight decay	$5\times 10^{-2}$	0
Per-device batch size	32	4
Gradient accumulation	1	4

Appendix

Table 4: Training hyperparameters. Identical to LLaVA-1.5.

Dataset	Used for	License	Source
Training data
LCS-558k	Stage-I pretrain	CC-BY-4.0	[ 2 ]
ShareGPT-1.2M	Stage-I pretrain	CC-BY-NC-4.0	[ 44 ]
LLaVA-665k	Stage-II finetune	CC-BY-4.0	[ 2 ]
Cambrian-737k	PORTAL image pool	Apache-2.0	[ 7 ]
COCO val2017	PORTAL image pool	CC-BY-4.0	[ 45 ]

Appendix

Table 5: Datasets used in our experiments.

Category	Benchmark	Samples	Metric
Knowledge	AI2D [ 46 ]	3,088	Exact match
	ScienceQA [ 47 ]	2,017	Exact match
	MathVista [ 48 ]	1,000	LLM-as-judge
	MMMU [ 49 ]	900	Accuracy
General	GQA [ 50 ]	12,578	Exact match
	MME [ 51 ]	2,374	Perception / 20

Appendix

Table 6: Evaluation benchmarks grouped by Cambrian-1 category.

{}^{\text{L}}

= Lite subset (500 samples).

Pretrain	AI2D	SciQA	MathV	MMMU	GQA	MME	SEED ${}^{\text{L}}$	MMB	TxtVQA	OCR-Bv2	DocVQA	ChartQA	CV-2D	CV-3D	BLINK	RealWQ	Avg	%BL
	Knowledge				General				OCR				Vision
0%	34.5	47.4	25.2	26.6	51.2	49.7	41.6	10.1	17.0	7.0	8.9	10.3	36.6	51.7	35.3	41.6	30.9	80.9
10%	37.5	53.0	24.3	28.0	56.5	56.4	58.6	39.3	32.1	7.8	13.1	10.6	43.7	53.2	35.7	44.4	37.1	97.1
20%	39.1	55.4	24.4	27.6	56.8	58.7	59.0	39.0	33.6	8.3	14.2	11.3	44.6	51.2	35.5	44.2	37.7	98.5
30%	39.1	53.7	25.8	29.3	56.6	57.6	58.6	37.7	34.8	7.6	14.9	10.6	44.6	51.8	35.7	44.3	37.7	98.5
40%	39.2	55.2	25.9	29.1	56.7	58.2	59.2	41.8	35.2	8.1	15.0	10.9	45.6	52.9	34.8	44.6	38.3	100.1

Appendix

Table 7: Per-benchmark results for each LCS-558k data fraction. Architecture: CLIP ViT-L/14 (336 px) and MobileLLaMA 1.4B [ 61 ] , with a two-layer MLP projector (

1024\to 2048\to 2048

). Rows: fraction of the

558

k-sample LCS corpus used for Stage-I pretraining (

0\%

corresponds to no Stage-I, random projector init). Columns:

16

benchmarks grouped into four categories adapted from the Cambrian-1 taxonomy (Appendix C ), followed by the overall average and its ratio to the

100\%

reference. All scores are normalized to

0

–

100

(MME is divided by

20

, others are already percentages).

{}^{\text{L}}

500

-sample Lite subset from lmms-eval [ 42 ] .

Pretrain	AI2D	SciQA	MathV	MMMU	GQA	MME	SEED ${}^{\text{L}}$	MMB	TxtVQA	OCR-Bv2	DocVQA	ChartQA	CV-2D	CV-3D	BLINK	RealWQ	Avg	%BL
	Knowledge				General				OCR				Vision
0%	34.5	47.4	25.2	26.6	51.2	49.7	41.6	10.1	17.0	7.0	8.9	10.3	36.6	51.7	35.3	41.6	30.9	81.4
10%	38.4	53.7	24.3	28.1	55.1	54.5	53.5	50.0	29.6	6.9	12.5	11.2	40.8	52.4	35.2	43.1	36.8	97.0
20%	37.9	54.2	24.1	28.1	55.7	55.1	55.4	54.5	30.7	6.5	12.7	11.1	41.7	53.8	32.3	44.1	37.4	98.4
30%	38.7	53.9	24.4	28.3	56.4	55.7	56.4	51.5	31.1	6.8	13.1	11.2	40.5	52.9	35.6	45.2	37.6	99.0
40%	38.0	53.9	24.8	28.4	56.4	57.5	57.2	50.8	31.4	6.8	13.2	11.6	44.4	52.8	35.3	46.4	38.1	100.2

Appendix

Table 8: Per-benchmark results for each ShareGPT-1.2M data fraction. Layout matches Table 7 but with the ShareGPT-1.2M corpus used for Stage-I pretraining.

Initialization	Avg16	% of Baseline
Stage-I trained baseline	48.03	100.0
Random	44.90	93.5
CORAL	46.36	96.5
Gromov-Wasserstein	47.06	98.0
PORTAL	47.43	98.8

Appendix

Table 9: Comparison with alternative alignment methods. All methods except the conventional Stage-I baseline require no paired image-caption samples.

Method	Paired Samples	Projector Training	Construction
Random	No	No	Random initialization
CORAL	No	No	Closed form
GW	No	No	Iterative optimization
PORTAL	No	No	Closed form

Appendix

Table 10: Characteristics of the unpaired initialization methods considered in Table 9 .

LLM	Vision Encoder	Knowledge	General	OCR	Vision	Overall
Vicuna 1.5 7B	CLIP	100.3	97.3	96.6	100.5	98.8
Vicuna 1.5 7B	SigLIP	100.6	98.1	97.6	101.0	99.3
LLaMA 3.1 8B	CLIP	103.6	100.7	103.9	103.8	102.7
LLaMA 3.2 1B	CLIP	102.5	109.8	112.1	103.1	106.2
Qwen 2.5 7B	CLIP	98.5	99.4	92.6	99.3	98.3
Qwen 2.5 0.5B	CLIP	100.9	113.2	121.1	97.9	106.1

Appendix

Table 11: PORTAL performance as a percentage of the corresponding Stage-I-trained baseline, broken down by task category. The last row averages the retention rate across the seven (vision encoder, LLM) configurations.

Category	Baseline	PORTAL	$\boldsymbol{\Delta}$	95% CI
Knowledge	45.95	46.68	+0.72	$[-0.40,\,+1.88]$
General	59.60	59.88	+0.27	$[-0.71,\,+1.27]$
OCR	30.28	31.45	+1.18	$[+0.67,\,+1.68]$
Vision	50.12	51.28	+1.16	$[-0.13,\,+2.46]$
Overall	46.49	47.32	+0.83	$\mathbf{[+0.36,\,+1.35]}$

Appendix

Table 12: Performance under large-scale Stage-II multimodal instruction tuning with 3.2M samples. The confidence interval reports the 95% CI for the difference between PORTAL and the Stage-I-trained baseline.

LLM	Vision Encoder	Baseline	Random	PORTAL
Vicuna 1.5 7B	CLIP	85.5	85.3	85.7 $(+0.2)$
Qwen 2.5 7B	CLIP	85.7	85.5	86.4 $(+0.7)$
LLaMA 3.1 8B	CLIP	84.0	84.2	84.9 $(+0.9)$
Vicuna 1.5 7B	SigLIP	84.6	79.1	85.1 $(+0.5)$
OLMo 2 1B	CLIP	85.3	84.8	85.5 $(+0.2)$
LLaMA 3.2 1B	CLIP	84.1	84.0	86.2 $(+2.1)$

Appendix

Table 13: POPE F1 scores across all seven (vision encoder, LLM) configurations. Numbers in parentheses indicate the improvement of PORTAL over the conventionally Stage-I-trained baseline.

Task	Baseline	Random	PORTAL
AI2D	49.3	46.7	48.0
ScienceQA	68.2	65.2	66.2
MathVista	26.9	25.9	26.2
MMMU	32.3	34.6	33.9
GQA	60.6	59.7	60.7
MME-P	60.8	59.6	65.4

Appendix

Table 14: Performance with the higher-capacity C-Abstractor connector. Despite substantially increasing connector capacity and introducing explicit token mixing and sequence compression, PORTAL remains essentially indistinguishable from conventional Stage-I training.

Type	Distribution	$n$	$d$	$d_{\mathrm{eff}}$	Mardia $b_{1}$	Exc. kurt.	AD rej.
Source	CLIP ViT-L/14 patches	5,000	1,024	4.5	0.0022	0.487	100%
Source	SigLIP SO400M patches	5,000	1,152	50.9	0.0026	0.278	100%
Target	Vicuna 7B embedding table	5,000	4,096	1,429.6	0.0005	0.071	84%
Target	LLaMA 3.1 8B embedding table	5,000	4,096	1,713.2	0.0005	0.131	98%
Target	Qwen 2.5 7B embedding table	5,000	3,584	830.1	0.0004	0.114	100%

Appendix

Table 15: Distributional audit on top

k{=}128

principal components. The columns most relevant to the argument in Section 4 are

d_{\mathrm{eff}}

and Mardia

b_{1}

. The Anderson-Darling rejection rate (AD rej.) and the excess kurtosis identify where the deviation from Gaussianity is concentrated.

Encoder	LLM	Scale	Source	Random	PORTAL	Target	PORTAL $d_{\mathrm{eff}}$	Random Avg	PORTAL Avg	Trained Avg
CLIP	Vicuna 7B	pop	0.234	0.234	0.100	0.056	266.7	44.9	47.4	48.0
CLIP	Vicuna 7B	intra	0.252	0.251	0.235	0.056	266.7	44.9	47.4	48.0
CLIP	LLaMA 3.1 8B	pop	0.234	0.234	0.101	0.057	257.4	50.4	52.2	50.8
CLIP	LLaMA 3.1 8B	intra	0.252	0.251	0.234	0.057	257.4	50.4	52.2	50.8
CLIP	Qwen 2.5 7B	pop	0.234	0.234	0.094	0.054	289.3	48.6	51.7	52.6
CLIP	Qwen 2.5 7B	intra	0.252	0.251	0.245	0.054	289.3	48.6	51.7	52.6

Appendix

Table 16: Cosine dispersion

\omega

of projector outputs and downstream average. For each (encoder, LLM) pair, Source is

\omega

of the raw source patches, Random and PORTAL are

\omega

after applying the corresponding projector, and Target is

\omega

of the LLM’s input embedding table restricted to the top-

512

principal subspace that PORTAL matches. Random and PORTAL denote the first layer

\mathbf{W}_{1}\mathbf{x}+\mathbf{b}_{1}

of each projector. PORTAL

d_{\mathrm{eff}}

reports the effective dimension of the PORTAL output cloud. Downstream Avg columns from Table 1 .

LLM	Method	$\\|\Delta\boldsymbol{\mu}\\|$	Rel.	Spec. err.	$d_{\mathrm{eff}}$	$\mathrm{cv}$	$\omega$ (pop)
Vicuna 7B	target	0.000	0.000	0.000	319.7	0.123	0.056
	Random+GELU	7.144	47.33	24.25	3.6	0.682	0.212
	PORTAL (preact)	0.021	0.142	0.066	266.7	0.281	0.100
	PORTAL+GELU	0.077	0.508	0.470	266.9	0.281	0.100
	Trained Stage-I	33.18	219.8	142.4	2.4	1.047	0.232
LLaMA 3.1 8B	target	0.000	0.000	0.000	308.2	0.135	0.057

Appendix

Table 17: Post-projection statistics on CLIP patch tokens. Distributional fit to the target distribution for four projector constructions, plus a target row for reference. Smaller

\|\Delta\boldsymbol{\mu}\|

and spectrum error indicate a closer fit to the target. The

d_{\mathrm{eff}}

and

\omega

values should be read against the target row of the same LLM.

Pool	Src	#Img	AI2D	SciQA	MathV	MMMU	GQA	MME	SEED ${}^{\text{L}}$	MMB	OCR-Bv2	DocVQA	ChartQA	CV-2D	CV-3D	BLINK	RealWQ	Avg	%BL
			Knowledge				General				OCR			Vision
Baseline †	LCS	558k	46.5	66.2	26.9	33.6	59.7	58.4	56.2	46.1	10.1	12.8	10.8	53.5	56.4	37.2	47.3	41.4	100.0
Random	–	0	46.6	66.9	26.3	34.2	59.8	58.1	56.0	45.2	10.4	12.8	11.0	54.2	54.9	37.4	46.9	41.4	99.8
PORTAL	Cambrian	4096	48.2	66.7	26.9	34.3	62.0	64.3	59.6	53.6	10.8	16.8	12.0	53.5	55.8	37.7	49.4	43.4	104.8
PORTAL	COCO	1024	47.6	67.1	29.2	33.7	62.1	66.0	59.6	54.2	11.4	16.3	12.0	54.6	57.3	38.9	49.7	44.0	106.1

Appendix

Table 18: Image-pool ablation for the PORTAL construction, per-benchmark breakdown. Setup: LLaMA 3.2 1B/CLIP, Stage-II on LLaVA-665k. Both PORTAL variants use rank

k{=}512

and require no paired captions. Bold : best between the two PORTAL rows per benchmark. Avg and %BL are computed over the

15

benchmarks shown.

Projector	Init	AI2D	SciQA	MathV	MMMU	GQA	MME	SEED ${}^{\text{L}}$	MMB	TxtVQA	OCR-Bv2	DocVQA	ChartQA	CV-2D	CV-3D	BLINK	RealWQ	Avg	%BL
		Knowledge				General				OCR				Vision
MLP-2x-GELU	Baseline †	46.5	66.2	26.9	33.6	59.7	58.4	56.2	46.1	43.7	10.1	12.8	10.8	53.5	56.4	37.2	47.3	41.6	100.0
	Random	46.6	66.9	26.3	34.2	59.8	58.1	56.0	45.2	43.8	10.4	12.8	11.0	54.2	54.9	37.4	46.9	41.5	99.9
	PORTAL	47.6	67.1	29.2	33.7	62.1	66.0	59.6	54.2	47.1	11.4	16.3	12.0	54.6	57.3	38.9	49.7	44.2	106.2
Linear	Baseline †	46.5	65.5	27.1	33.3	54.7	53.1	47.3	49.8	42.6	10.1	11.0	10.0	50.1	54.4	36.8	46.7	39.9	100.0
	Random	46.0	63.2	24.6	30.9	53.0	52.8	46.1	43.8	36.8	7.9	7.3	8.8	45.6	52.8	37.7	44.4	37.6	94.2

Appendix

Table 19: Linear versus MLP-2x-GELU projector, per-benchmark breakdown. Setup: LLaMA 3.2 1B/CLIP, Stage-II on LLaVA-665k. Each architecture is anchored to its own trained baseline for %BL. Bold : best between Random and PORTAL within an architecture. Underline : second best.

Figure 5: PCA rank sensitivity on LLaMA 3.2 1B/CLIP. Average over the 16 benchmarks as a function of the rank

k

used in the PORTAL construction, with identical Stage-II tuning. All ranks outperform the trained baseline (

41.6

k{=}512

is optimal (

44.2

Multimodal Large Language Models

Multimodal large language models combine language models with image, video, audio, and other inputs for grounded understanding and generation.

See all 1,091 papers

Projector Is All You Train
Aug 20, 2026Nyx Iskandar, Saathvik Selvan, Slater VictoroffMultimodal Large Language Models VLM Adaptation
Fitting Vision Adapters at Frontier Scales
Oct 5, 2026Jaehoon Lee, Harry Partridge, Mudith Jayasekara +3Vision-Language Models VLM Adaptation
Base Labs
Multimodal LLMs under Pairwise Modalities
May 20, 2026Yan Li, Yunlong Deng, Yuewen Sun +3Cross-Modal Learning Representation Identifiability
Mohamed bin Zayed University of Artificial Intelligence · Carnegie Mellon University

Aug 20, 2026J/K move · Enter open · S save

Projector Is All You Train

Nyx Iskandar, Saathvik Selvan, Slater Victoroff

Multimodal Large Language Models VLM Adaptation Parameter-Efficient Fine-Tuning 3D VLMs

Open paper AskPDF arXiv

The typical training process of a multimodal large language model (MLLM) involves adapting both the language model backbone and the projector between the backbone and a modality-specific encoder. We ask whether fine-tuning the backbone of an MLLM is necessary to adapt it to a new modality. Through experiments on 3D MLLMs, we find that training only the projector is sufficient to achieve strong multimodal performance relative to existing baseline models and our jointly trained MLLMs with the same encoder and backbone. We also show that joint training leads to undesirable drift in existing capabilities of the language model, which projector-only training avoids by definition. Furthermore, projector-only training has approximately twice the training sample throughput of joint training. We validate our findings across different language model backbones via 3D classification and captioning benchmarks as well as standard benchmarks evaluating language, vision, and spatial reasoning capabilities.

Abstract

Multimodal Large Language Models

Explore similar work

Projector Is All You Train

Fitting Vision Adapters at Frontier Scales

Multimodal LLMs under Pairwise Modalities