Projector Is All You Train

Nyx Iskandar; Saathvik Selvan; Slater Victoroff

cs.CLAug 20, 2026

Projector Is All You Train

Authors: Nyx Iskandar, Saathvik Selvan, Slater Victoroff

Abstract

The typical training process of a multimodal large language model (MLLM) involves adapting both the language model backbone and the projector between the backbone and a modality-specific encoder. We ask whether fine-tuning the backbone of an MLLM is necessary to adapt it to a new modality. Through experiments on 3D MLLMs, we find that training only the projector is sufficient to achieve strong multimodal performance relative to existing baseline models and our jointly trained MLLMs with the same encoder and backbone. We also show that joint training leads to undesirable drift in existing capabilities of the language model, which projector-only training avoids by definition. Furthermore, projector-only training has approximately twice the training sample throughput of joint training. We validate our findings across different language model backbones via 3D classification and captioning benchmarks as well as standard benchmarks evaluating language, vision, and spatial reasoning capabilities.

Figures & tables

Figure 1: Architecture of 3D MLLMs in this paper. This figure depicts projector-only training, where all parameters are frozen except those of the projector. An alternative training regime is joint training, where all parameters are frozen except those of the projector and LoRA adapters of the LLM backbone.

ID	LM Backbone	Projector Trained	LoRA Fine-tuned
P-Llama8B	Llama-3.1-8B-Instruct	Yes	No
J-Llama8B	Llama-3.1-8B-Instruct	Yes	Yes
P-Qwen4B	Qwen3.5-4B	Yes	No
J-Qwen4B	Qwen3.5-4B	Yes	Yes
P-Qwen9B	Qwen3.5-9B	Yes	No
J-Qwen9B	Qwen3.5-9B	Yes	Yes

Table 1: Configurations of MLLM variants. All variants are trained for the full 16 GPU hours. We save checkpoints at 2, 4, 8, 12, and 16 hours.

Figure 2: Evaluation results of projector-only-trained MLLMs against baseline PointLLM models. Chart (a) shows generative 3D object classification results on ModelNet40 (M40.), Objaverse (Obj.), and OmniObject3D (Omni.) objects under a zero-shot setting. There are two prompt types per benchmark: an instruction-style prompt (I, “What is this?”) and a completion-style prompt (C, “This is an object of”). Each entry reports accuracy judged by GPT-5.6 Luna [ 28 ] . Chart (b) shows the aggregate precision score on 3D object captioning tasks under the three judge LLMs. More details are found in Appendix B .

Model	M40. (I)	M40. (C)	Obj. (I)	Obj. (C)	Omni. (I)	Omni. (C)
PointLLM-7B [ 43 ]	54.94	55.55	59.83	60.43	34.78	35.70
PointLLM-13B [ 43 ]	57.66	56.81	61.73	60.90	34.63	34.01
P-Llama8B	61.95	63.57	65.60	62.33	49.19	40.69
P-Qwen4B	59.56	59.56	60.90	60.83	41.07	40.25
P-Qwen9B	59.28	62.80	74.07	67.77	48.33	38.76
Mean	58.68	59.66	64.43	62.45	41.60	37.88

Table 2: Generative 3D object classification results on ModelNet40 (M40.), Objaverse (Obj.), and OmniObject3D (Omni.) under a zero-shot setting as reported in Figure 2 (a).

	GPT-5.6 Luna			Claude Haiku 4.5			Gemini 3.5 Flash-Lite
Model	C	H $\downarrow$	P	C	H $\downarrow$	P	C	H $\downarrow$	P
PointLLM-7B [ 43 ]	4.26	2.12	66.75	3.37	1.95	63.30	2.75	1.99	58.08
PointLLM-13B [ 43 ]	4.35	2.10	67.43	3.47	1.91	64.46	2.82	1.92	59.49
P-Llama8B	5.61	2.17	72.12	3.76	1.75	68.19	3.29	1.87	63.75
P-Qwen4B	6.41	1.83	77.81	4.28	1.57	73.11	3.72	1.65	69.22
P-Qwen9B	6.58	1.95	77.19	4.30	1.68	71.90	3.76	1.71	68.72

Table 3: 3D object captioning results on Objaverse as reported in Figure 2 (b). C refers to correctness, H to hallucination, and P to precision (aggregate of C and H).

Panel of Figure 3 — Figure 3: M40. (top) and Obj. (bottom) mean accuracy against GPU hours. Projector-only training generally scores higher than joint training across all GPU hours.

Panel of Figure 4 — Figure 4: M40. (top) and Obj. (bottom) mean accuracy against training samples. Plotted points in these charts correspond to the same GPU-hour thresholds that in turn correspond to the plotted points in Figure 3 . Projector-only training sees more samples than joint training at each GPU-hour threshold as it is approximately twice as fast.

Language
	Llama-3.1-8B-Instruct		Qwen3.5-4B		Qwen3.5-9B
Benchmark	P	J	P	J	P	J
MMLU-Pro [ 39 ]	37.32	11.21 ↓	45.04	45.48	51.38	51.30
MMLU-Redux [ 15 ]	60.37	22.83 ↓	69.13	70.53	74.57	74.27
GPQA Diamond [ 35 ]	33.33	24.24 ↓	33.33	41.41 ↑	45.96	49.49 ↑
IFEval [ 51 ]	72.46	10.35 ↓	80.41	38.82 ↓	83.55	67.84 ↓

Table 4: Results for language, vision, and spatial reasoning benchmarks. The LoRA fine-tuned backbones are compared against the corresponding base backbone.

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

Backbone	Encoder dim $c$	Tokens $m$	Backbone dim $c^{\prime}$	Proj. params	LoRA params
Llama-3.1-8B-Instruct	384	513	4096	$10.89$ M	$41.94$ M
Qwen3.5-4B	384	513	2560	$7.74$ M	$32.46$ M
Qwen3.5-9B	384	513	4096	$10.89$ M	$43.28$ M

Appendix

Table 5: Projector and LoRA-adapter parameter counts, and projector input/output shapes, for each backbone.

System message
Answer the question or follow the instruction regarding the given 3D object.
Answer the following question or carry out the instruction about the provided 3D object.
Given a 3D object, respond to the question or instruction about it.
Consider the 3D object and answer the question or follow the instruction that follows.
Respond to the question or instruction concerning the presented 3D object.
Using the given 3D object, answer the question or complete the instruction.

Appendix

Table 6: The set of system message paraphrases, drawn uniformly at random during training.

Hyperparameter	Value
Optimizer	AdamW, $\beta=(0.9,0.999)$ , $\epsilon=10^{-8}$
Learning-rate schedule	constant, no warmup
Projector learning rate	$2\times 10^{-3}$ ( $1\times 10^{-3}$ for Llama8B)
Weight decay	$0.0$
Gradient clipping	global norm $1.0$
Micro-batch / accumulation / effective	$12$ / $2$ / $24$

Appendix

Table 7: Training and evaluation hyperparameters, shared across every configuration except the learning rates.

ID	Total samples	Avg. throughput (h -1 )	Avg. step rate (h -1 )
P-Llama8B	432,936	27,056	1,127
J-Llama8B	200,640	12,537	522
P-Qwen4B	480,768	30,045	1,252
J-Qwen4B	245,592	15,346	639
P-Qwen9B	325,176	20,321	847
J-Qwen9B	174,720	10,918	455

Appendix

Table 8: Total number of training samples seen by each MLLM variant, average throughput, and average step rate after 16 GPU hours.

Panel of Figure 5 — Figure 5: The judge prompt for generative zero-shot classification on Objaverse and OmniObject3D. Placeholders in braces are substituted at evaluation time; {gold} and {generation} refer to the ground-truth caption and the model’s response, respectively.

Panel of Figure 6 — Figure 6: The judge prompt for generative zero-shot classification on ModelNet40. The placeholder {generation} refers to the model’s response.

Panel of Figure 7 — Figure 7: The judge prompt for object captioning on Objaverse. Placeholders in braces are substituted at evaluation time; {generation} refers to the model’s generated caption. The judge additionally receives 8 rendered views of the object alongside this text.

Model	M40. (I)	M40. (C)	Obj. (I)	Obj. (C)	Omni. (I)	Omni. (C)
ShapeLLM-7B [ 32 ]	18.76	17.95	30.23	31.37	15.28	18.75
ShapeLLM-13B [ 32 ]	22.45	21.60	40.67	38.90	28.66	33.33
PointLLM-7B [ 43 ]	54.94	55.55	59.83	60.43	34.78	35.70
PointLLM-13B [ 43 ]	57.66	56.81	61.73	60.90	34.63	34.01
PointLLM-R [ 3 ]	62.28	62.64	65.23	65.40	38.81	38.69
MiniGPT-3D [ 37 ]	63.41	62.84	63.97	63.00	41.10	39.27

Appendix

Table 9: Generative 3D object classification results on ModelNet40 (M40.), Objaverse (Obj.), and OmniObject3D (Omni.) under a zero-shot setting. Extension of Table 2 .

	GPT-5.6 Luna			Claude Haiku 4.5			Gemini 3.5 Flash-Lite
Model	C	H $\downarrow$	P	C	H $\downarrow$	P	C	H $\downarrow$	P
ShapeLLM-7B [ 32 ]	2.23	1.52	59.47	1.65	1.46	53.15	1.03	1.69	37.85
ShapeLLM-13B [ 32 ]	2.68	1.84	59.32	2.04	1.68	54.93	1.39	2.02	40.81
PointLLM-7B [ 43 ]	4.26	2.12	66.75	3.37	1.95	63.30	2.75	1.99	58.08
PointLLM-13B [ 43 ]	4.35	2.10	67.43	3.47	1.91	64.46	2.82	1.92	59.49
PointLLM-R [ 3 ]	3.38	1.18	74.10	2.50	1.10	69.42	2.48	1.17	67.94

Appendix

Table 10: 3D object captioning results on Objaverse. Extension of Table 3 .

	M40. (Avg.)					Obj. (Avg.)
Model	2 h	4 h	8 h	12 h	16 h	2 h	4 h	8 h	12 h	16 h
P-Qwen4B	54.23	54.76	58.33	63.53	59.56	58.10	62.52	67.25	64.90	60.87
J-Qwen4B	46.84	51.64	51.03	52.09	58.93	58.20	66.50	73.28	75.05	69.90
P-Qwen9B	51.38	53.99	57.03	61.06	61.04	57.47	63.07	67.80	67.35	70.92
J-Qwen9B	39.49	46.84	54.52	58.53	55.67	51.20	60.48	71.08	67.90	70.73
P-Llama8B	53.85	57.22	57.80	56.14	62.76	51.48	57.58	66.10	70.07	63.97

Appendix

Table 11: Generative zero-shot classification accuracy against wall-clock hours. This is calculated as the average of the I and C variants of the ModelNet40 and Objaverse benchmarks, directly comparable to Figure 3 .

Object
PointLLM-7B	The 3D model represents a distinctive, green toy car that stands out due to its unconventional feature - a cornered driving unit. Designed for play, this model is aesthetically appealing with an eye-catching bright green color that would attract young children. The unique driving unit located at the corner provides an interesting twist to the traditional car design, promoting creative play and imaginative scenarios. However, despite its playful design, it is still a toy and should be handled with care.	The 3D object is a lively representation of a blue whale, which is the largest mammal on Earth. It is depicted in a playful cartoon style with an eye-catching cobalt-blue hue. The whale features proportional body parts, including a small dorsal fin, enormous mouth, and flat, wide head. It seems to be designed for entertainment or educational purposes, possibly for children, given its cartoon-like attributes and the significance of the whale in marine life.
PointLLM-13B	The 3D object model represents a small toy car, with the primary colour being a vibrant green. It features four black wheels that contrast well with the green body. Two red lights are situated at the front, possibly to mimic the headlights of an actual car. This model, being a toy, is typically used for children’s play, aiding in their motor skill development and imagination. The car’s simplistic design might be suitable for younger age groups.	This is a 3D model of a blue whale, the largest mammal on Earth, depicted in a cartoon style. The whale is distinguished by its small dorsal fin, enormous mouth, and flat, wide head. Despite the cartoon aesthetic, the model accurately represents the unique features of a blue whale. It could be used for educational purposes or in animated movies or documentaries.
P-Llama8B	The object is a green and white van with a boxy shape, featuring a prominent front grille, round headlights, and a rear window. It has a flat roof and a side door with a handle. The van has a distinctive green and white color scheme with black accents. The wheels are circular and black, and the vehicle has a distinctive front bumper.	The object is a sleek, streamlined shark with a pointed snout and a series of sharp, triangular fins along its back. The body is predominantly blue, with a lighter underside and a distinctive pattern of darker spots. The fins are a lighter shade, and the tail is long and narrow, tapering to a sharp point. The shark has a prominent dorsal fin and a smaller pectoral fin, with a series of gill slits visible along the sides.
J-Llama8B	The object is a green vehicle with a boxy design, featuring a large windshield, a rear window, and a front grille. It has a roof rack and a spare tire mounted on the rear. The vehicle appears to be a van, with a prominent front bumper and a rear section that includes a license plate. The wheels are circular, and the vehicle has a simple, utilitarian appearance.	The object is a stylized representation of a fish, characterized by a streamlined body with a pointed snout and a forked tail. The body is primarily blue with a white underbelly, and the fins are white. The fish has a smooth, abstract appearance with a minimalistic design.

Appendix

Table 12: Example generations on long captioning tasks. The MLLMs are given the prompt “Caption this 3D model in detail.” All prompts are independently provided to the MLLMs. The answers shown in this table were generated by P-Llama8B and J-Llama8B at 16 hours of training, as well as the two PointLLM models.

Multimodal Large Language Models

Multimodal large language models combine language models with image, video, audio, and other inputs for grounded understanding and generation.

See all 1,091 papers

Rethinking Projector Training in Multimodal LLMs
Oct 13, 2026Fabian Gröger, Naël Ouerghemi, Shuo Wen +1Multimodal Large Language Models Vision-Language Alignment
EPFL · University of Basel · HSLU
Fitting Vision Adapters at Frontier Scales
Oct 5, 2026Jaehoon Lee, Harry Partridge, Mudith Jayasekara +3Vision-Language Models VLM Adaptation
Base Labs
ProjLens: Unveiling the Role of Projectors in Multimodal Model Safety
Apr 21, 2026Kun Wang, Cheng Qian, Miao Yu +6LLM Backdoor Attacks Multimodal Large Language Models
3Nanyang Technological University · 2Beijing University of Aeronautics and Astronautics · University of Science and Technology of China +2

Oct 13, 2026J/K move · Enter open · S save

Rethinking Projector Training in Multimodal LLMs

Fabian Gröger, Naël Ouerghemi, Shuo Wen, Maria Brbić

EPFL · University of Basel · HSLU

Multimodal Large Language Models Vision-Language Alignment

Open paper AskPDF arXiv

Multimodal LLMs are typically trained by first aligning a vision encoder with a language model via projector training on large image-caption datasets, followed by joint training of the full multimodal model. Despite its widespread use, the role of the projector training stage remains poorly understood. We find that (i) using only

10

20\%

of the data for training the projector already recovers most downstream performance gains, and (ii) projectors trained on different datasets can be linearly interpolated while largely preserving performance. These results suggest that projector training mainly performs a coarse alignment, which many different projectors achieve equally well. Motivated by these findings, we propose PORTAL, a training-free projector initialization that requires neither paired image-caption data nor gradient-based optimization, but only per-modality summary statistics. PORTAL computes an optimal-transport-based map between Gaussian approximations of the vision and language feature distributions on a shared principal subspace, so that the projector training stage can be skipped entirely. Without any paired data, PORTAL matches the standard pipeline trained on 558k image-caption pairs across six LLM backbones, two vision encoders, and 16 benchmarks. It outperforms that pipeline in

3

7

(vision encoder, LLM) settings and reaches an average

98\%

of its score across the other four.

Abstract

Multimodal Large Language Models

Explore similar work

Rethinking Projector Training in Multimodal LLMs

Fitting Vision Adapters at Frontier Scales

ProjLens: Unveiling the Role of Projectors in Multimodal Model Safety