More Than Seeing: Vision as Scalable Anchors in Native Multimodal Pre-Training

Haoyuan Wu; Aoqi Wu; Jiani Hu; Bei Yu

cs.CLOct 14, 2026

More Than Seeing: Vision as Scalable Anchors in Native Multimodal Pre-Training

Authors: Haoyuan Wu, Aoqi Wu, Jiani Hu, Bei Yu

Organizations: The Chinese University of Hong Kong · Foundation Model Department, Tencent

Abstract

Native multimodal pre-training trains a single model from scratch on text and multimodal data, enabling it to understand both language and visual inputs. However, how visual signals shape scaling behavior and support capabilities beyond visual understanding remains unclear. We investigate this question through controlled visual-content ablations across model sizes and training budgets. Specifically, we replace image embeddings with content-free padding token embeddings. Retaining visual content improves text prediction on image-text examples without materially changing the lowest text-only loss at a given compute budget, the optimal balance of model size and training data, or aggregate language scores. The relative multimodal loss benefit grows approximately as a power law of compute. To test benefits beyond understanding images, we evaluate spatial reasoning from text descriptions alone. Both conditions improve with model and data scaling. During training of the largest model, visual gains are most sustained in localization, navigation, and symbolic planning, which involve maintaining object configurations or composing spatial updates; gains in spatial relations, geometry, and physical commonsense are less consistent. These findings identify visual content as a scalable anchor for multimodal learning, with benefits that extend beyond visual inputs while preserving aggregate language performance.

Figures & tables

Figure 1: IsoFLOP curves (language objective). (a,b) At each target budget

C_{\mathrm{text}}

, we read the

\widetilde{L}_{\mathrm{text}}^{\mathrm{pad}}

and

\widetilde{L}_{\mathrm{text}}^{\mathrm{img}}

across various model size at the token count satisfying

6ND_{\mathrm{text}}=C_{\mathrm{text}}

and plot it against

\log N

. (c,d) Optimal model size

\widehat{N}_{\mathrm{text},\mathrm{iso}}^{q}

and optimal token count

\widehat{D}_{\mathrm{text},\mathrm{iso}}^{q}

as a function of the compute budget

C_{\mathrm{text}}

Figure 2: Training curve envelope (language objective). (a,b) Training curves span the evaluated model sizes. The lower envelope gives the lowest observed loss at each budget

C_{\mathrm{text}}

. The fitted loss frontier summarizes this envelope. (c,d) Optimal model size

\widehat{N}_{\mathrm{text},\mathrm{env}}^{q}

and optimal token count

\widehat{D}_{\mathrm{text},\mathrm{env}}^{q}

as a function of the compute budget

C_{\mathrm{text}}

Figure 3: IsoFLOP curves (multimodal objective). (a,b) At each target budget

C_{\mathrm{mm}}

, we read the

\widetilde{L}_{\mathrm{mm}}^{\mathrm{pad}}

and

\widetilde{L}_{\mathrm{mm}}^{\mathrm{img}}

across various model size at the token count satisfying

6ND_{\mathrm{mm}}=C_{\mathrm{mm}}

and plot it against

\log N

. (c,d) Optimal model size

\widehat{N}_{\mathrm{mm},\mathrm{iso}}^{q}

and optimal token count

\widehat{D}_{\mathrm{mm},\mathrm{iso}}^{q}

as a function of the compute budget

C_{\mathrm{mm}}

Figure 4: Training curve envelope (multimodal objective). (a,b) Training curves span the evaluated model sizes. The lower envelope gives the lowest observed loss at each budget

C_{\mathrm{mm}}

. The fitted loss frontier summarizes this envelope. (c,d) Optimal model size

\widehat{N}_{\mathrm{mm},\mathrm{env}}^{q}

and optimal token count

\widehat{D}_{\mathrm{mm},\mathrm{env}}^{q}

as a function of the compute budget

C_{\mathrm{mm}}

Figure 5: Visual anchoring strengthens with multimodal compute while the text gap stays near zero. (a) Paired text gaps, with a shaded

\pm 0.1\%

reference band. (b) Paired multimodal gaps, colored by model size. (c) Gap profiles at six fixed compute budgets. Stars mark maxima of quadratic fits in

\log N

, not loss minima. (d) Per-model gap trajectories, paired gaps along the img envelope, and their power-law fit (black,

\gamma=0.256

). All gaps are relative loss increases under ablation, expressed as percentages.

Figure 6: Aggregate language performance. Average scores over 15 text benchmarks for img and pad (a) Text performance across model sizes

N

. (b) A3B text performance over

D

Figure 7: Per-category language performance. Average scores evaluated using A3B models for img and pad conditions.

Figure 8: Aggregate spatial reasoning performance. Scores for img and pad , averaged equally across six task groups, with equal benchmark weights within each group. (a) Final performance across model sizes

N

at a nominal budget of

D=325\mathrm{B}

tokens. (b) A3B performance over nominal training tokens

D

Figure 9: Scores across model sizes for each task group.

Figure 10: Scores under data scaling for each task group.

Figure 11: Growth of symbolic tower-building performance in the benchmark Blocksworld. Final-checkpoint scores for (a) short plans with 3-4 blocks (106 problems) and (b) long plans with 5-12 blocks (385 problems) in the tower-building task.

Figure 12: Execution outcomes across scales for (a) img and (b) pad . Length-limited responses are classified first; others are executed until the first invalid instruction or unmet precondition, with a goal check after legal plans. Labels give percentages for segments of at least 10%.

Appendix figures & tables12 assets

Supplementary material from the paper’s appendix.

Appendix

	A71M	A128M	A340M	A590M	A874M	A3B
Architecture Configurations
layers	11	15	19	23	27	48
hidden size	640	768	1152	1280	1536	2048
FFN hidden size	2048	2048	3072	4096	4096	6912
attention heads	8	12	16	20	24	32
query groups	4

Appendix

Table 1: Pre-training hyperparameters. We detail the hyperparameters used for pre-training different model configurations to derive scaling laws.

Capability	Benchmark	Items	Answer type
Spatial Relations	StepGame	3,000	Direction label
	SpaRTUN	3326	Relation label
	bAbI(positional reasoning)	627	Yes/no
	FoREST	1998	Direction label
Localization	SpartQA	1994	Choice label
	SpatialMap	997	Choice label

Appendix

Table 2: Spatial evaluation inventory. The 15 benchmarks cover six task groups.

Scale	Outcome or subset	img	pad
A71M	Truncation	1,115	1,069
A3B	Truncation	14	0
A3B	Legal plan, goal missed	523	1,121
A3B	Precondition violation	882	427
A3B	Too short among legal goal misses	231/523 (44.2%)	823/1,121 (73.4%)

Appendix

Table 3: Supplementary execution statistics. The last row is conditional on legal plans that miss the goal; its denominator differs from the outcome counts above.

	IMG		PAD
Scale	Accepted	C / L / S / U	Accepted	C / L / S / U
A71M	3	0 / 0 / 3 / 0	1	1 / 0 / 0 / 0
A128M	1	1 / 0 / 0 / 0	3	1 / 0 / 2 / 0
A340M	3	1 / 0 / 1 / 1	2	0 / 0 / 1 / 1
A590M	2	1 / 0 / 0 / 1	2	1 / 1 / 0 / 0
A874M	1	1 / 0 / 0 / 0	3	2 / 0 / 1 / 0

Appendix

Table 4: Explanation quality among accepted answers in eight fixed problems per condition. C: consistent derivation; L: local verbal error with the solution intact; S: substantive reasoning error; U: unresolved boundary between local and substantive error. The four counts sum to Accepted, which is out of eight first responses. This manual case audit is separate from the all-response planning analysis and does not estimate population frequencies.

	# Pre-Trained Tokens	25B	50B	75B	100B	125B	150B	175B	200B	225B	250B	275B	300B	325B
Aggregate	MMLU-Redux (Acc.)	27.05	38.81	44.74	48.35	51.19	51.37	54.49	54.21	55.05	56.00	57.17	57.09	57.98
	MMLU-Pro (Acc.)	10.85	12.99	17.73	19.19	19.86	20.21	22.03	23.84	24.61	24.92	25.33	27.53	27.68
	AGIEval en (Acc.)	20.49	22.71	27.88	29.27	32.13	32.66	36.03	36.58	35.02	35.99	37.10	36.92	37.73
	SuperGPQA (Acc.)	10.78	13.48	14.92	16.88	18.11	18.27	18.69	19.63	20.17	20.63	21.08	21.53	22.00
Coding	HumanEval+ (Pass@1)	6.10	11.59	15.24	14.63	15.24	15.24	13.41	18.90	19.51	15.85	17.68	21.34	21.34
Coding	MBPP+ (Pass@1)	18.52	26.98	31.48	37.83	41.27	47.09	46.03	48.94	49.47	48.94	49.73	53.44	54.23

Appendix

Table 5: A3B IMG language scores across various

D

	Model Sizes ( $N$ )	A71M	A128M	A340M	A590M	A874M	A3B
Aggregate	MMLU-Redux (Acc.)	25.40	32.10	38.91	44.95	49.03	57.98
	MMLU-Pro (Acc.)	10.68	11.63	13.87	14.97	17.29	27.68
	AGIEval en (Acc.)	20.64	20.45	25.54	26.29	29.67	37.73
	SuperGPQA (Acc.)	8.00	10.31	11.81	14.16	14.90	22.00
Coding	HumanEval+ (Pass@1)	4.88	7.93	12.20	16.46	15.85	21.34
Coding	MBPP+ (Pass@1)	7.94	13.23	26.45	30.95	37.57	54.23

Appendix

Table 6: img language scores across various

N

	# Pre-Trained Tokens	25B	50B	75B	100B	125B	150B	175B	200B	225B	250B	275B	300B	325B
Aggregate	MMLU-Redux (Acc.)	25.89	38.30	44.72	48.49	50.44	51.63	52.74	54.63	54.40	55.74	56.72	56.32	57.75
	MMLU-Pro (Acc.)	11.17	12.20	15.96	16.94	18.53	18.59	20.05	21.95	21.62	22.29	22.20	24.36	25.49
	AGIEval en (Acc.)	19.16	23.64	28.36	31.37	33.51	33.49	34.94	35.63	37.57	35.60	36.13	37.86	37.42
	SuperGPQA (Acc.)	10.43	12.63	14.66	15.33	15.04	16.57	17.68	17.84	19.89	19.44	19.61	20.87	20.17
Coding	HumanEval+ (Pass@1)	6.71	10.37	9.76	13.41	18.29	14.02	18.90	20.12	19.51	12.80	15.85	16.46	18.29
Coding	MBPP+ (Pass@1)	14.29	28.04	35.71	34.39	36.77	45.77	45.50	45.24	53.70	52.65	46.30	54.50	52.12

Appendix

Table 7: A3B pad language scores across various

D

	Model Sizes ( $N$ )	A71M	A128M	A340M	A590M	A874M	A3B
Aggregate	MMLU-Redux (Acc.)	25.44	30.68	38.51	45.79	47.60	57.75
	MMLU-Pro (Acc.)	11.18	11.63	13.22	16.48	16.70	25.49
	AGIEval en (Acc.)	21.42	21.73	24.94	27.60	30.52	37.42
	SuperGPQA (Acc.)	8.39	9.90	12.65	13.55	13.83	20.17
Coding	HumanEval+ (Pass@1)	7.32	7.93	10.98	16.46	13.41	18.29
Coding	MBPP+ (Pass@1)	8.99	12.17	22.75	31.48	35.98	52.12

Appendix

Table 8: pad language scores across various

N

	# Pre-Trained Tokens	25B	50B	75B	100B	125B	150B	175B	200B	225B	250B	275B	300B	325B
Spatial Relations	StepGame	16.34	19.31	21.07	24.62	27.58	26.99	27.02	29.59	29.45	29.04	29.68	31.56	33.11
	SpaRTUN	19.97	26.72	24.69	29.19	27.83	31.30	27.44	30.37	27.64	32.22	32.34	32.25	32.17
	bAbI(positional reasoning)	47.49	47.85	47.05	48.52	52.07	54.19	53.79	54.27	51.20	52.31	59.41	51.36	58.41
	FoREST	39.18	42.62	45.78	46.11	50.10	48.01	46.03	46.91	49.54	46.01	47.00	48.65	52.41
Localization	SpartQA	22.93	33.09	32.35	31.86	31.81	33.55	29.61	29.41	27.62	33.38	31.01	33.35	31.78
	SpatialMap	20.91	26.45	29.01	28.91	33.45	33.30	33.27	33.10	31.47	34.25	35.21	32.22	33.25

Appendix

Table 9: A3B img spatial scores across various

D

	Model Sizes ( $N$ )	A71M	A128M	A340M	A590M	A874M	A3B
Spatial Relations	StepGame	15.46	14.20	16.97	20.64	21.67	33.11
	SpaRTUN	19.76	24.11	24.95	25.17	24.68	32.17
	bAbI(positional reasoning)	28.51	48.09	47.73	47.77	47.73	58.41
	FoREST	34.23	35.70	40.04	40.90	47.75	52.41
Localization	SpartQA	25.93	33.35	32.72	31.01	30.58	31.78
	SpatialMap	8.78	18.61	22.19	24.35	27.63	33.25

Appendix

Table 10: img spatial scores across various

N

	# Pre-Trained Tokens	25B	50B	75B	100B	125B	150B	175B	200B	225B	250B	275B	300B	325B
Spatial Relations	StepGame	15.72	19.23	20.70	21.46	26.37	27.28	26.95	27.72	29.63	31.77	31.73	29.41	30.02
	SpaRTUN	21.41	25.37	25.14	26.45	28.83	29.83	30.60	30.49	31.64	31.95	32.33	31.20	33.32
	bAbI(positional reasoning)	46.53	47.81	47.89	50.32	48.41	51.59	52.43	49.96	52.07	50.48	52.19	57.50	52.75
	FoREST	33.95	46.01	46.95	44.89	49.06	51.50	48.79	48.21	51.64	50.10	49.41	49.00	48.69
Localization	SpartQA	31.23	31.96	29.11	30.20	31.68	29.00	30.20	29.53	30.12	29.35	28.84	30.03	28.97
	SpatialMap	16.22	22.77	28.18	25.95	30.07	31.04	30.62	30.44	27.76	31.19	30.74	34.05	31.92

Appendix

Table 11: A3B pad spatial scores across various

D

	Model Sizes ( $N$ )	A71M	A128M	A340M	A590M	A874M	A3B
Spatial Relations	StepGame	13.50	13.51	17.14	20.94	22.34	30.02
	SpaRTUN	16.03	19.69	25.82	24.59	25.98	33.32
	bAbI(positional reasoning)	47.33	47.05	48.37	48.41	48.92	52.75
	FoREST	27.69	36.37	45.08	42.93	47.61	48.69
Localization	SpartQA	22.81	26.62	31.80	31.03	32.33	28.97
	SpatialMap	8.32	18.33	18.88	24.85	26.63	31.92

Appendix

Table 12: pad spatial scores across various

N

Multimodal Pretraining

Multimodal pretraining learns reusable representations by training models on paired or mixed data from multiple modalities, such as text, images, audio, and signals.

See all 182 papers

Scaling Native Multimodal Pre-Training From Scratch
Jul 24, 2026Haoyuan Wu, Aoqi Wu, Hai Wang +3Cross-Modal Learning Multimodal Pretraining
The Chinese University of Hong Kong · LLM Department, Tencent
Beyond Language Modeling: An Exploration of Multimodal Pretraining
Mar 3, 2026Shengbang Tong, David Fan, John Nguyen +18Multimodal Pretraining Unified Multimodal Models
FAIR, Meta · New York University
Caption First, VQA Second: Knowledge Density, Not Task Format, Drives Multimodal Scaling
Mar 17, 2026Hongjian Zou, Yue Ge, Qi Ding +2Multimodal Large Language Models Multimodal Learning
vivo AI Lab, Shenzhen, China · Wuhan University, Wuhan, China

Jul 24, 2026J/K move · Enter open · S save

Scaling Native Multimodal Pre-Training From Scratch

Haoyuan Wu, Aoqi Wu, Hai Wang, Jiajia Wu+2

The Chinese University of Hong Kong · LLM Department, Tencent

Cross-Modal Learning Multimodal Pretraining Scaling Laws Multimodal Foundation Models

Open paper AskPDF arXiv

Although large language models (LLMs) exhibit remarkable reasoning capabilities, their reliance on text-only pre-training restricts the perception of the multimodal physical world. Native multimodal pre-training avoids this limitation by training models from scratch on multimodal inputs, thereby achieving deep cross-modal integration and mitigating optimization asymmetries inherent to traditional late-fusion architectures. Despite these advantages, the scaling properties of this paradigm remain incompletely characterized. To address this gap, we investigate the optimal model size and token count for training a Transformer-based vision-language model under a fixed computational budget. Our study demonstrates that minimal objective loss adheres to a predictable compute law, whereas compute-optimal model sizes and token counts scale as power laws. Notably, language and multimodal objectives manifest distinct allocation trends. The language allocation exponents lie in a similar range across the different data mixtures. The multimodal model-allocation exponent decreases modestly with the multimodal token ratio, indicating a relative shift toward token allocation. Additionally, our scaling analysis yields a budget-compensation rule. Specifically, an additional multimodal-token budget can offset the text-efficiency penalty caused by incorporating visual information into native multimodal pre-training under a fixed compute budget. Downstream evaluations further reveal that native multimodal pre-training is associated with improved spatial reasoning and multimodal few-shot learning. Generally, this empirical research establishes the essential groundwork for predictably scaling multimodal foundation models.

Abstract

Multimodal Pretraining

Explore similar work

Scaling Native Multimodal Pre-Training From Scratch

Beyond Language Modeling: An Exploration of Multimodal Pretraining

Caption First, VQA Second: Knowledge Density, Not Task Format, Drives Multimodal Scaling