Beyond Language Modeling: An Exploration of Multimodal Pretraining

Shengbang Tong; David Fan; John Nguyen; Ellis Brown; Gaoyue Zhou; Shengyi Qian; Boyang Zheng; Théophane Vallaeys

cs.CVMar 3, 2026

Beyond Language Modeling: An Exploration of Multimodal Pretraining

Authors: Shengbang Tong, David Fan, John Nguyen, Ellis Brown, Gaoyue Zhou, Shengyi Qian, Boyang Zheng, Théophane Vallaeys,

Organizations: FAIR, Meta · New York University

Abstract

The visual world offers a critical axis for advancing foundation models beyond language. Despite growing interest in this direction, the design space for native multimodal models remains opaque. We provide empirical clarity through controlled, from-scratch pretraining experiments, isolating the factors that govern multimodal pretraining without interference from language pretraining. We adopt the Transfusion framework, using next-token prediction for language and diffusion for vision, to train on diverse data including text, video, image-text pairs, and even action-conditioned video. Our experiments yield four key insights: (i) Representation Autoencoder (RAE) provides an optimal unified visual representation by excelling at both visual understanding and generation; (ii) visual and language data are complementary and yield synergy for downstream capabilities; (iii) unified multimodal pretraining leads naturally to world modeling, with capabilities emerging from general training; and (iv) Mixture-of-Experts (MoE) enables efficient and effective multimodal scaling while naturally inducing modality specialization. Through IsoFLOP analysis, we compute scaling laws for both modalities and uncover a scaling asymmetry: vision is significantly more data-hungry than language. We demonstrate that the MoE architecture harmonizes this scaling asymmetry by providing the high model capacity required by language while accommodating the data-intensive nature of vision, paving the way for truly unified multimodal models.

Figures & tables

Figure 1: Overview of our study. Top: High-level model architecture. We train a single autoregressive model with next-token prediction for text and next-visual state prediction with flow matching. Bottom: We study five axes: visual representations, data, world modeling, architecture, and scaling.

Figure 2: Examples of training data. We train on diverse data including text, videos, image-text, and action-conditioned video.

Figure 3: Shared

\rightarrow

Modality-specific FFN. Modality separation improves multimodal learning over shared FFN. We explore MoE design in Section 6.1 .

Figure 4: RAE outperforms VAEs for both generation and understanding. RAE (SigLIP 2) achieves the best performance on DPGBench, GenEval, and VQA while maintaining text perplexity comparable to the text-only baseline.

Figure 7: I/T data sources exhibit complementary trade-offs. Each source of I/T data has distinct strengths across text perplexity, generation, and VQA. Combining MetaCLIP (I2T) with Shutterstock (T2I) captures the strengths of both, suggesting that strategically using different I/T sources can outperform using a single source of I/T data.

Figure 10: Multimodal pretraining is better for VQA. Multimodal pretraining (dark bars) consistently outperforms text-only baselines (light bars) after finetuning on VQA.

Figure 11: Navigation actions are encoded directly as text tokens. We utilize data from Navigation World Model (NWM), where a training sequence consists of four context frames and a navigation action formatted as natural language text. The model is trained to predict the resulting target view.

Figure 14: Zero-shot model controllability via natural language. Given four context frames ( ctx0 – ctx3 ), the model predicts eight future frames ( p1 - p8 ) conditioned on WASD-style or natural language navigation actions.

Figure 15: Fine-grained experts improve performance;

x

-pred benefits high-dimensional diffusion. Increasing granularity improves all metrics up to

G{=}16

. For RAE (SigLIP 2),

x

-pred (solid) outperforms

v

-pred (dashed) on image generation; for VAE (FLUX.1),

v

-pred is better.

Figure 16: Sparsity scales multimodal performance. At fixed active compute (16 experts), increasing total expert count from 32 to 1008 (and thus total parameter count) consistently improves both language (lower PPL) and vision (lower diffusion loss, higher GenEval score).

Figure 17: Semantic representations scale better with sparsity. We fix 16 active experts and increase the total expert count. Text loss improves consistently regardless of the vision encoder, but diffusion loss saturates for VAE (FLUX.1) while continuing to improve for RAE (SigLIP 2).

Configuration	DCLM PPL $\downarrow$	Notes PPL $\downarrow$	Diff. Loss $\downarrow$	GenEval $\uparrow$
No Shared Expert	14.802	27.392	0.484	0.360
Global Shared Expert	14.794	27.249	0.483	0.364
Per-Modality Shared Expert	14.785	27.161	0.483	0.367

Table 2: Per-Modality Shared Experts outperform Global Shared Experts. We compare standard Top-16 routing against Global and Per-Modality shared expert strategies. Per-modality yields the best results.

Figure 20: Visual generation and understanding share the same experts. We compare the expert selection rates for image generation (x-axis) versus image understanding (y-axis) across different network depths. We observe very high Pearson correlation (

\geq r=0.90

) across all layers.

Figure 21: Modality-specific FFN, SigLIP 2, MoE, and

x

-pred achieve best unified performance. Starting from a Transfusion baseline, we progressively stack the best design choices for FFN, vision encoder, capacity separation, and prediction target. Each row shows Perplexity (

\downarrow

) and DPG score (

\uparrow

) .

Figure 22: Semantic encoders and MoE capture more world knowledge. WISE scores across six knowledge categories and overall. Models with semantic encoders (MoE, MoT, SigLIP 2) consistently outperform VAE-based models (SD-VAE, FLUX.1) by 3–4

\times

, with MoE achieving the highest overall score.

Figure 23: Scaling laws for unified dense models. We compute IsoFLOP curves across compute budgets. We project the optimal model size and token count for each FLOP budget, and find that the vision modality is more data-hungry, while text follows Chinchilla-like scaling.

Figure 24: Scaling laws and efficiency benchmarks for unified dense models. We fit

L(C)=A\cdot C^{-\alpha}+E

to each model class. Dense Multimodal outperforms both the unimodal Text-Only and T2I-Only baselines across all four metrics.

Figure 25: Scaling laws for multimodal MoE models. MoE narrows the vision-language exponent gap from 0.10 (dense) to 0.05 (

D_{\text{opt}}\propto C^{0.64}

for vision vs.

C^{0.59}

for language).

Figure 26: Scaling laws and efficiency benchmarks for unified MoE models. MoE Multimodal closely tracks unimodal MoE baselines across all four metrics, with near-identical FID and competitive language perplexity at

10^{21}

FLOPs.

Appendix figures & tables13 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 27: Qualitative reconstruction results across various SigLIP 2 layers using RAE. Observations indicate that earlier layers yield higher-fidelity reconstructions. We trained our own decoder over 3M images for this ablation.

Figure 28: SigLIP 2 encoder

	DPG Score ( $\uparrow$ )		Perplexity ( $\downarrow$ )
Encoder	Uncentered	Centered	Uncentered	Centered
SigLIP 2	0.570	0.612 (+.042)	15.06	15.36 (+.30)
FLUX.1	0.498	0.510 (+.012)	14.96	15.14 (+.18)
SD-VAE	0.470	0.480 (+.010)	15.13	15.35 (+.22)

Appendix

Table 3: Effect of loss centering on generation quality (DPG

\uparrow

) and text perplexity (PPL

\downarrow

) across visual encoders. Centering consistently improves DPG at a modest PPL cost.

Figure 31: Multimodal co-training exceeds unimodal performance. We compare multimodal models (solid lines) against unimodal baselines (dashed lines) across varying budgets.

Type	Encoder	Resolution	Patch Size	Grid Size	Spatial Adaptation	Tokens
Raw Pixel	Raw Pixel	$224^{2}$	14 $\times$ 14	$16{\times}16$	—	256
RAE	SigLIP 2 So400m	$224^{2}$	14 $\times$ 14	$16{\times}16$	—	256
RAE	DINOv2-L	$224^{2}$	14 $\times$ 14	$16{\times}16$	—	256
	WebSSL-L	$224^{2}$	14 $\times$ 14	$16{\times}16$	—	256
VAE	SD-VAE	$256^{2}$	$8{\times}$ 8	$32{\times}32$	Pixel[Un]shuffle(2)	256
VAE	FLUX.1 VAE	$256^{2}$	$8{\times}$ 8	$32{\times}32$	Pixel[Un]shuffle(2)	256

Appendix

Table 4: Vision encoder configurations. All encoders produce 256 tokens (

16{\times}16

grid) per image. For VAE encoders that natively produce

32{\times}32=1024

latent patches, we apply PixelUnshuffle(

2

) ( Shi et al., 2016 ) to rearrange the spatial grid into channels, reducing to

16{\times}16=256

tokens while increasing the channel dimension by

4{\times}

. To apply the diffusion loss, we then apply PixelShuffle(

2

) to upsample the feature map back to the original spatial resolution.

Configuration	Total Params	Active Params
Dense Baseline	1.50B	1.50B
$E=32$	2.30B	1.50B
$E=64$	3.92B	1.50B
$E=128$	7.14B	1.50B
$E=256$	13.59B	1.51B
$E=512$	26.48B	1.52B

Appendix

Table 5: Hyperparameters and parameter counts for the sparsity (IsoFLOP) sweep. We scale total parameters by increasing the expert pool

E

while keeping active computation constant. The slight increase in active params is due to the router.

Figure 32: More navigation examples with natural language commands (I).

Figure 33: More navigation examples with natural language commands (II).

Figure 34: More navigation examples with natural language commands (III).

Figure 35: More navigation examples with natural language commands (IV).

Figure 36: More navigation examples with natural language commands (V).

Figure 37: Counterfactual navigation trajectories from the same context. The model is able to generate counterfactual trajectories following different action sequences.

Data Type	Modality	Source
Text	${\color[rgb]{0.3633,0.5547,0.9922}\texttt{T}}\rightarrow{\color[rgb]{0.3633,0.5547,0.9922}\texttt{T}}$	DCLM ( Li et al., 2024b )
Video	${\color[rgb]{0.9648,0.332,0.3125}\texttt{I}}\rightarrow{\color[rgb]{0.9648,0.332,0.3125}\texttt{I}}$	YouTube-Temporal 1B ( Zellers et al., 2022 ) , SomethingSomething V2 ( Goyal et al., 2017 ) , Kinetics ( Kay et al., 2017 )
Image-Text Pairs	${\color[rgb]{0.9648,0.332,0.3125}\texttt{I}}\rightarrow{\color[rgb]{0.3633,0.5547,0.9922}\texttt{T}},{\color[rgb]{0.3633,0.5547,0.9922}\texttt{T}}\rightarrow{\color[rgb]{0.9648,0.332,0.3125}\texttt{I}}$	MetaCLIP ( Xu et al., 2024 ) , in-house Shutterstock
Action-Conditioned	${\color[rgb]{0.9648,0.332,0.3125}\texttt{I}}+{\color[rgb]{0.3633,0.5547,0.9922}\texttt{T}}\rightarrow{\color[rgb]{0.9648,0.332,0.3125}\texttt{I}}$	NWM ( Bar et al., 2025 ) , in-house Text-Annotated Videos

Appendix

Table 6: Training data sources and supervision signals.

Multimodal Pretraining

Multimodal pretraining learns reusable representations by training models on paired or mixed data from multiple modalities, such as text, images, audio, and signals.

See all 182 papers

Towards Physics of Multimodal Pretraining: Knowledge Flow, Modality Synergy, Early Unification, and Recipes
Aug 5, 2026Junlin Han, Shengbang Tong, David Fan +4Unified Multimodal Models Multimodal Pretraining
1FAIR · University of Oxford · 2Reality Labs
Scaling Native Multimodal Pre-Training From Scratch
Jul 24, 2026Haoyuan Wu, Aoqi Wu, Hai Wang +3Cross-Modal Learning Multimodal Pretraining
The Chinese University of Hong Kong · LLM Department, Tencent
More Than Seeing: Vision as Scalable Anchors in Native Multimodal Pre-Training
Oct 14, 2026Haoyuan Wu, Aoqi Wu, Jiani Hu +1Multimodal Pretraining Cross-Modal Learning
The Chinese University of Hong Kong · Foundation Model Department, Tencent

Aug 5, 2026J/K move · Enter open · S save

Towards Physics of Multimodal Pretraining: Knowledge Flow, Modality Synergy, Early Unification, and Recipes

Junlin Han, Shengbang Tong, David Fan, Minghao Chen+3

1FAIR · University of Oxford · 2Reality Labs

Unified Multimodal Models Multimodal Pretraining Cross-Modal Representation Transfer Transfer Learning Multimodal Fusion

Open paper AskPDF arXiv

Vision offers a critical axis for advancing foundation models, driving a shift towards natively unified multimodal pretraining. Despite this momentum, the design space and the fundamental mechanisms of how modalities interact during unified training remain underexplored. We provide empirical clarity through a systematic exploration of multimodal pretraining. Our controlled experiments on both synthetic and large-scale real-world datasets yield four key insights into the physics of multimodal pretraining: (i) Knowledge Flow: We disentangle how language, visual understanding, and visual generation transfer knowledge across modalities, revealing distinct patterns of influence and asymmetry; (ii) Synergy vs. Competition: We show that data "complexity" largely determines whether modalities are synergistic, identify architectural choices that promote synergy: such as shared attention and normalization with modality-specific feed-forward layers, and find that these behaviors generalize across different visual tokenizer designs; (iii) Early Unification: Unifying modalities from the very early stages and training them jointly is shown to be more effective than late alignment or sequential training. This process uncovers a vision laziness phenomenon, where delayed integration leads models to rely on language priors; (iv) Recipes: We derive efficient pretraining recipes that achieve strong generative performance using only 5% of the compute budget. These core findings are subsequently validated at scale by training multiple 13.5B MoE models on 2T tokens. We hope this study provides a principled foundation for understanding and scaling multimodal pretraining.

Abstract

Multimodal Pretraining

Explore similar work

Towards Physics of Multimodal Pretraining: Knowledge Flow, Modality Synergy, Early Unification, and Recipes

Scaling Native Multimodal Pre-Training From Scratch

More Than Seeing: Vision as Scalable Anchors in Native Multimodal Pre-Training