Beyond Language Modeling: An Exploration of Multimodal Pretraining
Authors: Shengbang Tong, David Fan, John Nguyen, Ellis Brown, Gaoyue Zhou, Shengyi Qian, Boyang Zheng, Théophane Vallaeys, Junlin Han, Rob Fergus, Naila Murray, Marjan Ghazvininejad, Mike Lewis, Nicolas Ballas, Amir Bar, Michael Rabbat, Jakob Verbeek, Luke Zettlemoyer, Koustuv Sinha, Yann LeCun, Saining Xie
The visual world offers a critical axis for advancing foundation models beyond language. Despite growing interest in this direction, the design space for native multimodal models remains opaque. We provide empirical clarity through controlled, from-scratch pretraining experiments, isolating the factors that govern multimodal pretraining without interference from language pretraining. We adopt the Transfusion framework, using next-token prediction for language and diffusion for vision, to train on diverse data including text, video, image-text pairs, and even action-conditioned video. Our experiments yield four key insights: (i) Representation Autoencoder (RAE) provides an optimal unified visual representation by excelling at both visual understanding and generation; (ii) visual and language data are complementary and yield synergy for downstream capabilities; (iii) unified multimodal pretraining leads naturally to world modeling, with capabilities emerging from general training; and (iv) Mixture-of-Experts (MoE) enables efficient and effective multimodal scaling while naturally inducing modality specialization. Through IsoFLOP analysis, we compute scaling laws for both modalities and uncover a scaling asymmetry: vision is significantly more data-hungry than language. We demonstrate that the MoE architecture harmonizes this scaling asymmetry by providing the high model capacity required by language while accommodating the data-intensive nature of vision, paving the way for truly unified multimodal models.
Figures & tables
Figure 1: Overview of our study. Top: High-level model architecture. We train a single autoregressive model with next-token prediction for text and next-visual state prediction with flow matching. Bottom: We study five axes: visual representations, data, world modeling, architecture, and scaling.
Figure 2: Examples of training data. We train on diverse data including text, videos, image-text, and action-conditioned video.
Figure 3: Shared → Modality-specific FFN. Modality separation improves multimodal learning over shared FFN. We explore MoE design in Section 6.1 .
Figure 4: RAE outperforms VAEs for both generation and understanding. RAE (SigLIP 2) achieves the best performance on DPGBench, GenEval, and VQA while maintaining text perplexity comparable to the text-only baseline.
Figure 5
Figure 7: I/T data sources exhibit complementary trade-offs. Each source of I/T data has distinct strengths across text perplexity, generation, and VQA. Combining MetaCLIP (I2T) with Shutterstock (T2I) captures the strengths of both, suggesting that strategically using different I/T sources can outperform using a single source of I/T data.
Figure 7
Figure 10: Multimodal pretraining is better for VQA. Multimodal pretraining (dark bars) consistently outperforms text-only baselines (light bars) after finetuning on VQA.
Figure 11: Navigation actions are encoded directly as text tokens. We utilize data from Navigation World Model (NWM), where a training sequence consists of four context frames and a navigation action formatted as natural language text. The model is trained to predict the resulting target view.
Figure 10
Figure 14: Zero-shot model controllability via natural language. Given four context frames ( ctx0 – ctx3 ), the model predicts eight future frames ( p1 - p8 ) conditioned on WASD-style or natural language navigation actions.
Figure 15: Fine-grained experts improve performance; x -pred benefits high-dimensional diffusion. Increasing granularity improves all metrics up to G=16 . For RAE (SigLIP 2), x -pred (solid) outperforms v -pred (dashed) on image generation; for VAE (FLUX.1), v -pred is better.
Figure 16: Sparsity scales multimodal performance. At fixed active compute (16 experts), increasing total expert count from 32 to 1008 (and thus total parameter count) consistently improves both language (lower PPL) and vision (lower diffusion loss, higher GenEval score).
Figure 17: Semantic representations scale better with sparsity. We fix 16 active experts and increase the total expert count. Text loss improves consistently regardless of the vision encoder, but diffusion loss saturates for VAE (FLUX.1) while continuing to improve for RAE (SigLIP 2).
Configuration
DCLM PPL ↓
Notes PPL ↓
Diff. Loss ↓
GenEval ↑
No Shared Expert
14.802
27.392
0.484
0.360
Global Shared Expert
14.794
27.249
0.483
0.364
Per-Modality Shared Expert
14.785
27.161
0.483
0.367
Table 2: Per-Modality Shared Experts outperform Global Shared Experts. We compare standard Top-16 routing against Global and Per-Modality shared expert strategies. Per-modality yields the best results.
Figure 16
Figure 20: Visual generation and understanding share the same experts. We compare the expert selection rates for image generation (x-axis) versus image understanding (y-axis) across different network depths. We observe very high Pearson correlation ( ≥r=0.90 ) across all layers.
Figure 21: Modality-specific FFN, SigLIP 2, MoE, and x -pred achieve best unified performance. Starting from a Transfusion baseline, we progressively stack the best design choices for FFN, vision encoder, capacity separation, and prediction target. Each row shows Perplexity ( ↓ ) and DPG score ( ↑ ) .
Figure 22: Semantic encoders and MoE capture more world knowledge. WISE scores across six knowledge categories and overall. Models with semantic encoders (MoE, MoT, SigLIP 2) consistently outperform VAE-based models (SD-VAE, FLUX.1) by 3–4 × , with MoE achieving the highest overall score.
Figure 23: Scaling laws for unified dense models. We compute IsoFLOP curves across compute budgets. We project the optimal model size and token count for each FLOP budget, and find that the vision modality is more data-hungry, while text follows Chinchilla-like scaling.
Figure 24: Scaling laws and efficiency benchmarks for unified dense models. We fit L(C)=A⋅C−α+E to each model class. Dense Multimodal outperforms both the unimodal Text-Only and T2I-Only baselines across all four metrics.
Figure 25: Scaling laws for multimodal MoE models. MoE narrows the vision-language exponent gap from 0.10 (dense) to 0.05 ( Dopt∝C0.64 for vision vs. C0.59 for language).
Figure 26: Scaling laws and efficiency benchmarks for unified MoE models. MoE Multimodal closely tracks unimodal MoE baselines across all four metrics, with near-identical FID and competitive language perplexity at 1021 FLOPs.
Appendix figures & tables13 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 27: Qualitative reconstruction results across various SigLIP 2 layers using RAE. Observations indicate that earlier layers yield higher-fidelity reconstructions. We trained our own decoder over 3M images for this ablation.
Figure 28: SigLIP 2 encoder
DPG Score ( ↑ )
Perplexity ( ↓ )
Encoder
Uncentered
Centered
Uncentered
Centered
SigLIP 2
0.570
0.612 (+.042)
15.06
15.36 (+.30)
FLUX.1
0.498
0.510 (+.012)
14.96
15.14 (+.18)
SD-VAE
0.470
0.480 (+.010)
15.13
15.35 (+.22)
Appendix
Table 3: Effect of loss centering on generation quality (DPG ↑ ) and text perplexity (PPL ↓ ) across visual encoders. Centering consistently improves DPG at a modest PPL cost.
Figure 31: Multimodal co-training exceeds unimodal performance. We compare multimodal models (solid lines) against unimodal baselines (dashed lines) across varying budgets.
Type
Encoder
Resolution
Patch Size
Grid Size
Spatial Adaptation
Tokens
Raw Pixel
Raw Pixel
2242
14 × 14
16×16
—
256
RAE
SigLIP 2 So400m
2242
14 × 14
16×16
—
256
DINOv2-L
2242
14 × 14
16×16
—
256
WebSSL-L
2242
14 × 14
16×16
—
256
VAE
SD-VAE
2562
8× 8
32×32
Pixel[Un]shuffle(2)
256
FLUX.1 VAE
2562
8× 8
32×32
Pixel[Un]shuffle(2)
256
Appendix
Table 4: Vision encoder configurations. All encoders produce 256 tokens ( 16×16 grid) per image. For VAE encoders that natively produce 32×32=1024 latent patches, we apply PixelUnshuffle( 2 ) ( Shi et al., 2016 ) to rearrange the spatial grid into channels, reducing to 16×16=256 tokens while increasing the channel dimension by 4× . To apply the diffusion loss, we then apply PixelShuffle( 2 ) to upsample the feature map back to the original spatial resolution.
Configuration
Total Params
Active Params
Dense Baseline
1.50B
1.50B
E=32
2.30B
1.50B
E=64
3.92B
1.50B
E=128
7.14B
1.50B
E=256
13.59B
1.51B
E=512
26.48B
1.52B
Appendix
Table 5: Hyperparameters and parameter counts for the sparsity (IsoFLOP) sweep. We scale total parameters by increasing the expert pool E while keeping active computation constant. The slight increase in active params is due to the router.
Figure 32: More navigation examples with natural language commands (I).
Figure 33: More navigation examples with natural language commands (II).
Figure 34: More navigation examples with natural language commands (III).
Figure 35: More navigation examples with natural language commands (IV).
Figure 36: More navigation examples with natural language commands (V).
Figure 37: Counterfactual navigation trajectories from the same context. The model is able to generate counterfactual trajectories following different action sequences.
Multimodal pretraining learns reusable representations by training models on paired or mixed data from multiple modalities, such as text, images, audio, and signals.
Vision offers a critical axis for advancing foundation models, driving a shift towards natively unified multimodal pretraining. Despite this momentum, the design space and the fundamental mechanisms of how modalities interact during unified training remain underexplored. We provide empirical clarity through a systematic exploration of multimodal pretraining. Our controlled experiments on both synthetic and large-scale real-world datasets yield four key insights into the physics of multimodal pretraining: (i) Knowledge Flow: We disentangle how language, visual understanding, and visual generation transfer knowledge across modalities, revealing distinct patterns of influence and asymmetry; (ii) Synergy vs. Competition: We show that data "complexity" largely determines whether modalities are synergistic, identify architectural choices that promote synergy: such as shared attention and normalization with modality-specific feed-forward layers, and find that these behaviors generalize across different visual tokenizer designs; (iii) Early Unification: Unifying modalities from the very early stages and training them jointly is shown to be more effective than late alignment or sequential training. This process uncovers a vision laziness phenomenon, where delayed integration leads models to rely on language priors; (iv) Recipes: We derive efficient pretraining recipes that achieve strong generative performance using only 5% of the compute budget. These core findings are subsequently validated at scale by training multiple 13.5B MoE models on 2T tokens. We hope this study provides a principled foundation for understanding and scaling multimodal pretraining.
Although large language models (LLMs) exhibit remarkable reasoning capabilities, their reliance on text-only pre-training restricts the perception of the multimodal physical world. Native multimodal pre-training avoids this limitation by training models from scratch on multimodal inputs, thereby achieving deep cross-modal integration and mitigating optimization asymmetries inherent to traditional late-fusion architectures. Despite these advantages, the scaling properties of this paradigm remain incompletely characterized. To address this gap, we investigate the optimal model size and token count for training a Transformer-based vision-language model under a fixed computational budget. Our study demonstrates that minimal objective loss adheres to a predictable compute law, whereas compute-optimal model sizes and token counts scale as power laws. Notably, language and multimodal objectives manifest distinct allocation trends. The language allocation exponents lie in a similar range across the different data mixtures. The multimodal model-allocation exponent decreases modestly with the multimodal token ratio, indicating a relative shift toward token allocation. Additionally, our scaling analysis yields a budget-compensation rule. Specifically, an additional multimodal-token budget can offset the text-efficiency penalty caused by incorporating visual information into native multimodal pre-training under a fixed compute budget. Downstream evaluations further reveal that native multimodal pre-training is associated with improved spatial reasoning and multimodal few-shot learning. Generally, this empirical research establishes the essential groundwork for predictably scaling multimodal foundation models.
Haoyuan Wu, Aoqi Wu, Hai Wang +3
The Chinese University of Hong Kong · LLM Department, Tencent
Native multimodal pre-training trains a single model from scratch on text and multimodal data, enabling it to understand both language and visual inputs. However, how visual signals shape scaling behavior and support capabilities beyond visual understanding remains unclear. We investigate this question through controlled visual-content ablations across model sizes and training budgets. Specifically, we replace image embeddings with content-free padding token embeddings. Retaining visual content improves text prediction on image-text examples without materially changing the lowest text-only loss at a given compute budget, the optimal balance of model size and training data, or aggregate language scores. The relative multimodal loss benefit grows approximately as a power law of compute. To test benefits beyond understanding images, we evaluate spatial reasoning from text descriptions alone. Both conditions improve with model and data scaling. During training of the largest model, visual gains are most sustained in localization, navigation, and symbolic planning, which involve maintaining object configurations or composing spatial updates; gains in spatial relations, geometry, and physical commonsense are less consistent. These findings identify visual content as a scalable anchor for multimodal learning, with benefits that extend beyond visual inputs while preserving aggregate language performance.
Haoyuan Wu, Aoqi Wu, Jiani Hu +1
The Chinese University of Hong Kong · Foundation Model Department, Tencent