Frontier AI systems are increasingly natively multimodal, jointly pretrained on multiple modalities, such as text, vision, and audio. However, the scientific understanding of the multimodal data mixtures used to pretrain these systems --- how much of each modality to mix, and at what potential interference cost to the others --- is largely absent or gatekept. In this work, we derive data mixing scaling laws for three modalities: text, vision, and audio. Multimodal Large Language Models (MLLMs) increasingly adopt Mixture-of-Experts (MoE) architectures, as MoEs scale efficiently and partially limit cross-modal interference. We therefore use MoEs to conduct 268 experiments ranging from 457M to 8.3B parameters, on up to 150 billion tokens. Our work leads to multiple actionable insights into multimodal data mixing for training MLLMs. First, compute optima are modality-specific: audio modality is model-heavy, favoring scaling parameters over tokens, while text and vision require balanced scaling of model size and data. Second, repeating vision or audio data beyond 2x yields negligible benefits. Finally, cross-modal interactions are highly asymmetric: vision strongly interferes with audio, whereas audio's interference with learning text and vision is negligible. Overall, we provide a principled foundation for understanding the multimodal data mixtures needed to train frontier MLLMs.
Figures & tables
Figure 1: Predicted loss curves and observed values for OOD training scales and regimes: (i) Val Scratch : a 12.6B-A3.8B MoE trained for 200B tokens (185B text, 7.5B vision, 7.5B audio); (ii) Val non-text-heavy CPT : continual pretraining (CPT) of a text-only 8.3B-A2.6B MoE on 50B tokens (10B text, 20B (unique 10B × 2) each vision and audio); and (iii) Val text-heavy CPT : CPT of text-only 8.3B-A2.6B MoE on 40B text, 5B each of vision and audio. Observed points are very close to the predicted curves. The only exception is the text loss in (ii), where CPT causes a significant spike in text loss (Appendix H ) and does not stabilize, unlike (iii), which has a higher text proportion.
Figure 2: Validation loss as a function of model size for text+vision (left) and text+audio (right) training at 100 B total tokens, averaged across modality-specific evaluation sets. Solid curves correspond to training with 5 B– 25 B unique non-text tokens; dashed purple curves repeat 5 B unique tokens 2 – 5 times. At every model scale, repeating data 5 × underperforms training on roughly 15 B unique tokens, motivating the diminishing-returns form of ϕm(rm) in Equation 4 .
Modality / Reference
Em
αm
βm
Nopt∝Ca
Dopt∝Cb
Text
1.91
0.34
0.34
0.50
0.50
Vision
5.67
0.50
0.51
0.50
0.50
Audio
4.36
0.32
0.52
0.62
0.38
Chinchilla [ 19 ] (text-dense)
1.69
0.34
0.28
0.46
0.54
TGL [ 59 ] (text-MoE)
–
–
–
0.51
0.49
NMM [ 53 ] (vision-dense)
1.90
0.30
0.34
0.53
0.47
Table 1: Fitted scaling-law exponents for each modality in comparison to prior art: Chinchilla [ 19 ] and TGL [ 59 ] for text, NMM [ 53 ] for vision, and SODA [ 36 ] for audio. We report the irreducible loss Em , the scaling exponents αm,βm , and the induced compute-optimal exponents a,b .
Figure 3: (Left) Cross-Modal Transfer Matrix of directional log-transfer coefficients vk→m ; positive values (blue) indicate synergistic transfer and negative values (red) indicate interference. (Middle, Right) Marginal worth of each successive epoch for audio and vision, expressed as a fraction of one unique-data epoch. Repetition beyond two epochs yields steeply diminishing returns: by the third epoch, each new pass is worth ≤43.2% unique data for audio and ≤23.2% unique data for vision.
Appendix figures & tables16 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 4: Validation loss as a function of (left) training tokens D at fixed model sizes and (right) total parameter count N at fixed dataset sizes. Text has the lowest irreducible loss Em , while vision has the highest. Vision also exhibits large fitted scaling exponents for both data and model scale, indicating steep improvements with both additional data ( βvision ) and model parameters ( αvision ).
Figure 5: Training losses across different multipliers of optimum learning rates (LR) and batch sizes (BS) obtained using Step Law [ 28 ] . dmodel=384 and dmodel=2048 denote the smallest and largest models studied in our work, respectively. The predicted learning rate by Step Law is the best for our case as well, but the optimal batch size is twice the predicted value. It is very close to the best, if not the best, for all three modalities: text, vision, and audio.
dmodel
Active (M)
Total (M)
Embedding (M)
384
54.5
252.7
204.7
576
122.4
568.3
307.0
768
217.5
1010.2
409.3
1152
489.0
2272.7
614.0
1536
869.1
4040.0
818.7
2048
1544.7
7181.8
1091.6
Appendix
Table 2: Hidden size and parameter counts (millions) for all the models used in this study. The active and total parameter count only consists of the non-embedding parameters. Model hyperparameters other than the hidden dimension ( dmodel ) are mentioned in Section 3 .
Model name
Model
dmodel
ep
mbs
gbs
Learning Rate
Tokens (B)
M1
457M-A259M
384
1
4
256
1.73×10−3
5
256
2.15×10−3
10
256
2.43×10−3
15
256
2.65×10−3
20
256
2.84×10−3
25
256
3.28×10−3
40
Appendix
Table 3: Training hyperparameters for different models. Model labels follow the (total params)-A(active params) convention with both counts including untied embeddings. ep , mbs , and gbs denote the size of the expert parallel group, micro batch size, and the global batch size, respectively. Model hyperparameters other than the hidden dimension ( dmodel ) are mentioned in Section 3 .
Figure 6: Validation loss as a function of model size for the text+vision repetition experiments at 100 B total training tokens, on the text-only evaluation set (left) and averaged across the two vision evaluation sets (right). Solid green curves vary the unique-vision-token budget from 5 B to 25 B; dashed purple curves repeat a fixed 5 B unique-vision budget 2 – 5 times. Text loss is essentially independent of the vision repetition factor, while on the averaged vision panel the gap between repeated and unique-data curves widens with model scale, with 5B×5 ( 25 B sampled) falling below the 15 B unique-token curve at the largest scales.
Figure 7: Validation loss as a function of model size for the text+audio repetition experiments at 100 B total training tokens, on the text-only evaluation set (left) and averaged across the audio evaluation sets (right). Solid red curves vary the unique-audio-token budget from 5 B to 25 B; dashed purple curves repeat a fixed 5 B unique-audio budget 2 – 5 times. As with vision, text loss is essentially unaffected by the audio repetition factor, while on the averaged audio panel repeated data underperforms unique data more strongly at larger model scales: 5B×5 tracks roughly the 10 B unique-token curve, indicating that ∼60% of the additional compute spent on repetition produces little marginal value.
Figure 8: Validation loss as a function of model size for the vision–audio interference experiments under a fixed 150 B-token budget with text held constant at 100 B. The remaining 50 B tokens are traded between vision and audio in five splits ( 45 B/ 5 B, 41.5 B/ 8.5 B, 37.5 B/ 12.5 B, 33 B/ 17 B, 25 B/ 25 B), running from vision-dominant (green) to audio-dominant (red). Panels show loss on the text-only evaluation set (left), averaged across the audio evaluation sets (middle), and averaged across the vision evaluation sets (right).
Table 4: Refits across four interpolation leave-one-out holdouts (M2, M3, M4, M5, with the extremes M1 (smallest) and M6 (largest) always included in training). The Joint-fit column in (a) reports coefficients from the fit on all configurations (M1-M6), while Leave-one-scale-out coefficient estimates are averaged over the four leave-one-out folds and reported as mean ± standard deviation. The metrics in (b) are computed on runs at the scale held out in each fold.
Coefficient
Joint-fit
Random-subset Refits
Base law (per modality)
EText
1.910
2.010±0.085
αText
0.337
0.360±0.048
βText
0.337
0.401±0.062
EVision
5.670
5.675±0.068
αVision
0.568
0.535±0.067
Appendix
Table 5: Coefficients under random-subset resampling: five independent refits, each on a random 90% subset of the fitted configurations. The Joint-fit column reports coefficients fit on the full data. Random-subset Refit estimates are averaged over five 90% subsamples of runs and reported as mean ± standard deviation. Coefficients that contain zero within one standard deviation of the average value are marked † .
Table 6: Refits under training extrapolation regimes. Columns R{ x -1} (train M1- x ) denote the regime where runs corresponding to models M1, M2, … M x are used for fitting, whereas the Joint-fit column reports coefficients from the fit on all configurations (M1-M6). The metrics in (b) are validated using M6 runs only (the most distant out-of-sample scale), so the four regimes are directly comparable. Both RMSE log and Overall Maximum Relative Error decrease monotonically as more model scales are included in the fit.
Variant
k
RMSE log
Δ vs Full
RMSE Text
RMSE Vision
RMSE Audio
Max rel. err. (%)
Full law
26
0.0064
–
0.0066
0.0053
0.0073
2.16
No repetition
24
0.0096
+49.4%
0.0062
0.0112
0.0107
3.62
No transfer
20
0.0064
−0.7%
0.0061
0.0061
0.0070
2.61
No capacity
23
0.0089
+38.4%
0.0077
0.0074
0.0111
5.24
Appendix
Table 7: Component ablations of the joint fit on the full training set. The “ Δ vs Full” column shows the relative increase in overall RMSE log when the corresponding component is removed. k denotes the parameters to fit in the resulting scaling law. Per-modality RMSE log and the overall maximum relative error, both computed on the fitted set, are shown for context. The full law achieves the minimum in the Maximum Relative Error ( Max. rel. err. ) column.
Regime
ntrain
RMSE log
Maximum relative error (%)
Overall
Text
Vision
Audio
R1 (train M1-2)
80
0.0441
7.91
5.41
3.88
7.91
R2 (train M1-3)
120
0.0358
6.54
3.89
5.60
6.54
R3 (train M1-4)
160
0.0208
4.84
3.35
4.84
3.62
R4 (train M1-5)
200
0.0101
2.54
2.54
2.21
2.13
Appendix
Table 8: RMSE log and maximum relative errors (%) for the directional-capacity variant of our scaling law: a separate ck→m for each (source, target) pair instead of one cm per target in Eq. 2 . We use the same training extrapolation regimes as Table 6 where R{ x -1} (train M1- x ) denote the regime where runs corresponding to models M1, M2, … M x are used for fitting, and the metrics are computed on M6 only. ntrain denotes the number of training points in the respective regime. Even though the RMSE log and overall maximum relative errors are comparable, the vision errors are strictly worse than the non-directional version (Table 6(b) ).
RMSE log
Maximum relative error (%)
Variant
R1
R2
R3
R4
R1
R2
R3
R4
Full law
0.0430
0.0349
0.0217
0.0094
8.27
6.91
4.06
2.46
No transfer
0.0469
0.0379
0.0138
0.0095
10.55
7.94
3.71
2.22
No capacity
0.0436
0.0355
0.0212
0.0139
9.96
7.76
5.58
5.38
No repetition
0.0456
0.0362
0.0218
0.0127
9.30
6.74
4.28
2.70
Directional capacity
0.0441
0.0358
0.0208
0.0101
7.91
6.54
4.84
2.54
Appendix
Table 9: Metrics on the held-out M6 scale across the four training extrapolation regimes (R1: train M1-2, R2: train M1-3, R3: train M1-4, R4: train M1-5). Five variants of our law are compared: the full law, three variants with each modeling component individually removed, and the directional-capacity variant. Lower is better in every column. Under these extrapolation regimes, we observe that the full law and the directional capacity variant perform consistently well across the regimes.
Figure 9: The overall training loss and the text-only loss for (i) Continual run 1 : Continual pretraining (CPT) on a text-only checkpoint trained with 100B tokens. The continual training mixture consists of 10B text, 20B vision (10B repeated twice), and 20B audio (10B repeated twice), (ii) Continual Run 2 : Continual pretraining on the same checkpoint as (i) with 40B text, 5B vision, and 5B audio.
Modality
Observed
Predicted
Δ
Relative error
12.6B-A3.8B, from scratch on 200B tokens (185B text / 7.5B vision / 7.5B audio)
Table 10: Predicted vs. observed validation loss on three held-out runs. Predictions use the fitted coefficients from Table 1 and Section 5 . No parameters are re-tuned on these runs, and 8/9 losses are predicted to within 2.51%.
Text vs. vision
Text vs. audio
Model
95/5
75/25
95/5
75/25
M1
0.026
0.010
0.014
0.011
M2
0.007
0.005
0.005
0.005
M3
0.010
0.008
0.006
0.009
M4
0.006
0.006
0.007
0.007
M5
0.005
0.005
0.005
0.004
Appendix
Table 11: Mean per-layer Jensen–Shannon divergence between the expert-assignment distributions of two modalities, for every model scale. Mixture labels give the token split between the two modalities present in training. Values near zero indicate that the two modalities route to the same experts in the same proportions; the scale is normalized to [0,1] . Barring the smallest model M1, all the divergences are ≤0.01 , indicating negligible modality-wise expert segregation. This hints that modalities compete for representational capacity within the same experts leading to cm<0 .
Multimodal pretraining learns reusable representations by training models on paired or mixed data from multiple modalities, such as text, images, audio, and signals.
Although large language models (LLMs) exhibit remarkable reasoning capabilities, their reliance on text-only pre-training restricts the perception of the multimodal physical world. Native multimodal pre-training avoids this limitation by training models from scratch on multimodal inputs, thereby achieving deep cross-modal integration and mitigating optimization asymmetries inherent to traditional late-fusion architectures. Despite these advantages, the scaling properties of this paradigm remain incompletely characterized. To address this gap, we investigate the optimal model size and token count for training a Transformer-based vision-language model under a fixed computational budget. Our study demonstrates that minimal objective loss adheres to a predictable compute law, whereas compute-optimal model sizes and token counts scale as power laws. Notably, language and multimodal objectives manifest distinct allocation trends. The language allocation exponents lie in a similar range across the different data mixtures. The multimodal model-allocation exponent decreases modestly with the multimodal token ratio, indicating a relative shift toward token allocation. Additionally, our scaling analysis yields a budget-compensation rule. Specifically, an additional multimodal-token budget can offset the text-efficiency penalty caused by incorporating visual information into native multimodal pre-training under a fixed compute budget. Downstream evaluations further reveal that native multimodal pre-training is associated with improved spatial reasoning and multimodal few-shot learning. Generally, this empirical research establishes the essential groundwork for predictably scaling multimodal foundation models.
Haoyuan Wu, Aoqi Wu, Hai Wang +3
The Chinese University of Hong Kong · LLM Department, Tencent
Multimodal large language models (MLLMs) have achieved rapid progress, yet their scaling behavior remains less clearly characterized and often less predictable than that of text-only LLMs. Increasing model size and task diversity often yields diminishing returns. In this work, we argue that the primary bottleneck in multimodal scaling is not task format, but knowledge density in training data. We first show that task-specific supervision such as Visual Question Answering (VQA) contributes little incremental semantic information beyond image captions: VQA signals can be reconstructed from captions with negligible performance loss. We then demonstrate that increasing knowledge density -- through structured caption enrichment and cross-modal knowledge injection -- leads to consistent performance improvements across multimodal and downstream benchmarks. Across controlled experiments, performance correlates more strongly with semantic coverage than with task diversity. These findings suggest that current MLLMs fail to scale primarily because training data lacks sufficient knowledge coverage. We advocate for knowledge-centric multimodal training as a principled foundation for scalable multimodal models.
Hongjian Zou, Yue Ge, Qi Ding +2
vivo AI Lab, Shenzhen, China · Wuhan University, Wuhan, China
A data mixture refers to how different data sources are combined to train large language models, and selecting an effective mixture is crucial for optimal downstream performance. Existing methods either conduct costly searches directly on the target model or rely on mixture scaling laws that fail to extrapolate well to large model sizes. We address these limitations by introducing a compute-efficient pipeline for data mixture scaling. First, we propose CAMEL, a capacity-aware mixture law that models validation loss with the nonlinear interplay between model size and mixture. We also introduce a loss-to-benchmark prediction law that estimates benchmark accuracy from validation loss, enabling end-to-end performance prediction for the target model. Next, we study how to allocate a fixed compute budget across model scales to fit the law and reduce prediction error. Finally, we apply our method to Mixture-of-Experts models with up to 7B-A150M parameters to fit the law, and verify the optimal mixture derived from the law by extrapolating to a 55B-A1.2B target model. Compared to prior methods, we reduce mixture optimization costs by 50% and improves downstream benchmark performance by up to 3%.
Jingwei Li, Xinran Gu, Jingzhao Zhang
Institute for Interdisciplinary Information Sciences, Tsinghua University · Shanghai Qizhi Institute