cs.CLOct 10, 2026
SaveScaling Laws for Multimodal Data Mixtures
Organizations: MILA - Quebec AI Institute, McGill University · EPFL · ETH · University of Washington
Abstract
Frontier AI systems are increasingly natively multimodal, jointly pretrained on multiple modalities, such as text, vision, and audio. However, the scientific understanding of the multimodal data mixtures used to pretrain these systems --- how much of each modality to mix, and at what potential interference cost to the others --- is largely absent or gatekept. In this work, we derive data mixing scaling laws for three modalities: text, vision, and audio. Multimodal Large Language Models (MLLMs) increasingly adopt Mixture-of-Experts (MoE) architectures, as MoEs scale efficiently and partially limit cross-modal interference. We therefore use MoEs to conduct 268 experiments ranging from 457M to 8.3B parameters, on up to 150 billion tokens. Our work leads to multiple actionable insights into multimodal data mixing for training MLLMs. First, compute optima are modality-specific: audio modality is model-heavy, favoring scaling parameters over tokens, while text and vision require balanced scaling of model size and data. Second, repeating vision or audio data beyond 2x yields negligible benefits. Finally, cross-modal interactions are highly asymmetric: vision strongly interferes with audio, whereas audio's interference with learning text and vision is negligible. Overall, we provide a principled foundation for understanding the multimodal data mixtures needed to train frontier MLLMs.
Figures & tables
Figure 1: Predicted loss curves and observed values for OOD training scales and regimes: (i) Val Scratch : a 12.6B-A3.8B MoE trained for 200B tokens (185B text, 7.5B vision, 7.5B audio); (ii) Val non-text-heavy CPT : continual pretraining (CPT) of a text-only 8.3B-A2.6B MoE on 50B tokens (10B text, 20B (unique 10B 2) each vision and audio); and (iii) Val text-heavy CPT : CPT of text-only 8.3B-A2.6B MoE on 40B text, 5B each of vision and audio. Observed points are very close to the predicted curves. The only exception is the text loss in (ii), where CPT causes a significant spike in text loss (Appendix H ) and does not stabilize, unlike (iii), which has a higher text proportion.
Figure 2: Validation loss as a function of model size for text+vision (left) and text+audio (right) training at B total tokens, averaged across modality-specific evaluation sets. Solid curves correspond to training with B– B unique non-text tokens; dashed purple curves repeat B unique tokens – times. At every model scale, repeating data 5 underperforms training on roughly B unique tokens, motivating the diminishing-returns form of in Equation 4 .
| Modality / Reference | |||||
| Text | |||||
| Vision | |||||
| Audio | |||||
| Chinchilla [ 19 ] (text-dense) | |||||
| TGL [ 59 ] (text-MoE) | – | – | – | ||
| NMM [ 53 ] (vision-dense) |
Table 1: Fitted scaling-law exponents for each modality in comparison to prior art: Chinchilla [ 19 ] and TGL [ 59 ] for text, NMM [ 53 ] for vision, and SODA [ 36 ] for audio. We report the irreducible loss , the scaling exponents , and the induced compute-optimal exponents .
Figure 3: (Left) Cross-Modal Transfer Matrix of directional log-transfer coefficients ; positive values (blue) indicate synergistic transfer and negative values (red) indicate interference. (Middle, Right) Marginal worth of each successive epoch for audio and vision, expressed as a fraction of one unique-data epoch. Repetition beyond two epochs yields steeply diminishing returns: by the third epoch, each new pass is worth unique data for audio and unique data for vision.
Appendix figures & tables16 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 4: Validation loss as a function of (left) training tokens D at fixed model sizes and (right) total parameter count N at fixed dataset sizes. Text has the lowest irreducible loss , while vision has the highest. Vision also exhibits large fitted scaling exponents for both data and model scale, indicating steep improvements with both additional data ( ) and model parameters ( ).
Figure 5: Training losses across different multipliers of optimum learning rates (LR) and batch sizes (BS) obtained using Step Law [ 28 ] . and denote the smallest and largest models studied in our work, respectively. The predicted learning rate by Step Law is the best for our case as well, but the optimal batch size is twice the predicted value. It is very close to the best, if not the best, for all three modalities: text, vision, and audio.
| Active (M) | Total (M) | Embedding (M) | |
|---|---|---|---|
Table 2: Hidden size and parameter counts (millions) for all the models used in this study. The active and total parameter count only consists of the non-embedding parameters. Model hyperparameters other than the hidden dimension ( ) are mentioned in Section 3 .
| Model name | Model | ep | mbs | gbs | Learning Rate | Tokens (B) | |
|---|---|---|---|---|---|---|---|
| M1 | 457M-A259M | ||||||
Table 3: Training hyperparameters for different models. Model labels follow the (total params)-A(active params) convention with both counts including untied embeddings. ep , mbs , and gbs denote the size of the expert parallel group, micro batch size, and the global batch size, respectively. Model hyperparameters other than the hidden dimension ( ) are mentioned in Section 3 .
Figure 6: Validation loss as a function of model size for the text+vision repetition experiments at B total training tokens, on the text-only evaluation set (left) and averaged across the two vision evaluation sets (right). Solid green curves vary the unique-vision-token budget from B to B; dashed purple curves repeat a fixed B unique-vision budget – times. Text loss is essentially independent of the vision repetition factor, while on the averaged vision panel the gap between repeated and unique-data curves widens with model scale, with ( B sampled) falling below the B unique-token curve at the largest scales.
Figure 7: Validation loss as a function of model size for the text+audio repetition experiments at B total training tokens, on the text-only evaluation set (left) and averaged across the audio evaluation sets (right). Solid red curves vary the unique-audio-token budget from B to B; dashed purple curves repeat a fixed B unique-audio budget – times. As with vision, text loss is essentially unaffected by the audio repetition factor, while on the averaged audio panel repeated data underperforms unique data more strongly at larger model scales: tracks roughly the B unique-token curve, indicating that of the additional compute spent on repetition produces little marginal value.
Figure 8: Validation loss as a function of model size for the vision–audio interference experiments under a fixed B-token budget with text held constant at B. The remaining B tokens are traded between vision and audio in five splits ( B/ B, B/ B, B/ B, B/ B, B/ B), running from vision-dominant (green) to audio-dominant (red). Panels show loss on the text-only evaluation set (left), averaged across the audio evaluation sets (middle), and averaged across the vision evaluation sets (right).
Table 4: Refits across four interpolation leave-one-out holdouts (M2, M3, M4, M5, with the extremes M1 (smallest) and M6 (largest) always included in training). The Joint-fit column in (a) reports coefficients from the fit on all configurations (M1-M6), while Leave-one-scale-out coefficient estimates are averaged over the four leave-one-out folds and reported as mean standard deviation. The metrics in (b) are computed on runs at the scale held out in each fold.
| Coefficient | Joint-fit | Random-subset Refits |
|---|---|---|
| Base law (per modality) | ||
Table 5: Coefficients under random-subset resampling: five independent refits, each on a random subset of the fitted configurations. The Joint-fit column reports coefficients fit on the full data. Random-subset Refit estimates are averaged over five subsamples of runs and reported as mean standard deviation. Coefficients that contain zero within one standard deviation of the average value are marked † .
Table 6: Refits under training extrapolation regimes. Columns R{ x -1} (train M1- x ) denote the regime where runs corresponding to models M1, M2, … M x are used for fitting, whereas the Joint-fit column reports coefficients from the fit on all configurations (M1-M6). The metrics in (b) are validated using M6 runs only (the most distant out-of-sample scale), so the four regimes are directly comparable. Both RMSE and Overall Maximum Relative Error decrease monotonically as more model scales are included in the fit.
| Variant | RMSE | vs Full | RMSE | RMSE | RMSE | Max rel. err. (%) | |
|---|---|---|---|---|---|---|---|
| Full law | – | ||||||
| No repetition | |||||||
| No transfer | |||||||
| No capacity |
Table 7: Component ablations of the joint fit on the full training set. The “ vs Full” column shows the relative increase in overall RMSE when the corresponding component is removed. denotes the parameters to fit in the resulting scaling law. Per-modality RMSE and the overall maximum relative error, both computed on the fitted set, are shown for context. The full law achieves the minimum in the Maximum Relative Error ( Max. rel. err. ) column.
| Regime | RMSE | Maximum relative error (%) | ||||
|---|---|---|---|---|---|---|
| Overall | Text | Vision | Audio | |||
| R1 (train M1-2) | ||||||
| R2 (train M1-3) | ||||||
| R3 (train M1-4) | ||||||
| R4 (train M1-5) | ||||||
Table 8: RMSE and maximum relative errors (%) for the directional-capacity variant of our scaling law: a separate for each (source, target) pair instead of one per target in Eq. 2 . We use the same training extrapolation regimes as Table 6 where R{ x -1} (train M1- x ) denote the regime where runs corresponding to models M1, M2, … M x are used for fitting, and the metrics are computed on M6 only. denotes the number of training points in the respective regime. Even though the RMSE and overall maximum relative errors are comparable, the vision errors are strictly worse than the non-directional version (Table 6(b) ).
| RMSE | Maximum relative error (%) | |||||||
|---|---|---|---|---|---|---|---|---|
| Variant | R1 | R2 | R3 | R4 | R1 | R2 | R3 | R4 |
| Full law | ||||||||
| No transfer | ||||||||
| No capacity | ||||||||
| No repetition | ||||||||
| Directional capacity | ||||||||
Table 9: Metrics on the held-out M6 scale across the four training extrapolation regimes (R1: train M1-2, R2: train M1-3, R3: train M1-4, R4: train M1-5). Five variants of our law are compared: the full law, three variants with each modeling component individually removed, and the directional-capacity variant. Lower is better in every column. Under these extrapolation regimes, we observe that the full law and the directional capacity variant perform consistently well across the regimes.
Figure 9: The overall training loss and the text-only loss for (i) Continual run 1 : Continual pretraining (CPT) on a text-only checkpoint trained with 100B tokens. The continual training mixture consists of 10B text, 20B vision (10B repeated twice), and 20B audio (10B repeated twice), (ii) Continual Run 2 : Continual pretraining on the same checkpoint as (i) with 40B text, 5B vision, and 5B audio.
| Modality | Observed | Predicted | Relative error | |
| 12.6B-A3.8B, from scratch on 200B tokens (185B text / 7.5B vision / 7.5B audio) | ||||
| Text | ||||
| Vision | ||||
| Audio | ||||
| 8.3B-A2.6B, non-text-heavy continual pretraining: 10B text + 20B vision ( ) + 20B audio ( ) | ||||
| Text | ||||
Table 10: Predicted vs. observed validation loss on three held-out runs. Predictions use the fitted coefficients from Table 1 and Section 5 . No parameters are re-tuned on these runs, and 8/9 losses are predicted to within 2.51%.
| Text vs. vision | Text vs. audio | |||
|---|---|---|---|---|
| Model | 95/5 | 75/25 | 95/5 | 75/25 |
| M1 | ||||
| M2 | ||||
| M3 | ||||
| M4 | ||||
| M5 | ||||
Table 11: Mean per-layer Jensen–Shannon divergence between the expert-assignment distributions of two modalities, for every model scale. Mixture labels give the token split between the two modalities present in training. Values near zero indicate that the two modalities route to the same experts in the same proportions; the scale is normalized to . Barring the smallest model M1, all the divergences are , indicating negligible modality-wise expert segregation. This hints that modalities compete for representational capacity within the same experts leading to .
More in this topic
Multimodal Pretraining
Multimodal pretraining learns reusable representations by training models on paired or mixed data from multiple modalities, such as text, images, audio, and signals.
See all 182 papers