Paired Multimodal Scaling Laws

Marcus Ma; Shrikanth Narayanan

cs.LGSep 28, 2026

Paired Multimodal Scaling Laws

Authors: Marcus Ma, Shrikanth Narayanan

Organizations: University of Southern California

Abstract

Existing multimodal scaling laws fit multimodality terms empirically after testing and never vary how much data is multimodally paired at fixed data budgets. We investigate how, under the same total data per modality, changing the number of paired data affects loss curves in multimodal classification tasks. We train models in three different environments and run experiment sweeps varying data sizes and pairing budget. Pairing ratios have a dramatic impact on loss and this impact is directly tied to how much information synergy the task contains. Only paired data is able to reduce synergistic loss, while unpaired data can reduce redundant or unimodal information up until unimodal floors. Unlike traditional scaling laws where loss drops immediately in power law decay, synergy acquisition is gated, requiring a critical threshold of paired data before synergistic loss falls at all. We introduce a new family of multimodal scaling laws where total data-attributable loss is the sum of four individual power laws corresponding to the four different information channels of redundancy, a unique channel per modality, and synergy, and show how this law is both more theoretically sound and empirically valid across our experiments. This law predicts multimodal loss in our experiments more accurately than existing laws, with 3.2% error on fit tests versus 10.4% error for the best pairing extension of published laws.

Figures & tables

Figure 1: Illustration of each environment along with an example and tested architecture.

Experimental details
	Digits		VQA-v2		NLVR2
	image	audio	image	question	photos	sentence
architecture	CNN + 4-way head ( $476$ k)		Frozen CLIP + head ( $5.3$ M)		Finetuned LXMERT ( $210$ M)
chance loss $L_{0}$	$\ln 10\approx 2.303$		$\ln 3129\approx 8.048$		$\ln 2\approx 0.693$
$D$	$8$ k, $16$ k, $32$ k, $64$ k, $128$ k, $256$ k		$0$ , $4$ k … $256$ k		$4$ k, $8$ k, $16$ k, $32$ k, $64$ k
$C$ ( $k(D_{1}+D_{2})$ steps)	$k\in\{10,20,40\}$		$k\in\mathbb{Z_{+}}\leq 80$		$k\in\{1,2,3\}$

Table 1: Experimental details, unimodal data curve variables, and PID terms for each environment. Compute

C

represents number of training steps multiplied by batch size.

Law	$k$	$\Gamma$	$\partial\Gamma/\partial D\neq 0$	$L(N,D_{1},D_{2},n_{p})$	at $p=1$
Hoffmann Unimodal	4	$\leq 0$	$\checkmark$	$E_{N}+B\,g(D_{1}{+}D_{2})$	$E_{N}+B\,g(2D)$
Hoffmann on unique $D$	4	$\leq 0$	$\checkmark$	$E_{N}+B\,g(D_{\mathrm{tot}})$	$E_{N}+B\,g(D)$
Hoffmann on $D_{1}$ and $D_{2}$	7	$=0$	$\times$	$E_{N}+B_{1}g_{1}(D_{1})+B_{2}g_{2}(D_{2})$	$E_{N}+B_{1}g_{1}(D)+B_{2}g_{2}(D)$
Aghajanyan $+Cp$	8	$=0$	$\times$	$E_{N}+B_{1}g_{1}(D_{1})+B_{2}g_{2}(D_{2})+Cp$	$E_{N}+B_{1}g_{1}(D)+B_{2}g_{2}(D)+C$
Shukor mixture	6	$<0$	$\checkmark$	$E_{N}+(C_{1}p+C_{2}(1{-}p))^{-1}+B\,g(D_{\mathrm{tot}})$	$E_{N}+C_{1}^{-1}+B\,g(D)$
Ye mixture	7	any	$\checkmark$	$E_{N}+C\exp(\alpha_{2}n_{2}/D_{\mathrm{tot}}+\alpha_{p}n_{p}/D_{\mathrm{tot}})+B\,g(D_{\mathrm{tot}})$	$E_{N}+C\,e^{\alpha_{p}}+B\,g(D)$

Table 2: Extensions of existing scaling laws (Top) and the introduced four-channel law and variants (Bottom). Existing laws reduce to their original equations at

p=1

. Each of the variations of the four-channel law can be applied independently, resulting in 16 four-channel law combinations.

k

is the number of fitted parameters for the corresponding law,

D_{\mathrm{tot}}=D_{1}+D_{2}-n_{p}

n_{i}=D_{i}-n_{p}

, and

W_{a}

are measured PID weights.

\Gamma=-\partial^{2}L/\partial D_{1}\partial D_{2}

is taken at fixed

p

; for the four-channel family its sign is that of the base law unless a switch changes it.

\Gamma

and

\partial\Gamma/\partial D

are discussed in § 5.3 .

		Digits			VQA			NLVR2			Overall
Law	$k$	$D{\uparrow}$	$p{=}1$	$R^{2}$	$D{\uparrow}$	$p{=}1$	$R^{2}$	$D{\uparrow}$	$p{=}1$	$R^{2}$	$D{\uparrow}$	$p{=}1$	$R^{2}$
Hoffmann Unimodal	4	13.9%	30.4%	0.00	8.2%	8.1%	0.00	1.8%	6.6%	0.00	8.0%	15.0%	0.00
Hoffmann on unique $D$	4	14.9%	37.1%	-0.12	10.5%	13.1%	-1.26	1.7%	6.9%	-0.45	9.0%	19.0%	-0.61
Hoffmann on $D_{1}$ and $D_{2}$	7	11.5%	27.3%	0.00	2.7%	6.7%	0.00	1.8%	6.6%	0.00	5.3%	13.5%	0.00
Aghajanyan $+Cp$	8	11.5%	20.4%	0.39	2.3%	5.0%	0.40	1.5%	6.0%	0.22	5.1%	10.4%	0.34
Shukor mixture	6	9.6%	29.1%	0.46	8.2%	9.4%	0.02	1.4%	6.2%	-0.09	6.4%	14.9%	0.13

Table 3: Paired scaling law scores on the three fit tests: percent error for

D\uparrow

and

p=1

(lower is better), and within-cell

R^{2}

(higher is better).

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

		Digits		VQA		NLVR2		Overall
Law	$k$	LOO	Full	LOO	Full	LOO	Full	LOO	Full
Hoffmann Unimodal	4	5.9%	5.6%	3.9%	3.7%	1.0%	1.3%	3.6%	3.5%
Hoffmann on unique $D$	4	5.2%	4.5%	4.0%	3.8%	1.0%	1.2%	3.4%	3.1%
Hoffmann on $D_{1}$ and $D_{2}$	7	4.2%	3.6%	1.1%	1.0%	1.2%	1.3%	2.2%	2.0%
Aghajanyan $+Cp$	8	5.7%	4.6%	1.2%	1.0%	0.9%	0.8%	2.6%	2.2%
Shukor mixture	6	4.1%	3.9%	4.6%	4.3%	0.8%	1.1%	3.2%	3.1%

Appendix

Table 4: Paired scaling law scores of the two interpolation tests: Leave-One-Out and the Full test.

Explore similar work

SynIB: Informational Bottleneck for Maximizing Synergy in Multimodal Learning
May 12, 2026Konstantinos Kontras, Teodora Gagaleska, Thomas Strypsteen +4Multimodal Learning Cross-Modal
1KU Leuven · 2MIT
Multiplicity is an Inevitable and Inherent Challenge in Multimodal Learning
May 26, 2025Sanghyuk Chun, Olga RussakovskyMultimodal Learning Challenges
1Princeton University
When to Align, When to Predict: A Phase Diagram for Multimodal Learning
Jun 9, 2026Ilay Kamai, Hugues Van Assel, Aviv Regev +2Multimodal Learning Multimodal Representations
Technion · Genentech · Brown University

May 12, 2026, cs.LGJ/K move · Enter open · S save

SynIB: Informational Bottleneck for Maximizing Synergy in Multimodal Learning

Konstantinos Kontras, Teodora Gagaleska, Thomas Strypsteen, Christos Chatzichristos+3

1KU Leuven · 2MIT

Multimodal Learning Cross-Modal Synergy Information Bottleneck

Open paper AskPDF arXiv

A central objective in multimodal learning is to capture synergy: task-relevant information that arises only from the joint use of multiple modalities, and is not available from any single modality alone. While most approaches operate at the architectural level through larger or more complex fusion models, we propose a complementary axis: shaping the training objective itself. Standard training often emphasizes unimodal or redundant information, falling short on examples that require cross-modal reasoning. We formalize multimodal synergy through information theory and introduce the Synergistic Information Bottleneck (SynIB), a scalable objective that targets synergy directly. To prioritize learning synergy, SynIB motivates the model to predict accurately from all modalities while penalizing confidence when information from any modality is withheld. Alongside the standard task loss, the model runs forward passes with one modality masked at a time and is penalized for remaining confident, which would indicate reliance on unimodal cues rather than cross-modal interactions. We validate SynIB in two regimes. On synthetic XOR tasks where the ground-truth synergy is known by construction, standard training fails to recover it while SynIB does. On five real-world benchmarks, including three MultiBench affective tasks, Hateful Memes with CLIP-ViT and DeBERTa backbones, and a controllable irony extension of CREMA-D we introduce, SynIB improves accuracy on synergy-dependent examples by up to 7.8% and overall accuracy by up to 3.8%.

Abstract

Explore similar work

SynIB: Informational Bottleneck for Maximizing Synergy in Multimodal Learning

Multiplicity is an Inevitable and Inherent Challenge in Multimodal Learning

When to Align, When to Predict: A Phase Diagram for Multimodal Learning