Contrastive learning beyond two modalities now scores the whole tuple with multilinear products, volumes, and fusion heads, critics that can represent the joint distribution. In practice these critics still behave like bags of pairs. Their negatives break at least two of the three pairs, a single pairwise check rejects them, and when most samples can be solved by pairs the higher-order term receives almost no gradient. To solve this problem, we propose CHORD (Contrastive Higher-Order Residuals with Decoys), which trains a residual against decoys drawn from a calibrated pairwise model, negatives that the pairwise model finds plausible, so that rejecting them often needs all three modalities. Drawing decoys costs one inner-product query over a memory bank, and the optimal residual is the log-ratio between the data and the pairwise model, so the decoy loss also bounds what that model misses. On M3-Mix, where one in ten training samples needs all three modalities, our CHORD improves accuracy on those samples by 21.2 percentage points over the strongest of eight baselines. CHORD also gives the best two-to-one results on every real dataset, and its residual improves other retrievers without retraining.
Figures & tables
Figure 2: Overview of CHORD. (a) Encoders feed a calibrated bridge, which also serves one-to-one retrieval, a decoy sampler, and a residual head, and two-to-one retrieval adds the bridge and residual scores. (b) The bridge scores a memory bank with one query, and K independent Gumbel-max draws pick the decoys. Sharpness γ=1 draws from the bridge itself, between uniform ( γ=0 ) and hardest ( γ→∞ ) draws. (c) A zero-initialized product of projected features scores the positive against the decoys, and the decoy loss does not reach the bridge.
Synthetic
Symile-M3
Method
XOR
XOR-Mix
XNOR
w=2
w=5
w=10
Mix
Tri-CLIP
3.8
50.1
24.4
47.5
18.8
9.6
48.6
Symile
100.0
64.2
33.2
93.8
92.0
88.3
66.2
Gated Symile
99.8
66.0
87.1
94.1
92.0
88.6
67.0
H-CLIP
100.0
65.3
41.6
94.4
92.3
88.9
68.4
GRAM
13.8
50.3
48.3
48.1
19.3
9.9
48.9
Table 1: Top-1 accuracy (%) on controlled benchmarks. XOR-Mix and M3-Mix keep a pairwise path for most samples and need the triple for a fraction ρ=0.1 . Pairwise training reaches 50% on those samples (pairwise reference, Sec. A.3 ). The best results are shown in bold , and the second-best results are underlined .
MOSI
MOSEI
UR-FUNNY
MUStARD
Method
1 → 1
2 → 1
Decoy@16
1 → 1
2 → 1
Decoy@16
1 → 1
2 → 1
Decoy@16
1 → 1
2 → 1
Decoy@16
Tri-CLIP
22.7
23.1
8.4
6.8
8.9
8.0
8.0
12.6
7.7
41.8
59.0
9.9
Symile
n/a
17.0
9.6
n/a
7.1
8.6
n/a
11.8
9.3
n/a
46.5
13.1
Gated Symile
n/a
18.4
10.1
n/a
7.6
8.9
n/a
12.3
9.8
n/a
49.3
13.9
H-CLIP
n/a
18.1
9.9
n/a
7.4
8.8
n/a
12.1
9.6
n/a
48.1
13.4
GRAM
6.0
11.7
7.4
2.9
5.2
7.0
2.4
3.6
6.9
35.1
60.2
9.8
Table 2: Retrieval on MultiBench (Recall@10, %) and the decoy probe (Decoy@16, top-1 %, chance 6.25). 1 → 1 averages six directions and 2 → 1 three targets. n/a: the critic does not score pairs. The best results are shown in bold , and the second-best results are underlined .
Figure 3: Ablation of CHORD on M3-Mix, MUStARD (two-to-one and one-to-one), and MSR-VTT. Error bars show the standard deviation over five seeds, and the dashed line marks the full model.
Retrieval (T → VA)
Zero-shot (A+V)
Method
MSR-VTT
VATEX
YC2
SSW60
VB100
AV-MN.
Best single
46.0
69.5
12.4
70.2
20.5
64.8
Tri-CLIP
46.8
70.4
12.6
70.3
19.2
69.4
Symile
44.9
67.9
10.9
60.4
13.1
71.0
Gated Symile
45.6
68.8
11.2
63.1
17.2
71.5
H-CLIP
45.1
68.1
10.8
61.0
13.9
71.3
Table 3: Audio-visual tasks. Left: text → (video, audio) retrieval, R@1 (%), with frozen LanguageBind encoders. YC2 is zero-shot transfer from MSR-VTT. Right: fused audio-visual zero-shot accuracy (%).
MOSI
MUStARD
Mode
1 → 1
2 → 1
1 → 1
2 → 1
Tri-CLIP (reference)
22.7
23.1
41.8
59.0
Frozen encoders
22.7
23.4
41.8
62.6
Trainable heads only
22.7
23.5
41.8
63.2
End to end, shared encoders
22.8
23.7
43.6
64.4
Table 4: One-to-one and two-to-one retrieval (R@10, %) of CHORD under the three training modes.
M3-Mix
MUStARD
Residual
Unif.
Top- K
Decoys
Unif.
Top- K
Decoys
Pairwise (control)
48.8
49.0
49.3
61.3
61.5
61.9
Hadamard (H-CLIP)
66.9
79.6
89.6
61.4
62.6
64.4
Multilinear (Symile)
65.0
77.8
88.3
61.0
62.0
63.6
Gated (Gated Symile)
66.1
78.6
88.9
61.2
62.3
63.9
Fusion MLP (ConFu)
62.4
75.1
86.8
61.5
62.4
63.2
Table 5: Residual families and proposals on one calibrated bridge, with the same bank and K=32 . Uniform and decoy columns use noise-corrected logits and share the same target residual. M3-Mix reports accuracy on samples that need the triple, and MUStARD reports 2 → 1 R@10 (%). Hadamard with decoys is CHORD.
Dot product
Bilinear
Pairwise MLP
Method
URF
MUS
SSW
URF
MUS
SSW
URF
MUS
SSW
Tri-CLIP
7.7
9.9
9.2
8.0
10.2
9.5
7.8
10.0
9.4
Symile
9.3
13.1
10.8
9.2
12.9
10.7
9.0
12.5
10.5
Gated Symile
9.8
13.9
11.3
9.7
13.6
11.2
9.5
13.3
11.0
ConFu
9.0
12.0
12.1
8.9
11.8
11.9
8.7
11.5
11.7
CHORD
14.4
18.3
14.5
13.6
17.2
13.9
12.9
16.4
13.4
Table 6: Decoy@16 (top-1 %) with judges from three pairwise critic families on UR-FUNNY (URF), MUStARD (MUS), and SSW60 (SSW).
Figure 4: Sensitivity of CHORD: change from the default (shaded) with (a) the number of decoys, (b) their sharpness γ , with top- K mining shown apart (hollow), (c) the bank size, (d) the residual rank, whose default is 128 on MUStARD and 512 otherwise, and (e) the loss weight. Standard deviations over five seeds stay below 0.5%.
Figure 5: Diagnosing bags of pairs. (a) XOR-Mix accuracy on synergy samples as ρ falls from 1 (circle) via 1/10 (diamond) to 1/50 (arrow). (b) M3-Mix accuracy versus batch size with uniform negatives (CHORD: 32 decoys at every size). (c) Normalized gradient norm reaching the residual on M3-Mix, where each band spans ρ=1 to ρ=0.1 . (d) Decoy@16 accuracy on six real datasets as a multiple of chance.
Dataset
Sum
Bridge
CHORD
Res.
Plug-in
Δ BPI
MUStARD
59.0
61.2
64.4
+3.2 ± 1.2
+2.0
0.164
UR-FUNNY
12.6
13.0
14.5
+1.5 ± 0.5
+0.9
0.096
SSW60
70.3
71.0
72.3
+1.3 ± 0.5
+0.8
0.083
AV-MNIST
69.4
70.6
71.9
+1.3 ± 0.4
+0.8
0.051
MOSI
23.1
23.4
23.7
+0.3 ± 0.9
+0.3
0.034
MOSEI
8.9
9.2
9.4
+0.2 ± 0.4
+0.1
0.027
Table 7: Sum, bridge, and CHORD on real data, ordered by Δ BPI. Res.: gain of the residual over the bridge, mean ± std over seeds. Plug-in: mean gain when the same residual is added to the eight baselines without retraining them.
ρ=0 , I(3)=0
ρ=0.1 , I(3)=0.199
Bridge
BPI
Δ BPI
BPI
Δ BPI
Random projections
1.240
0.004
1.420
0.170
Frozen encoders
0.288
0.001
0.480
0.190
Trained heads (default)
0.011
0.001
0.205
0.193
Pairwise MLP
0.005
0.000
0.201
0.196
Table 8: BPI and Δ BPI against the oracle I(3) on XOR-Mix for bridges of increasing capacity (nats, K=64 ). With ρ=0 no sample needs the triple.
Complete triples
Modalities
Method
100%
10%
1%
M=3
M=4
M=5
Tri-CLIP
47.5
47.2
46.9
41.2
43.0
43.8
Symile
93.8
89.0
47.9
44.6
40.3
36.9
Gated Symile
94.1
89.5
48.2
45.3
42.9
40.6
H-CLIP
94.4
89.3
48.0
45.8
42.7
39.5
ConFu
92.6
85.1
47.6
45.1
46.2
46.0
Table 9: Left: Symile-M3-2 accuracy (%) when only a share of the training tuples is complete. Right: Vision&Touch top-1 accuracy (%) with M modalities.
Figure 6: What the residual learns. (a) Residual versus its target logp/qb under the trained bridge on synergy queries of XOR-Mix with a per-sample reliability, both centered per query. (b) BPI64 versus the oracle I(3) . Hollow markers: logw , an upper bound for Symile-M3. Dotted line: the log(K+1) cap.
Appendix figures & tables23 assets
Supplementary material from the paper’s appendix.
Appendix
Setting
Encoders
d
R
B
Epochs
XOR
affine
16
16
1000
100
XOR-Mix
linear
16
64
256
50
Symile-M3
frozen (Symile)
512
512
512
20
MultiBench
ConFu
ConFu
128
ConFu
ConFu
Bird-MML
ConFu
ConFu
256
ConFu
ConFu
LanguageBind
frozen, MLP heads
512
512
256
30
Appendix
Table S1 : Settings per benchmark ( d : embedding size, R : residual rank, B : batch size, ConFu: same as the ConFu configuration).
Time
Method
Dim.
Frozen
E2E
Mem. (GB)
Lat. (ms)
Tri-CLIP
512
1.00
1.00
0.0
0.9
Symile
512
1.04
1.02
0.1
1.0
Gated Symile
512
1.31
1.18
0.6
6.8
H-CLIP
512
1.03
1.02
0.1
1.0
GRAM
512
1.06
1.03
0.1
2.7
Appendix
Table S2 : Cost on MSR-VTT with one A100 80GB, batch 256, and a FIFO bank. Dim.: size of the retrieval embedding. Time: median time per step over 200 steps after 50 warm-up steps, including bank sampling, relative to Tri-CLIP, with frozen and with end-to-end encoders. Mem.: extra peak memory. Lat.: time to score 10k precomputed candidates for one query, excluding data transfer.
Method
XOR
XOR-Mix
Syn-XNOR
M3 ( w=2 )
M3 ( w=5 )
M3 ( w=10 )
M3-Mix
Tri-CLIP
3.8 ± 0.2
50.1 ± 0.4
24.4 ± 1.1
47.5 ± 0.1
18.8 ± 0.1
9.6 ± 0.1
48.6 ± 0.5
Symile
100.0 ± 0.0
64.2 ± 3.1
33.2 ± 1.4
93.8 ± 0.1
92.0 ± 0.1
88.3 ± 0.1
66.2 ± 2.8
Gated Symile
99.8 ± 0.1
66.0 ± 2.9
87.1 ± 0.9
94.1 ± 0.2
92.0 ± 0.2
88.6 ± 0.2
67.0 ± 2.6
H-CLIP
100.0 ± 0.0
65.3 ± 3.3
41.6 ± 1.6
94.4 ± 0.2
92.3 ± 0.2
88.9 ± 0.3
68.4 ± 2.9
GRAM
13.8 ± 1.2
50.3 ± 0.5
48.3 ± 1.8
48.1 ± 0.4
19.3 ± 0.4
9.9 ± 0.3
48.9 ± 0.6
TRIANGLE
12.9 ± 1.1
50.6 ± 0.5
60.7 ± 1.5
48.4 ± 0.4
19.6 ± 0.3
10.1 ± 0.3
49.2 ± 0.6
Appendix
Table S3 : Table 1 with standard deviations over five seeds.
Target
M1
M2
M3
Query
M2
M3
M23
M1
M3
M13
M1
M2
M12
MOSI
Tri-CLIP
23.3 ± 1.9
20.7 ± 1.6
22.4 ± 1.8
23.2 ± 3.6
23.1 ± 1.8
23.6 ± 2.0
21.1 ± 2.0
24.6 ± 2.1
23.4 ± 1.9
Symile
n/a
n/a
15.9 ± 0.9
n/a
n/a
18.5 ± 2.4
n/a
n/a
16.7 ± 1.7
Gated Symile
n/a
n/a
17.6 ± 1.2
n/a
n/a
19.2 ± 2.1
n/a
n/a
18.4 ± 1.6
H-CLIP
n/a
n/a
17.4 ± 1.1
n/a
n/a
18.9 ± 2.2
n/a
n/a
18.0 ± 1.8
Appendix
Table S4 : Per-direction Recall@10 (%) on all four MultiBench datasets, mean ± std over five seeds.
SSW60
VB100
AV-MNIST
Method
A
V
A+V
A
V
A+V
A
V
A+V
Tri-CLIP
30.9
68.9
70.3
3.7
20.5
19.2
39.8
62.6
69.4
Symile
n/a
n/a
60.4
n/a
n/a
13.1
n/a
n/a
71.0
Gated Symile
n/a
n/a
63.1
n/a
n/a
17.2
n/a
n/a
71.5
H-CLIP
n/a
n/a
61.0
n/a
n/a
13.9
n/a
n/a
71.3
GRAM
0.8
66.5
57.1
1.2
13.5
7.9
9.5
64.1
64.1
Appendix
Table S5 : Zero-shot accuracy (%) with audio only (A), vision only (V), and fused inputs (A+V).
MSR-VTT
VATEX
T → VA
VA → T
T → VA
VA → T
Method
R@1
R@5
R@10
R@1
R@5
R@10
R@1
R@5
R@10
R@1
R@5
R@10
Tri-CLIP
46.8
72.1
81.3
44.0
70.6
80.2
70.4
93.0
96.8
66.9
91.7
96.0
Symile
44.9
70.4
79.8
42.6
69.1
78.9
67.9
91.6
95.9
64.8
90.3
95.2
Gated Symile
45.6
71.0
80.4
43.1
69.8
79.5
68.8
92.1
96.2
65.5
90.9
95.5
H-CLIP
45.1
70.6
80.0
42.8
69.3
79.1
68.1
91.8
96.0
65.0
90.5
95.3
Appendix
Table S6 : Video retrieval with frozen LanguageBind encoders, Recall@ k (%). MSR-VTT: 1k-A test split, one caption per video, 1,000 candidates. VATEX: 1,500 test videos, one caption per video.
Table S8 : Details of Tab. 7 . Cal.: bridge minus sum. Res.: CHORD minus bridge with its bootstrap 95% interval over test queries. ‡ : p<0.05 in a paired t -test over seeds after Holm correction over the ten datasets. BPI and BPI pair : decoy-loss bounds of the product residual and of a pairwise residual on the same decoys (nats, K=64 ).
BPI (nats)
Accuracy
Draws
Gauss.
XOR-Mix
M3-Mix
MUStARD
Independent (CHORD)
0.002
0.205
89.6
64.4
No replacement
0.004
1.86
89.4
64.3
No repl., positive removed
0.006
3.95
89.1
64.0
Appendix
Table S9 : Decoy sampling schemes, applied in training and in the BPI readout. BPI uses the default trained bridge and K=64 as in Tab. S10 . The oracle I(3) is 0 on the Gaussian control and 0.199 nats on XOR-Mix with ρ=0.1 . Bottom: median effective support 1/∑nqb(n∣xS)2 of the bank distribution.
XOR-Mix, ρ=0
XOR-Mix, ρ=0.1
Gaussian control
MUStARD
Bridge
BPI
BPI pair
Δ BPI
BPI
BPI pair
Δ BPI
BPI
BPI pair
Δ BPI
BPI
BPI pair
Δ BPI
Random frozen projections
1.240
1.236
0.004
1.420
1.250
0.170
0.214
0.210
0.004
0.395
0.226
0.169
Frozen encoders, calibrated
0.288
0.287
0.001
0.480
0.290
0.190
0.031
0.029
0.002
0.262
0.101
0.161
Trained heads, calibrated (default)
0.011
0.010
0.001
0.205
0.012
0.193
0.002
0.002
0.000
0.214
0.050
0.164
Pairwise MLP critic
0.005
0.005
0.000
0.201
0.005
0.196
0.001
0.001
0.000
0.181
0.019
0.162
Appendix
Table S10 : BPI depends on the bridge, while Δ BPI depends on it much less. Bridges of increasing capacity on XOR-Mix with ρ=0 (oracle I(3)=0 ) and ρ=0.1 (oracle 0.199), the Gaussian control (oracle 0), and MUStARD (no oracle), in nats with K=64 . The XOR-Mix blocks repeat Tab. 8 with BPIpair .
Suppression
Proposal
M3-Mix
MSR-VTT
R@1
Uniform (bank)
0.04
0.07
72.4
Top- K hardest
1.21
0.86
69.8
Hardness reweighting
0.48
0.39
71.0
Decoys from the bridge
0.09
0.12
74.1
Appendix
Table S11 : Valid alternatives among the negatives. Suppression: mean r(positive)−r(valid alternative) in nats (lower is better), for same-class images on M3-Mix and other captions of the same video on MSR-VTT. R@1: MSR-VTT VA → T with all 20 captions of the 1,000 test videos as candidates, where any caption of the query video counts as correct, so it is higher than the one-caption protocol of Tab. S6 .
Figure S1 : (a) BPI (filled) and BPIpair of a pairwise residual on the same decoys (ring), whose gap is ΔBPI . (b) Gain of the residual over the calibrated bridge versus ΔBPI on the ten real datasets.
Figure S2 : XOR task of Symile and ConFu. (a) Accuracy versus synergy level p^ , mean ± std over three seeds, with the Bayes optimum dotted. (b) CHORD’s bridge alone, residual alone, and full score, with the Bayes optimum as bars.
Figure S3 : Training bridge and residual together on M3-Mix. (a) Normalized gradient norm reaching the residual. (b) Drift of the decoy proposal between steps 500 apart. Sequential follows CHORD until its bridge is frozen at 5k steps (dashed), after which the drift is zero.
Variant
M3-Mix
MUS 2 → 1
MSR-VTT
MUS 1 → 1
w/o Decoy
66.9 ± 2.7
61.0 ± 3.6
47.4 ± 0.5
43.4 ± 1.5
Full bank
82.1 ± 1.2
62.3 ± 3.4
47.9 ± 0.5
43.3 ± 1.5
Top-K
79.6 ± 1.8
62.6 ± 3.3
46.9 ± 0.6
42.9 ± 1.6
Reweighting
71.3 ± 2.4
61.8 ± 3.5
47.3 ± 0.5
43.1 ± 1.5
Self-decoy
87.2 ± 1.5
62.7 ± 3.4
46.8 ± 0.6
41.4 ± 1.7
w/o Stop-grad
87.8 ± 1.1
63.3 ± 3.2
48.1 ± 0.4
41.9 ± 1.7
Appendix
Table S12 : Values behind Fig. 3 , mean ± std over five seeds (%, MUS: MUStARD).
Retriever
M3-Mix
MUS
URF
SSW60
MSR-VTT
Tri-CLIP
72.9 +24.3
62.1 +3.1
13.9 +1.3
71.4 +1.1
47.2 +0.4
GRAM
71.0 +22.1
62.3 +2.1
4.3 +0.7
57.7 +0.6
47.6 +0.3
TRIANGLE
72.1 +22.9
52.4 +2.1
4.4 +0.8
65.0 +0.8
48.0 +0.2
PMRL
72.3 +23.5
59.8 +2.7
11.9 +1.1
67.1 +0.9
47.5 +0.4
Symile
80.4 +14.2
48.4 +1.9
12.8 +1.0
61.0 +0.6
45.2 +0.3
Gated Symile
81.6 +14.6
50.8 +1.5
13.1 +0.8
63.8 +0.7
45.8 +0.2
Appendix
Table S13 : The trained residual of CHORD added to other retrievers without retraining them ( α=1 ), with the gain over each retriever in small type. Bottom: mean gain over the eight retrievers for the residual and two controls. MUS: MUStARD, URF: UR-FUNNY (2 → 1 R@10).
Retriever
Score
M3-Mix
MUS
URF
SSW60
MSR-VTT
Tri-CLIP
Retriever alone
48.6
59.0
12.6
70.3
46.8
+ second seed
49.0 +0.4
59.6 +0.6
12.8 +0.2
70.6 +0.3
47.1 +0.3
+ pairwise residual
50.3 +1.7
60.1 +1.1
12.9 +0.3
70.6 +0.3
47.0 +0.2
+ residual
72.9 +24.3
62.1 +3.1
13.9 +1.3
71.4 +1.1
47.2 +0.4
+ residual, tuned α
73.8 +25.2
62.3 +3.3
14.0 +1.4
71.5 +1.2
47.3 +0.5
GRAM
Retriever alone
48.9
60.2
3.6
57.1
47.3
Appendix
Table S14 : Per-retriever results behind Tab. S13 , with the two controls and a weight α tuned on validation data. Small type: gain over the retriever alone.
Retriever
MOSI
MOSEI
AV-MNIST
VB100
VATEX
YC2
Tri-CLIP
23.4 +0.3
9.1 +0.2
70.6 +1.2
19.5 +0.3
70.8 +0.4
12.6 +0.0
GRAM
12.4 +0.7
5.3 +0.1
65.4 +1.3
8.3 +0.4
71.6 +0.4
11.5 +0.1
TRIANGLE
7.6 +0.1
6.0 +0.2
66.0 +0.9
12.3 +0.5
72.1 +0.3
10.7 +0.0
PMRL
21.9 +0.3
8.0 +0.2
67.1 +1.1
15.4 +0.3
71.4 +0.4
11.8 +0.0
Symile
17.4 +0.4
7.2 +0.1
71.4 +0.4
13.6 +0.5
68.2 +0.3
10.8 − 0.1
Gated Symile
18.5 +0.1
7.7 +0.1
71.8 +0.3
17.3 +0.1
69.1 +0.3
11.3 +0.1
Appendix
Table S15 : The residual added to other retrievers ( α=1 ) on the remaining two-to-one benchmarks. YC2 uses the residual trained on MSR-VTT. Small type: gain over the retriever alone.
k=10
k=50
k=100
All
MSR-VTT
47.9
48.3
48.4
48.4
VATEX
71.8
72.3
72.4
72.4
Appendix
Table S16 : Bridge as a first stage. Only the top- k candidates of the bridge receive the residual (T → VA R@1, %). The last column scores every candidate.
Method
Third-order
Four-way
Tri-CLIP
3.2
3.1
Symile (four-way product)
99.1
99.4
CHORD (triple residuals)
100.0
3.3
CHORD + four-way term
99.8
99.6
Appendix
Table S17 : Parity with four modalities (top-1 %, 32 candidates, chance 3.1). Third-order: x4=x1⊕x2 with x3 independent. Four-way: x4=x1⊕x2⊕x3 , where every pair and every triple is independent.
Figure S4 : Symile-M3-2 accuracy when only a share of the training tuples is complete (full curve of Tab. 9 , left).
MOSI
UR-FUNNY
MUStARD
Method
Rand
Decoy
Rand
Decoy
Rand
Decoy
Tri-CLIP
61.8
8.4
43.5
7.7
78.6
9.9
Symile
55.3
9.6
41.2
9.3
70.1
13.1
ConFu
59.4
9.3
44.0
9.0
80.2
12.0
CHORD
63.0
11.9
45.1
14.4
81.9
18.3
Appendix
Table S18 : Two-to-one top-1 accuracy (%) with 15 uniformly drawn candidates (Rand@16) and with 15 decoys from the held-out judge bridge (Decoy@16).
Figure S5 : t-SNE of two-to-one query embeddings on M3-Mix test samples that need the speech, colored by the target class. The bridge places a query between the two classes named in its caption, whereas CHORD separates queries by their target.
Multimodal contrastive learning aligns representations from different data types so models can connect related images, text, audio, signals, and other modalities.
Contrastive learning is effective for aligning paired views or modalities, but alignment beyond two modalities remains non-trivial and comparatively underexplored. Pairwise CLIP-style losses decompose multi-modal alignment into independent two-way comparisons and therefore do not explicitly model higher-order dependencies among multiple modalities. Recent beyond-pairwise objectives approach this problem from statistical or geometric perspectives, but arbitrary-modality alignment still lacks a principled criterion for defining what each modality should preserve and compress relative to the others. We revisit arbitrary-modality alignment through the Information Bottleneck principle. In multi-modal learning, sufficiency should preserve information predictable from the remaining modalities, while minimality should compress modality-specific information not supported by them. This naturally leads to a One-vs-All view, where each modality is characterized with respect to the remaining modalities. We propose OVA-IB, an Information Bottleneck framework for arbitrary-modality alignment. OVA-IB optimizes a tractable One-vs-All contrastive lower bound for sufficiency connected to a Dual Total Correlation-style objective, uses a parameter-free geometry-aware projection score, and derives a tractable upper-bound regularizer for minimality by bounding each representation's dependence on its own input with representation distributions induced by the remaining modalities. Experiments on classification, regression, modality-agnostic evaluation, and cross-modal retrieval benchmarks demonstrate strong and robust performance.
Tianchao Li, Shujian Yu, Xinrui Zu +4
Hong Kong University of Science and Technology, Hong Kong · Vrije Universiteit Amsterdam, Netherlands · UiT – The Arctic University of Norway, Norway +3
Multimodal contrastive learning is a dominant paradigm for learning transferable representations from unlabeled data, but standard objectives primarily capture information that is redundant between modalities. Partial Information Decomposition (PID) shows that task-relevant information in multimodal data decomposes into three components: redundancy shared between modalities, uniqueness specific to each modality, and synergy available only from their joint observation. Recent frameworks extend contrastive learning to capture all three components, yet synergy remains undertrained in practice. We propose SynCo (Synergy Contrastive Learning), a method that directly addresses synergy undertraining through dedicated supervision on an interaction residual. SynCo fits a linear projector to predict the fused representation from independently computed unimodal features, and the resulting interaction residual, which removes the linearly unimodal-predictable component, receives dedicated contrastive supervision at negligible computational cost. On the controlled Trifeature benchmark, SynCo achieves state-of-the-art synergy capture with a +5.98% gain over the baseline, and on real-world benchmarks from MultiBench, DARai, and MM-IMDb, SynCo consistently outperforms or matches prior methods across diverse modality combinations and task types. The method operates as a plug-in to existing contrastive multimodal frameworks without modifying the underlying fusion architecture and can further improve synergy capture when combined with other methods.
Yavuz Yarici, Ghassan AlRegib
OLIVES, CSIP, School of ECE Georgia Institute of Technology Atlanta, GA, USA
Despite the impressive results achieved by multimodal large language models (MLLMs), their training typically relies on jointly curated multimodal data, requiring substantial human effort to construct multi-way aligned datasets and thereby limiting scalability across domains. In this work, we explore training MLLMs by only leveraging multiple paired modalities as a surrogate for the full joint multimodal distribution. Specifically, we first provide a theoretical analysis of the conditions under which the representations are identifiable with only observing pairwise modalities. Building on this analysis, we propose a representation learning framework for aligning latent representations across modalities using only pairwise data. The framework consists of two stages: latent representation alignment and cross-modal recomposition. Specifically, in the first stage, we learn the shared latent space across modalities by both self-modal reconstruction and pair-wise contrastive learning. We also incorporate an inductive bias in the contrastive learning process by partially aligning and minimal latent specification. In stage two, we integrate the encoder of newly introduced modalities with the decoders of the pre-trained modalities to facilitate cross-modal transfer and generation. We evaluate our method by newly adding 3D point clouds and tactile modalities into pre-trained MLLMs with three modality pairs and show that, by learning an aligned latent representation space, our model achieves strong cross-modal performance.
Yan Li, Yunlong Deng, Yuewen Sun +3
Mohamed bin Zayed University of Artificial Intelligence · Carnegie Mellon University