Mitigating Strong-Modality Collapse in Multimodal Learning via Inverted Asymmetric Fusion

Mary Ogbuka Kenneth; Foaad Khosmood; Abbas Edalat

cs.LGAug 27, 2026

Mitigating Strong-Modality Collapse in Multimodal Learning via Inverted Asymmetric Fusion

Authors: Mary Ogbuka Kenneth, Foaad Khosmood, Abbas Edalat

Organizations: Algorithmic Human Development group, Department of Computing Imperial College London, United Kingdom

Abstract

Fusing multiple modalities is expected to improve model performance. However, on the MultiHuSE dataset, early, late, and symmetric attention fusion often fail to outperform the best unimodal baseline (text). Pathway isolation of a symmetric attention fusion model reveals that the text-pathway accuracy drops from 74.9% to 56.4% after fusion in one such setting, indicating that the dominant modality can be degraded during integration. We term this strong-modality collapse and argue that it helps explain why some multimodal models fail to surpass unimodal baselines. We propose Inverted Asymmetric Fusion (IAF), which avoids forcing mutual attention across modalities. The dominant modality is preserved by passing through fusion unchanged, while weaker modalities attend to it as a contextual anchor. Before fusion, weaker modalities are strengthened using Modality-Aware Knowledge Distillation. We evaluate IAF on three benchmarks with different modality hierarchies: text-dominant datasets (MultiHuSE, UR-FUNNY) and an audio-visual-dominant dataset (MUStARD). Pathway isolation shows that IAF preserves the dominant modality's internal accuracy at its unimodal ceiling across all tested configurations, whereas symmetric fusion degrades it by up to 18.5% on MultiHuSE. IAF improves over the strongest unimodal baseline by up to 8.25%.

Figures & tables

Panel of Figure 1 — Figure 1: Text-pathway accuracy on MultiHuSE (E5 encoder). Symmetric fusion degrades the dominant modality by 18.5 pp; IAF preserves it (Table 5 ).

Panel of Figure 2 — Figure 2: Overview of the proposed two-stage framework. Stage 1 performs Modality-Aware Knowledge Distillation, and Stage 2 applies the Inverted Asymmetric Fusion.

Modality / Feature	Base Acc(%)	Base F1(%)	Dist. Acc(%)	Dist. F1(%)
MultiHuSE — 5-class humor style \| Text $\gg$ Audio $\gg$ Video
Text: BERT	68.26 $\pm$ 2.25	68	70.21 $\pm$ 2.82	70
Text: E5 ⋆	74.86 $\pm$ 3.08	75	–	–
Audio: Dasheng	60.45 $\pm$ 1.58	60	62.90 $\pm$ 1.83	63
Video: PE-Core	43.66 $\pm$ 1.29	43	45.57 $\pm$ 1.22	46
Video: VideoMAE	40.13 $\pm$ 0.73	40	41.01 $\pm$ 1.60	41

Table 1: Unimodal results across all three benchmarks. Base : raw pre-trained; Dist. : knowledge-distilled. ⋆ = dominant modality. MultiHuSE and MUStARD: mean

\pm

SD over 5-fold CV; UR-FUNNY: single split.

Text	Video	Architecture	Acc	F1	$\Delta$
Audio: Dasheng used across all configurations
Text: BERT (base 68.26%)
BERT	VideoMAE	Early Fusion	67.22 $\pm$ 2.00	67	$-1.04$
		Late Fusion	68.05 $\pm$ 2.12	68	$-0.21$
		Symmetric Fusion	69.42 $\pm$ 1.48	69	$+1.16$
		Proposed (IAF)	73.16 $\pm$ 2.59	72	$+4.90$

Table 2: Trimodal fusion results on MultiHuSE.

\Delta

= gain over the strongest unimodal text baseline (BERT = 68.26%; E5 = 74.86%). All metrics in %.

Method	Context	Acc (%)	F1 (%)	$\Delta_{\text{uni}}$
Prior published methods
C-MFN Kamrul Hasan et al. (2019)	×	64.47	–	–
C-MFN Kamrul Hasan et al. (2019)	✓	65.23	–	–
AGM(Late fusion) Li et al. (2023a)	×	65.97	–	–
AGM(Early fusion) Li et al. (2023a)	×	66.07	–	–
MULTIBENCH Pu Liang et al. (2021)	✓	66.70	–	–

Table 3: Results on UR-FUNNY. All methods use GloVe/COVAREP/OpenFace features Kamrul Hasan et al. (2019) .

\Delta_{\text{uni}}

= gain over the GloVe baseline (62.47%) for our internal baselines. Context: ✓ = punchline + context; × = punchline only. † Seed 999; mean

69.70\pm 0.61\%

over 5 seeds.

Method	Context	Acc(%)	F1(%)	$\Delta_{\text{uni}}$
Prior published methods
MULTIBENCH Pu Liang et al. (2021)	✓	71.8 $\pm$ 0.3	—	–
SVM (BERT) Castro et al. (2019)	✓	—	71.5	–
SWIA Chauhan et al. (2020)	✓	—	72.6	–
CE (BART) Ray et al. (2022)	✓	—	74.2	–
MuLOT (BERT) Pramanick et al. (2022)	×	74.52	—	–

Table 4: Results on MUStARD.

\Delta_{\text{uni}}

= gain over Dasheng audio unimodal baseline (our internal baselines only).

Text	Video	Archi.	Full Fusion	Text Path	Audio Path	Video Path
MultiHuSE – Dominant Modality: TEXT – BERT=70.2%, E5=74.9%
BERT	VideoMAE	Symmetric	69.42 $\pm$ 1.48	61.3 $\pm$ 2.5	59.0 $\pm$ 4.5	60.1 $\pm$ 1.3
	VideoMAE	IAF	73.16 $\pm$ 2.59	70.2 $\pm$ 2.8	55.8 $\pm$ 7.4	57.5 $\pm$ 5.0
	PE-Core	Symmetric	72.00 $\pm$ 1.20	63.5 $\pm$ 1.6	63.6 $\pm$ 2.3	61.6 $\pm$ 7.2
	PE-Core	IAF	74.74 $\pm$ 1.71	70.2 $\pm$ 2.8	62.9 $\pm$ 2.5	65.6 $\pm$ 3.1
E5	VideoMAE	Symmetric	72.37 $\pm$ 2.73	56.4 $\pm$ 4.6	63.7 $\pm$ 9.2	67.4 $\pm$ 3.3

Table 5: Pathway isolation analysis. Each score reflects accuracy using only the corresponding modality pathway, bypassing fusion aggregation. Bold = dominant modality pathway. All metrics in %.

Dataset	Config	Feat.	Acc(%) $\pm$ SD	F1(%)	$\Delta$ (%)
MultiHuSE	BERT+PEC	Base	72.33 $\pm$ 2.02	72	-2.41
	BERT+PEC	Dist	74.74 $\pm$ 1.71	74	–
	E5+PEC	Base	75.90 $\pm$ 6.17	76	-2.16
	E5+PEC	Dist	78.06 $\pm$ 3.82	78	–
MUStARD	BERT+PEC	Base	81.59 $\pm$ 3.85	82	-1.02
	BERT+PEC	Dist	82.61 $\pm$ 3.92	83	–

Table 6: Distillation ablation: raw (Base) vs. distilled (Dist.) features within IAF.

\Delta

= Dist

-

Base.

Appendix figures & tables12 assets

Supplementary material from the paper’s appendix.

Appendix

Hyperparameter	MultiHuSE	UR-FUNNY	MUStARD
Optimiser	AdamW	Adam	AdamW
Learning rate	$6\times 10^{-5}$	$1\times 10^{-5}$	$1\times 10^{-3}$
Weight decay	$10^{-3}$	$10^{-3}$	$10^{-2}$
Max epochs	100	80	100
Early stopping patience	35	20	40
Gradient clipping	1.0	—	1.0

Appendix

Table 7: Dataset-specific hyperparameters for the fusion

Hyperparameter	Value
KD loss weights $(\alpha,\beta,\gamma)$	0.4, 0.35, 0.25
Temperature $T$	3.5
Optimiser	AdamW
Weight decay	$10^{-3}$
Text/Audio LR	$10^{-4}$ – $10^{-3}$
Video LR	$9\times 10^{-5}$

Appendix

Table 8: Distillation stage hyperparameters.

Configuration	Pure	Attends	Acc(%) $\pm$ SD	F1(%)
MultiHuSE — Text dominant (E5, Dasheng, PE-Core)
Full Trimodal	Text	Audio, Video	78.06 $\pm$ 3.82	78
T+V	Text	Video	76.61 $\pm$ 4.58	77
T+A	Text	Audio	76.57 $\pm$ 2.92	76
A+V (Text-Free)	Audio	Video	65.85 $\pm$ 1.63	65
UR-FUNNY — Text dominant (GloVe, COVAREP, OpenFace)

Appendix

Table 9: Bimodal ablation of E5 and GloVe-based configuration for IAF

Configuration	Pure	Attends	Acc (%) $\pm$ SD	F1 (%)
MultiHuSE — Text dominant (BERT, Dasheng, PE-Core)
Full Trimodal	Text	Audio, Video	74.74 $\pm$ 1.71	74
T+V	Text	Video	72.45 $\pm$ 3.27	72
T+A	Text	Audio	72.08 $\pm$ 2.71	72
MUStARD — Audio/Video dominant (BERT, Dasheng, PE-Core)
Full Trimodal	Audio, Video	Text	82.61 $\pm$ 3.92	83

Appendix

Table 10: Bimodal fusion for BERT configurations on MultiHuSE and MUStARD.

Mixup	Curriculum	Acc (%) $\pm$ SD	F1	$\Delta$ (%)
MultiHuSE (BERT + Dasheng + PE-Core)
✓	✓	74.74 $\pm$ 1.71	74	–
✗	✓	73.49 $\pm$ 3.15	73	$-1.25$
✓	✗	72.58 $\pm$ 3.80	72	$-2.16$
✗	✗	70.75 $\pm$ 2.67	70	$-3.99$
MultiHuSE (E5 + Dasheng + PE-Core)

Appendix

Table 11: Regularisation ablation across MultiHuSE, UR-FUNNY, and MUStARD. ✓✓ = both Mixup and Curriculum Dropout enabled (full model);

\Delta

= accuracy change relative to the full model. All metrics in %.

Config	Gate Type	Acc(%) $\pm$ SD	$\Delta$ (%)
MultiHuSE Dataset (Hierarchy: T=0.50, A=0.30, V=0.20)
BERT + PE-Core	Learnable	74.74 $\pm$ 1.71	–
	Hierarchy	74.99 $\pm$ 2.68	$+0.25$
	Uniform	74.86 $\pm$ 3.31	$+0.12$
E5 + PE-Core	Learnable	78.06 $\pm$ 3.82	–
	Hierarchy	78.52 $\pm$ 3.64	$+0.46$

Appendix

Table 12: Gate mechanism ablation across all three datasets. Learnable = sample-adaptive weights; Hierarchy = fixed weights derived from modality ranking; Uniform = equal weights (T=A=V=0.33).

\Delta

= accuracy difference relative to the Learnable gate.

Figure 3: Dominant-modality pathway accuracy for all configurations (data from Table 5 ). Shaded regions group MultiHuSE (blue) and MUStARD (purple) configurations.

Figure 4: Joint t-SNE of features used at the final classification step for the text pathway under IAF ( left ) and symmetric fusion ( right ), fold 4 of MultiHuSE (E5 + PE-Core, distilled). Green = correctly classified by that pathway; red = misclassified.

Text	Feat.	$\bm{\Delta}$ (%)	$\bar{w}_{T}$	$\bar{w}_{A}$	$\bar{w}_{V}$
MultiHuSE (Text pure $\|$ Dasheng+PE-Core)
BERT+PE-Core	Base	$+4.07^{[\text{B}]}$	0.474	0.355	0.171
BERT+PE-Core	Dist.	$+6.48^{[\text{B}]}$	0.355	0.206	0.439
E5+PE-Core	Base	$+1.04^{[\text{E}]}$	0.431	0.425	0.144
E5+PE-Core	Dist.	$+3.20^{[\text{E}]}$	0.412	0.197	0.391
UR-FUNNY (Text pure $\|$ COVAREP + OpenFace)

Appendix

Table 13: Mean learned gate weights across CV folds at the best checkpoint.

\Delta

= gain over dominant unimodal baseline (MultiHuSE: BERT = 68.26%, E5 = 74.86%; UR-FUNNY: GloVe = 62.47%; MUStARD: Dasheng = 78.12%). [B] / [E] : gap from BERT/E5 baseline. Base = raw pre-trained; Dist. = knowledge-distilled.

Sample text	True	Pred	$w_{T}$	$w_{A}$	$w_{V}$
(i) High text weight, correctly classified
Joe Kennedy III reveals how his GOP counterparts really feel about Trump’s tweets.	Neu	Neu	0.970	0.006	0.024	✓
Katy Perry wears American flag dress to kids in overall concerts.	Neu	Neu	0.961	0.018	0.020	✓
Porn actress confirms Trump affair after unpublished 2011 interview.	Neu	Neu	0.959	0.020	0.021	✓
(ii) Low text weight, correctly classified
The only positive thing about you is your HIV status.	Agg	Agg	0.023	0.548	0.429	✓

Appendix

Table 14: Representative IAF gate weight profiles (E5 + PE-Core, MultiHuSE, all folds). This table contains potentially offensive or sensitive example text reproduced from the dataset for analysis. Class labels: Aff = Affiliative, Agg = Aggressive, Neu = Neutral, S-enh = Self-enhancing.

w_{T}{+}w_{A}{+}w_{V}=1

per sample. ✓ correct; ✗ incorrect.

Setting	Coeff. ( $\alpha$ / $\beta$ / $\gamma$ )	BERT Dist.	Audio Dist.	Video Dist.	IAF (BERT)	IAF (E5)
Default †	0.40 / 0.35 / 0.25	70.21 $\pm$ 2.82	62.90 $\pm$ 1.83	45.57 $\pm$ 1.22	74.74 $\pm$ 1.71	78.06 $\pm$ 3.82
Equal weights	0.33 / 0.33 / 0.33	69.92 $\pm$ 2.79	63.11 $\pm$ 1.54	44.66 $\pm$ 1.57	74.37 $\pm$ 2.33	78.06 $\pm$ 3.36
CE-heavy	0.60 / 0.25 / 0.15	69.63 $\pm$ 2.75	62.78 $\pm$ 1.18	45.95 $\pm$ 0.98	74.03 $\pm$ 2.48	76.86 $\pm$ 4.32
Soft-target-heavy	0.25 / 0.55 / 0.20	70.17 $\pm$ 2.58	62.86 $\pm$ 1.10	44.74 $\pm$ 1.35	73.33 $\pm$ 2.32	76.90 $\pm$ 4.18
Range	–	0.58	0.33	1.29	1.41	1.20

Appendix

Table 15: Distillation coefficient sensitivity on MultiHuSE (5-fold CV, mean

\pm

SD %).

\dagger

= setting used in all main experiments. IAF columns use distilled students; Audio = Dasheng, Video = PE-Core throughout. “Range” reports max

-

min across the four settings.

Method	Stage	Teacher	Purpose	Procedure
Cross-M KD Albanie et al. (2018) Thoker and Gall (2019)	Pre-fusion †	Paired modality	Unimodal transfer	Sequential
Wang et al. Wang et al. (2020a)	Within fusion	Full multimodal	Dropout robustness	Joint
MMANet Wei et al. (2023)	Within fusion	Full multimodal	Missing-modality	Joint
DMD Li et al. (2023b)	Within fusion	Subspace teachers	Repr. disentanglement	Concurrent
MTMD Lin and Hu (2024)	Within fusion	Unimodal + fused	Heterogeneity reduction	Concurrent
MAKD (Ours)	Pre-fusion	Dom. unimodal	Encoder strengthening	Sequential

Appendix

Table 16: Design comparison of multimodal knowledge distillation methods. † Not designed in the context of a subsequent fusion stage.

Multimodal Fusion

Multimodal fusion combines information from different data types to improve prediction, perception, and decision-making.

See all 771 papers

Segregate, Refine, Integrate: Decomposing Multimodal Fusion for Sentiment Analysis
Jul 14, 2026Alexios Filippakopoulos, Elias Kallioras, Nikolaos Xiros +2Cross-Modal Representation Learning Multimodal Large Language Models
National Technical University of Athens, Greece · Athena Research Center, Greece · University of Bern, Switzerland +2
Feature Alignment Determines Fusion Strategy: A Comparative Study of Cross-Attention and Concatenation in Multimodal Learning
May 31, 2026Zhiqiang Zhou, Xuezhen XieCross-Attention Multimodal Fusion
Hunan Chemical Industry Vocational and Technical College
MuteBench: Modality Unavailability Tolerance Evaluation for Incomplete Multimodal Fusion
May 13, 2026Wugeng Zheng, Ziwen Kan, Tianyu Hu +3Missing-Modality Learning Multimodal Robustness
University of Central Florida · University of North Carolina at Chapel Hill

Jul 14, 2026J/K move · Enter open · S save

Segregate, Refine, Integrate: Decomposing Multimodal Fusion for Sentiment Analysis

Alexios Filippakopoulos, Elias Kallioras, Nikolaos Xiros, Efthymios Georgiou+1

National Technical University of Athens, Greece · Athena Research Center, Greece · University of Bern, Switzerland+2

Cross-Modal Representation Learning Multimodal Large Language Models Multimodal Sentiment Analysis Multimodal Fusion Data Fusion

Open paper AskPDF arXiv

Multimodal fusion must simultaneously refine modality-specific signals and model cross-modal interactions; two competing objectives typically entangled within the same operation. We propose SeRIn (Segregate, Refine, Integrate), a multimodal LM fusion scheme that enforces this separation as an architectural prior. Modality-specific representations evolve along isolated pathways, each refined against its respective encoder context, while a dedicated cross-modal pathway accumulates their joint evolution without contaminating unimodal streams. Full cross-modal interaction is deferred to a final prediction step - ablations confirm that structured interactions, not added capacity, drive the gains; gate analysis under visual corruption reveals emergent modality reweighting without explicit supervision. SeRIn achieves state-of-the-art results on CH-SIMS and CMU-MOSEI, improving all metrics on both benchmarks.

Abstract

Multimodal Fusion

Explore similar work

Segregate, Refine, Integrate: Decomposing Multimodal Fusion for Sentiment Analysis

Feature Alignment Determines Fusion Strategy: A Comparative Study of Cross-Attention and Concatenation in Multimodal Learning

MuteBench: Modality Unavailability Tolerance Evaluation for Incomplete Multimodal Fusion