Fusion or Confusion? Multimodal Complexity Is Not All You Need
Authors: Tillmann Rheude, Roland Eils, Benjamin Wild
Organizations: Berlin Institute of Health, Charité - Universitätsmedizin Berlin · Department of Mathematics and Computer Science, Freie Universität Berlin · Intelligent Medicine Institute, Fudan University
Multimodal learning has become a prominent research area, with the potential of substantial performance gains by combining information across modalities. At the same time, model development has trended toward increasingly complex deep learning architectures, motivated by the assumption that multimodal-specific methods improve performance. We challenge this assumption through a large-scale empirical study by reimplementing 19 high-impact multimodal methods across nine diverse datasets with up to 23 modalities. Under standardized experimental conditions, including hyperparameter tuning, weight initialization, cross-validation, and statistical testing, increased multimodal complexity often yields confusion rather than effective fusion of data modalities. Accordingly, complex multimodal architectures do not reliably outperform unimodal baselines and a Simple Baseline for Multimodal Learning (SimBaMM). Through a focused case study, we further demonstrate concrete methodological shortcomings even in top-tier multimodal learning publications, underscoring the need for standardized evaluation practices. In summary, we argue for a shift in focus for multimodal learning: away from the pursuit of architectural novelty and toward methodological rigor.
Figures & tables
Figure 1: Taxonomy of multimodal learning methods exemplified by two modalities. An LF baseline ( SimBaMM ) processes a multimodal dataset D with up to M modalities through encoders Em , combines them via fusion, and produces predictions through a classification head. Categorized into five groups, we evaluate 19 SoTA methods across nine datasets based on their architectural innovations on top of SimBaMM : Missing Data handling for incomplete modalities , e.g., a missing ECG , Encoder -level modifications, Fusion mechanisms for combining representations, Head -based cross-modal architectures, and Gradient -based optimization techniques. Our work reveals that none of these complex extensions reliably outperforms SimBaMM , suggesting that reported performance gains are often attributable to experimental confounders rather than methodological novelty.
Dataset
Size
Modalities
Raw Input
Encoders
Crema-D
7,442
2 (Video, Audio)
✓
2× ResNet
MIMIC HAIM
45,050
2 (Laboratory, X-ray)
✓
MLP , ViT
INSPECT
22,449
2 ( CT , EHR )
✗
MLPs
MIMIC Symile
10,345
3 (Laboratory, X-ray, ECG)
✓
MLP , ViT , Transformer
CMU-MOSI
2,199
3 (Video, Audio, Language)
✗
2× Transformer, BERT
CMU-MOSEI
22,856
3 (Video, Audio, Language)
✗
2× Transformer, BERT
Table 1: Overview of datasets and the respective encoders used in our evaluation. Details in Appendix F .
Evaluation Protocol
% of Methods ↑
Hyperparameter Tuning
0
Statistical Significance
0
Cross-Validation
11
Error Bars
21
Unimodal Baselines
82
Table 2: Overview of evaluation protocols used by recent multimodal methods. Values are the percentage of surveyed multimodal papers using each protocol.
Figure 2: Aggregated forest plot comparing each method against SimBaMM -CLS across all datasets. Points show the mean performance difference relative to SimBaMM -CLS, horizontal lines indicate uncertainty, and the shaded region marks the dataset-specific ±1% ROPE , rather than the generalized ROPE used in the other tables. Unimodal represents the best-performing modality. No method shows a consistent aggregate advantage over SimBaMM -CLS across datasets. Details in Appendix D .
Method
P(base > other)
P( ≈ )
P(other > base)
E[δ0]
95% CI
Best Unimodal
0.53
0.06
0.41
0.002
[-0.031,0.031]
C-Mamba [ 39 ]
0.03
0.67
0.30
-0.006
[-0.021,0.006]
MMPareto [ 81 ]
0.13
0.67
0.19
-0.002
[-0.019,0.022]
ARL [ 80 ]
0.30
0.47
0.23
0.002
[-0.019,0.028]
GBlend-On [ 75 ]
0.36
0.48
0.16
0.005
[-0.018,0.045]
BMML [ 82 ]
0.75
0.05
0.19
0.013
[-0.021,0.051]
Table 3: Comparison with SimBaMM CLS ( base in this table). No method achieves a high probability of meaningfully outperforming SimBaMM CLS . Details in Appendix H .
Figure 4: Missing modality analysis for IMDer [ 77 ] , MMP [ 56 ] , ShaSpec [ 74 ] , and SimBaMM. Methods perform similarly across missing rates.
Figure 5: Case study on the CREMA-D dataset. Under a corrected data protocol with subject-independent splits and consistent hyperparameter tuning, the relative performance of methods changes.
Appendix figures & tables15 assets
Supplementary material from the paper’s appendix.
Appendix
Method
HAIM ↑
Symile ↑
INSPECT ↑
UKB ↑
Best Unimodal
0.7042±.020
0.6453±.030
0.6568±.003
0.7560±.010
Missing Data-Based
IMDer [ 77 ]
0.6900±.018
0.6266±.016
0.6385±.007
0.7969±.003
MMP [ 56 ]
0.6611±.016
0.5824±.054
0.6289±.018
0.7984±.004
ShaSpec [ 74 ]
0.6863±.019
0.5823±.052
0.6348±.006
0.7968±.007
SimBaMM CLS
0.6558±.031
0.5925±.045
0.6378±.008
0.7957±.008
Appendix
Table 4: Performance in terms of AUROC. Details including full unimodal results in Appendix F . No method is highlighted because performance differences fall within the reported mean ± SDs , summarized in Table 3 .
Method
MOSI ↑
MOSEI ↑
CH-SIMS ↑
CH-SIMS 2 ↑
Crema-D ↑
Unimodal
Language
0.4084±.024
0.5144±.006
0.5458±.022
0.4280±.014
n/a
Vision
0.2556±.038
0.4285±.007
0.3593±.014
0.2497±.010
0.3192±.027
Audio
0.2401±.011
0.4197±.005
0.4298±.021
0.2686±.031
0.6081±.032
Missing Data-Based
IMDer [ 77 ]
0.2810±.058
0.4273±.039
0.4541±.079
0.3951±.022
0.6840±.032
Appendix
Table 5: Performance in terms of Acc7 (MOSI/MOSEI), Acc5 (CH-SIMS/CH-SIMS2) and Acc6 (Crema-D). No method is highlighted because performance differences fall within the reported mean ± SDs , summarized in Table 3 .
Group
Method
#Parameters (M) ↓
FLOPs (G) ↓
Training Time (sec/epoch) ↓
Data
IMDer [ 77 ]
103.0
408.23
203.50
MMP [ 56 ]
98.1
12.57
15.42
ShaSpec [ 74 ]
103.0
11.14
17.45
Encoder
RegBN [ 19 ]
97.4
11.15
43.21
AUG [ 30 ]
97.4
11.10
12.15
Fusion
MBT [ 55 ]
99.4
11.11
18.62
Appendix
Table 6: Tabular representation of the efficiency of the methods w.r.t. the number of parameters, training time, and FLOPS .
Figure 6: Visual representation of the efficiency of the methods w.r.t. the number of parameters, training time, and FLOPS .
Modality
AUROC ↑
HAIM
X-ray
0.7042±0.020
Laboratory
0.6654±0.038
Symile
X-ray
0.6453±0.030
Laboratory
0.5597±0.010
Appendix
Table 7: Unimodal performance of the modalities for the healthcare datasets. For the UKB , if unimodal data is not available for a sample, it is zero-imputed to ensure a fair comparison to the multimodal performance.
Modality
AUROC ↑
Metabolomics
0.7118±0.004
EHR
0.7728±0.005
Proteomics
0.7588±0.018
PRS
0.6009±0.009
Blood biochemistry
0.7183±0.004
Baseline characteristics
0.7150±0.006
Appendix
Table 8: Unimodal performance of the modalities for the UKB without zero imputation , i.e., only the available modality subset is used.
Figure 7: Alternative configurations for AUG in the case study setting with the corrected data protocol , i.e., with a linear head (AUG-L) and another optimizer (AUG-AdamW).
Method
P(base > other)
P( ≈ )
P(other > base)
E[δ0]
95% CI
CoupledMamba
0.18
0.10
0.72
-0.010
[-0.038,0.024]
SimBaMM-CLS
0.41
0.06
0.53
-0.002
[-0.031,0.031]
MMPareto
0.66
0.08
0.26
0.013
[-0.026,0.066]
ARL
0.19
0.14
0.67
-0.008
[-0.034,0.024]
GBlend-On
0.61
0.10
0.29
0.011
[-0.029,0.062]
BMML
0.59
0.03
0.38
0.006
[-0.035,0.056]
Appendix
Table 9: Comparison with the best unimodal baseline.
Method
P(base > other)
P( ≈ )
P(other > base)
E[δ0]
95% CI
CoupledMamba
0.07
0.28
0.64
-0.006
[-0.021,0.006]
Best unimodal
0.56
0.00
0.44
0.002
[-0.031,0.031]
MMPareto
0.23
0.32
0.45
-0.002
[-0.019,0.022]
ARL
0.45
0.18
0.37
0.002
[-0.019,0.028]
GBlend-On
0.48
0.23
0.29
0.005
[-0.018,0.045]
BMML
0.79
0.00
0.21
0.013
[-0.021,0.051]
Appendix
Table 10: Comparison with SimBaMM-CLS ( ROPE ±0.5% ).
Method
P(base > other)
P( ≈ )
P(other > base)
E[δ0]
95% CI
CoupledMamba
0.01
0.95
0.04
-0.006
[-0.021,0.006]
Best unimodal
0.25
0.53
0.22
0.002
[-0.031,0.031]
MMPareto
0.05
0.91
0.04
-0.002
[-0.019,0.022]
ARL
0.11
0.84
0.06
0.002
[-0.019,0.028]
GBlend-On
0.21
0.74
0.05
0.005
[-0.018,0.045]
BMML
0.51
0.38
0.11
0.013
[-0.021,0.051]
Appendix
Table 11: Comparison with SimBaMM-CLS ( ROPE ±2% ).
Method
P(base > other)
P( ≈ )
P(other > base)
E[δ0]
95% CI
CoupledMamba
0.20
0.02
0.79
-0.010
[-0.038,0.024]
SimBaMM-CLS
0.44
0.00
0.56
-0.002
[-0.031,0.031]
MMPareto
0.69
0.02
0.29
0.013
[-0.026,0.066]
ARL
0.23
0.02
0.76
-0.008
[-0.034,0.024]
GBlend-On
0.65
0.02
0.33
0.011
[-0.029,0.062]
BMML
0.60
0.00
0.40
0.006
[-0.035,0.056]
Appendix
Table 12: Comparison with the best unimodal baseline ( ROPE ±0.5% ).
Method
P(base > other)
P( ≈ )
P(other > base)
E[δ0]
95% CI
CoupledMamba
0.10
0.52
0.37
-0.010
[-0.038,0.024]
SimBaMM-CLS
0.22
0.53
0.25
-0.002
[-0.031,0.031]
MMPareto
0.52
0.34
0.14
0.013
[-0.026,0.066]
ARL
0.09
0.63
0.28
-0.008
[-0.034,0.024]
GBlend-On
0.46
0.38
0.16
0.011
[-0.029,0.062]
BMML
0.43
0.34
0.23
0.006
[-0.035,0.056]
Appendix
Table 13: Comparison with the best unimodal baseline ( ROPE ±2% ).
Baseline
Metric
ρ=1/(K−1)
sˉ×100
SimBaMM-CLS
max ∣ΔP∣ (win)
0.034
0.015
max ∣ΔP∣ ( ≈ )
0.053
0.024
max ∣ΔP∣ (loss)
0.024
0.024
max ∣ΔE[δ0]∣
0.001
0.001
Best unimodal
max ∣ΔP∣ (win)
0.036
0.012
max ∣ΔP∣ ( ≈ )
0.035
0.008
Appendix
Table 14: Sensitivity of baseline comparisons to ρ and prior bounds.
Baseline
max R^
min ESS (bulk/tail)
div /td
max P(∣δnew∣>1)
SimBaMM-CLS
1.010
439/108
0/0
7.07e-04
Best unimodal
1.010
501/320
0/0
3.86e-04
Appendix
Table 15: Sampler diagnostics for baseline comparisons.
Method
M8 AUROC ↑
M16 AUROC ↑
Best Unimodal
0.6869±0.005 (ECG)
0.7118±0.004 (Metabolomics)
Missing Data-Based
IMDer [ 77 ]
0.7187±0.003
0.7689±0.004
MMP [ 56 ]
0.7203±0.001
0.7716±0.002
ShaSpec [ 74 ]
0.6831±0.015
0.7640±0.013
SimBaMM CLS
0.7211±0.003
0.7660±0.003
Appendix
Table 16: Performance for eight and 16 modalities on the UKB .
Multimodal learning has seen remarkable progress, particularly with large-scale pre-training across various modalities. Most current approaches are built on the assumption of a deterministic one-to-one alignment between modalities. However, this oversimplifies real-world multimodal relationships, where their nature is inherently many-to-many. The many-to-many property, or multiplicity, is not a side-effect of noise or annotation error, but an inevitable outcome of intra-modal variability, representational asymmetry, and task-dependent ambiguity in multimodal tasks. We argue that multiplicity is a fundamental bottleneck that affects all stages of the multimodal learning pipeline: from data construction to model training and evaluation benchmarks. By formalizing its causes and consequences, we demonstrate how ignoring multiplicity leads to training uncertainty, unreliable evaluation, and degraded dataset quality. This position paper calls for new research directions on multimodal learning, including multiplicity-aware learning frameworks and dataset construction and evaluation protocols.
Fusing multiple modalities is expected to improve model performance. However, on the MultiHuSE dataset, early, late, and symmetric attention fusion often fail to outperform the best unimodal baseline (text). Pathway isolation of a symmetric attention fusion model reveals that the text-pathway accuracy drops from 74.9% to 56.4% after fusion in one such setting, indicating that the dominant modality can be degraded during integration. We term this strong-modality collapse and argue that it helps explain why some multimodal models fail to surpass unimodal baselines. We propose Inverted Asymmetric Fusion (IAF), which avoids forcing mutual attention across modalities. The dominant modality is preserved by passing through fusion unchanged, while weaker modalities attend to it as a contextual anchor. Before fusion, weaker modalities are strengthened using Modality-Aware Knowledge Distillation. We evaluate IAF on three benchmarks with different modality hierarchies: text-dominant datasets (MultiHuSE, UR-FUNNY) and an audio-visual-dominant dataset (MUStARD). Pathway isolation shows that IAF preserves the dominant modality's internal accuracy at its unimodal ceiling across all tested configurations, whereas symmetric fusion degrades it by up to 18.5% on MultiHuSE. IAF improves over the strongest unimodal baseline by up to 8.25%.
Mary Ogbuka Kenneth, Foaad Khosmood, Abbas Edalat
Algorithmic Human Development group, Department of Computing Imperial College London, United Kingdom
Multimodal learning (MML) falls into the optimization dilemma due to the modality imbalance phenomenon, leading to suboptimal overall performance in practice. While many attempts primarily focus on balancing the optimization dynamics across modalities to address this issue, we identify a subtle yet critical flaw: optimization yields asymmetric gains in predictive certainty, with the strong modality more confident than the weak one, driving imbalanced modality contributions. In this paper, our analysis reveals that this flaw stems from unimodal characteristics rather than multimodal learning, and this confidence discrepancy can be corrected by positive cross-modal intervention. Based on this insight, we propose multimodal Max Confidence Regularization (MaxCR) to dynamically intervene in modality semantic confidence. Specifically, the semantic confidence of each modality is tracked using a nonlinear sparsity measure. We then design max suppression and max excitation based on this measure to regularize strong and weak modalities, respectively. They penalize and encourage the top-1 confidence, thereby constraining multimodal prediction. To this end, strong and weak modalities are expected to make calibrated confidence, thereby improving the overall performance. Empirical experiments on widely used datasets reveal the superiority of our method through comparison with various state-of-the-art (SOTA) multimodal learning baselines.
Longfei Huang, Xiangyu Wu, Yang Yang
Nanjing University of Science and Technology · Alibaba Group