In multi-task settings, combining multiple modalities allows task representations to draw on signals that a single modality alone would not capture. However, individual tasks may depend disproportionately on particular modalities, a structure that conventional fusion-based architectures do not model explicitly, instead enforcing a unified representation space. Consequently, a task's gradients can affect parameters tied to modalities it barely relies on, leaving features that compromise between competing demands. Although prior work has explored task-weighting and loss-balancing strategies, their effectiveness is limited by the absence of explicit modality separation, preventing models from optimising each modality according to its task-specific importance. To address these limitations, we propose MOSAIC (MOdality-Separated Architecture with Inter-modal Cross-attention), which preserves unimodal representations while enabling controlled interaction between them, and uses modality-specific losses to route gradients to their corresponding partitions. On the Corpus of Everyday Japanese Conversation (CEJC), MOSAIC improves average macro-F1 by 7.4 points across four tasks with a 72% reduction in backbone parameters relative to a fusion-based baseline. Our results demonstrate that modality partitioning yields consistent gains across tasks while requiring far fewer backbone parameters than a fusion-based design.
Figures & tables
Figure 1: MOSAIC’s Architecture. Dotted lines indicate separation between partitions.
Model
Style
Place
Activity
Relationship
Avg.
Video (DINOv2)
0.562
0.674
0.289
0.561
0.522
Audio (HuBERT)
0.620
0.641
0.262
0.591
0.529
Text (BERT-Japanese)
0.548
0.303
0.201
0.558
0.403
AVL-BERT
0.648
0.592
0.238
0.684
0.541
AVL-BERT (Upgraded)
0.692
0.676
0.251
0.712
0.583
MOSAIC (Single-Task)
0.676
0.763
0.476
0.689
0.651
Table 1: Macro-averaged F1 scores on the CEJC test set across four classification tasks. Best results are shown in bold and second-best results are underlined .
Model
Style
Place
Activity
Relationship
Avg.
MOSAIC (full)
0.727
0.739
0.441
0.720
0.657
w/o Modality-Specific Attention
0.699
0.678
0.322
0.604
0.576
w/o Block-Diagonal MLP
0.729
0.710
0.358
0.664
0.615
w/o Cross-Attention Layers
0.687
0.718
0.341
0.603
0.587
w/o Modality-Specific Losses
0.719
0.708
0.348
0.628
0.601
Table 2: Ablation study on CEJC. Each row replaces one component of the full MOSAIC architecture with its standard BERT-Base counterpart (see text). Macro-averaged F1 scores reported.
Model
Total Backbone Params
Trainable Params
Avg F1
AVL-BERT (upgraded)
109,482,240
109,482,240
0.583
MOSAIC (no fine-tuning)
30,813,723
610,587
0.639
MOSAIC (Best)
30,813,723
11,846,400
0.657
Table 3: Parameter counts and average F1 for AVL-BERT and MOSAIC variants.
Multimodal fusion must simultaneously refine modality-specific signals and model cross-modal interactions; two competing objectives typically entangled within the same operation. We propose SeRIn (Segregate, Refine, Integrate), a multimodal LM fusion scheme that enforces this separation as an architectural prior. Modality-specific representations evolve along isolated pathways, each refined against its respective encoder context, while a dedicated cross-modal pathway accumulates their joint evolution without contaminating unimodal streams. Full cross-modal interaction is deferred to a final prediction step - ablations confirm that structured interactions, not added capacity, drive the gains; gate analysis under visual corruption reveals emergent modality reweighting without explicit supervision. SeRIn achieves state-of-the-art results on CH-SIMS and CMU-MOSEI, improving all metrics on both benchmarks.
Alexios Filippakopoulos, Elias Kallioras, Nikolaos Xiros +2
National Technical University of Athens, Greece · Athena Research Center, Greece · University of Bern, Switzerland +2
Fusing multiple modalities is expected to improve model performance. However, on the MultiHuSE dataset, early, late, and symmetric attention fusion often fail to outperform the best unimodal baseline (text). Pathway isolation of a symmetric attention fusion model reveals that the text-pathway accuracy drops from 74.9% to 56.4% after fusion in one such setting, indicating that the dominant modality can be degraded during integration. We term this strong-modality collapse and argue that it helps explain why some multimodal models fail to surpass unimodal baselines. We propose Inverted Asymmetric Fusion (IAF), which avoids forcing mutual attention across modalities. The dominant modality is preserved by passing through fusion unchanged, while weaker modalities attend to it as a contextual anchor. Before fusion, weaker modalities are strengthened using Modality-Aware Knowledge Distillation. We evaluate IAF on three benchmarks with different modality hierarchies: text-dominant datasets (MultiHuSE, UR-FUNNY) and an audio-visual-dominant dataset (MUStARD). Pathway isolation shows that IAF preserves the dominant modality's internal accuracy at its unimodal ceiling across all tested configurations, whereas symmetric fusion degrades it by up to 18.5% on MultiHuSE. IAF improves over the strongest unimodal baseline by up to 8.25%.
Mary Ogbuka Kenneth, Foaad Khosmood, Abbas Edalat
Algorithmic Human Development group, Department of Computing Imperial College London, United Kingdom
Multi-modal semantic segmentation benefits from complementary sensors, but this benefit is fragile when some modalities are degraded or missing at test time. Many existing methods rely on dedicated fusion modules or multi-branch cross-modal encoders, and the fusion often becomes unreliable when modalities are absent, leading to substantial performance drops. We propose FuseAdapt, a parameter-efficient framework that performs fusion within the adaptation space of a frozen vision foundation model (VFM), exploring whether VFM priors can serve as an effective shared backbone for multi-modal segmentation. Instead of attaching a standalone fusion network, FuseAdapt assigns each modality a lightweight adaptation branch and performs adaptation-space fusion by keeping Q/V low-rank adaptation private to each modality and routing only FFN low-rank residual activations across modalities inside the frozen backbone. The pretrained VFM remains fixed while modality-specific corrections and explicit cross-modal interaction are learned through the adaptation paths. To handle missing modalities, we further introduce Predictive Modality Modeling (PMM), a direct representation-level objective for cross-modal redundancy. During training, PMM masks a target modality and predicts its embedding from the visible ones, supervised by a teacher target computed under full-modality input. PMM is used only during training and adds no inference-time overhead. Across three multi-modal segmentation benchmarks (MCubeS, DELIVER, MUSES), FuseAdapt consistently improves both modality-complete and modality-incomplete segmentation with a small trainable budget, yielding up to +3.01 mIoU under full modalities and +11.25 mIoU under missing modalities over prior state-of-the-art. Project page: https://devinxzhang.github.io/FuseAdapt-Page/