cs.LGMar 23, 2026

SSAM: Singular Subspace Alignment for Merging Multimodal Large Language Models

Authors: Md Kaykobad Reza, Ameya Patil, Edward Ayrapetian, M. Salman Asif

Organizations: University of California Riverside · Amazon

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Multimodal large language models (MLLMs) achieve strong performance by jointly processing inputs from multiple modalities, such as vision, audio, and language. However, building such models or extending them to new modalities often requires large paired datasets and substantial computational resources. Since many pretrained MLLMs (e.g., vision-language or audio-language) are publicly available, we ask whether we can merge them into a single MLLM that can handle multiple modalities? Merging MLLMs with different input modalities remains challenging, partly because of differences in the learned representations and interference between their parameter spaces. To address these challenges, we propose Singular Subspace Alignment and Merging (SSAM), a training-free model merging framework that unifies independently trained specialist MLLMs into a single model capable of handling any combination of input modalities. SSAM maintains modality-specific parameter updates separately and identifies a shared low-rank subspace for language-related parameter updates, aligns them within this subspace, and merges them to preserve complementary knowledge while minimizing parameter interference. Without using any multimodal training data, SSAM achieves state-of-the-art performance across four datasets, surpassing prior training-free merging methods and even jointly trained multimodal models. These results demonstrate that aligning models in parameter space provides a scalable and resource-efficient alternative to conventional joint multimodal training.

Figures & tables

Figure 1: Comparison of multimodal learning paradigms. (a) Conventional methods train a single MLLM on paired multimodal datasets. (b) ImageBind-LLM [ 12 ] aligns all modalities to shared image embeddings. (c) SSAM offers a training-free method to merge independently trained specialist MLLMs into a single, unified model via subspace projection.
Figure 2: Overview of the proposed SSAM method. (a) Subspace projection: each language vector Δi\Delta_{i} is projected onto a shared low-rank subspace prior to merging. (b) SSAM constructs covariance matrices AA and BB from all Δi\Delta_{i} , extracts the shared left and right bases UcU_{c} and VcV_{c} , projects language vectors, and merge them into Δmerged\Delta_{\text{merged}} .
DatasetMethodV+LV+I+LV+A+LV+I+A+LAverage
MUSIC-AVQAModels finetuned on paired multimodal data
ChatBridge-13B [ 51 ]--43.00-43.00
OneLLM-7B [ 11 ]--47.60-47.60
ImageBind-LLM [ 12 ]37.2438.7639.7238.1638.47
X-InstructBLIP [ 27 ]45.8341.2348.3447.3945.70
Proj-Only [ 3 ]44.9346.6446.1750.2146.99
Table 1: Performance comparison on MUSIC-AVQA [ 17 ] and AVQA [ 45 ] datasets. SSAM achieves the best average performance across all modality combinations and consistently outperforms both training-free model merging baselines and models finetuned on paired multimodal data.
MethodMCUB-3MCUB-4
I+V+P+LA+V+P+LI+A+P+LI+A+V+LAverageI+A+V+P+L
Models finetuned on paired multimodal data
ImageBind-LLM [ 12 ]31.8031.4033.4035.2032.9532.93
X-InstructBLIP [ 27 ]29.4025.2021.2041.4029.3027.94
Proj-Only [ 3 ]42.6043.8042.8047.4044.1543.00
Training-free model merging methods
Table 2: Performance comparison on MCUB benchmark [ 3 ] . SSAM achieves the best average performance across all modality combinations and consistently outperforms both training-free merging baselines and models finetuned on paired multimodal data.
Figure 3: Qualitative examples on commonality understanding on the MCUB-4 dataset.
Table 6
Figure 4: Qualitative examples on the AVQA dataset.
Figure 8
SubspaceMCUB-3MCUB-4
Rank (k)I+V+P+LA+V+P+LI+A+P+LI+A+V+LAverageI+A+V+P+L
6458.2061.4065.0056.4060.2562.34
12859.4061.0065.4055.6060.3562.27
25659.2059.2064.8056.2059.8561.66
38458.4057.8064.4054.4058.7561.47
51258.6058.2065.2055.4059.3561.52
Table S1 : Performance of SSAM with different subspace ranks ( kk ) on the MCUB-3 and MCUB-4 datasets. I, A, V, P, and L denote Image, Audio, Video, Point Cloud, and Language, respectively. Rank k=128k=128 empirically yields overall better performance.
Figure S1 : Qualitative results on the MCUB-4 [ 3 ] dataset with four input modalities (image, video, audio, and point cloud) paired with natural-language queries. SSAM demonstrates strong cross-modal reasoning and provides coherent answers in complex multimodal settings, even though none of the specialist models were trained on this modality combination.
Figure S2 : Qualitative results on the MUSIC-AVQA [ 17 ] and AVQA [ 45 ] datasets with video and audio as the input modalities paired with natural-language queries. SSAM provides consistent responses, demonstrating stable performance on multimodal queries despite the specialist models not being trained on paired video-audio-language data.
Figure S3 : Qualitative results on the AVQA [ 45 ] dataset with video and audio as input modalities paired with natural-language queries. SSAM effectively combines information from both audio and video to produce more accurate and consistent responses, demonstrating stable performance on multimodal queries compared to individual specialist models.
More in this topic

Multimodal Large Language Models

Multimodal large language models combine language models with image, video, audio, and other inputs for grounded understanding and generation.

See all 1,069 papers

Explore similar work

CardsList
Apr 18, 2026

PivotMerge: Bridging Heterogeneous Multimodal Pre-training via Post-Alignment Model Merging

Multimodal Large Language Models (MLLMs) rely on multimodal pre-training over diverse data sources, where different datasets often induce complementary cross-modal alignment capabilities. Model merging provides a cost-effective mechanism for integrating multiple expert MLLMs with complementary strengths into a unified model. However, existing model merging research mainly focuses on post-finetuning scenarios, leaving the pre-training stage largely unexplored. We argue that the core of MLLM pre-training lies in establishing effective cross-modal alignment, which bridges visual and textual representations into a unified semantic space. Motivated by this insight, we introduce the post-alignment merging task, which aims to integrate cross-modal alignment capabilities learned from heterogeneous multimodal pre-training. This setting introduces two key challenges: cross-domain parameter interference, where parameter updates learned from different data distributions conflict during merging, and layer-wise alignment contribution disparity, where different layers and projectors contribute unevenly to cross-modal alignment. To address them, we propose PivotMerge, a post-alignment merging framework for cross-modal projectors. PivotMerge incorporates two key components: Shared-space Decomposition and Filtering, which disentangles shared alignment patterns from domain-specific variations and suppresses conflicting directions, and Alignment-guided Layer-wise Merging, which assigns layer-specific merging weights based on differing alignment contributions. We construct systematic CC12M-based post-alignment merging scenarios for evaluation. Extensive experiments on multiple multimodal benchmarks show that PivotMerge consistently outperforms existing baselines, demonstrating its effectiveness and generalization ability.

Zibo Shao, Baochen Xiong, Xiaoshan Yang +4

State Key Laboratory of Multimodal Artificial Intelligence Systems, Institute of Automation, Chinese Academy of Sciences, Beijing 100190, China · Pengcheng Laboratory, Shenzhen 518066, China · School of Artificial Intelligence, University of Chinese Academy Sciences, Beijing 100049, China +1

Cross-Modal LearningCross-Modal AlignmentMultimodal PretrainingMultimodal Large Language Models
AskPDF
Mar 15, 2026

ES-Merging: Biological MLLM Merging via Embedding Space Signals

Biological multimodal large language models (MLLMs) have emerged as powerful foundation models for scientific discovery. However, existing models are specialized to a single modality, limiting their ability to solve inherently cross-modal scientific problems. While model merging is an efficient method to combine the different modalities into a unified MLLM, existing methods rely on input-agnostic parameter space heuristics that fail to faithfully capture modality specialization. To overcome this limitation, we propose the Embedding-Signal-based MLLM Merging (ES-Merging), a framework that estimates merging coefficients from embedding space signals, moving the merging paradigm from the parameter signals to the embedding signals. ES-Merging exploits coarse-grained and fine-grained signals from embedding space to estimate the layer-wise and element-wise merging coefficients, respectively, which are jointly combined for complementary coefficient estimation. Through extensive experiments, we demonstrate that ES-Merging outperforms existing merging methods not only on the cross-modal reasoning but also on the single-modal knowledge preservation, establishing that embedding space signals provide a principled and effective foundation for MLLM merging.

Wonbin Lee, Dongki Kim, Sung Ju Hwang

KAIST

Language Model MergingMultimodal Large Language Models
AskPDF
Oct 3, 2026

Strong Helps Weak: Directional Cross-Modal Alignment Transfer in Multi-modal LLMs

Multi-modal large language models (MLLMs) achieve strong modality understanding by pairing a large language model (LLM) with an encoder for a target modality such as vision, video, or audio. However, improving an MLLM's capability for a given modality typically requires additional training on large modality-specific datasets, incurring substantial data collection and compute costs. Model merging offers an alternative, but it is often infeasible for data-scarce, large per-sample size, or domain-specific modalities (e.g., audio and video), where same-modality model variants are rarely available. In this work, we characterize an intriguing asymmetric phenomenon: merging a well-aligned, data-rich source-modality MLLM into a data-scarce target-modality MLLM substantially improves the target on its own benchmarks. Our theoretical and empirical analyses show that this gain stems from enhanced alignment between modality-specific and textual tokens, induced by the stronger donor modality. Specifically, we derive a mutual-information lower bound that is monotonic in alignment-related quantities and strongly correlated with downstream MLLM performance. Building on this principle, we propose Directional Cross-modal Alignment Transfer (DCAT), a novel framework that transfers textual alignment from a strong, well-aligned source (donor) modality to a weak target (recipient) modality, boosting target-modality performance without further fine-tuning. We further show that the alignment-enhancing objective admits a closed-form weight-space solution computed from only a small calibration set. DCAT outperforms existing model-merging methods, offering an efficient path toward cross-modal alignment transfer. Project page with code is available at https://seohoiki3215.github.io/DCAT_project_page

Hoigi Seo, Byung Hyun Lee, Minjun Kim +3

Dept. of ECE · INMC & AIIS · IPAI, Seoul National University, Republic of Korea

Cross-Modal AlignmentMultimodal Large Language ModelsCross-Modal Representation TransferModel Merging
AskPDF