SSAM: Singular Subspace Alignment for Merging Multimodal Large Language Models

Md Kaykobad Reza; Ameya Patil; Edward Ayrapetian; M. Salman Asif

cs.LGMar 23, 2026

SSAM: Singular Subspace Alignment for Merging Multimodal Large Language Models

Authors: Md Kaykobad Reza, Ameya Patil, Edward Ayrapetian, M. Salman Asif

Organizations: University of California Riverside · Amazon

Abstract

Multimodal large language models (MLLMs) achieve strong performance by jointly processing inputs from multiple modalities, such as vision, audio, and language. However, building such models or extending them to new modalities often requires large paired datasets and substantial computational resources. Since many pretrained MLLMs (e.g., vision-language or audio-language) are publicly available, we ask whether we can merge them into a single MLLM that can handle multiple modalities? Merging MLLMs with different input modalities remains challenging, partly because of differences in the learned representations and interference between their parameter spaces. To address these challenges, we propose Singular Subspace Alignment and Merging (SSAM), a training-free model merging framework that unifies independently trained specialist MLLMs into a single model capable of handling any combination of input modalities. SSAM maintains modality-specific parameter updates separately and identifies a shared low-rank subspace for language-related parameter updates, aligns them within this subspace, and merges them to preserve complementary knowledge while minimizing parameter interference. Without using any multimodal training data, SSAM achieves state-of-the-art performance across four datasets, surpassing prior training-free merging methods and even jointly trained multimodal models. These results demonstrate that aligning models in parameter space provides a scalable and resource-efficient alternative to conventional joint multimodal training.

Figures & tables

Figure 1: Comparison of multimodal learning paradigms. (a) Conventional methods train a single MLLM on paired multimodal datasets. (b) ImageBind-LLM [ 12 ] aligns all modalities to shared image embeddings. (c) SSAM offers a training-free method to merge independently trained specialist MLLMs into a single, unified model via subspace projection.

Figure 2: Overview of the proposed SSAM method. (a) Subspace projection: each language vector $\Delta_{i}$ is projected onto a shared low-rank subspace prior to merging. (b) SSAM constructs covariance matrices $A$ and $B$ from all $\Delta_{i}$ , extracts the shared left and right bases $U_{c}$ and $V_{c}$ , projects language vectors, and merge them into $\Delta_{\text{merged}}$ .

MUSIC-AVQA	Models finetuned on paired multimodal data
Dataset	Method	V+L	V+I+L	V+A+L	V+I+A+L	Average
	ChatBridge-13B [ 51 ]	-	-	43.00	-	43.00
	OneLLM-7B [ 11 ]	-	-	47.60	-	47.60
	ImageBind-LLM [ 12 ]	37.24	38.76	39.72	38.16	38.47
	X-InstructBLIP [ 27 ]	45.83	41.23	48.34	47.39	45.70
	Proj-Only [ 3 ]	44.93	46.64	46.17	50.21	46.99

Table 1: Performance comparison on MUSIC-AVQA [ 17 ] and AVQA [ 45 ] datasets. SSAM achieves the best average performance across all modality combinations and consistently outperforms both training-free model merging baselines and models finetuned on paired multimodal data.

Models finetuned on paired multimodal data
Method	MCUB-3					MCUB-4
Method	I+V+P+L	A+V+P+L	I+A+P+L	I+A+V+L	Average	I+A+V+P+L
ImageBind-LLM [ 12 ]	31.80	31.40	33.40	35.20	32.95	32.93
X-InstructBLIP [ 27 ]	29.40	25.20	21.20	41.40	29.30	27.94
Proj-Only [ 3 ]	42.60	43.80	42.80	47.40	44.15	43.00
Training-free model merging methods

Table 2: Performance comparison on MCUB benchmark [ 3 ] . SSAM achieves the best average performance across all modality combinations and consistently outperforms both training-free merging baselines and models finetuned on paired multimodal data.

Figure 3: Qualitative examples on commonality understanding on the MCUB-4 dataset.

Table 6

Figure 4: Qualitative examples on the AVQA dataset.

Subspace	MCUB-3					MCUB-4
Rank (k)	I+V+P+L	A+V+P+L	I+A+P+L	I+A+V+L	Average	I+A+V+P+L
64	58.20	61.40	65.00	56.40	60.25	62.34
128	59.40	61.00	65.40	55.60	60.35	62.27
256	59.20	59.20	64.80	56.20	59.85	61.66
384	58.40	57.80	64.40	54.40	58.75	61.47
512	58.60	58.20	65.20	55.40	59.35	61.52

Table S1 : Performance of SSAM with different subspace ranks (

k

) on the MCUB-3 and MCUB-4 datasets. I, A, V, P, and L denote Image, Audio, Video, Point Cloud, and Language, respectively. Rank

k=128

empirically yields overall better performance.

Figure S1 : Qualitative results on the MCUB-4 [ 3 ] dataset with four input modalities (image, video, audio, and point cloud) paired with natural-language queries. SSAM demonstrates strong cross-modal reasoning and provides coherent answers in complex multimodal settings, even though none of the specialist models were trained on this modality combination.

Figure S2 : Qualitative results on the MUSIC-AVQA [ 17 ] and AVQA [ 45 ] datasets with video and audio as the input modalities paired with natural-language queries. SSAM provides consistent responses, demonstrating stable performance on multimodal queries despite the specialist models not being trained on paired video-audio-language data.

Figure S3 : Qualitative results on the AVQA [ 45 ] dataset with video and audio as input modalities paired with natural-language queries. SSAM effectively combines information from both audio and video to produce more accurate and consistent responses, demonstrating stable performance on multimodal queries compared to individual specialist models.

Multimodal Large Language Models

Multimodal large language models combine language models with image, video, audio, and other inputs for grounded understanding and generation.

See all 1,069 papers

PivotMerge: Bridging Heterogeneous Multimodal Pre-training via Post-Alignment Model Merging
Apr 18, 2026Zibo Shao, Baochen Xiong, Xiaoshan Yang +4Cross-Modal Learning Cross-Modal Alignment
State Key Laboratory of Multimodal Artificial Intelligence Systems, Institute of Automation, Chinese Academy of Sciences, Beijing 100190, China · Pengcheng Laboratory, Shenzhen 518066, China · School of Artificial Intelligence, University of Chinese Academy Sciences, Beijing 100049, China +1
ES-Merging: Biological MLLM Merging via Embedding Space Signals
Mar 15, 2026Wonbin Lee, Dongki Kim, Sung Ju HwangLanguage Model Merging Multimodal Large Language Models
KAIST
Strong Helps Weak: Directional Cross-Modal Alignment Transfer in Multi-modal LLMs
Oct 3, 2026Hoigi Seo, Byung Hyun Lee, Minjun Kim +3Cross-Modal Alignment Multimodal Large Language Models
Dept. of ECE · INMC & AIIS · IPAI, Seoul National University, Republic of Korea

Apr 18, 2026J/K move · Enter open · S save

PivotMerge: Bridging Heterogeneous Multimodal Pre-training via Post-Alignment Model Merging

Zibo Shao, Baochen Xiong, Xiaoshan Yang, Yaguang Song+3

State Key Laboratory of Multimodal Artificial Intelligence Systems, Institute of Automation, Chinese Academy of Sciences, Beijing 100190, China · Pengcheng Laboratory, Shenzhen 518066, China · School of Artificial Intelligence, University of Chinese Academy Sciences, Beijing 100049, China+1

Cross-Modal Learning Cross-Modal Alignment Multimodal Pretraining Multimodal Large Language Models

Open paper AskPDF arXiv

Multimodal Large Language Models (MLLMs) rely on multimodal pre-training over diverse data sources, where different datasets often induce complementary cross-modal alignment capabilities. Model merging provides a cost-effective mechanism for integrating multiple expert MLLMs with complementary strengths into a unified model. However, existing model merging research mainly focuses on post-finetuning scenarios, leaving the pre-training stage largely unexplored. We argue that the core of MLLM pre-training lies in establishing effective cross-modal alignment, which bridges visual and textual representations into a unified semantic space. Motivated by this insight, we introduce the post-alignment merging task, which aims to integrate cross-modal alignment capabilities learned from heterogeneous multimodal pre-training. This setting introduces two key challenges: cross-domain parameter interference, where parameter updates learned from different data distributions conflict during merging, and layer-wise alignment contribution disparity, where different layers and projectors contribute unevenly to cross-modal alignment. To address them, we propose PivotMerge, a post-alignment merging framework for cross-modal projectors. PivotMerge incorporates two key components: Shared-space Decomposition and Filtering, which disentangles shared alignment patterns from domain-specific variations and suppresses conflicting directions, and Alignment-guided Layer-wise Merging, which assigns layer-specific merging weights based on differing alignment contributions. We construct systematic CC12M-based post-alignment merging scenarios for evaluation. Extensive experiments on multiple multimodal benchmarks show that PivotMerge consistently outperforms existing baselines, demonstrating its effectiveness and generalization ability.

Abstract

Multimodal Large Language Models

Explore similar work

PivotMerge: Bridging Heterogeneous Multimodal Pre-training via Post-Alignment Model Merging

ES-Merging: Biological MLLM Merging via Embedding Space Signals

Strong Helps Weak: Directional Cross-Modal Alignment Transfer in Multi-modal LLMs