ES-Merging: Biological MLLM Merging via Embedding Space Signals

Wonbin Lee; Dongki Kim; Sung Ju Hwang

cs.LGMar 15, 2026

ES-Merging: Biological MLLM Merging via Embedding Space Signals

Authors: Wonbin Lee, Dongki Kim, Sung Ju Hwang

Organizations: KAIST

Abstract

Biological multimodal large language models (MLLMs) have emerged as powerful foundation models for scientific discovery. However, existing models are specialized to a single modality, limiting their ability to solve inherently cross-modal scientific problems. While model merging is an efficient method to combine the different modalities into a unified MLLM, existing methods rely on input-agnostic parameter space heuristics that fail to faithfully capture modality specialization. To overcome this limitation, we propose the Embedding-Signal-based MLLM Merging (ES-Merging), a framework that estimates merging coefficients from embedding space signals, moving the merging paradigm from the parameter signals to the embedding signals. ES-Merging exploits coarse-grained and fine-grained signals from embedding space to estimate the layer-wise and element-wise merging coefficients, respectively, which are jointly combined for complementary coefficient estimation. Through extensive experiments, we demonstrate that ES-Merging outperforms existing merging methods not only on the cross-modal reasoning but also on the single-modal knowledge preservation, establishing that embedding space signals provide a principled and effective foundation for MLLM merging.

Figures & tables

Panel of Figure 3 — Figure 3: Probe input template.

Figure 4: Overview of ES-Merging. (A) Layer-wise global merging coefficients are computed from the coarse-grained embedding signals, which are the layer-wise differences of distribution distances between mean pooled representations of the base LLM and a specialized MLLM. (B) Element-wise local merging coefficients are assigned based on the fine-grained embedding signals by computing the gradients from the embedding-wise distances.

Base LLM and Specialized MLLMs
	Molecule-Protein Interaction								Molecule-Cell Interaction
	BindingDB		BioSNAP		Human		Avg.		DrugComb		GDSC2		Avg.
	Acc.	F1	Acc.	F1	Acc.	F1	Acc.	F1	Acc.	F1	Acc.	F1	Acc.	F1
LLaMA-3.1-8B-Instruct	51.9	51.7	61.5	60.2	59.0	58.8	57.5	56.9	73.7	73.6	84.8	84.7	79.3	79.1
Mol-LLaMA [ 29 ]	55.8	54.7	66.4	64.0	61.5	61.4	61.2	60.0	64.1	64.1	55.5	55.5	59.8	59.8
Prot2Text-V2 [ 17 ]	59.2	59.2	55.3	54.1	47.2	47.2	53.9	53.5	65.6	65.6	59.7	57.9	62.6	61.7

Table 1: Performance comparison of ES-Merging with the base, specialized MLLMs, and merging methods on the instance-varying interaction prediction tasks. We report accuracy and macro-F1 across each subset. Bold and underline indicate the best and second best performances, respectively.

Base LLM and Specialized MLLMs
	CYP Inhibition												CYP Substrate
	CYP1A2		CYP2C19		CYP2C9		CYP2D6		CYP3A4		Avg.		CYP2C9		CYP2D6		CYP3A4		Avg.
	Acc.	F1	Acc.	F1	Acc.	F1	Acc.	F1	Acc.	F1	Acc.	F1	Acc.	F1	Acc.	F1	Acc.	F1	Acc.	F1
LLaMA-3.1-8B-Instruct	58.9	55.3	56.0	50.6	55.9	47.4	61.2	56.4	55.1	43.1	57.4	50.6	57.7	47.8	34.2	30.1	55.6	48.5	49.2	42.1
Mol-LLaMA	68.6	67.1	65.8	64.3	63.3	63.3	64.7	59.2	64.3	64.0	65.3	63.6	62.7	49.7	61.7	57.4	55.2	53.7	59.9	53.6
Prot2Text-V2	59.4	55.8	56.8	52.5	67.4	60.1	76.3	55.9	59.6	50.3	63.9	54.9	54.5	47.5	53.4	51.4	56.0	53.8	54.6	50.9

Table 2: Performance comparison of ES-Merging with the base, specialized MLLMs, and merging methods on the target-fixed functionality prediction tasks. We report accuracy and macro-F1 across each subset. Bold indicates the best and underline indicates the second best.

Table 6

Figure 5: Computed layer-wise merging coefficient visualization of each specialized MLLM. The $l$ -th column corresponds to the merging coefficient of the $l$ -th layer, $\alpha^{l}_{m_{j}}$ .

Appendix figures & tables13 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 7: Label distribution across datasets. Each panel shows positive and negative sample counts for Molecule-Protein Interaction, CYP Inhibition, CYP Substrate, and Molecule-Cell Interaction tasks.


Molecule Name: Paramethasone
SMILES: C[C@@H]1C[C@H]2[C@@H]3CC@@HF
Protein: Glucocorticoid receptor
Protein Sequence: MDSKESLTPGREENPSSVLAQERGDVMDFYKTLRGGATVKVSASSPSLAVASQSDSKQRRLLVDFPKGSVSNAQQPDLSKAVSLSMGLYMGETETKVMGNDLGFPQQGQISLSSGETDLKLLEESIANLNRSTSVPENPKSSASTAVSAAPTEKEFPKTHSDVSSEQQHLKGQTGTNGGNVKLYTTDQSTFDILQDLEFSSGSPGKETNESPWRSDLLIDENCLLSPLAGEDDSFLLEGNSNEDCKPLILPDTKPKIKDNGDLVLSSPSNVTLPQVKTEKEDFIELCTPGVIKQEKLGTVYCQASFPGANIIGNKMSAISVHGVSTSGGQMYHYDMNTASLSQQQDQKPIFNVIPPIPVGSENWNRCQGSGDDNLTSLGTLNFPGRTVFSNGYSSPSMRPDVSSPPSSSSTATTGPPPKLCLVCSDEASGCHYGVLTCGSCKVFFKRAVEGQHNYLCAGRNDCIIDKIRRKNCPACRYRKCLQAGMNLEARKTKKKIKGIQQATTGVSQETSENPGNKTIVPATLPQLTPTLVSLLEVIEPEVLYAGYDSSVPDSTWRIMTTLNMLGGRQVIAAVKWAKAIPGFRNLHLDDQMTLLQYSWMFLMAFALGWRSYRQSSANLLCFAPDLIINEQRMTLPCMYDQCKHMLYVSSELHRLQVSYEEYLCMKTLLLLSSVPKDGLKSQELFDEIRMTYIKELGKAIVKREGNSSQNWQRFYQLTKLLDSMHEVVENLLNYCFQTFLDKTMSIEFPEMLAEIITNQIPKYSNGNIKKLLFHQK
User: Determine whether the given molecule interacts with the protein by following the example.
Ground Truth: Interacts

Appendix

Table 7: Case study on Molecule-Protein Interaction prediction.


Drug Name: Gefitinib
SMILES: COC1=C(C=C2C(=C1)N=CN=C2NC3=CC(=C(C=C3)F)Cl)OCCCN4CCOCC4
Cell Line: OVCA420
Top 50 Genes: RPL35, FTL, MT-ND4, GAPDH, AKR1B10, UBC, RPS6, RPL24, EIF1, GNB2L1, SPP1, RPS11, AKR1C1, RPL19, GNG5, AKR1B10P1, ACTG1, MT-CO3, RPL5, FAU, RPS15A, TXN, RPL8, MIF, LDHB, KRT19, EIF4H, RPL11, ALDH1A1, MGST1, RPS12, CLIC1, SNRPD2, RPL30, H2AFX, NQO1, PRDX1, RPS5, RPLP1, H2AFZ, TUBB4B, FAUP1, NACA, RPL14, HIST1H2BK, MYL12B, SHFM1P1, HINT1P1, RPL27, HIST1H4C
User: Determine whether the given cell is sensitive or resistant to the given molecule based on the ranked gene-expression list.
Ground Truth: Sensitive

Appendix

Table 8: Case study on Molecule-Cell Interaction prediction.

Size	Human	GDSC2	CYP2C9 Inh.	CYP2C9 Sub.
32	60.7	90.6	68.2	53.7
256	62.0	93.1	68.9	48.5
1024	62.0	94.1	72.5	64.2

Appendix

Table 9: Additional studies on projection size of SWD for ES-Merging across Molecule-Protein and Molecule-Cell Interaction tasks.

	Molecule	Protein	Cell
Mol-LLaMA [ 29 ]	75.68	88.02	0.00
Prot2Text-V2 [ 17 ]	39.80	85.25	0.00
Cell-o1 [ 15 ]	60.89	77.81	77.84

Appendix

Table 10: Performance comparison of different modality-specialized LLMs, evaluating their knowledge and capabilities not only on tasks within their target modality but also across other modalities. We report accuracy across each subset.

	Molecule-Protein Interaction				Molecule-Cell Interaction			CYP Inhibition						CYP Substrate
Coefficient Type	BindingDB	BioSNAP	Human	Avg.	DrugComb	GDSC2	Avg.	CYP1A2	CYP2C19	CYP2C9	CYP2D6	CYP3A4	Avg.	CYP2C9	CYP2D6	CYP3A4	Avg.
Layer-wise	65.5	68.2	57.0	63.6	80.1	90.2	85.2	76.3	71.4	72.3	79.2	70.2	73.9	61.2	54.9	55.2	57.1
Element-wise	66.5	66.6	61.4	64.9	79.3	94.1	86.7	76.0	69.1	70.7	77.6	69.8	72.7	65.7	57.6	58.2	60.5
Layer $\times$ Element Mixed	66.0	69.1	62.0	65.7	80.7	94.1	87.4	77.4	70.6	72.5	80.7	71.3	74.5	64.2	60.9	60.5	61.9

Appendix

Table 11: Full results of the ablation studies on merging coefficient. We report accuracy across all datasets and their averages. Bold indicates the best and underline indicates the second best.

Figure 8: Performance with diverse temperatures $\tau$ for the layer-wise and element-wise coefficients in ES-Merging. Each cell reports the average accuracy (%) across CYP2D6 Inhibition, CYP2D6 Substrate, Human, and GDSC2.

Table 16

Size	Human	GDSC2	CYP2D6 Inh.	CYP2D6 Sub.
50	61.0	91.2	78.7	59.4
600	61.2	92.4	79.1	60.9
330 (Another subset)	61.8	93.9	79.6	60.5
330	62.0	94.1	80.7	60.9

Appendix

Table 14: Additional studies of probe dataset size and sampling for ES-Merging across Molecule-Protein and Molecule-Cell Interaction tasks.

Panel of Figure 9 — Figure 9: Embedding visualization of the last transformer block for each specialized LLM and the merged model with our method for (a) molecule, (b) protein, and (c) cell, (d) image, (e) video, and (f) audio tokens.

Panel of Table 16 — Table 16: In-context learning prompt template for the Molecule-Cell Interaction prediction task. Angle-bracketed tokens are replaced as follows: target molecule token <mol> and <molecule k> for each of the $k$ few-shot examples are substituted with the corresponding encoder embedding, while the gene tokens <gene> are replaced with the corresponding cell line string. The {label k} is replaced with the ground-truth label string for each example pair in the few-shot context.

Figure 10: Element-wise merging coefficients of the LoRA parameters for self_attn.q/k/v/o_proj modules at layers 0, 10, 20, and 30. Rows correspond to LoRA-A and LoRA-B, and columns correspond to Molecule, Protein, and Cell modalities.

Panel of Figure 11 — Figure 11: Element $\times$ Layer merging coefficients of the LoRA parameters for self_attn.q/k/v/o_proj modules at Layers 0, 10, 20, and 30, respectively.

Language Model Merging

Language model merging combines capabilities from multiple models or adapters by composing their weights, updates, or outputs.

See all 106 papers

SSAM: Singular Subspace Alignment for Merging Multimodal Large Language Models
Mar 23, 2026Md Kaykobad Reza, Ameya Patil, Edward Ayrapetian +1Multimodal Large Language Models Language Model Merging
University of California Riverside · Amazon
Model Merging in the Era of Large Language Models: Methods, Applications, and Future Directions
Mar 10, 2026Mingyang Song, Mao ZhengLanguage Model Merging Model Merging
Large Language Model Department, Tencent, China
PivotMerge: Bridging Heterogeneous Multimodal Pre-training via Post-Alignment Model Merging
Apr 18, 2026Zibo Shao, Baochen Xiong, Xiaoshan Yang +4Cross-Modal Learning Cross-Modal Alignment
State Key Laboratory of Multimodal Artificial Intelligence Systems, Institute of Automation, Chinese Academy of Sciences, Beijing 100190, China · Pengcheng Laboratory, Shenzhen 518066, China · School of Artificial Intelligence, University of Chinese Academy Sciences, Beijing 100049, China +1

Mar 23, 2026J/K move · Enter open · S save

SSAM: Singular Subspace Alignment for Merging Multimodal Large Language Models

Md Kaykobad Reza, Ameya Patil, Edward Ayrapetian, M. Salman Asif

University of California Riverside · Amazon

Multimodal Large Language Models Language Model Merging

Open paper AskPDF arXiv

Multimodal large language models (MLLMs) achieve strong performance by jointly processing inputs from multiple modalities, such as vision, audio, and language. However, building such models or extending them to new modalities often requires large paired datasets and substantial computational resources. Since many pretrained MLLMs (e.g., vision-language or audio-language) are publicly available, we ask whether we can merge them into a single MLLM that can handle multiple modalities? Merging MLLMs with different input modalities remains challenging, partly because of differences in the learned representations and interference between their parameter spaces. To address these challenges, we propose Singular Subspace Alignment and Merging (SSAM), a training-free model merging framework that unifies independently trained specialist MLLMs into a single model capable of handling any combination of input modalities. SSAM maintains modality-specific parameter updates separately and identifies a shared low-rank subspace for language-related parameter updates, aligns them within this subspace, and merges them to preserve complementary knowledge while minimizing parameter interference. Without using any multimodal training data, SSAM achieves state-of-the-art performance across four datasets, surpassing prior training-free merging methods and even jointly trained multimodal models. These results demonstrate that aligning models in parameter space provides a scalable and resource-efficient alternative to conventional joint multimodal training.

Abstract

Language Model Merging

Explore similar work

SSAM: Singular Subspace Alignment for Merging Multimodal Large Language Models

Model Merging in the Era of Large Language Models: Methods, Applications, and Future Directions

PivotMerge: Bridging Heterogeneous Multimodal Pre-training via Post-Alignment Model Merging