Strong Helps Weak: Directional Cross-Modal Alignment Transfer in Multi-modal LLMs

Hoigi Seo; Byung Hyun Lee; Minjun Kim; Dohyun Mah; Jongho Lee; Se Young Chun

cs.AIOct 3, 2026

Strong Helps Weak: Directional Cross-Modal Alignment Transfer in Multi-modal LLMs

Authors: Hoigi Seo, Byung Hyun Lee, Minjun Kim, Dohyun Mah, Jongho Lee, Se Young Chun

Organizations: Dept. of ECE · INMC & AIIS · IPAI, Seoul National University, Republic of Korea

Abstract

Multi-modal large language models (MLLMs) achieve strong modality understanding by pairing a large language model (LLM) with an encoder for a target modality such as vision, video, or audio. However, improving an MLLM's capability for a given modality typically requires additional training on large modality-specific datasets, incurring substantial data collection and compute costs. Model merging offers an alternative, but it is often infeasible for data-scarce, large per-sample size, or domain-specific modalities (e.g., audio and video), where same-modality model variants are rarely available. In this work, we characterize an intriguing asymmetric phenomenon: merging a well-aligned, data-rich source-modality MLLM into a data-scarce target-modality MLLM substantially improves the target on its own benchmarks. Our theoretical and empirical analyses show that this gain stems from enhanced alignment between modality-specific and textual tokens, induced by the stronger donor modality. Specifically, we derive a mutual-information lower bound that is monotonic in alignment-related quantities and strongly correlated with downstream MLLM performance. Building on this principle, we propose Directional Cross-modal Alignment Transfer (DCAT), a novel framework that transfers textual alignment from a strong, well-aligned source (donor) modality to a weak target (recipient) modality, boosting target-modality performance without further fine-tuning. We further show that the alignment-enhancing objective admits a closed-form weight-space solution computed from only a small calibration set. DCAT outperforms existing model-merging methods, offering an efficient path toward cross-modal alignment transfer. Project page with code is available at https://seohoiki3215.github.io/DCAT_project_page

Figures & tables

Figure 1: Conceptual diagram of DCAT. DCAT transfers alignment from a strong donor modality model to a relatively weak recipient modality model, thereby improving the recipient model’s modality-specific performance.

	Music	Sound	Speech	Overall	Align γ
Audio only	54.79	60.66	31.23	48.90	0.0091
A+V (TA)	55.69	64.56	39.34	53.20	0.0103
A+V (ISO)	58.68	66.67	34.83	53.40	0.0950
A+V (TSV)	55.99	66.67	35.44	52.70	0.0100
A+V (Opt.)	59.58	64.26	35.44	53.10	0.0099

Table 1: Quantitative comparison between modality-specific models and merged models. We report (Left) the MMAU Tyagi et al. (2025) performance obtained by merging an audio-only model with an audio-vision merged model (A+V), and (Right) the Video-MME Fu et al. (2025) performance obtained by merging a video-only model with a video-vision merged model (Vd+V). Results are reported for Task Arithmetic (TA), ISO-C (ISO), TSV, and OptMerge (Opt.). In all cases, merging is applied only to the LLM and the Alignment score was measured at the 21st layer of the Vicuna model. We set

\gamma=0.8

Figure 2: Empirical analysis of our alignment metric and performance. We analyze the correlation between the Alignment score and modality performance across a total of $N=114$ model checkpoints. We use MMAU Tyagi et al. (2025) for audio, Video-MME Fu et al. (2025) for video, and TextVQA Singh et al. (2019) for vision performance. Checkpoints are obtained from a range of merging methods Ilharco et al. (2023) ; Yadav et al. (2023) ; Marczak et al. (2025) ; Gargiulo et al. (2025) ; Cheng et al. (2025) ; Wei et al. (2025) , combined with several variations of the merged weights. The red dotted line refers to the performance of original model. For all experiments, we chose $k=32$ and $\gamma=0.8$ , and measure performance on 300 questions per benchmark across categories. Text embeddings are extracted from the questions.

Figure 3: Layer-wise Pearson correlation between Alignment score and performance across MLLM combinations and modalities. We measure alignment in a layer-wise manner for each modality across multiple modality pairs. A consistently strong positive correlation between the Alignment score and performance emerges from the middle layers onward.

		MMAU				AIR-Bench				MuCho	Clotho	Vocal	Average
		Music	Sound	Speech	Overall	Music	Sound	Speech	Overall	MuCho	Clotho	Vocal	Average
Vicuna-based	Original	54.79	60.66	31.23	48.90	38.44	46.76	44.62	43.17	26.96	61.23	48.98	45.85
	TA Ilharco et al. (2023)	55.69	64.56	39.34	53.20	45.64	57.48	47.03	49.09	50.04	81.00	44.81	55.63
	TIES Yadav et al. (2023)	54.49	62.46	36.04	51.00	44.86	55.92	47.51	48.68	38.42	74.97	50.93	52.80
	ISO Marczak et al. (2025)	58.68	66.67	34.83	53.40	45.82	60.77	46.72	49.79	52.82	78.57	44.31	55.78
	TSV Gargiulo et al. (2025)	55.99	66.67	35.44	52.70	47.03	58.51	48.69	50.51	50.80	81.21	47.54	56.55

Table 2: Quantitative comparison of DCAT. We evaluate DCAT against various model merging methods across benchmarks, using a vision model as donor and an audio/video model as recipient. (Top) In Audio+Vision, DCAT transfers vision-model alignment to the audio model, achieving gains up to 100.92% (MuCho-Music) over the original. (Bottom) Video+Vision shows a similar trend, with improvements up to 53.88% (MLVU). ( ∗ Since our LLaMA 3.1-based audio model does not tune the language backbone, several merging methods reduce to TA, yielding identical outputs.)

Figure 4: SEO, SD and token norm ratio before and after applying DCAT. We compare SEO, SD, and the norm ratio of modality tokens before and after applying DCAT. As depicted in first and second column, SEO and SD increase to a statistically significant degree ( $p<0.05$ ), while the relative norm remains close to 1.0, confirming that our DCAT formulation behaves as intended.

ID	Study subject	MMAU				Video-MME
ID	Study subject	Music	Sound	Speech	Overall	Short	Medium	Long	Overall
A0	Original	54.79	60.66	31.23	48.90	38.33	33.67	30.67	34.22
A1	Base (DCAT)	60.22	69.06	41.74	56.20	46.33	39.67	34.33	40.11
A2	w/o SEO encouraging term in $\mathcal{L}_{\mathrm{self}}(\tau)$	57.49	66.97	34.83	53.10	45.33	34.00	30.33	36.56
A3	w/o SD encouraging term in $\mathcal{L}_{\mathrm{self}}(\tau)$	55.09	64.26	34.53	51.30	35.67	25.00	25.00	28.56
A4	w/o $\mathcal{L}_{\mathrm{cross}}(\tau)$	25.45	27.63	24.62	25.90	3.33	4.67	2.67	3.56

Table 3: Ablation study on DCAT. We ablate the individual components of DCAT. (A2–A5) isolate the contribution of each term in the DCAT objective by either dropping a term or fixing the

\alpha

scheduler, while (A6–A9) examine the sensitivity of DCAT to its key hyperparameters.

Appendix figures & tables19 assets

Supplementary material from the paper’s appendix.

Appendix

Audio+Vision			Video+Vision
$\mathrm{SD}$	$\mathrm{SD}_{\det}$	$\delta_{\mathrm{eff}}\leq$	$\mathrm{SD}$	$\mathrm{SD}_{\det}$	$\delta_{\mathrm{eff}}\leq$
0.2839 $\pm 0.0273$	0.2752 $\pm 0.0228$	0.9704 $\pm 0.0224$	0.2015 $\pm 0.0178$	0.2226 $\pm 0.0197$	1.1046 $\pm 0.0140$

Appendix

Table A1 : Empirical validation on

\mathrm{SD}

\mathrm{SD}_{\det}

, and

\delta_{\mathrm{eff}}

. To assess the tightness of the bound, we compute the entropy-based spectral diversity

\mathrm{SD}

, the proof-side spectral diversity

\mathrm{SD}_{\det}

, and spectral conversion factor

\delta_{\mathrm{eff}}

for Vicuna-based models in Fig. 2 . The results show that

\delta_{\mathrm{eff}}

remains close to one, indicating that the entropy-based effective-rank SD introduces only a mild constant-level relaxation in the MI lower bound.

Component	Vicuna+Audio	Vicuna+Video	LLaMA+Audio	LLaMA+Video
Backbone	Vicuna-7B-v1.5	Vicuna-7B-v1.5	LLaMA-3.1-8B-Instruct	LLaMA-3.1-8B-Instruct
Calibration source	AVQA (audio track)	AVQA (video track)	AVQA (audio track)	AVQA (video track)
# calibration samples	256	256	256	256
$k$ (text subspace dim)	128	128	256	256
$\beta$	0.30	0.30	0.95	0.95
$\eta$	$1{\times}10^{5}$	$1{\times}10^{5}$	$1{\times}10^{6}$	$1{\times}10^{6}$

Appendix

Table A2 : Full implementation configuration of DCAT for the four headline settings. A blank entry under Evaluation seed indicates that the benchmark is small enough to be evaluated in full and therefore does not require subsampling.

Figure A1 : Layer-wise Pearson correlation between Alignment score and modality performance on Vicuna backbone. We measure the Alignment score in a layer-wise manner for vision, audio, and video benchmark on Audio+Vision and Video+Vision Vicuna backbone. Across all cases, the correlation between Alignment score and accuracy is strongest in the middle layers.

Figure A2 : Layer-wise Pearson correlation between Alignment score and modality performance on LLaMA-3.1. We measure the Alignment score in a layer-wise manner for vision, audio, and video on Audio+Vision and Video+Vision LLaMa backbone. Across all cases, the correlation between the Alignment score and accuracy is strongest in the middle layers.

Figure A3 : Correlation between Alignment score and modality performance on the 21st layer of Vicuna backbone. We analyze the correlation between the Alignment score and modality performance across checkpoints obtained from merging methods Ilharco et al. (2023) ; Yadav et al. (2023) ; Marczak et al. (2025) ; Gargiulo et al. (2025) ; Cheng et al. (2025) ; Wei et al. (2025) , and their SVD-derived variants. The red dotted line refers to the performance of original model. Across all results, the Alignment score shows a positive correlation with modality performance under both Pearson and Spearman measures.

Figure A4 : Correlation between Alignment score and modality performance on the 21st layer of LLaMA-3.1 backbone. We analyze the correlation between the Alignment score and modality performance across checkpoints obtained from merging methods Ilharco et al. (2023) ; Yadav et al. (2023) ; Marczak et al. (2025) ; Gargiulo et al. (2025) ; Cheng et al. (2025) ; Wei et al. (2025) , and their SVD-derived variants. The red dotted line refers to the performance of original model. Across all results, the Alignment score shows a positive correlation with modality performance under both Pearson and Spearman measures.

Figure A5 : Layer-wise SEO and SD of individual transformer submodules before and after applying DCAT on the Audio+Vision Vicuna. We compare SEO and SD for the MLP projections ( $W_{\mathrm{gate}}$ , $W_{\mathrm{down}}$ , and $W_{\mathrm{up}}$ ) and the attention projections ( $W_{k}$ , $W_{q}$ , $W_{v}$ , and $W_{o}$ ) across all Vicuna layers. In all examined components, applying DCAT consistently increases both SEO and SD.

Figure A6 : Layer-wise SEO and SD of individual transformer submodules before and after applying DCAT on the Video+Vision Vicuna. We compare SEO and SD for the MLP projections ( $W_{\mathrm{gate}}$ , $W_{\mathrm{down}}$ , and $W_{\mathrm{up}}$ ) and the attention projections ( $W_{k}$ , $W_{q}$ , $W_{v}$ , and $W_{o}$ ) across all Vicuna layers. In all examined components, applying DCAT consistently increases both SEO and SD.

Figure A7 : Layer-wise effective rank and scheduling of

\alpha

. DCAT schedules

\alpha

inversely with the text effective rank, as small text effective rank with large

\alpha

could introduce noisy alignment between text and modality-token activations.

		MMAU		AIR-Bench		MuCho		Clotho		Vocal
		$\Delta$ (%p)	$p$ -value	$\Delta$ (%p)	$p$ -value	$\Delta$ (%p)	$p$ -value	$\Delta$ (%p)	$p$ -value	$\Delta$ (%p)	$p$ -value
Vicuna-based	TA	+4.30	1.71e-02	+5.91	3.62e-61	+23.08	8.57e-29	+19.76	4.14e-36	-4.18	1.75e-07
	TIES	+2.10	1.78e-01	+5.50	3.74e-78	+11.46	7.79e-17	+13.73	1.83e-24	+1.95	1.55e-02
	ISO	+4.50	5.57e-03	+6.62	3.53e-88	+25.86	1.64e-41	+17.34	2.99e-33	-4.68	2.86e-13
	TSV	+3.80	2.80e-02	+7.34	3.15e-112	+23.84	1.65e-33	+19.97	4.21e-37	-1.45	4.50e-02
	WUDI	+4.80	6.74e-03	+6.06	5.07e-64	+22.49	1.35e-27	+19.49	1.09e-34	-2.01	9.10e-03

Appendix

Table A3 :

p

-value of DCAT on audio+vision

		Video-MME		TempCompass		MVBench		MLVU
		$\Delta$ (%p)	$p$ -value	$\Delta$ (%p)	$p$ -value	$\Delta$ (%p)	$p$ -value	$\Delta$ (%p)	$p$ -value
Vicuna-based	TA	+2.11	7.31e-02	-0.07	9.47e-01	+5.20	1.30e-07	+12.37	3.29e-20
	TIES	-6.70	7.38e-16	-0.83	1.81e-01	-5.70	2.99e-17	+0.41	6.36e-01
	ISO	+3.44	8.96e-04	-0.07	9.43e-01	+4.53	1.74e-06	+13.06	2.53e-21
	TSV	+4.04	1.33e-04	+1.79	1.13e-02	+6.10	2.25e-10	+12.97	1.32e-28
	WUDI	+1.56	1.81e-01	-0.52	5.39e-01	+4.83	8.67e-07	+12.05	1.21e-19

Appendix

Table A4 :

p

-value of DCAT on video+vision

		MVBench
		Action Antonym	Action Count	Action Localization	Action Prediction	Action Sequence	Character Order	Counterfactual Inference	Egocentric Navigation	Episodic Reasoning	Fine Grained Action
Vicuna-based	Original	51.50	38.00	27.50	29.00	35.00	47.00	34.00	27.00	32.00	40.00
	TA	62.00	32.50	36.00	46.00	48.50	45.00	34.00	34.00	40.50	38.00
	TIES	55.50	16.50	26.00	24.50	32.00	28.50	31.50	25.00	22.50	30.50
	ISO	54.50	38.00	28.00	49.00	46.00	46.50	41.00	34.00	37.50	41.00
	TSV	60.50	36.50	31.00	48.00	50.00	46.00	38.50	34.00	41.00	40.50

Appendix

Table A5 : Detailed results for MVBench benchmark. We report the specific performance measured from subcategories on MVBench.

		MLVU								TempCompass - Multi Choice
		Count	Topic Reasoning	Ego	Order	PlotQA	Needle	Anomaly Reco.	Average	Action	Attr.	Dir.	Order	Speed	Overall
Vicuna-based	Original	27.18	57.79	28.41	23.17	30.80	33.80	27.00	32.59	70.41	35.07	33.13	37.75	32.81	42.28
	TA	34.47	65.40	34.66	23.17	50.09	45.07	61.00	44.84	82.84	45.14	37.01	38.74	35.33	48.29
	TIES	25.73	58.56	27.27	23.17	32.47	31.83	33.00	33.15	63.61	33.33	32.54	35.76	30.60	39.56
	ISO	30.10	67.30	42.33	22.39	49.35	44.23	61.50	45.31	81.66	46.18	35.52	39.40	35.65	48.10
	TSV	33.98	70.72	42.90	23.94	48.98	42.25	53.50	45.18	86.69	47.57	36.42	42.38	35.02	50.06

Appendix

Table A6 : Detailed results for MLVU, TempCompass benchmark and other benchmarks. We report the specific performance measured from subcategories on MLVU, TempCompass(Multi Choice, Caption Matching, Yes/No), and other two benchmarks(EgoSchema, LongVideoBench).

	GQA		MMBench		MMStar		POPE		Text-VQA		Average
Original	61.91		63.83		35.44		84.31		47.76		58.65
Merging Method	A+V	Vd+V	A+V	Vd+V	A+V	Vd+V	A+V	Vd+V	A+V	Vd+V	A+V	Vd+V
TA	61.76	58.08	62.97	60.40	35.62	34.69	85.34	84.83	49.10	45.39	58.96	56.68
TIES	43.89	53.02	57.99	59.36	33.98	33.61	79.56	85.06	42.99	46.26	51.68	55.46
ISO	58.75	57.62	61.43	60.74	34.07	32.48	85.31	82.51	48.15	46.89	57.54	56.05
TSV	60.92	60.75	63.83	63.66	34.26	33.58	86.02	85.58	49.86	47.87	58.98	58.29

Appendix

Table A7 : Results on vision benchmarks (Vicuna-based). The performance of Vicuna based audio-vision(A+V) and video-vision(Vd+V) models are measured from five different benchmarks: GQA, MMBench-Dev(EN), MMStar, POPE, TextVQA-Val.

Model	Music	Sound	Speech	Overall
Original	54.79	60.66	31.23	48.90
DCAT	57.49	66.07	37.24	53.60

Appendix

Table A8 : Audio-video cross-modal transfer results. We report DCAT performance when transferring alignment between audio and video models. The left table evaluates video-to-audio transfer on MMAU Tyagi et al. (2025) , while the right table evaluates audio-to-video transfer on Video-MME Fu et al. (2025) .

Method	MMAU				Video-MME
Method	Music	Sound	Speech	Overall	Short	Medium	Long	Overall
Original	54.8	60.7	31.2	48.9	39.9	34.2	30.9	35.0
LoRA fine-tuned	53.6	62.2	33.0	49.6	44.2	35.7	31.3	37.1
RegMean Jin et al. (2023)	58.5	66.5	37.9	54.3	45.8	38.4	34.8	39.7
RegMean++ Huu-Tien et al. (2025)	58.8	66.5	35.5	53.6	46.9	39.3	34.6	40.3
DCAT (ours)	60.2	69.1	41.7	56.2	48.0	40.5	36.2	41.6

Appendix

Table A9 : Calibration-matched comparison. We compare DCAT with LoRA fine-tuning, RegMean Jin et al. (2023) , and RegMean++ Huu-Tien et al. (2025) under a matched calibration budget of 256 AVQA samples. MMAU is evaluated with Audio+Vision Vicuna-based models, and Video-MME with Video+Vision Vicuna-based models. LoRA is trained with rank 256, batch size 8, and 128 steps. The best results are in bold , and the second-best are underlined .

Seed	MMAU				Video-MME
Seed	Music	Sound	Speech	Overall	Short	Medium	Long	Overall
0	59.38	68.57	40.04	56.00	48.64	40.53	35.53	41.57
1	59.08	68.27	39.74	55.70	48.76	39.98	35.53	41.42
2	60.88	67.67	39.74	56.10	48.09	40.31	35.76	41.39
3	59.68	66.47	40.34	55.50	47.64	41.09	35.31	41.35
4	57.89	67.67	40.34	55.30	48.76	40.98	35.20	41.64

Appendix

Table A10 : Robustness to calibration sampling. We randomly sample 256 AVQA training examples with 10 different random seeds and apply DCAT to the Audio+Vision (MMAU) and Video+Vision (Video-MME) settings. We report per-seed results, together with the mean and standard deviation over the 10 calibration subsets.

Task	Audio+Vision			Video+Vision
Task	Before	After	$\Delta$	Before	After	$\Delta$
MMLU	0.492	0.491	$-$ 0.001	0.501	0.510	$+$ 0.009
ARC	0.550	0.514	$-$ 0.036	0.506	0.521	$+$ 0.015
HellaSwag	0.763	0.745	$-$ 0.018	0.777	0.775	$-$ 0.002
WinoGrande	0.731	0.710	$-$ 0.021	0.722	0.722	0.000
PIQA	0.787	0.784	$-$ 0.003	0.778	0.786	$+$ 0.008

Appendix

Table A11 : Language ability before and after DCAT. We evaluate general language ability on 13 text benchmarks for the Vicuna-based Audio+Vision and Video+Vision models before and after applying DCAT.

\Delta

denotes After

-

Before. McNemar’s test shows no significant change for Audio+Vision (

p=0.506

) and a significant improvement for Video+Vision (

p\ll 0.01

Study subject	MMAU				Video-MME
Study subject	Music	Sound	Speech	Overall	Short	Medium	Long	Overall
Original	54.79	60.66	31.23	48.90	38.33	33.67	30.67	34.22
Base (DCAT)	60.22	69.06	41.74	56.20	46.33	39.67	34.33	40.11
$\beta=0.1$ in Eq. 6	50.90	60.66	34.53	48.70	37.33	29.00	26.33	30.89
$\beta=0.2$ in Eq. 6	61.08	66.97	39.64	55.10	49.33	36.67	34.67	40.22
$\beta=0.4$ in Eq. 6	60.18	67.27	39.34	54.80	49.33	37.00	37.67	41.33

Appendix

Table A12 : Hyperparameter study on DCAT. We conduct a hyperparameter study for DCAT. We evaluate a range of options for

\beta

\eta

k

, and the calibration sample size, and provide the rationale behind our hyperparameter choices.

Cross-Modal Alignment

Cross-modal alignment maps representations from different data types into compatible spaces so models can compare, transfer, and combine information.

See all 433 papers

Multimodal LLMs under Pairwise Modalities
May 20, 2026Yan Li, Yunlong Deng, Yuewen Sun +3Cross-Modal Learning Representation Identifiability
Mohamed bin Zayed University of Artificial Intelligence · Carnegie Mellon University
SSAM: Singular Subspace Alignment for Merging Multimodal Large Language Models
Mar 23, 2026Md Kaykobad Reza, Ameya Patil, Edward Ayrapetian +1Multimodal Large Language Models Language Model Merging
University of California Riverside · Amazon
PivotMerge: Bridging Heterogeneous Multimodal Pre-training via Post-Alignment Model Merging
Apr 18, 2026Zibo Shao, Baochen Xiong, Xiaoshan Yang +4Cross-Modal Learning Cross-Modal Alignment
State Key Laboratory of Multimodal Artificial Intelligence Systems, Institute of Automation, Chinese Academy of Sciences, Beijing 100190, China · Pengcheng Laboratory, Shenzhen 518066, China · School of Artificial Intelligence, University of Chinese Academy Sciences, Beijing 100049, China +1

May 20, 2026J/K move · Enter open · S save

Multimodal LLMs under Pairwise Modalities

Yan Li, Yunlong Deng, Yuewen Sun, Gongxu Luo+2

Mohamed bin Zayed University of Artificial Intelligence · Carnegie Mellon University

Cross-Modal Learning Representation Identifiability Cross-Modal Alignment Multimodal Large Language Models Multimodal Understanding Multimodal Generation+3

Open paper AskPDF arXiv

Despite the impressive results achieved by multimodal large language models (MLLMs), their training typically relies on jointly curated multimodal data, requiring substantial human effort to construct multi-way aligned datasets and thereby limiting scalability across domains. In this work, we explore training MLLMs by only leveraging multiple paired modalities as a surrogate for the full joint multimodal distribution. Specifically, we first provide a theoretical analysis of the conditions under which the representations are identifiable with only observing pairwise modalities. Building on this analysis, we propose a representation learning framework for aligning latent representations across modalities using only pairwise data. The framework consists of two stages: latent representation alignment and cross-modal recomposition. Specifically, in the first stage, we learn the shared latent space across modalities by both self-modal reconstruction and pair-wise contrastive learning. We also incorporate an inductive bias in the contrastive learning process by partially aligning and minimal latent specification. In stage two, we integrate the encoder of newly introduced modalities with the decoders of the pre-trained modalities to facilitate cross-modal transfer and generation. We evaluate our method by newly adding 3D point clouds and tactile modalities into pre-trained MLLMs with three modality pairs and show that, by learning an aligned latent representation space, our model achieves strong cross-modal performance.

Abstract

Cross-Modal Alignment

Explore similar work

Multimodal LLMs under Pairwise Modalities

SSAM: Singular Subspace Alignment for Merging Multimodal Large Language Models

PivotMerge: Bridging Heterogeneous Multimodal Pre-training via Post-Alignment Model Merging