MSP: Modality Self-Play for Training Multimodal LLMs Without Paired Cross-Modal Data

Robert Moseley; Md Kaykobad Reza; Ameya Patil; Edward Ayrapetian; Salman Asif

cs.AIOct 14, 2026

MSP: Modality Self-Play for Training Multimodal LLMs Without Paired Cross-Modal Data

Authors: Robert Moseley, Md Kaykobad Reza, Ameya Patil, Edward Ayrapetian, Salman Asif

Organizations: University of California, Riverside · Amazon

Abstract

Large multimodal models have achieved remarkable progress by scaling model capacity and training on massive paired multimodal data. However, this paradigm introduces a fundamental bottleneck: aligned multimodal datasets are expensive to curate, difficult to scale across diverse modality combinations, and increasingly constrain further progress. We introduce Modality Self-Play (MSP), a training framework for multimodal foundation models based on proxy-mediated modality composition. Instead of relying on paired cross-modal data, MSP uses anchor-supporting proxy textual descriptions of unseen modalities as semantic bridges during training. Given an observed modality (e.g., image or audio), the model generates proxy descriptions for complementary unseen modalities, which are then encoded and aligned with the observed modality through learned bridges, enabling cross-modal interaction without real paired non-text supervision. Our central hypothesis is that proxy text, combined with contrastively trained pretrained encoders, provides sufficient semantic structure for multimodal composition to emerge. We show that MSP enables zero-shot modality composition: models trained only with a real anchor modality and accompanying proxy descriptions can integrate and reason over multiple modalities at inference time despite never observing real paired non-text data during training. Empirically, MSP achieves state-of-the-art or competitive performance on multiple multimodal benchmarks. Ablation studies further demonstrate that learned bridges improve alignment, while anchor-supporting proxy descriptions enable effective cross-modal composition. MSP removes the need for real paired non-text (e.g., audio-vision) supervision, while continuing to rely on modality-text pairs, pretrained contrastive encoders, and modality-distribution bridge calibration.

Figures & tables

Figure 1: MSP training pipeline. (1) Phase 1 trains one projector per modality using modality–text pairs only. (2) The frozen Phase-1 model then generates a text proxy for the modality it does not observe. (3) Phase 2 trains on one real modality together with the proxy for the missing one. No training sample ever contains two real non-text modalities. A component-level view of the same architecture, with the bridge formulation and the full token sequence, is given in Figure 2 .

Method	Parameters	Performance
ImageBind-LLM ( Han et al., 2023 )	7B	39.72
X-InstructBLIP ( Panagopoulou et al., 2023 )	7B	28.10
X-InstructBLIP ( Panagopoulou et al., 2023 )	13B	44.50
ChatBridge ( Zhao et al., 2023 )	13B	43.00
OneLLM ( Han et al., 2024 )	7B	47.60
Qwen2.5-VL ( Bai et al., 2025 )	7B	47.75

Table 1: Performance comparison across different multimodal models on the MUSIC-AVQA test split. Primarily, we wanted to compare against models that were allowed to use paired data but did not specifically finetune for performance on the listed benchmarks.

Method	Parameters	Performance
PSAC ( Li et al., 2019b )	-	78.60
ACRTransformer ( Zhang et al., 2020 )	-	81.70
HME ( Fan et al., 2019 )	-	81.80
LADNet ( Li et al., 2019a )	-	81.90
SSAM ( Reza et al., 2026 )	7B	82.12
HGA ( Jiang and Han, 2020 )	-	82.20

Table 2: Performance comparison across different multimodal models on the AVQA test split. Primarily, we wanted to compare against models that were allowed to use paired data but did not specifically finetune for performance on the listed benchmarks.

Method	Parameters	Performance
AnyGPT ( Zhan et al., 2024 )	7B	18.04
UnifiedIO2-large ( Lu et al., 2024 )	1.1B	27.06
Baichuan-Omni ( Li et al., 2024a )	7B	32.20
VITA ( Fu et al., 2024 )	8 $\times$ 7B	33.10
VITA-1.5 ( Fu et al., 2025 )	7B	33.40
UnifiedIO2-xxlarge ( Lu et al., 2024 )	6.8B	33.98

Table 3: Performance comparison across different multimodal models on OmniBench test split. Primarily, we wanted to compare against models that were allowed to use paired data but did not specifically finetune for performance on the listed benchmarks.

Method	AVQA			MUSIC-AVQA
Method	Audio	Video	Audio-Video	Audio	Video	Audio-Video
Without proxy and bridge	59.55	63.91	70.21	44.03	44.64	45.75
With proxy without bridge	78.53	79.50	82.91	45.85	54.49	58.60
With proxy and bridge	80.24	81.33	85.58	57.26	57.11	65.65

Table 4: Proxy and bridge help improve model performance. Every cell is at

r{=}512

on the test splits.

Modality	residual cos ( $L_{0}/L_{\text{mid}}/L_{\text{last}}$ )	vocab cos ( $L_{0}/L_{\text{mid}}/L_{\text{last}}$ )	$\Delta$ at last layer
Audio	$.956\,/\,.889\,/\,.778$	$.953\,/\,.913\,/\,.944$	$\mathbf{+0.166}$
Vision	$.821\,/\,.883\,/\,.805$	$.828\,/\,.799\,/\,.929$	$\mathbf{+0.124}$

Table 5: Paired residual and output-layer cosine similarity. Each proxy is matched to its nearest real counterpart by last-layer residual cosine, and similarities are computed over those pairs. Here,

\Delta=\text{vocab cos}_{L_{\text{last}}}-\text{residual cos}_{L_{\text{last}}}

. These are not comparable to the trajectory cosines in Table 10 , which measure a different quantity.

Modality	$L_{0}$	$L_{\text{mid}}$	$L_{\text{last}}$
Audio	0.953	0.970	0.963
Vision	1.000	1.000	0.998

Table 6: Real and proxy hidden states remain distinguishable. A logistic-regression probe predicts whether a hidden state came from a real feature or a bridged-proxy feature at three layers;

n{=}400

per cell. Values are probe accuracies, so higher values mean real and proxy states are easier to separate. The high accuracies show that MSP does not make the hidden states identical; Table 5 tests whether the LM-head readout is nevertheless aligned.

(a) Real modalities vs. proxy modalities at inference
Audio	Vision	AVQA	MUSIC-AVQA
real	real	85.58	65.65
imagined	real	81.19	57.34
real	imagined	79.28	62.87
imagined	imagined	75.75	41.33

Table 7: Real modalities vs. proxy modalities at inference and token-matched controls. Qwen3-8B LoRA

r{=}512

, test splits: AVQA (

n{=}16{,}910

), MUSIC-AVQA (

n{=}9{,}185

Appendix figures & tables12 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 2: MSP architecture, component-level view. MSP uses frozen audio and vision encoders with their associated text encoders, trainable per-modality projectors, a frozen text-to-modality bridge for Phase 2, and a Qwen3-8B language model with LoRA adapters.

Dataset	Question type	Accuracy
AVQA	Come From (sound localization)	90.9
AVQA	Where (spatial)	80.3
MUSIC-AVQA	Counting	72.6
MUSIC-AVQA	Location	57.2

Appendix

Table 8: Reported question-type and overall modality results at

r{=}512

. The question-type block reports the four categories recomputed for this checkpoint.

Dataset	Modality	Pre-bridge	Pre-projector	Post-projector
Dataset	Modality	(encoder only)	(encoder+bridge)	(encoder+bridge+projector)
AVQA	Vision	0.307	0.786 (+0.479)	0.889 (+0.582)
AVQA	Audio	0.218	0.548 (+0.330)	0.697 (+0.479)
MUSIC-AVQA	Vision	0.333	0.823 (+0.490)	0.893 (+0.560)
MUSIC-AVQA	Audio	0.422	0.752 (+0.330)	0.821 (+0.399)

Appendix

Table 9: Bridge and projector progressively align proxy with real-modality features.

Modality	bridged (ours)	zero	Gaussian	wrong-mod	shuffled-real
audio	0.291	0.097	0.136	0.309	0.316
vision	0.321	0.208	0.237	0.391	0.432

Appendix

Table 10: Valid features engage the placeholder. We compare real features with surrogate inputs using the last-layer residual cosine of the whole hidden-state trajectory against the unmodified real-input run on the same example (

n{=}200

pairs). This is a different quantity from the nearest-pair cosine in Table 5 and the two should not be compared directly. Higher is closer to the real trajectory. Zero and Gaussian inputs reduce engagement; valid features remain closer.

Vision encoder	Mean cosine similarity
MetaCLIP B/16	0.407
OpenAI CLIP B/16	0.347
DFN2B B/16	0.331
SigLIP2 B/16	0.152
SigLIP1 B/16	0.143

Appendix

Table 11: Mean cosine similarity between all COCO images and their text captions, used to inform encoder selection.

Panel of Figure 3 — Figure 3: Examples of anchor-supporting proxy generation. Each panel shows a real anchor in one modality and the proxy text generated for the other modality. The held-out ground truth is shown only for context; it is not used to generate the proxy.

Panel of Figure 4 — Figure 4: Anchor-supporting proxy examples. Generated vision proxies can differ from the held-out scene while remaining compatible with the real audio anchor.

Bridge	Audio	Vision
identity (no bridge)	0.94	8.22
mean-shift	0.86	5.36
residual MLP	0.78	2.12
linear (ridge) $\checkmark$	0.70	0.86

Appendix

Table 12: Bridge calibration residuals. Held-out bridge error from text-tower to modality-encoder space; lower is better. The linear bridge gives the lowest residual for audio and vision.

Stage	AVQA	MUSIC-AVQA
Phase 1 only (per-modality projectors)	70.21	45.75
+ QA supervision only, no proxies, no bridge	77.46	55.18
+ full MSP composition	85.58	65.65
QA-supervision contribution	+7.25	+9.43
composition contribution	+8.12	+10.47

Appendix

Table 13: QA supervision and MSP composition. All numbers are at

r{=}512

on the test splits.

Split	$n$	MSP	Majority class
head	138,153	67.26	35.60
tail	73,419	47.76	31.67
overall	211,572	60.49	—

Appendix

Table 14: MUSIC-AVQA-R results on the full

211{,}572

-question test set.

Audio proxy quality (vision real)	AVQA ( $r{=}512$ )
real audio	85.58
scene-grounded (imagined from real vision)	81.19
question-derived (observes nothing)	76.15

Appendix

Table 15: Audio proxy quality at test time with real vision.

Method	AVQA accuracy
Paired upper bound (fully paired data)	87.45
MSP (ours)	86.00
Interleaved unimodal (alternating modalities by sample)	81.46
Vision only	80.72
Audio only	75.35

Appendix

Table 16: Paired data at a smaller scale: a text-only Qwen3 4B model finetuned on AVQA.

Multimodal Learning

Multimodal learning studies how models combine and reason over different data types, such as images, text, audio, signals, and structured records.

See all 331 papers

Multimodal LLMs under Pairwise Modalities
May 20, 2026Yan Li, Yunlong Deng, Yuewen Sun +3Cross-Modal Learning Representation Identifiability
Mohamed bin Zayed University of Artificial Intelligence · Carnegie Mellon University
Mixture of Probes: Learning from Privileged Modalities in Multimodal LLMs Through Probing
Jul 9, 2026Dominick Reilly, Qiyu Wu, Hiromi Wakaki +2Missing-Modality Learning Cross-Modal Representation Learning
Sony Group Corporation · University of North Carolina at Charlotte · Sony AI
SMA: Submodular Modality Aligner For Data Efficient Multimodal Learning
May 13, 2026Truong Pham, Anay Majee, Rishabh IyerCross-Modal Learning Cross-Modal Alignment
The University of Texas at Dallas

May 20, 2026J/K move · Enter open · S save

Multimodal LLMs under Pairwise Modalities

Yan Li, Yunlong Deng, Yuewen Sun, Gongxu Luo+2

Mohamed bin Zayed University of Artificial Intelligence · Carnegie Mellon University

Cross-Modal Learning Representation Identifiability Cross-Modal Alignment Multimodal Large Language Models Multimodal Understanding Multimodal Generation+3

Open paper AskPDF arXiv

Despite the impressive results achieved by multimodal large language models (MLLMs), their training typically relies on jointly curated multimodal data, requiring substantial human effort to construct multi-way aligned datasets and thereby limiting scalability across domains. In this work, we explore training MLLMs by only leveraging multiple paired modalities as a surrogate for the full joint multimodal distribution. Specifically, we first provide a theoretical analysis of the conditions under which the representations are identifiable with only observing pairwise modalities. Building on this analysis, we propose a representation learning framework for aligning latent representations across modalities using only pairwise data. The framework consists of two stages: latent representation alignment and cross-modal recomposition. Specifically, in the first stage, we learn the shared latent space across modalities by both self-modal reconstruction and pair-wise contrastive learning. We also incorporate an inductive bias in the contrastive learning process by partially aligning and minimal latent specification. In stage two, we integrate the encoder of newly introduced modalities with the decoders of the pre-trained modalities to facilitate cross-modal transfer and generation. We evaluate our method by newly adding 3D point clouds and tactile modalities into pre-trained MLLMs with three modality pairs and show that, by learning an aligned latent representation space, our model achieves strong cross-modal performance.

Abstract

Multimodal Learning

Explore similar work

Multimodal LLMs under Pairwise Modalities

Mixture of Probes: Learning from Privileged Modalities in Multimodal LLMs Through Probing

SMA: Submodular Modality Aligner For Data Efficient Multimodal Learning