EmergentBridge: Improving Zero-Shot Cross-Modal Transfer in Unified Multimodal Embedding Models

Jincheng Xie; Xingchen Xiao; Runheng Liu; Zhongyi Huang; Yu Zheng; Heyan Huang

cs.AIApr 13, 2026

EmergentBridge: Improving Zero-Shot Cross-Modal Transfer in Unified Multimodal Embedding Models

Authors: Jincheng Xie, Xingchen Xiao, Runheng Liu, Zhongyi Huang, Yu Zheng, Heyan Huang

Organizations: School of Computer Science and Technology, Beijing Institute of Technology, Beijing, China · JD iCity, JD Technology, JD Intelligent Cities Research, Beijing, China

Abstract

Unified multimodal embedding spaces underpin practical applications such as cross-modal retrieval and zero-shot recognition. In many real deployments, however, supervision is available only for a small subset of modality pairs (e.g., image--text), leaving unpaired modality pairs (e.g., audio $\leftrightarrow$ depth, infrared $\leftrightarrow$ audio) weakly connected and thus performing poorly on zero-shot transfer. Addressing this sparse-pairing regime is therefore essential for scaling unified embedding systems to new tasks without curating exhaustive pairwise data. We propose EmergentBridge, an embedding-level bridging framework that improves performance on these unpaired pairs without requiring exhaustive pairwise supervision. Our key observation is that naively aligning a new modality to a synthesized proxy embedding can introduce gradient interference, degrading the anchor-alignment structure that existing retrieval/classification relies on. EmergentBridge addresses this by (i) learning a mapping that produces a noisy bridge anchor (a proxy embedding of an already-aligned modality) from an anchor embedding, and (ii) enforcing proxy alignment only in the subspace orthogonal to the anchor-alignment direction, preserving anchor alignment while strengthening non-anchor connectivity. Across nine datasets spanning multiple modalities, EmergentBridge consistently outperforms prior binding baselines on zero-shot classification and retrieval, demonstrating strong emergent alignment.

Figures & tables

Figure 1. Zero-Shot Language-Related Task Performance: EmergentBrideg with image anchor modality is demonstrated as a powerful emergent ability for indirect alignment modalities.

Figure 2. EmergentBridge vs. ImageBind and LanguageBind: The left image illustrates ImageBind’s approach, where modalities are indirectly aligned through direct alignment with the image modality. The center image depicts LanguageBind’s strategy of augmenting the model’s zero-shot capabilities by generating data. In contrast, the right image demonstrates EmergentBridge, which uses the image as the core modality to enhance emergent zero-shot capabilities by generating embeddings rather than additional data.

Figure 3. An Example Overview of EmergentBridge. $\mathcal{M}_{a},\mathcal{M}_{b}$ and $\mathcal{C}$ correspond to text, audio, and image, respectively. (Step 1) We align the anchor modality (image) with an already-aligned modality (text) using InfoNCE. (Step 2) We train a proxy predictor that maps image embeddings to proxy text embeddings, while keeping the image/text encoders frozen. (Step 3) We align audio with both the image embedding and the synthesized text proxy, using InfoNCE together with an orthogonal-subspace regularizer that reduces interference with anchor alignment. In practice, when $\mathcal{M}_{a}$ is already anchor-aligned in a pretrained binding model, Step 1 can be skipped and we only train the proxy predictor and the new modality encoder.

Panel of Figure 4 — Figure 4. Orthogonal-subspace regularization. (a) $T_{\bar{c}_{i}}$ projects $x$ onto the orthogonal complement of the anchor-alignment direction $\bar{c}_{i}$ and normalizes it. (b) The InfoNCE objective pulls $x_{i}^{b}$ toward $c_{i}$ (black arrow), while the orthogonal-subspace regularizer guides $x_{i}^{b}$ toward $\hat{x}_{i}^{a}$ within directions orthogonal to $\bar{c}_{i}$ , reducing interference with anchor alignment.

Method	Anchor	Infrared	Depth		Audio
	Modality	LLVIP	NYU-D	SUN	AudioSet	ESC-50	VGG-S
OpenCLIP ( Cherti et al., 2023 )	–	82.2	45.4	25.4	–	–	–
DepthSwin ( Girdhar et al., 2022 )	–	–	72.5	63.1	–	–	–
JointCRF ( Wang et al., 2017 )	–	–	65.8	63.6	–	–	–
DFCR ( Cao et al., 2018 )	–	–	65.3	56.3	–	–	–
AudioCLIP ( Guzhov et al., 2021 )	–	–	–	–	28.4	68.6	47.4

Table 1. X–Language classification. ∗ denotes emergent zero-shot (image anchor). We report top-1 accuracy (%) for all datasets except AudioSet (mAP). Best results are in bold.

Figure 5. Experimental Results on Classification and Retrieval Tasks After Varying Hyperparameters

\lambda

: The first row of figures, arranged from left to right, presents the results on the VGG-S, LLVIP, and NYU-D datasets, with images serving as the core modality. The second row of figures, similarly ordered, displays the results on the same datasets, but with text serving as the core modality.

Method	Anchor	Clotho		AudioCaps
Method	Modality	R@1	R@10	R@1	R@10
AVFIC ( Nagrani et al., 2022 )	–	3.0	17.5	8.7	37.7
AudioCLIP ( Guzhov et al., 2021 )	–	3.20	20.3	3.53	31.6
WAV2CLIP ( Wu et al., 2021 )	–	0.78	12.1	0.88	15.3
C-MCR ( Wang et al., 2023b )	–	8.37	36.7	15.76	48.1
LanguageBind	Text	12.1	44.0	12.2	53.2

Table 2. Zero-shot audio–language retrieval. ∗ denotes emergent zero-shot (image anchor). We report Recall@K.

Dataset	Method	Anchor	R@1
AVE	ImageBind	Image	36.9
	EmergentBridge	Image	37.0
	LanguageBind ∗	Text	10.6
	EmergentBridge ∗	Text	15.1
VGG-S	ImageBind	Image	28.7
	EmergentBridge	Image	30.1

Table 3. RGB

\rightarrow

X retrieval. ∗ denotes the emergent setting with text as the anchor; we report Recall@1. Best results are in bold.

Dataset	Task	Image Anchor					Text Anchor
Dataset	Task	Diff	ResNet	VAE	Noise	C-MCR	Diff	ResNet	VAE	Noise	C-MCR
VGG-S	A $\rightarrow$ T ∗	36.3	30.7	32.1	23.7	26.7	29.3	27.2	28.3	26.4	29.1
VGG-S	RGB $\rightarrow$ A	30.1	29.3	28.1	27.7	28.3	15.1 ∗	12.3 ∗	13.5 ∗	8.1 ∗	9.5 ∗
LLVIP	I $\rightarrow$ T ∗	73.7	68.1	64.3	59.1	61.7	85.1	83.1	87.5	86.2	86.9
LLVIP	RGB $\rightarrow$ I	27.4	26.5	27.1	26.1	25.9	12.8 ∗	10.3 ∗	9.1 ∗	5.2 ∗	8.9 ∗
NYU-D	D $\rightarrow$ T ∗	70.1	67.2	69.3	50.2	58.4	65.8	65.7	64.8	64.1	65.4

Table 4. Comparison of proxy embedding synthesis methods (Step 2). We instantiate the synthesis network

N_{\Theta}

with different architectures to generate proxy embeddings

\hat{x}^{a}

from anchor embeddings

c

. We report top-1 accuracy (%) on

X\!\rightarrow\!T

and Recall@1 (%) on

\mathrm{RGB}\!\rightarrow\!X

. Diff denotes conditional diffusion-based synthesis. ∗ marks emergent zero-shot tasks (i.e., unpaired modality pairs). Best results are in bold.

Dataset	Task	Anchor	Proxy Alignment
Dataset	Task	Anchor	OSR	Direct InfoNCE
VGG-S	RGB $\rightarrow$ A	Image	30.1	25.2
VGG-S	A $\rightarrow$ T	Text	29.3	22.2
AVE	RGB $\rightarrow$ A	Image	37.0	32.3
NYU-D	RGB $\rightarrow$ D	Image	35.2	29.1
NYU-D	D $\rightarrow$ T	Text	65.8	55.7

Table 5. Replacing OSR with direct InfoNCE for proxy alignment. We replace OSR by removing the projection

T_{\bar{c}_{i}}(\cdot)

and directly applying InfoNCE between

x^{b}

and the proxy embedding

\hat{x}^{a}

(keeping

\lambda{=}1

). We report Recall@1 for image-anchored retrieval and top-1 accuracy (%) for text-anchored classification.

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Config	Audio	Depth	Infrared
Encoder	ViT-Huge
Number of Heads	12	8	12
Optimizer	AdamW
Optimizer Momentum	$\beta_{1}=0.9,\beta_{2}=0.95$
Epochs	8	2	2
Learning rate	5e-4	5e-4	1e-4

Appendix

Table 6. Training Settings in ImageCore Model

Config	Audio	Depth	Infrared
Encoder	ViT-Huge
Number of Heads	12	8	12
Optimizer	AdamW
Optimizer Momentum	$\beta_{1}=0.9,\beta_{2}=0.95$
Epochs	8	4	4
Learning rate	1e-4	5e-4	1e-4

Appendix

Table 7. Training Settings in LanguageCore Model

Figure 6. CDF Curves of Similarity on VGG-S and SUN: The top and bottom rows respectively illustrate the cumulative distribution functions (CDFs) of cosine similarity between generated text/image embeddings (produced by different methods in Step 2 ) and the corresponding actual text/image embeddings. Results are presented with image and text as the core modalities. The solid, dashed, dash-dotted, and dotted lines correspond to the diffusion, C-MCR, ResNet, and VAE methods, respectively.

Cross-Modal Learning

Cross-modal learning connects information across different data types, enabling models to align, transfer, and reason across modalities.

See all 143 papers

MSP: Modality Self-Play for Training Multimodal LLMs Without Paired Cross-Modal Data
Oct 14, 2026Robert Moseley, Md Kaykobad Reza, Ameya Patil +2Multimodal Learning Multimodal Foundation Models
University of California, Riverside · Amazon
Shared Geometry As A Rosetta Stone: Cross-Modal Alignment Without Paired Data
Oct 7, 2026Dominik Schnaus, Thomas Dagès, Daniel Cremers +2Cross-Modal Alignment Cross-Modal Representation Transfer
TU Munich · MCML · Ulm University +2
Gemini Embedding 2: A Native Multimodal Embedding Model from Gemini
May 26, 2026Madhuri Shanbhogue, Zhe Li, Shanfeng Zhang +86Cross-Modal Learning Unified Multimodal Models

Oct 14, 2026J/K move · Enter open · S save

MSP: Modality Self-Play for Training Multimodal LLMs Without Paired Cross-Modal Data

Robert Moseley, Md Kaykobad Reza, Ameya Patil, Edward Ayrapetian+1

University of California, Riverside · Amazon

Multimodal Learning Multimodal Foundation Models Multimodal Pretraining Cross-Modal Alignment Multimodal Generation

Open paper AskPDF arXiv

Large multimodal models have achieved remarkable progress by scaling model capacity and training on massive paired multimodal data. However, this paradigm introduces a fundamental bottleneck: aligned multimodal datasets are expensive to curate, difficult to scale across diverse modality combinations, and increasingly constrain further progress. We introduce Modality Self-Play (MSP), a training framework for multimodal foundation models based on proxy-mediated modality composition. Instead of relying on paired cross-modal data, MSP uses anchor-supporting proxy textual descriptions of unseen modalities as semantic bridges during training. Given an observed modality (e.g., image or audio), the model generates proxy descriptions for complementary unseen modalities, which are then encoded and aligned with the observed modality through learned bridges, enabling cross-modal interaction without real paired non-text supervision. Our central hypothesis is that proxy text, combined with contrastively trained pretrained encoders, provides sufficient semantic structure for multimodal composition to emerge. We show that MSP enables zero-shot modality composition: models trained only with a real anchor modality and accompanying proxy descriptions can integrate and reason over multiple modalities at inference time despite never observing real paired non-text data during training. Empirically, MSP achieves state-of-the-art or competitive performance on multiple multimodal benchmarks. Ablation studies further demonstrate that learned bridges improve alignment, while anchor-supporting proxy descriptions enable effective cross-modal composition. MSP removes the need for real paired non-text (e.g., audio-vision) supervision, while continuing to rely on modality-text pairs, pretrained contrastive encoders, and modality-distribution bridge calibration.

Abstract

Cross-Modal Learning

Explore similar work

MSP: Modality Self-Play for Training Multimodal LLMs Without Paired Cross-Modal Data

Shared Geometry As A Rosetta Stone: Cross-Modal Alignment Without Paired Data

Gemini Embedding 2: A Native Multimodal Embedding Model from Gemini