Organizations: School of Computer Science and Technology, Beijing Institute of Technology, Beijing, China · JD iCity, JD Technology, JD Intelligent Cities Research, Beijing, China
Unified multimodal embedding spaces underpin practical applications such as cross-modal retrieval and zero-shot recognition. In many real deployments, however, supervision is available only for a small subset of modality pairs (e.g., image--text), leaving unpaired modality pairs (e.g., audio↔depth, infrared↔audio) weakly connected and thus performing poorly on zero-shot transfer. Addressing this sparse-pairing regime is therefore essential for scaling unified embedding systems to new tasks without curating exhaustive pairwise data. We propose EmergentBridge, an embedding-level bridging framework that improves performance on these unpaired pairs without requiring exhaustive pairwise supervision. Our key observation is that naively aligning a new modality to a synthesized proxy embedding can introduce gradient interference, degrading the anchor-alignment structure that existing retrieval/classification relies on. EmergentBridge addresses this by (i) learning a mapping that produces a noisy bridge anchor (a proxy embedding of an already-aligned modality) from an anchor embedding, and (ii) enforcing proxy alignment only in the subspace orthogonal to the anchor-alignment direction, preserving anchor alignment while strengthening non-anchor connectivity. Across nine datasets spanning multiple modalities, EmergentBridge consistently outperforms prior binding baselines on zero-shot classification and retrieval, demonstrating strong emergent alignment.
Figures & tables
Figure 1. Zero-Shot Language-Related Task Performance: EmergentBrideg with image anchor modality is demonstrated as a powerful emergent ability for indirect alignment modalities.
Figure 2. EmergentBridge vs. ImageBind and LanguageBind: The left image illustrates ImageBind’s approach, where modalities are indirectly aligned through direct alignment with the image modality. The center image depicts LanguageBind’s strategy of augmenting the model’s zero-shot capabilities by generating data. In contrast, the right image demonstrates EmergentBridge, which uses the image as the core modality to enhance emergent zero-shot capabilities by generating embeddings rather than additional data.
Figure 3. An Example Overview of EmergentBridge. Ma,Mb and C correspond to text, audio, and image, respectively. (Step 1) We align the anchor modality (image) with an already-aligned modality (text) using InfoNCE. (Step 2) We train a proxy predictor that maps image embeddings to proxy text embeddings, while keeping the image/text encoders frozen. (Step 3) We align audio with both the image embedding and the synthesized text proxy, using InfoNCE together with an orthogonal-subspace regularizer that reduces interference with anchor alignment. In practice, when Ma is already anchor-aligned in a pretrained binding model, Step 1 can be skipped and we only train the proxy predictor and the new modality encoder.
Figure 4. Orthogonal-subspace regularization. (a) Tcˉi projects x onto the orthogonal complement of the anchor-alignment direction cˉi and normalizes it. (b) The InfoNCE objective pulls xib toward ci (black arrow), while the orthogonal-subspace regularizer guides xib toward x^ia within directions orthogonal to cˉi , reducing interference with anchor alignment.
Method
Anchor
Infrared
Depth
Audio
Modality
LLVIP
NYU-D
SUN
AudioSet
ESC-50
VGG-S
OpenCLIP ( Cherti et al., 2023 )
–
82.2
45.4
25.4
–
–
–
DepthSwin ( Girdhar et al., 2022 )
–
–
72.5
63.1
–
–
–
JointCRF ( Wang et al., 2017 )
–
–
65.8
63.6
–
–
–
DFCR ( Cao et al., 2018 )
–
–
65.3
56.3
–
–
–
AudioCLIP ( Guzhov et al., 2021 )
–
–
–
–
28.4
68.6
47.4
Table 1. X–Language classification. ∗ denotes emergent zero-shot (image anchor). We report top-1 accuracy (%) for all datasets except AudioSet (mAP). Best results are in bold.
Figure 5. Experimental Results on Classification and Retrieval Tasks After Varying Hyperparameters λ : The first row of figures, arranged from left to right, presents the results on the VGG-S, LLVIP, and NYU-D datasets, with images serving as the core modality. The second row of figures, similarly ordered, displays the results on the same datasets, but with text serving as the core modality.
Table 3. RGB → X retrieval. ∗ denotes the emergent setting with text as the anchor; we report Recall@1. Best results are in bold.
Dataset
Task
Image Anchor
Text Anchor
Diff
ResNet
VAE
Noise
C-MCR
Diff
ResNet
VAE
Noise
C-MCR
VGG-S
A → T ∗
36.3
30.7
32.1
23.7
26.7
29.3
27.2
28.3
26.4
29.1
RGB → A
30.1
29.3
28.1
27.7
28.3
15.1 ∗
12.3 ∗
13.5 ∗
8.1 ∗
9.5 ∗
LLVIP
I → T ∗
73.7
68.1
64.3
59.1
61.7
85.1
83.1
87.5
86.2
86.9
RGB → I
27.4
26.5
27.1
26.1
25.9
12.8 ∗
10.3 ∗
9.1 ∗
5.2 ∗
8.9 ∗
NYU-D
D → T ∗
70.1
67.2
69.3
50.2
58.4
65.8
65.7
64.8
64.1
65.4
Table 4. Comparison of proxy embedding synthesis methods (Step 2). We instantiate the synthesis network NΘ with different architectures to generate proxy embeddings x^a from anchor embeddings c . We report top-1 accuracy (%) on X→T and Recall@1 (%) on RGB→X . Diff denotes conditional diffusion-based synthesis. ∗ marks emergent zero-shot tasks (i.e., unpaired modality pairs). Best results are in bold.
Dataset
Task
Anchor
Proxy Alignment
OSR
Direct InfoNCE
VGG-S
RGB → A
Image
30.1
25.2
A → T
Text
29.3
22.2
AVE
RGB → A
Image
37.0
32.3
NYU-D
RGB → D
Image
35.2
29.1
D → T
Text
65.8
55.7
Table 5. Replacing OSR with direct InfoNCE for proxy alignment. We replace OSR by removing the projection Tcˉi(⋅) and directly applying InfoNCE between xb and the proxy embedding x^a (keeping λ=1 ). We report Recall@1 for image-anchored retrieval and top-1 accuracy (%) for text-anchored classification.
Appendix figures & tables3 assets
Supplementary material from the paper’s appendix.
Appendix
Config
Audio
Depth
Infrared
Encoder
ViT-Huge
Number of Heads
12
8
12
Optimizer
AdamW
Optimizer Momentum
β1=0.9,β2=0.95
Epochs
8
2
2
Learning rate
5e-4
5e-4
1e-4
Appendix
Table 6. Training Settings in ImageCore Model
Config
Audio
Depth
Infrared
Encoder
ViT-Huge
Number of Heads
12
8
12
Optimizer
AdamW
Optimizer Momentum
β1=0.9,β2=0.95
Epochs
8
4
4
Learning rate
1e-4
5e-4
1e-4
Appendix
Table 7. Training Settings in LanguageCore Model
Figure 6. CDF Curves of Similarity on VGG-S and SUN: The top and bottom rows respectively illustrate the cumulative distribution functions (CDFs) of cosine similarity between generated text/image embeddings (produced by different methods in Step 2 ) and the corresponding actual text/image embeddings. Results are presented with image and text as the core modalities. The solid, dashed, dash-dotted, and dotted lines correspond to the diffusion, C-MCR, ResNet, and VAE methods, respectively.
Large multimodal models have achieved remarkable progress by scaling model capacity and training on massive paired multimodal data. However, this paradigm introduces a fundamental bottleneck: aligned multimodal datasets are expensive to curate, difficult to scale across diverse modality combinations, and increasingly constrain further progress. We introduce Modality Self-Play (MSP), a training framework for multimodal foundation models based on proxy-mediated modality composition. Instead of relying on paired cross-modal data, MSP uses anchor-supporting proxy textual descriptions of unseen modalities as semantic bridges during training. Given an observed modality (e.g., image or audio), the model generates proxy descriptions for complementary unseen modalities, which are then encoded and aligned with the observed modality through learned bridges, enabling cross-modal interaction without real paired non-text supervision. Our central hypothesis is that proxy text, combined with contrastively trained pretrained encoders, provides sufficient semantic structure for multimodal composition to emerge. We show that MSP enables zero-shot modality composition: models trained only with a real anchor modality and accompanying proxy descriptions can integrate and reason over multiple modalities at inference time despite never observing real paired non-text data during training. Empirically, MSP achieves state-of-the-art or competitive performance on multiple multimodal benchmarks. Ablation studies further demonstrate that learned bridges improve alignment, while anchor-supporting proxy descriptions enable effective cross-modal composition. MSP removes the need for real paired non-text (e.g., audio-vision) supervision, while continuing to rely on modality-text pairs, pretrained contrastive encoders, and modality-distribution bridge calibration.
Multimodal representations enable zero-shot classification and retrieval, but aligning independently trained models usually requires large amounts of paired data. Yet, the Platonic Representation Hypothesis suggests that models trained on different modalities may converge spontaneously toward a shared representation geometry. But then, do we even need paired examples for cross-modal alignment? Remarkably, we show that paired examples are unnecessary for coarse cross-modal alignment. Our simple Wasserstein Procrustes method with a coarse geometric initialization aligns two disjoint embedding sets by estimating a single orthogonal map without seeing any pairs. Across datasets, modalities, and unimodal models, we show that we can consistently align independently trained representations without pairs, and standard geometric alignment metrics accurately predict when this is possible. Nevertheless, we can naturally benefit from paired examples. In the very few-pair regime, our method substantially outperforms existing ones, while staying competitive with pair-based methods with more added examples. Finally, we demonstrate that the resulting alignments can enable text-to-image generation without paired examples. These results show that independently trained models often share enough geometry to establish cross-modal correspondence with little or no paired data.
We introduce Gemini Embedding 2, a native multimodal embedding model that allows embedding video, audio, image, and text modalities in a unified representation space. We leverage the multimodal capabilities of Gemini to produce embeddings for arbitrary combinations of interleaved inputs across all these modalities that generalize well across a wide variety of tasks. Applying large-scale contrastive learning in a multi-task multi-stage training setup, we achieve state-of-the-art performance on key embedding benchmarks including unimodal, cross-modal, and multimodal retrieval spanning a diverse set of tasks. We show that our embedding model demonstrates strong performance (with a score of 62.9 R@1 on MSCOCO, 68.8 NDCG@10 on Vatex, 69.9 on MTEB multilingual and 84.0 on MTEB Code) across a variety of tasks surpassing the performance of specialized models. These unified capabilities make Gemini Embedding 2 a promising candidate for downstream use cases such as RAG, recommendation and search. Furthermore, its robust zero-shot performance across distinct fields - from astronomy and bioscience to fine arts and the culinary arts - establishes it as a highly reliable, out-of-the-box representation even for specialized domains.