From Fragmentation to Fusion: A Survey and Roadmap for Omnimodal Embeddings
Organizations: Ant Group · Shanghai Jiao Tong University · Australian National University · Tongji University ·
Abstract
Embedding models underpin virtually every modern AI system, from search engines to recommendation systems. Despite shared mathematical foundations, research on embedding models remains fragmented by modality: the communities develop largely in isolation, seldom exchanging insights. We argue that this fragmentation obscures a deeper convergence. Driven by contrastive learning, Transformer architectures, and a granularity shift, embedding research across modalities is converging toward a common set of principles, even as modality-specific challenges remain underexplored. To reason across this convergence without conflating it, we propose a framework of three progressively harder capabilities: coverage, comparability, and compositionality. Three paths toward unification satisfy these in turn: transformation broadens coverage, alignment enables comparability, and joint embedding, the terminus we advocate, aims at compositionality. This position paper traces the resulting evolution, providing the first unified, cross-cutting treatment of omnimodal embedding research.
Figures & tables
| Family | Examples and type | Structure to preserve | Representative evidence |
|---|---|---|---|
| Language and symbols | Natural language and code (R); tables and markup (S) | Discrete sequence, hierarchy and references; code adds syntax, flow and execution | BERT, CodeBERT, GraphCodeBERT, Meta-Transformer ( Devlin et al., 2019 ; Feng et al., 2020 ; Guo et al., 2021 ; Zhang et al., 2023c ) |
| Visible and dynamic vision | RGB image, video and event streams (R); flow, segmentation, keypoints and boxes (D) | 2D topology; video/events add order, rate, sparse time and motion | VideoMAE, UniBind, 4M-21 ( Tong et al., 2022 ; Lyu et al., 2024 ; Bachmann et al., 2024 ) |
| Geometry and 3D | Depth, LiDAR (R/D); point sets, meshes, voxels, poses and fields (S/D) | Metric coordinates, surface topology and SE(3) behavior | ImageBind, ViT-Lens, PointNet++, MeshCNN ( Girdhar et al., 2023 ; Lei et al., 2024 ; Qi et al., 2017b ; Hanocka et al., 2019 ) |
| Spectral and ranging sensors | Thermal, hyperspectral, SAR, radar, sonar, location streams (R) | Band identity, calibration, sparsity and sensor noise | SatMAE, BEVFusion ( Cong et al., 2022 ; Liu et al., 2023d ) |
| Audio and acoustics | Speech, music, environmental sound, waveforms (R) | Sampling rate, phase, mixtures, content, speaker and environment | wav2vec 2.0, CLAP, MAEB ( Baevski et al., 2020 ; Elizalde et al., 2023 ; Assadi et al., 2026a ) |
| Embodied and physiological sensing | IMU, force, tactile, EEG, ECG, EMG, PPG (R) | Body/sensor topology, units, synchronization, privacy | ImageBind, CodeBind, MultiBench ( Girdhar et al., 2023 ; Chen et al., 2026f ; Liang et al., 2021 ) |
| System | Inputs tested | Mechanism/output | What remains unproven | |||
|---|---|---|---|---|---|---|
| Meta-Transformer ( Zhang et al., 2023c ) | Twelve reported data types | Modality tokenizers plus shared frozen backbone; task outputs | D | – | – | Reusing parameters does not show that outputs share a retrieval space |
| ImageBind ( Girdhar et al., 2023 ) | Image, text, audio, depth, thermal, IMU | Image-anchor alignment; independent global vectors | D | D | – | Only image–other pairs are trained directly; comparing two non-image inputs is not fusion |
| 4M-21 ( Bachmann et al., 2024 ) | Twenty-one visual, semantic, geometric, feature and metadata channels | Discrete any-to-any token prediction | D | – | – | Supports broad conditional generation, but not a general dense retrieval vector |
| ViT-Lens / UniBind ( Lei et al., 2024 ; Lyu et al., 2024 ) | Multiple 2D, 3D, video, audio and sensor inputs | Adapters map new inputs into existing aligned spaces | D | D | – | Evaluation covers only a subset of possible input pairs |
| LCO-Embedding ( Xiao et al., 2025a ) | Language-centered image, video and related retrieval tasks | MLLM state alignment plus contrastive refinement | D | D | – | Improved cross-modal comparison does not establish arbitrary multi-input composition |
| jina-embeddings-v4 ( Günther et al., 2025 ) | Multilingual text and images | Universal retrieval vectors | D | D | P | Tested image–text combinations do not establish broader multi-input composition |
| System | Multi-input case tested | Mechanism / output | What remains unproven | |||
|---|---|---|---|---|---|---|
| Everything at Once ( Shvetsova et al., 2022 ) | Available subsets of video, audio and text | Input-agnostic fusion Transformer | D | D | P | Tests missing-input subsets, but not all baselines or both retrieval roles |
| VISTA ( Zhou et al., 2024b ) | Text, image, and image–text queries | Visual tokens injected into a text encoder | D | D | P | Tests composition directly, but only for vision and language |
| UniIR / M-BEIR ( Wei et al., 2024a ) | Text, image and image–text queries/candidates | Instruction-conditioned score or feature fusion | D | D | P | Varies query/candidate roles, but uses only two modalities |
| VLM2Vec ( Jiang et al., 2025 ) | Text, image and instructed combinations | Pooled MLLM hidden states | D | D | P | Rewriting the prompt does not show that both inputs affect the result |
| MM-Embed / GME ( Lin et al., 2025 ; Zhang et al., 2025c ) | Vision–language retrieval and combinations | Negative sampling by modality and fused training | D | D | P | Does not test arbitrary input subsets |
| UniME-V2 ( Gu et al., 2026 ) | Universal vision–language embedding tasks | Model-generated semantic supervision | D | D | P | Richer supervision; evaluation remains limited to vision and language |
Appendix figures & tables9 assets
Supplementary material from the paper’s appendix.
Appendix
Multimodal Embedding
Multimodal embedding maps text, images, audio, video, and other data into representations that support comparison and retrieval across modalities.
See all 168 papers