Multimodal Embedding

Ask about this topic

Overview

Multimodal embedding studies how to represent different kinds of data so that related items are close in a shared space, enabling tasks such as cross-modal search, recommendation, and classification. Common approaches include contrastive dual encoders, instruction-tuned multimodal language models, and alignment methods based on graphs or optimal transport; multi-vector and late-interaction representations can retain finer detail than a single vector. Benchmarks such as MMEB, ViDoRe, and audio- and video-retrieval suites test these models across tasks and modalities, while newer methods explore compressed representations and latent reasoning to improve quality without costly inference. A central challenge is balancing broad, zero-shot capability with fine-grained, task-specific information, while keeping representations efficient and robust when modalities are missing or unevenly informative.

Latest papers

CardsList

160 papers19 papers in the last four weeks, up 6% on the four weeks before. 0.1% of all new papers.

  1. EmbeddingGemma 2: Unified and Efficient Multimodal Representations

    Oct 27, 2026Henrique Schechter Vera, Sahil Dua, Jovan Andonov +70Multimodal EmbeddingUnified Multimodal Models

  2. Beyond Post-Hoc Rewards: Privileged Reasoning Exploration for Multimodal Embeddings

    Oct 23, 2026Tianyu Yang, Yijun Tian, Lars Benedikt Kaesberg +4Multimodal EmbeddingCross-Modal Retrieval

    University of Göttingen · University of Notre Dame · State and University Library of Göttingen

  3. OlmoEarth v1.3: Embeddings

    Oct 23, 2026Gabriel Tseng, Favyen Bastani, Patrick Alan Johnson +13Geospatial Representation LearningMultimodal Embedding

    Allen Institute for AI

  4. pplx-embed-v2-late Technical Report

    Oct 21, 2026Markus Krimmel, Louis Milliken, Ekaterina Glazkova +8Multimodal EmbeddingMultimodal IR

    Perplexity AI

  5. Inducing Fourier Geometry: A Smoothness Prior Imposes Approximate Translation Equivariance in Multimodal Stellar Embeddings

    Oct 16, 2026Ilay Kamai, Alex M. Bronstein, Hagai B. PeretsGeometric Representation LearningEquivariant Representation Learning

    Technion - Israel Institute of Technology · ISTA - Institute of Science and Technology Austria

  6. Domain-specific language alignment of multimodal embeddings towards explainable predictions

    Oct 16, 2026Gabrielė Tijūnaitytė, Thijs L. van der Plas, Mohamed-Bachir Belaid +3Multimodal EmbeddingConcept-Based Explanations

    Artificial Intelligence Group, Wageningen University & Research, Wageningen, the Netherlands · Machine Learning in Earth Observation Lab, University of Bonn, Germany · NILU, Climate and environmental research institute, Kjeller, Norway +1

  7. VTrajEmbed: Aligning Text with Video Object Trajectories via Spatiotemporal Embeddings

    Oct 16, 2026Ruizhi Chen, Zining Fan, Yudi Guo +5Multimodal EmbeddingVision-Language Retrieval

    Xingchen AGI Lab, China Telecom Artificial Intelligence Technology (Beijing) Co., Ltd. · Peking University · Nanjing University +1

  8. TWIN-MoE: Two-stage Intra-Expert Imputation Network for Incomplete Multimodal Learning

    Oct 15, 2026Liang Nathan Zheng, Jiyuan Liu, Wenhao Liang +3Missing-Modality LearningMultimodal Learning

    School of Computer Science and Information Technology, Adelaide University, Adelaide, SA 5000, Australia

  9. Stuck on Shuffle: On Attribute-Object Binding in Vision-Language Models

    Oct 15, 2026Netanel Y. Tamir, Michael Green, Issar Tzachor +4Vision-Language RetrievalMultimodal Embedding

    OriginAI · NVIDIA · Tel Aviv University

  10. CIM-Embed: Learning Instruction-Conditioned Multimodal Embeddings for Clustering

    Oct 15, 2026Yinghua Xu, Pengyang Xie, Yanxiang Zhou +4ClusteringMultimodal Embedding

    College of Control Science and Engineering, Zhejiang University · Alibaba International Digital Commerce

  11. What Really Works for Universal Multimodal Embeddings? A Controlled Study of Contrastive Training Strategies

    Oct 10, 2026Duo Li, Haojun Pan, Zuhao Yang +4Multimodal EmbeddingMultimodal Contrastive Learning

    CCDS, Nanyang Technological University · Alibaba Group · Universität Hamburg +1

  12. EarthAlign: Learning Language-Aligned Representations of Earth System Data

    Oct 10, 2026Fadillah Adamsyah Maani, Numan Saeed, Jiaqi Ma +4Cross-Modal Representation LearningMultimodal Embedding

    Mohamed bin Zayed University of Artificial Intelligence · ADIA Lab · University of California San Diego

  13. Syn-Omni: Structured Specialization and Progressive Collaboration for Omnimodal Embeddings

    Oct 8, 2026Youngtaek Oh, Qiyu Wu, Hiromi Wakaki +2Multimodal EmbeddingCross-Modal Representation Learning

    KAIST · Sony Group Corporation

  14. Autoregressive Retriever: Improving Query Understanding from Item Feedback for Universal Multimodal Retrieval

    Oct 8, 2026Jianfei Zhao, Yifan Wang, Feng Zhang +7Information RetrievalCross-Modal Retrieval

    School of Computer Science and Technology, Beijing Institute of Technology · Zhongguancun Academy · Xiaohongshu +2

  15. Enhancing Long-Video VLM Embeddings with Query-Aware Streaming Latent Reasoning

    Oct 4, 2026Haozhe Chi, Song Jin, Yang Jin +1Multimodal EmbeddingVision-Language Retrieval

    Peking University · Renmin University of China

  16. World Embedding Benchmark

    Oct 2, 2026Yiqi Liu, Ruifeng Yuan, Yang Wang +7Physical Reasoning in VideoVideo Representation Learning

    World-Embedding Team

  17. Omni-Embed-Mini: Binding Modalities Without Forgetting via Dense Distillation

    Oct 1, 2026Mohammed Irfan Kurpath, Jaseel Muhammad Kaithakkodan, Sahal Shaji Mullappilly +2Cross-Modal Knowledge DistillationMultimodal Embedding

    Mohamed Bin Zayed University of Artificial Intelligence (MBZUAI)

  18. Mutual Equilibrium: Multimodal Representation Learning through Reciprocal Feedback

    Sep 30, 2026Ho-min Park, Byungkon KangMultimodal GroundingMultimodal Embedding

    Data Science Center, Texas Children’s Hospital Baylor College of Medicine Houston, Texas 77030, USA · Department of Computer Science SUNY Korea Incheon, Republic of Korea

  19. GATE-ST: Gene-Aware Text-image Encoder for Spatial Transcriptomics

    Sep 30, 2026Lucas Ni, Jian Luo, Wentao Huang +1Spatial TranscriptomicsMultimodal Embedding

    Ridge High School Basking Ridge, United States · Stony Brook University Stony Brook, United States

  20. ResComEmb: Effective and Efficient Multimodal Embedding via Residual Homogeneity Compression

    Sep 29, 2026Zijing Cai, Yuzhe Wang, Jingxian Zhu +2Multimodal IRMultimodal Embedding

    University of Science and Technology of China · Hefei University of Technology · National University of Singapore

  21. Binding Multiple Modalities via Multimodal Wasserstein Barycenter

    Sep 27, 2026Xiaole Tang, Jiayi Xu, Xiang Gu +2Cross-Modal AlignmentMultimodal Embedding

    Xi’an Jiaotong University

  22. Learning Multimodal Embeddings with Evidence-Aligned Readout

    Sep 27, 2026Zirong Chen, Fuda Ye, Enjun Du +8Multimodal Large Language ModelsMultimodal Embedding

    The Hong Kong University of Science and Technology (Guangzhou) · Tencent Yuanbao · Tsinghua University +3

  23. VaME: Exploring Variational Latent Reasoning for Multimodal Embeddings

    Sep 27, 2026Peixi Wu, Mingzhou Jiang, Feipeng Ma +11Multimodal IREfficient Multimodal Inference

    University of Science and Technology of China · Tsinghua University · Kuaishou +1

  24. BAT-CLIP: Trimodal Alignment of Brain, Audio and Text

    Sep 25, 2026Suhyun Kim, Jinmo Han, Danny Dongyeop Han +9Cross-Modal AlignmentMultimodal Embedding

    Yonsei University, Seoul, Republic of Korea · Seoul National University, Seoul, Republic of Korea · Dartmouth College, Hanover, NH, USA

  25. LAYERSCOPE: A Layerwise Characterization of Video and Multimodal Learned Representations

    Sep 23, 2026Sandra Arcos-Holzinger, Debashish Chakraborty, Rohita Mocharla +7Representation GeometryRepresentation Learning

    Johns Hopkins University · University of Melbourne · Human Language Technology Center of Excellence +2

  26. Ovis-Embedding: Pushing the Frontiers of Universal Omni-Modal Embeddings

    Sep 21, 2026Ovis-Embedding TeamUnified Multimodal ModelsMultimodal Embedding

    Alibaba Token Hub, Alibaba Group

  27. SCOUT: Sim-to-Real Text-Based Person Retrieval by Embedding-Space Prediction over Frozen Video Features

    Sep 16, 2026Abdarahmane Traoré, Andy Couturier, Éric HervetText-Based Person SearchSynthetic-to-Real Domain Adaptation

    Embia, Computer Science Department, Faculty of Science, Université de Moncton, Moncton, NB, Canada

  28. Multi-modal Knowledge Preserving Adapter for Embedding Backward Compatibility

    Sep 15, 2026Jaeseok Byun, Gukyeong Kwon, Han-Kai Hsu +4Multimodal Large Language ModelsAdapter Tuning

    1Seoul National University, South Korea · 2Amazon AGI, USA

  29. Hypergraph-Regularized Gramian Volumes for Multimodal Retrieval

    Sep 14, 2026Anindya Nag, Ambuj Mehrish, Sebastiano VasconMultimodal IRCross-Modal Alignment

    Department of Environmental Science, Informatics and Statistics Ca’ Foscari University of Venice, Italy

  30. Reason What Matters: Retrieval-Grounded Reasoning for Universal Multimodal Embeddings

    Sep 14, 2026Mingzhou Jiang, Peixi Wu, Hang Cheng +7Multimodal IREfficient Multimodal Inference

    Tsinghua Shenzhen International Graduate School, Tsinghua University · School of Artificial Intelligence and Data Science, USTC · Kuaishou Technology +1