Multimodal Embedding

Ask about this topic

Overview

Multimodal embedding studies how to represent different kinds of data so that related items are close in a shared space, enabling tasks such as cross-modal search, recommendation, and classification. Common approaches include contrastive dual encoders, instruction-tuned multimodal language models, and alignment methods based on graphs or optimal transport; multi-vector and late-interaction representations can retain finer detail than a single vector. Benchmarks such as MMEB, ViDoRe, and audio- and video-retrieval suites test these models across tasks and modalities, while newer methods explore compressed representations and latent reasoning to improve quality without costly inference. A central challenge is balancing broad, zero-shot capability with fine-grained, task-specific information, while keeping representations efficient and robust when modalities are missing or unevenly informative.

Latest papers

CardsList

160 papers19 papers in the last four weeks, up 6% on the four weeks before. 0.1% of all new papers.

  1. Combating Visual Neglect and Semantic Drift in Large Multimodal Models for Enhanced Cross-Modal Retrieval

    Apr 28, 2026Guosheng Zhang, Linkai Liu, Keyao Wang +3Multimodal GroundingCross-Modal Alignment

    Baidu Inc.

  2. Robust Audio-Text Retrieval via Cross-Modal Attention and Hybrid Loss

    Apr 25, 2026Meizhu Liu, Matthew Rowe, Amit Agarwal +8Multimodal RobustnessMultimodal Embedding

  3. Beyond Chain-of-Thought: Rewrite as a Universal Interface for Generative Multimodal Embeddings

    Apr 24, 2026Peixi Wu, Ke Mei, Feipeng Ma +15Cross-Modal AlignmentMultimodal Embedding

    WeChat Vision, Tencent Inc. · Zhejiang University · Tsinghua University +1

  4. FLiP: Towards understanding and interpreting multimodal multilingual sentence embeddings

    Apr 20, 2026Santosh Kesiraju, Bolaji Yusuf, Šimon Sedláček +2Linear ProbingSentence Embeddings

  5. mEOL: Training-Free Instruction-Guided Multimodal Embedder for Vector Graphics and Image Retrieval

    Apr 18, 2026Kyeong Seon Kim, Baek Seong-Eun, Lee Jung-Mok +1Multimodal IRMultimodal Large Language Models

    1KAIST · 2POSTECH

  6. SIMMER: Cross-Modal Food Image--Recipe Retrieval via MLLM-Based Embedding

    Apr 17, 2026Keisuke Gomi, Keiji YanaiCross-Modal LearningImage-Text Retrieval

    The University of Electro-Communications

  7. SLQ: Bridging Modalities via Shared Latent Queries for Retrieval with Frozen MLLMs

    Apr 15, 2026Haoran Lou, Ziyan Liu, Chunxiao Fan +6Multimodal IRCross-Modal Retrieval

    School of Electronic Engineering, Beijing University of Posts and Telecommunications, China · Xiaohongshu Inc., China

  8. UNIGEOCLIP: Unified Geospatial Contrastive Learning

    Apr 13, 2026Guillaume Astruc, Eduard Trulls, Jan Hosang +2Geospatial Representation LearningMultimodal Embedding

    LASTIG, Univ Gustave Eiffel, IGN, ENSG, France · CNES, France · LIGM, CNRS, Univ Gustave Eiffel, ENPC, Institut Polytechnique de Paris, Marne-la-Vallée, France +1

  9. Bottleneck Tokens for Unified Multimodal Retrieval

    Apr 13, 2026Siyu Sun, Jing Ren, Zhaohe Liao +8Multimodal IRMultimodal Embedding

    Shanghai Jiao Tong University · ByteDance

  10. EmergentBridge: Improving Zero-Shot Cross-Modal Transfer in Unified Multimodal Embedding Models

    Apr 13, 2026Jincheng Xie, Xingchen Xiao, Runheng Liu +3Cross-Modal LearningMultimodal Embedding

    School of Computer Science and Technology, Beijing Institute of Technology, Beijing, China · JD iCity, JD Technology, JD Intelligent Cities Research, Beijing, China

  11. MMEmb-R1: Reasoning-Enhanced Multimodal Embedding with Pair-Aware Selection and Adaptive Control

    Apr 7, 2026Yuchi Wang, Dingkang Yang, Haiyang Yu +5Multimodal EmbeddingMultimodal Reasoning

    MMLab, The Chinese University of Hong Kong · ByteDance Inc. · Fudan University

  12. Hierarchical Co-Embedding of Font Shapes and Impression Tags

    Apr 5, 2026Yugo Kubota, Kaito Shiku, Seiichi UchidaMultimodal EmbeddingVision-Language Retrieval

    Kyushu University, Fukuoka, Japan

  13. PLUME: Latent Reasoning Based Universal Multimodal Embedding

    Apr 2, 2026Chenwei He, Xiangzhao Hao, Tianyu Yang +6Multimodal EmbeddingMultimodal IR

    Southeast University · Institute of Automation, Chinese Academy of Sciences · University of Chinese Academy of Sciences

  14. Assessing Multimodal Chronic Wound Embeddings with Expert Triplet Agreement

    Mar 31, 2026Fabian Kabus, Julia Hindel, Jelena Bratulić +6Medical Image RetrievalMultimodal Embedding

    Institute of Medical Biometry and Statistics (IMBI), Medical Faculty and Medical Center, University of Freiburg · Department of Computer Science, Faculty of Engineering, University of Freiburg · Department of Dermatology, Medical Faculty and Medical Center, University of Freiburg

  15. Is the Modality Gap a Bug or a Feature? A Robustness Perspective

    Mar 30, 2026Rhea Chowers, Oshri Naparstek, Udi Barzelay +1Modality Gap in VLMsMultimodal Embedding

    Hebrew University, Jerusalem, Israel · IBM Research, Haifa, Israel

  16. Gaussian Joint Embeddings For Self-Supervised Representation Learning

    Mar 26, 2026Yongchao HuangRepresentation LearningMultimodal Embedding

  17. BioVITA: Biological Dataset, Model, and Benchmark for Visual-Textual-Acoustic Alignment

    Mar 25, 2026Risa Shinoda, Kaede Shiohara, Nakamasa Inoue +3Cross-Modal AlignmentMultimodal Embedding

    The University of Osaka · The University of Tokyo · Institute of Science Tokyo +1

  18. VLM2Rec: Resolving Modality Collapse in Vision-Language Model Embedders for Multimodal Sequential Recommendation

    Mar 18, 2026Junyoung Kim, Woojoo Kim, Wonbin Kweon +3Multimodal RecommendationMultimodal Embedding

    Pohang University of Science and Technology · University of Illinois Urbana-Champaign

  19. Anchored Alignment: Preventing Positional Collapse in Multimodal Recommender Systems

    Mar 13, 2026Yonghun Jeong, David Yoon Suk Kang, Yeon-Chang LeeMultimodal RecommendationCross-Modal Alignment

    UNIST, Ulsan, Korea · Chungbuk National University, Cheongju, Korea

  20. VLM4Rec: Multimodal Semantic Representation for Recommendation with Large Vision-Language Models

    Mar 13, 2026Ty Valencia, Burak Barlas, Varun Singhal +2Multimodal RecommendationMultimodal Embedding

    University of Southern California Los Angeles, USA

  21. Multimodal Graph Representation Learning with Dynamic Information Pathways

    Mar 10, 2026Xiaobin Hong, Mingkai Lin, Xiaoli Wang +2Multimodal Graph LearningMultimodal Embedding

    State Key Laboratory for Novel Software Technology, Nanjing University · Nanjing Forest University, College of Information Science and Technology · Zhejiang Provincial Seaport Investment & Operation Group Co. Ltd

  22. TSEmbed: Unlocking Task Scaling in Universal Multimodal Embeddings

    Mar 5, 2026Yebo Wu, Feng Liu, Ziwei Xie +4Multimodal EmbeddingMultimodal Disentangled Representation Learning

    State Key Laboratory of IOTSC, University of Macau · OPPO Research Institute

  23. PinCLIP: Large-scale Foundational Multimodal Representation at Pinterest

    Mar 3, 2026Josh Beal, Eric Kim, Jinfeng Rao +3Multimodal EmbeddingMultimodal IR

    Pinterest Inc., San Francisco, CA, USA

  24. TRACE: Task-Adaptive Reasoning and Representation Learning for Universal Multimodal Retrieval

    Mar 3, 2026Xiangzhao Hao, Shijie Wang, Tianyu Yang +3Multimodal IRMultimodal Embedding

    Institute of Automation, Chinese Academy of Sciences · University of Chinese Academy of Sciences

  25. Efficient and High-Fidelity Omni Modality Retrieval

    Mar 2, 2026Chuong Huynh, Manh Luong, Abhinav ShrivastavaMultimodal IRCross-Modal Retrieval

    University of Maryland, College Park, USA · Monash University, Australia

  26. Beyond the Grid: Layout-Informed Multi-Vector Retrieval with Parsed Visual Document Representations

    Mar 2, 2026Yibo Yan, Mingdong Ou, Yi Cao +6Visual Document RetrievalMulti-Vector Retrieval

    The Hong Kong University of Science and Technology (Guangzhou) · Alibaba Cloud Computing · The Hong Kong University of Science and Technology

  27. IDProxy: CTR Prediction with Multimodal LLMs for Cold-Start Recommendation at Xiaohongshu

    Mar 2, 2026Yubin Zhang, Haiming Xu, Guillaume Salha-Galvan +6Cold-Start RecommendationMultimodal Embedding

    Xiaohongshu Inc. Shanghai, China · Shanghai Jiao Tong University Shanghai, China · Fudan University Shanghai, China

  28. Reconstructing Content with Collaborative Attention for Universal Multimodal Representation Learning

    Mar 2, 2026Jiahan Chen, Da Li, Hengran Zhang +6Multimodal EmbeddingMultimodal Learning

    State Key Laboratory of AI Safety, Institute of Computing Technology, Chinese Academy of Sciences, University of Chinese Academy of Sciences, Beijing, China · Baidu Inc., Beijing, China

  29. SOTAlign: Semi-Supervised Alignment of Unimodal Vision and Language Models via Optimal Transport

    Feb 26, 2026Simon Roschmann, Paul Krzakala, Sonia Mazelet +2Cross-Modal LearningMultimodal Embedding

    Helmholtz Munich · Technical University of Munich · Munich Center for Machine Learning +3

  30. Unlocking Multimodal Document Intelligence: From Current Triumphs to Future Frontiers of Visual Document Retrieval

    Feb 23, 2026Yibo Yan, Jiahao Huo, Guanbo Feng +12Multimodal RAGMultimodal IR

    Hong Kong University of Science and Technology (Guangzhou) · Alibaba Cloud Computing · Hong Kong University of Science and Technology +1