Multimodal Embedding

Follow this topic Ask about this topic

Multimodal embedding studies how to represent different kinds of data so that related items are close in a shared space, enabling tasks such as cross-modal search, recommendation, and classification. Common approaches include contrastive dual encoders, instruction-tuned multimodal language models, and alignment methods based on graphs or optimal transport; multi-vector and late-interaction representations can retain finer detail than a single vector. Benchmarks such as MMEB, ViDoRe, and audio- and video-retrieval suites test these models across tasks and modalities, while newer methods explore compressed representations and latent reasoning to improve quality without costly inference. A central challenge is balancing broad, zero-shot capability with fine-grained, task-specific information, while keeping representations efficient and robust when modalities are missing or unevenly informative.

Latest papers

160 papers19 papers in the last four weeks, up 6% on the four weeks before. 0.1% of all new papers.

CREM: Compression-Driven Representation Enhancement for Multimodal Retrieval and Comprehension
Feb 22, 2026Lihao Liu, Yan Wang, Biao Yang +10Multimodal Large Language Models Cross-Modal Retrieval
Tsinghua University · Kuaishou Technology
Canonicalizing Multimodal Contrastive Representation Learning
Feb 19, 2026Sharut Gupta, Sanyam Kansal, Stefanie Jegelka +2Multimodal Embedding Representation Alignment
Aalto University, YaiYai Ltd
Orthogonalized Multimodal Contrastive Learning with Asymmetric Masking for Structured Representations
Feb 16, 2026Carolin Cissee, Raneen Younis, Zahra AhmadiMultimodal Embedding Multimodal Contrastive Learning
Peter L. Reichertz Institute for Medical Informatics of TU Braunschweig and Hannover Medical School, Hannover, Germany · Lower Saxony Center for AI and Causal Methods in Medicine (CAIMed), Hannover, Germany
Magic-MM-Embedding: Towards Visual-Token-Efficient Universal Multimodal Embedding with MLLMs
Feb 5, 2026Qi Li, Yanzhe Zhao, Yongxin Zhou +4Efficient Multimodal Inference Multimodal Large Language Models
Honor Device Co., Ltd., China
CausalEmbed: Auto-Regressive Multi-Vector Generation in Latent Space for Visual Document Embedding
Jan 29, 2026Jiahao Huo, Yu Huang, Yibo Yan +7Multimodal Embedding Multi-Vector Retrieval
The Hong Kong University of Science and Technology (Guangzhou) · Alibaba Cloud Computing · The Hong Kong University of Science and Technology +1
VLM2GeoVec: Toward Universal Multimodal Embeddings for Remote Sensing
Dec 12, 2025Emanuel Sánchez Aimar, Gulnaz Zhambulova, Fahad Shahbaz Khan +2Multimodal Understanding Cross-Modal Learning
Linköping University, Sweden
Image Recognition with Vision and Language Embeddings of VLMs
Sep 11, 2025Illia Volkov, Nikita Kisel, Klara Janouskova +1Image Classification VLM Evaluation
Visual Recognition Group, Faculty of Electrical Engineering, Czech Technical University in Prague Prague, Czechia
EGRA:Toward Enhanced Behavior Graphs and Representation Alignment for Multimodal Recommendation
Aug 22, 2025Xiaoxiong Zhang, Xin Zhou, Zhiwei Zeng +2Cross-Modal Alignment Multimodal Embedding
College of Computing and Data Science, Nanyang Technological University, Singapore
Multimodal Representation Alignment for Cross-modal Information Retrieval
Jun 10, 2025Fan Xu, Luis A. LeivaCross-Modal Learning Multimodal IR
aDepartment of Computer Science, University of Luxembourg, 6, avenue de la Fonte, Esch-sur-Alzette, L-4364, Luxembourg
X-SG $^2$ S: Safe and Generalizable Gaussian Splatting with X-dimensional Watermarks
Feb 13, 2025Zihang Cheng, Wentao Bao, Huiping Zhuang +63D Gaussian Splatting Multimodal Embedding
South China University of Technology · Shenzhen MSU-BIT University · Peking University +1

Feb 22, 2026J/K move · Enter open · S save

CREM: Compression-Driven Representation Enhancement for Multimodal Retrieval and Comprehension

Lihao Liu, Yan Wang, Biao Yang, Da Li+9

Tsinghua University · Kuaishou Technology

Multimodal Large Language Models Cross-Modal Retrieval Multimodal Embedding Cross-Modal Representation Learning Multimodal Understanding Multimodal Token Compression

Open paper AskPDF arXiv

Multimodal Large Language Models (MLLMs) have shown remarkable success in comprehension tasks such as visual description and visual question answering. However, their direct application to embedding-based tasks like retrieval remains challenging due to the discrepancy between output formats and optimization objectives. Previous approaches often employ contrastive fine-tuning to adapt MLLMs for retrieval, but at the cost of losing their generative capabilities. We argue that both generative and embedding tasks fundamentally rely on shared cognitive mechanisms, specifically cross-modal representation alignment and contextual comprehension. To this end, we propose CREM (Compression-driven Representation Enhanced Model), with a unified framework that enhances multimodal representations for retrieval while preserving generative ability. Specifically, we introduce a compression-based prompt design with learnable chorus tokens to aggregate multimodal semantics and a compression-driven training strategy that integrates contrastive and generative objectives through compression-aware attention. Extensive experiments demonstrate that CREM achieves state-of-the-art retrieval performance on MMEB while maintaining strong generative performance on multiple comprehension benchmarks. Our findings highlight that generative supervision can further improve the representational quality of MLLMs under the proposed compression-driven paradigm.