Multimodal Embedding
Overview
Multimodal embedding studies how to represent different kinds of data so that related items are close in a shared space, enabling tasks such as cross-modal search, recommendation, and classification. Common approaches include contrastive dual encoders, instruction-tuned multimodal language models, and alignment methods based on graphs or optimal transport; multi-vector and late-interaction representations can retain finer detail than a single vector. Benchmarks such as MMEB, ViDoRe, and audio- and video-retrieval suites test these models across tasks and modalities, while newer methods explore compressed representations and latent reasoning to improve quality without costly inference. A central challenge is balancing broad, zero-shot capability with fine-grained, task-specific information, while keeping representations efficient and robust when modalities are missing or unevenly informative.
Latest papers
160 papers19 papers in the last four weeks, up 6% on the four weeks before. 0.1% of all new papers.