Multimodal IR

IR: Information Retrieval

Multimodal information retrieval studies how to find relevant items when queries and collections contain text, images, video, audio, or structured documents. Common approaches learn shared representations with contrastive training, use multi-vector or sparse indexes for fine-grained matching, and add rerankers or agents that gather and verify evidence. Applications range from visual document and product search to medical, news, and video retrieval, with benchmarks testing both cross-modal matching and multi-step search. A central tension is how to capture detailed, grounded evidence and reasoning without making indexing, storage, or inference too costly; current systems still struggle with long contexts, noisy inputs, and complex constraints.

Latest papers

128 papers9 papers in the last four weeks, level with the four weeks before. 0.1% of all new papers.

CodeMMR: Bridging Natural Language, Code, and Image for Unified Retrieval
Apr 17, 2026Jiahui Geng, Qing Li, Fengyu Cai +1Multimodal IR Unified Multimodal Models
MBZUAI, UAE · Linköping University, Sweden · University of Groningen, Netherlands +1
SLQ: Bridging Modalities via Shared Latent Queries for Retrieval with Frozen MLLMs
Apr 15, 2026Haoran Lou, Ziyan Liu, Chunxiao Fan +6Multimodal IR Cross-Modal Retrieval
School of Electronic Engineering, Beijing University of Posts and Telecommunications, China · Xiaohongshu Inc., China
MERRIN: A Benchmark for Multimodal Evidence Retrieval and Reasoning in Noisy Web Environments
Apr 15, 2026Han Wang, David Wan, Hyunji Lee +6AI Agent Benchmarks Multimodal Search Agents
UNC Chapel Hill · Virginia Tech · University of Texas at Austin
Human-Inspired Context-Selective Multimodal Memory for Social Robots
Apr 13, 2026Hangyeol Kang, Slava Voloshynovskiy, Nadia Magnenat ThalmannMultimodal Memory Social Robotics
Department of Computer Science, University of Geneva, Geneva, Switzerland · MIRALab, University of Geneva, Geneva, Switzerland
Bottleneck Tokens for Unified Multimodal Retrieval
Apr 13, 2026Siyu Sun, Jing Ren, Zhaohe Liao +8Multimodal IR Multimodal Embedding
Shanghai Jiao Tong University · ByteDance
Task-Adaptive Retrieval over Agentic Multi-Modal Web Histories via Learned Graph Memory
Apr 9, 2026Saman Forouzandeh, Kamal Berahmand, Mahdi JaliliAdaptive Retrieval Multimodal IR
School of Engineering, Royal Melbourne Institute of Technology University, Melbourne, Australia
BRIDGE: Multimodal-to-Text Retrieval via Reinforcement-Learned Query Alignment
Apr 8, 2026Mohamed Darwish Mounis, Mohamed Mahmoud, Shaimaa Sedek +4Multimodal IR Cross-Modal Retrieval
High institute for computer & information systems · Chungbuk National University · Assiut University +1
PLUME: Latent Reasoning Based Universal Multimodal Embedding
Apr 2, 2026Chenwei He, Xiangzhao Hao, Tianyu Yang +6Multimodal Embedding Multimodal IR
Southeast University · Institute of Automation, Chinese Academy of Sciences · University of Chinese Academy of Sciences
BioVITA: Biological Dataset, Model, and Benchmark for Visual-Textual-Acoustic Alignment
Mar 25, 2026Risa Shinoda, Kaede Shiohara, Nakamasa Inoue +3Cross-Modal Alignment Multimodal Embedding
The University of Osaka · The University of Tokyo · Institute of Science Tokyo +1
ForeSea: AI Forensic Search with Multi-modal Queries for Video Surveillance
Mar 24, 2026Hyojin Park, Yi Li, Janghoon Cho +8Video Surveillance Multimodal IR
Qualcomm AI Research, San Diego, CA, USA · Kyunghee University, Gyeonggi, South Korea
ReBOL: Retrieval via Bayesian Optimization with Batched LLM Relevance Observations and Query Reformulation
Mar 20, 2026Anton Korikov, Scott SannerInformation Retrieval LLM Reranking
University of Toronto
Evo-Retriever: LLM-Guided Curriculum Evolution with Viewpoint-Pathway Collaboration for Multimodal Document Retrieval
Mar 17, 2026Weiqing Li, Jinyue Guo, Yaqi Wang +4Multimodal IR Cross-Modal Retrieval
Alibaba Cloud Computing
VisBrowse-Bench: Benchmarking Visual-Native Search for Multimodal Browsing Agents
Mar 17, 2026Zhengbo Zhang, Jinbo Su, Zhaowen Zhou +14Multimodal Search Agents AI Agent Benchmarks
CASIA · Ant Digital Technologies, Ant Group · RUC +4
AMES: Approximate Multi-modal Enterprise Search via Late Interaction Retrieval
Mar 13, 2026Tony Joseph, Carlos Pareja, David Lopes Pegna +1Multimodal IR Information Retrieval
Apple, Cupertino, CA, USA
LITTA: Late-Interaction and Test-Time Alignment for Visually-Grounded Multimodal Retrieval
Mar 10, 2026Seonok KimMultimodal IR Visual Document Retrieval
Mazelone
DEO: Training-Free Direct Embedding Optimization for Negation-Aware Retrieval
Mar 10, 2026Taegyeong Lee, Jiwon Park, Seunghyun Hwang +1Dense Retrieval Multimodal IR
Department of Industrial Engineering, Hanyang University · Department of Applied Data Science, Sungkyunkwan University
MultiHaystack: Benchmarking Multimodal Retrieval and Reasoning over 40K Images, Videos, and Documents
Mar 5, 2026Dannong Xu, Zhongyu Yang, Jun Chen +6Multimodal IR Multimodal Model Evaluation
INSAIT, Sofia University “St. Kliment Ohridski” · Lanzhou University · King Abdullah University of Science and Technology +3
NaiLIA: Multimodal Nail Design Retrieval Based on Dense Intent Descriptions and Palette Queries
Mar 5, 2026Kanon Amemiya, Daichi Yashima, Kei Katsumata +4Image Retrieval Multimodal IR
Keio University
Retrieval or Representation? Reassessing Benchmark Gaps in Multilingual and Visually Rich RAG
Mar 4, 2026Martin Asenov, Kenza Benkirane, Dan Goldwater +1RAG Evaluation Multimodal IR
Parexel AI Labs London, United Kingdom · Parexel AI Labs San Francisco, United-States
PinCLIP: Large-scale Foundational Multimodal Representation at Pinterest
Mar 3, 2026Josh Beal, Eric Kim, Jinfeng Rao +3Multimodal Embedding Multimodal IR
Pinterest Inc., San Francisco, CA, USA
TRACE: Task-Adaptive Reasoning and Representation Learning for Universal Multimodal Retrieval
Mar 3, 2026Xiangzhao Hao, Shijie Wang, Tianyu Yang +3Multimodal IR Multimodal Embedding
Institute of Automation, Chinese Academy of Sciences · University of Chinese Academy of Sciences
LLandMark: A Multi-Agent Framework for Landmark-Aware Multimodal Interactive Video Retrieval
Mar 3, 2026Minh-Chi Phung, Thien-Bao Le, Cam-Tu Tran-Thi +2Multimodal IR Multimodal Search Agents
University of Information Technology, Ho Chi Minh City, Vietnam · Vietnam National University, Ho Chi Minh City, Vietnam
VSearcher: Long-Horizon Multimodal Search Agent via Reinforcement Learning
Mar 3, 2026Ruiyang Zhang, Qianguo Sun, Chao Song +2Multimodal Search Agents Multimodal Tool Use
FST and ICI, University of Macau, China · IDEA
Efficient and High-Fidelity Omni Modality Retrieval
Mar 2, 2026Chuong Huynh, Manh Luong, Abhinav ShrivastavaMultimodal IR Cross-Modal Retrieval
University of Maryland, College Park, USA · Monash University, Australia
Cognitive Prosthetic: An AI-Enabled Multimodal System for Episodic Recall in Knowledge Work
Mar 2, 2026Lawrence Obiuwevwi, Krzysztof J. Rechowicz, Vikas Ashok +2Multimodal Memory Episodic Memory
Old Dominion University Norfolk, Virginia, United States
PhotoBench: Beyond Visual Matching Towards Personalized Intent-Driven Photo Retrieval
Mar 2, 2026Tianyi Xu, Rong Shan, Junjie Wu +11Multimodal IR Image Retrieval
Shanghai Jiao Tong University, Shanghai, China · OPPO, Shenzhen, China
Beyond Global Similarity: Towards Fine-Grained, Multi-Condition Multimodal Retrieval
Mar 1, 2026Xuan Lu, Kangle Li, Haohang Huang +3Multimodal IR Cross-Modal Retrieval
Shanghai Jiao Tong University · Institute of Digital Twin, Eastern Institute of Technology, Ningbo · Ningbo Key Laboratory of Spatial Intelligence and Digital Derivative +1
RETLLM: Training and Data-Free MLLMs for Multimodal Information Retrieval
Feb 25, 2026Dawei Su, Dongsheng WangMultimodal IR Multimodal Reranking
College of Computer Science and Software Engineering, Shenzhen University, Shenzhen, China
Multi-Vector Index Compression in Any Modality
Feb 24, 2026Hanxiang Qin, Alexander Martin, Rohan Jha +3Multi-Vector Retrieval Embedding Compression
Johns Hopkins University, Baltimore, MD, USA
E-MMKGR: A Unified Multimodal Knowledge Graph Framework for E-commerce Applications
Feb 24, 2026Jiwoo Kang, Yeon-Chang LeeMultimodal KGs Multimodal Recommendation
UNIST Ulsan, Korea

Apr 17, 2026J/K move · Enter open · S save

CodeMMR: Bridging Natural Language, Code, and Image for Unified Retrieval

Jiahui Geng, Qing Li, Fengyu Cai, Fakhri Karray

MBZUAI, UAE · Linköping University, Sweden · University of Groningen, Netherlands+1

Multimodal IR Unified Multimodal Models

Open paper AskPDF arXiv

Code search, framed as information retrieval (IR), underpins modern software engineering and increasingly powers retrieval-augmented generation (RAG), improving code discovery, reuse, and the reliability of LLM-based coding. Yet existing code IR models remain largely text-centric and often overlook the visual and structural aspects inherent in programming artifacts such as web interfaces, data visualizations, SVGs, schematic diagrams, and UML. To bridge this gap, we introduce MMCoIR, the first comprehensive benchmark for evaluating multimodal code IR across five visual domains, eight programming languages, eleven libraries, and show the challenge of the task through extensive evaluation. Therefore, we then propose CodeMMR, a unified retrieval model that jointly embeds natural language, code, and images into a shared semantic space through instruction-based multimodal alignment. CodeMMR achieves strong generalization across modalities and languages, outperforming competitive baselines (e.g., UniIR, GME, VLM2Vec) by an average of 10 points on nDCG@10. Moreover, integrating CodeMMR into RAG enhances code generation fidelity and visual grounding on unseen code generation tasks, underscoring the potential of multimodal retrieval as a core enabler for next-generation intelligent programming systems. Datasets are available at HuggingFace.