Multimodal IR

IR: Information Retrieval

Ask about this topic

Overview

Multimodal information retrieval studies how to find relevant items when queries and collections contain text, images, video, audio, or structured documents. Common approaches learn shared representations with contrastive training, use multi-vector or sparse indexes for fine-grained matching, and add rerankers or agents that gather and verify evidence. Applications range from visual document and product search to medical, news, and video retrieval, with benchmarks testing both cross-modal matching and multi-step search. A central tension is how to capture detailed, grounded evidence and reasoning without making indexing, storage, or inference too costly; current systems still struggle with long contexts, noisy inputs, and complex constraints.

Latest papers

CardsList

128 papers9 papers in the last four weeks, level with the four weeks before. 0.1% of all new papers.

  1. PailitaoMMSearch2.0 Technical Report: Scaling Generative Retrieval to Unified End-to-End Multimodal Search

    Oct 27, 2026Xiaohan Ye, Zihan Gong, Jingqian Zhao +9Multimodal IRGenerative Retrieval

    Taobao & Tmall Group of Alibaba

  2. MTA-Zero: Self-Improving Multimodal Deep Search via Online QA-Solver Co-training

    Oct 24, 2026Xiangyu Peng, Jiaxin Zhang, An Yan +4Multimodal Search AgentsCurriculum Learning

  3. A Broader Look into MLLM's Retrieval Heads

    Oct 23, 2026Jia Wang, Yuqing Li, Jiangnan Li +2Multimodal Large Language ModelsMultimodal IR

    Institute of Information Engineering, Chinese Academy of Sciences · School of Cyber Security, University of Chinese Academy of Sciences · Tencent

  4. pplx-embed-v2-late Technical Report

    Oct 21, 2026Markus Krimmel, Louis Milliken, Ekaterina Glazkova +8Multimodal EmbeddingMultimodal IR

    Perplexity AI

  5. Enhancing RAG-based Medical VQA with Gated Multimodal Representations

    Oct 16, 2026Daniela Jimenez-Pano, Hugo Jair Escalante, Delia Irazú Hernández-FaríasGated Multimodal FusionMedical VQA

    LabTL - INAOE, Puebla, Mexico · Institute for AAII, UTEP, El Paso, TX, USA · Department of Computer Science, UTEP, El Paso, TX, USA

  6. MMSearch-OPD: Easy Made Hard, Hard Made Learnable for Long-Horizon Multimodal Search Agents

    Oct 15, 2026Huanyao Zhang, Jiepeng Zhou, Jiaye Lin +9Multimodal Search AgentsMultimodal IR

    Peking University · HKUST(GZ) · Tsinghua University +3

  7. Cascaded Retrieval with Cross-Modal Query Refinement for Efficient Visual Document Search

    Oct 14, 2026Diana Turmakhan, Aseem Srivastava, Vivek Gupta +2Information RetrievalVisual Document Retrieval

    Mohamed bin Zayed University of Artificial Intelligence · Arizona State University · Ubiquitous Knowledge Processing Lab (UKP Lab), Department of Computer Science

  8. HyperBrowseComp: A Multilingual and Multimodal Stress Test for Web-Browsing Agents

    Oct 2, 2026Alham Fikri Aji, Faiz Rizki Ramadhan, Zayd M. K. Zuhri +14Web Agent BenchmarksAgent Benchmarks

    Mohamed bin Zayed University of Artificial Intelligence · Mila – Quebec Artificial Intelligence Institute · Inception AI +2

  9. AiSearch: Interactive Multi-Modal Search with VLMs

    Oct 1, 2026Ali Koksal, Mei Chee Leong, Vicky Sintunata +2Multimodal IRVision-Language Models

    Institute of Advanced Intelligence and Computing (IAIC), Agency for Science, Technology and Research (A*STAR), Singapore

  10. ResComEmb: Effective and Efficient Multimodal Embedding via Residual Homogeneity Compression

    Sep 29, 2026Zijing Cai, Yuzhe Wang, Jingxian Zhu +2Multimodal IRMultimodal Embedding

    University of Science and Technology of China · Hefei University of Technology · National University of Singapore

  11. VaME: Exploring Variational Latent Reasoning for Multimodal Embeddings

    Sep 27, 2026Peixi Wu, Mingzhou Jiang, Feipeng Ma +11Multimodal IREfficient Multimodal Inference

    University of Science and Technology of China · Tsinghua University · Kuaishou +1

  12. UniK: Universal Knowledge Perception for Digital and Physical AI

    Sep 21, 2026Nirmit Desai, Kunal Sawarkar, Aditya Mahakali +3Multimodal RAGMultimodal IR

    AIntropy AI

  13. Integrating knowledge from case reports: a medical ontology based multimodal information system with structured summary

    Sep 17, 2026Shuyu Guo, Lan Huang, Yichen Liu +2Multimodal IRRare Disease Diagnosis

    College of Computer Science and Technology Jilin University Changchun · College of Software Jilin University Changchun · Department of Laboratory Medicine Sichuan Provincial People’s Hospital, School of Medicine, University of Electronic Science and Technology of China Chengdu

  14. Less Is More: Graph-free Multimodal RAG via Multi-signal Late Fusion

    Sep 16, 2026Tithi Rakshit, Hongkuan Zhou, Lavdim Halilaj +1Late FusionMultimodal RAG

    University of Tübingen, Tübingen, Germany · Corporate Research, Robert Bosch GmbH, Renningen, Germany · University of Stuttgart, Stuttgart, Germany

  15. RegRet: Enhancing Region-Level Retrieval in Large Multimodal Models

    Sep 15, 2026Xun Liang, Honghui Yang, Weihang Pan +6Multimodal IRMultimodal Large Language Models

    State Key Lab of CAD&CG, Zhejiang University · Xiaohongshu Inc. · School of Software Technology, Zhejiang University

  16. Hypergraph-Regularized Gramian Volumes for Multimodal Retrieval

    Sep 14, 2026Anindya Nag, Ambuj Mehrish, Sebastiano VasconMultimodal IRCross-Modal Alignment

    Department of Environmental Science, Informatics and Statistics Ca’ Foscari University of Venice, Italy

  17. Reason What Matters: Retrieval-Grounded Reasoning for Universal Multimodal Embeddings

    Sep 14, 2026Mingzhou Jiang, Peixi Wu, Hang Cheng +7Multimodal IREfficient Multimodal Inference

    Tsinghua Shenzhen International Graduate School, Tsinghua University · School of Artificial Intelligence and Data Science, USTC · Kuaishou Technology +1

  18. ReGround: Grounding Reviewer Comments in Multimodal Evidence

    Sep 10, 2026Serwar Basch, Lizhen Qu, Iryna GurevychMultimodal IRScholarly Peer Review

    Ubiquitous Knowledge Processing Lab (UKP Lab), Department of Computer Science and Hessian Center for AI (hessian.AI), TU Darmstadt · Department of Data Science & AI, Monash University, Australia

  19. MIDR: Enrichment-Augmented Indexing for Multimodal Document Retrieval

    Sep 1, 2026Debanjan Mahata, Atharva Tendle, Daniel Preotiuc-Pietro +2Multimodal IRVisual Document Retrieval

    Bloomberg, NYC, USA

  20. SnapBench: Benchmarking Snap-and-Ask Multimodal Retrieval for Mobile Interactions

    Aug 30, 2026Zirong Chen, Fuda Ye, Kuan Zhang +7Multimodal IRInformation Retrieval

    The Hong Kong University of Science and Technology (Guangzhou) · Tencent Yuanbao · Tsinghua University +2

  21. WeAgent-MMSearch: Native Text-Vision Interaction for Multimodal Search Agents

    Aug 28, 2026Zongkai Liu, Hui Zhang, Liqiang Niu +7Multimodal IRMultimodal Search Agents

    Weixin AI, Tencent · Sun Yat-sen University

  22. Omni-Interactive Universal Embedder

    Aug 27, 2026Wei-Yao Wang, Kazuya Tateishi, Shuyang Cui +4Multimodal EmbeddingMultimodal IR

    Sony Group Corporation · Sony AI

  23. Is Reasoning Always Useful? Rethinking Reasoning Utility in Universal Multimodal Embeddings

    Aug 26, 2026Wenxiao Fan, Jingling Fu, Luohang Liu +6Multimodal EmbeddingMultimodal IR

    School of Computer Science and Technology, Beijing Institute of Technology · Institute of Artificial Intelligence and Robotics, Xi’an Jiaotong University

  24. AdaptiveEmbed: Sample-Adaptive Multi-Vector Representation for Multimodal Retrieval

    Aug 26, 2026Xinze Liu, Lei Yang, Dayan Wu +7Multimodal IRMulti-Vector Retrieval

    Institute of Information Engineering, Chinese Academy of Sciences · School of Cyber Security, University of Chinese Academy of Sciences

  25. Multi-Modal Semantic Expansion with Constrained LLM Reranking for Conversational Music Recommendation

    Aug 24, 2026Naman Garg, Sarika Jain, George FazekasMusic RecommendationMultimodal Recommendation

    National Institute of Technology Kurukshetra India · Queen Mary University of London United Kingdom

  26. EviRank: Structured Relevance Evidence for Multimodal Image Re-ranking

    Aug 21, 2026Enjun Du, Siyi Liu, Zirong Chen +8Multimodal RerankingMultimodal IR

    The Hong Kong University of Science and Technology (Guangzhou) · Tencent Yuanbao · The University of Hong Kong

  27. KoViDoRe: A Benchmark for Korean Visual Document Retrieval

    Aug 21, 2026Yongbin Choi, Yongwoo Song, Mujeen SungVisual Document RetrievalMultimodal IR

    Kyung Hee University

  28. When Images Look Right and Retrieve Wrong: Coverage-Guided Cross-Scale Re-Indexing for Knowledge-Faithful Generative Perception

    Aug 21, 2026Guangyuan Dong, Chuang Liu, Haoyu Wang +10Multimodal IRRemote Sensing

    1National University of Singapore · 2Wuhan University · 4Nankai University +6

  29. VisDocAgentBench: Benchmarking Agents for Visually Rich Document Retrieval

    Aug 18, 2026Lexiang Hu, Yanzhao Zhang, Mingxin Li +5Visual Document RetrievalAgentic Retrieval

    State Key Lab of General AI, School of Intelligence Science and Technology, Peking University

  30. Hypergraph-based Multimodal Retrieval-Augmented Generation with Incremental Refinement

    Aug 17, 2026Shenao Chen, Yidan Xu, Xiangmin Han +5Multimodal RAGMultimodal IR

    Hangzhou Dianzi University Hangzhou, China · Beijing University of Technology Beijing, China · Xi’an Jiaotong University Xi’an, China +1