Multimodal IR

IR: Information Retrieval

Overview

Multimodal information retrieval studies how to find relevant items when queries and collections contain text, images, video, audio, or structured documents. Common approaches learn shared representations with contrastive training, use multi-vector or sparse indexes for fine-grained matching, and add rerankers or agents that gather and verify evidence. Applications range from visual document and product search to medical, news, and video retrieval, with benchmarks testing both cross-modal matching and multi-step search. A central tension is how to capture detailed, grounded evidence and reasoning without making indexing, storage, or inference too costly; current systems still struggle with long contexts, noisy inputs, and complex constraints.

Latest papers

CardsList

135 papers10 papers in the last four weeks, up 25% on the four weeks before. 0.1% of all new papers.

  1. HiVe-Mem: Interleaving Horizontal Memory Expansion and Vertical Evidence Composition for Multimodal Agents

    Nov 9, 2026Haozhen Zhang, Bohan Liu, Jianzhu Bao +5Multimodal MemoryEpisodic Memory for LLM Agents

  2. TRIAD: Multimodal Retrieval across Abstraction, Constraints, and State

    Nov 9, 2026Xiangyang Wu, Chen Ju, Xu Chen +4Multimodal IRInformation Retrieval

  3. STARE: Scene Text-Aware multimodal Retrieval for Everyday objects

    Nov 4, 2026Kento Tokura, Ryosuke Korekata, Daichi Yashima +4Multimodal IRRobotic Object Search

  4. GeoDIVE: Geospatial Descent Informed by Value Estimation

    Nov 3, 2026Hamza Iqbal, Dan Cher, Eric Xing +4Image RetrievalMultimodal IR

  5. MedHEB: Benchmarking Medical Embeddings Across Heterogeneous Clinical Evidence

    Nov 2, 2026Yingshu Li, Shaoyang Zhou, Zhanyu Wang +6Multimodal IRMultimodal Embedding

  6. Parameter-Efficient Fine-Tuning of Transformer Encoders for Audio-Image-Symbolic Music Retrieval

    Oct 30, 2026Eliseo Fuentes-Martinez, Antonio Hidalgo-Centeno, Jorge Calvo-Zaragoza +1Multimodal IRParameter-Efficient Fine-Tuning

  7. A Broader Look into MLLM's Retrieval Heads

    Oct 23, 2026Jia Wang, Yuqing Li, Jiangnan Li +2Multimodal Large Language ModelsMultimodal IR

  8. pplx-embed-v2-late Technical Report

    Oct 21, 2026Markus Krimmel, Louis Milliken, Ekaterina Glazkova +8Multimodal EmbeddingMultimodal IR

  9. Enhancing RAG-based Medical VQA with Gated Multimodal Representations

    Oct 16, 2026Daniela Jimenez-Pano, Hugo Jair Escalante, Delia Irazú Hernández-FaríasGated Multimodal FusionMedical Visual Question Answering

  10. MMSearch-OPD: Easy Made Hard, Hard Made Learnable for Long-Horizon Multimodal Search Agents

    Oct 15, 2026Huanyao Zhang, Jiepeng Zhou, Jiaye Lin +9Multimodal Search AgentsMultimodal IR

  11. Cascaded Retrieval with Cross-Modal Query Refinement for Efficient Visual Document Search

    Oct 14, 2026Diana Turmakhan, Aseem Srivastava, Vivek Gupta +2Information RetrievalVisual Document Retrieval

  12. HyperBrowseComp: A Multilingual and Multimodal Stress Test for Web-Browsing Agents

    Oct 2, 2026Alham Fikri Aji, Faiz Rizki Ramadhan, Zayd M. K. Zuhri +14Web Agent BenchmarksAgent Benchmarks

  13. AiSearch: Interactive Multi-Modal Search with VLMs

    Oct 1, 2026Ali Koksal, Mei Chee Leong, Vicky Sintunata +2Multimodal IRVision-Language Models

  14. ResComEmb: Effective and Efficient Multimodal Embedding via Residual Homogeneity Compression

    Sep 29, 2026Zijing Cai, Yuzhe Wang, Jingxian Zhu +2Multimodal IRMultimodal Embedding

  15. VaME: Exploring Variational Latent Reasoning for Multimodal Embeddings

    Sep 27, 2026Peixi Wu, Mingzhou Jiang, Feipeng Ma +11Multimodal IREfficient Multimodal Inference

  16. UniK: Universal Knowledge Perception for Digital and Physical AI

    Sep 21, 2026Nirmit Desai, Kunal Sawarkar, Aditya Mahakali +3Multimodal RAGMultimodal IR

  17. Integrating knowledge from case reports: a medical ontology based multimodal information system with structured summary

    Sep 17, 2026Shuyu Guo, Lan Huang, Yichen Liu +2Multimodal IRRare Disease Diagnosis

  18. Less Is More: Graph-free Multimodal RAG via Multi-signal Late Fusion

    Sep 16, 2026Tithi Rakshit, Hongkuan Zhou, Lavdim Halilaj +1Late FusionMultimodal RAG

  19. RegRet: Enhancing Region-Level Retrieval in Large Multimodal Models

    Sep 15, 2026Xun Liang, Honghui Yang, Weihang Pan +6Multimodal IRMultimodal Large Language Models

  20. Hypergraph-Regularized Gramian Volumes for Multimodal Retrieval

    Sep 14, 2026Anindya Nag, Ambuj Mehrish, Sebastiano VasconMultimodal IRCross-Modal Alignment

  21. Reason What Matters: Retrieval-Grounded Reasoning for Universal Multimodal Embeddings

    Sep 14, 2026Mingzhou Jiang, Peixi Wu, Hang Cheng +7Multimodal IREfficient Multimodal Inference

  22. ReGround: Grounding Reviewer Comments in Multimodal Evidence

    Sep 10, 2026Serwar Basch, Lizhen Qu, Iryna GurevychMultimodal IRScholarly Peer Review

  23. MIDR: Enrichment-Augmented Indexing for Multimodal Document Retrieval

    Sep 1, 2026Debanjan Mahata, Atharva Tendle, Daniel Preotiuc-Pietro +2Multimodal IRVisual Document Retrieval

  24. SnapBench: Benchmarking Snap-and-Ask Multimodal Retrieval for Mobile Interactions

    Aug 30, 2026Zirong Chen, Fuda Ye, Kuan Zhang +7Multimodal IRInformation Retrieval

  25. WeAgent-MMSearch: Native Text-Vision Interaction for Multimodal Search Agents

    Aug 28, 2026Zongkai Liu, Hui Zhang, Liqiang Niu +7Multimodal IRMultimodal Search Agents

  26. Omni-Interactive Universal Embedder

    Aug 27, 2026Wei-Yao Wang, Kazuya Tateishi, Shuyang Cui +4Multimodal EmbeddingMultimodal IR

  27. Is Reasoning Always Useful? Rethinking Reasoning Utility in Universal Multimodal Embeddings

    Aug 26, 2026Wenxiao Fan, Jingling Fu, Luohang Liu +6Multimodal EmbeddingMultimodal IR

  28. AdaptiveEmbed: Sample-Adaptive Multi-Vector Representation for Multimodal Retrieval

    Aug 26, 2026Xinze Liu, Lei Yang, Dayan Wu +7Multimodal IRMulti-Vector Retrieval