Multimodal RAG

RAG: Retrieval-Augmented Generation

Multimodal RAG studies how to retrieve external evidence across modalities and use it to produce grounded answers, reports, or decisions. Systems combine dense or hybrid search with techniques such as OCR and layout-aware document indexing, image-region or video-event retrieval, and graph-based evidence linking; rerankers and agents can refine what is retrieved and how it is used. This supports knowledge-based visual question answering, long-document and video understanding, and specialized applications where evidence and citations matter. Benchmarks increasingly separate retrieval quality from answer quality, while a central challenge remains ensuring that models use relevant evidence faithfully and efficiently despite noisy, biased, or poisoned context.

Latest papers

158 papers13 papers in the last four weeks, down 7% on the four weeks before. 0.1% of all new papers.

VIGiA: Instructional Video Guidance via Dialogue Reasoning and Retrieval
Feb 22, 2026Diogo Glória-Silva, David Semedo, João MaglhãesVLM Reasoning Multimodal RAG
NOVA LINCS, NOVA School of Science and Technology, Portugal
Concept-Enhanced Multimodal RAG: Towards Interpretable and Accurate Radiology Report Generation
Feb 17, 2026Marco Salmè, Federico Siciliano, Fabrizio Silvestri +3Radiology Report Generation Radiology VLMs
Department of Engineering, Research Unit of Artificial Intelligence and Computer Systems, Università Campus Bio-Medico of Roma, Rome, Italy. · Department of Computer, Control and Management Engineering, Sapienza University of Rome, Rome, Italy. · Department of Diagnostics and Intervention, Radiation Physics, Biomedical Engineering, Umeå University, Umeå, Sweden. +1
Event-Grounded Question Answering over Long Audio via Structured Retrieval
Feb 16, 2026Kartik Hegde, Arvind Krishna Sridhar, Naveen Vakada +2Multimodal RAG Audio QA
Qualcomm Technologies, Inc., India · Qualcomm Technologies, Inc., USA
PixSearch: Region-Grounded Retrieval for Knowledgeable Large Multimodal Models
Jan 27, 2026Jeonghwan Kim, Renjie Tao, Sanat Sharma +8Multimodal RAG Visual Question Answering
Meta Reality Labs · University of Illinois Urbana-Champaign
R3G: A Reasoning-Retrieval-Reranking Framework for Vision-Centric Answer Generation
Jan 25, 2026Zhuohong Chen, Zhengxian Wu, Zirui Liao +6Multimodal RAG Visual Question Answering
1The Shenzhen International Graduate School, Tsinghua University, China · School of Computer Science and Information Engineering, Hefei University of Technology, China · 2State Key Laboratory of Nuclear Power Safety Technology and Equipment, China
Seeing Through the MiRAGE: Evaluating Multimodal Retrieval Augmented Generation
Oct 28, 2025Alexander Martin, William Walden, Reno Kriz +5Multimodal RAG RAG Evaluation
Johns Hopkins University · Human Language Technology Center of Excellence
ArtSeek: Deep artwork understanding via multimodal in-context reasoning and late interaction retrieval
Jul 29, 2025Nicola Fanelli, Gennaro Vessio, Giovanna CastellanoMultimodal RAG Multimodal IR
Department of Computer Science, University of Bari Aldo Moro, Bari, Italy
RS-Agent: Automating Remote Sensing Tasks through Intelligent Agent
Jun 11, 2024Wenjia Xu, Zijian Yu, Boyang Mu +3Multimodal RAG Multimodal Large Language Models
State Key Laboratory of Networking and Switching Technology, Beijing University of Posts and Telecommunications, Beijing 100876, China · City University of HongKong, Hong Kong 999077, China · School of Geographic Sciences, Hunan Normal University, Changsha 410081, China

Feb 22, 2026J/K move · Enter open · S save

VIGiA: Instructional Video Guidance via Dialogue Reasoning and Retrieval

Diogo Glória-Silva, David Semedo, João Maglhães

NOVA LINCS, NOVA School of Science and Technology, Portugal

VLM Reasoning Multimodal RAG Knowledge-Based VQA

Open paper AskPDF arXiv

We introduce VIGiA, a novel multimodal dialogue model designed to understand and reason over complex, multi-step instructional video action plans. Unlike prior work which focuses mainly on text-only guidance, or treats vision and language in isolation, VIGiA supports grounded, plan-aware dialogue that requires reasoning over visual inputs, instructional plans, and interleaved user interactions. To this end, VIGiA incorporates two key capabilities: (1) multimodal plan reasoning, enabling the model to align uni- and multimodal queries with the current task plan and respond accurately; and (2) plan-based retrieval, allowing it to retrieve relevant plan steps in either textual or visual representations. Experiments were done on a novel dataset with rich Instructional Video Dialogues aligned with Cooking and DIY plans. Our evaluation shows that VIGiA outperforms existing state-of-the-art models on all tasks in a conversational plan guidance setting, reaching over 90% accuracy on plan-aware VQA.