Multimodal IR

Multimodal information retrieval studies how to find relevant items when queries and collections contain text, images, video, audio, or structured documents. Common approaches learn shared representations with contrastive training, use multi-vector or sparse indexes for fine-grained matching, and add rerankers or agents that gather and verify evidence. Applications range from visual document and product search to medical, news, and video retrieval, with benchmarks testing both cross-modal matching and multi-step search. A central tension is how to capture detailed, grounded evidence and reasoning without making indexing, storage, or inference too costly; current systems still struggle with long contexts, noisy inputs, and complex constraints.