cs.IRMar 10, 2026

LITTA: Late-Interaction and Test-Time Alignment for Visually-Grounded Multimodal Retrieval

Authors: Seonok Kim

Organizations: Mazelone

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Retrieving relevant evidence from visually rich documents such as textbooks, technical reports, and manuals is challenging due to long context, complex layouts, and weak lexical overlap between user questions and supporting pages. We propose LITTA, a query-expansion-centric retrieval framework for evidence page retrieval that improves multimodal document retrieval without retriever retraining. Given a user query, LITTA generates complementary query variants using a large language model and retrieves candidate pages for each variant using a frozen vision retriever with late-interaction scoring. Candidates from expanded queries are then aggregated through reciprocal rank fusion to improve evidence coverage and reduce sensitivity to any single phrasing. This simple test-time strategy significantly improves retrieval robustness while remaining compatible with existing multimodal embedding indices. We evaluate LITTA on visually grounded document retrieval tasks across three domains: computer science, pharmaceuticals, and industrial manuals. Multi-query retrieval consistently improves top-k accuracy, recall, and MRR compared to single-query retrieval, with particularly large gains in domains with high visual and semantic variability. Moreover, the accuracy-efficiency trade-off is directly controllable by the number of query variants, making LITTA practical for deployment under latency constraints. These results demonstrate that query expansion provides a simple yet effective mechanism for improving visually grounded multimodal retrieval.

Figures & tables

Figure 1: Overview of LITTA. Left: we pre-compute and index page embeddings for a multi-page document collection. Middle: a query is expanded into complementary variants. Right: each expanded query retrieves top- kk candidate pages via late-interaction scoring; we aggregate candidates across expansions using reciprocal rank fusion, optionally re-rank a small subset, and pass top- KK evidence pages to the generator.
ModelSizeC.S.Phar.Ind.Avg.
Textual Retrievers
BGE-M30.6B58.052.028.546.2
Jina-v4 (Textual)3B64.354.938.452.5
Qwen38B71.759.240.457.1
Qwen3.58B73.767.042.661.1
Visual Retrievers
Table 1: Retrieval performance (NDCG@10) across three domains. Multi-query retrieval with query variants ( q>1q>1 ) consistently improves ranking quality over single-query retrieval ( q=1q=1 ), with the largest gains in Industrial manuals where terminology and layout variability reduce lexical overlap. Bold indicates best.
 C.S.Phar.Ind.Avg.
MethodsR@5R@10R@5R@10R@5R@10R@5R@10
ColEmbed-v267.0782.8072.1377.2459.8173.9666.3478.00
LITTA ( qq =1)67.9582.6472.1177.7443.8965.5261.3275.30
LITTA ( qq =3)68.2283.3273.3377.2757.6373.7666.3978.45
LITTA ( qq =5)65.7483.1172.6577.7460.3773.7666.2578.01
Table 2: Recall@5 and Recall@10 (%). ColEmbed-v2: base retriever; LITTA ( qq = kk ): multi-query retrieval with kk expanded queries. Bold indicates the best value per column; ties are bolded.
Figure 2: Top- kk accuracy (Hit@ kk ) curves for multi-query settings Q∈{1,3,5}Q\in\{1,3,5\} across domains and the average.
Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

CorpusDomain(s)# Docs# Pages# Queries ∗Main modalities
Computer Science TextbooksComputer Science / Education21360215Books
FDA ReportsPharmaceuticals522313364Slides, Books
USAF Technical OrdersIndustrial Maintenance275244283Manuals
Appendix
Table 3: Dataset statistics for the three evaluation corpora used in our experiments.
Figure 3: Prompt templates used for query expansion (query variant generation) and answer evaluation.
More in this topic

Multimodal IR

Multimodal information retrieval finds relevant images, videos, documents, audio, or other media using queries that may combine different modalities.

See all 126 papers

Explore similar work

CardsList
Jun 22, 2026

LightSTAR: Efficient Visual Document Retrieval via Lightweight Selection with Vision-Adaptive Refinement

Visual document retrieval requires rapidly locating relevant pages from large multi-modal corpora in response to user queries. While recent methods powered by Multi-modal Large Language Models (MLLMs) show competitive accuracy, they suffer from prohibitive computational costs by applying intensive MLLM encoding to every single page. Meanwhile, we observe that user queries are typically keyword-anchored, containing semantically rich words that are expected to appear directly in the visible text of relevant pages, offering an efficient cue for quickly narrowing down candidate pages. Building on this insight, we propose LightSTAR, an efficient framework that decomposes visual document retrieval into: 1) LLM-free Visual Selection, which utilizes content-grounded query encoding to focus on informative words and employs LLM-free visual embeddings to produce a high-recall candidate set; and 2) Vision-adaptive Semantic Refinement, which further performs fine-grained semantic matching exclusively on these top candidates via adaptive region-wise feature fusion to effectively combine textual and layout cues, optimized through a hardness-aware contrastive objective. Experimental results demonstrate that LightSTAR achieves state-of-the-art retrieval accuracy while reducing end-to-end latency by several-fold, offering a highly practical solution to the accuracy-efficiency trade-off in visual document retrieval. Code is available at https://github.com/bokufa/LightSTAR.

Tongkun Guan, Haocheng Wang, Wei Shen +1

MoE Key Lab of Artificial Intelligence, AI Institute, School of Computer Science, Shanghai Jiao Tong University

Information RetrievalVisual Document Retrieval
AskPDF
Sep 1, 2026

MIDR: Enrichment-Augmented Indexing for Multimodal Document Retrieval

Retrieval over visually rich documents has a representation problem: important content often lives in tables, charts, figures, and layout relations that plain OCR linearizes, corrupts, or omits. ColPali-family visual retrievers address this with patch-level multi-vector indexes and late-interaction scoring, keeping image-derived retrieval on the query-time serving path. We introduce MIDR (Multimodal Indexing for Document Retrieval), a training-free framework for enrichment-augmented indexing that shifts multimodal reasoning to index time. During ingestion, a multimodal LLM converts rendered pages into verified textual fields that are indexed with BM25F and optionally fused with dense retrieval, enabling text-centric serving over multimodally grounded evidence. On ViDoRe V3, MIDR Hybrid achieves 0.6219 average nDCG across five English domains, a 23.0% relative gain over BM25, remaining competitive with ColQwen2.5. On two French-document domains, enrichment bridges English queries and French page text, lifting BM25 from 0.1532 to 0.5448 nDCG and outperforming ColQwen2.5. Across all seven domains, MIDR leads ColQwen2.5 on four while using approximately 9x smaller index memory and approximately 2x lower query latency. These results establish index-time multimodal reasoning as a compelling accuracy-deployment alternative to serving-time visual late interaction.

Debanjan Mahata, Atharva Tendle, Daniel Preotiuc-Pietro +2

Bloomberg, NYC, USA

Multimodal IRVisual Document Retrieval
AskPDF
Feb 5, 2026

V-Retrver: Evidence-Driven Agentic Reasoning for Universal Multimodal Retrieval

Multimodal Large Language Models (MLLMs) have recently been applied to universal multimodal retrieval, where Chain-of-Thought (CoT) reasoning improves candidate reranking. However, existing approaches remain largely language-driven, relying on static visual encodings and lacking the ability to actively verify fine-grained visual evidence, which often leads to speculative reasoning in visually ambiguous cases. We propose V-Retrver, an evidence-driven retrieval framework that reformulates multimodal retrieval as an agentic reasoning process grounded in visual inspection. V-Retrver enables an MLLM to selectively acquire visual evidence during reasoning via external visual tools, performing a multimodal interleaved reasoning process that alternates between hypothesis generation and targeted visual verification.To train such an evidence-gathering retrieval agent, we adopt a curriculum-based learning strategy combining supervised reasoning activation, rejection-based refinement, and reinforcement learning with an evidence-aligned objective. Experiments across multiple multimodal retrieval benchmarks demonstrate consistent improvements in retrieval accuracy (with 23.0% improvements on average), perception-driven reasoning reliability, and generalization.

Dongyang Chen, Chaoyang Wang, Dezhao Su +6

Tsinghua University · University of Central Florida · Fudan University +4

Multimodal IRMultimodal GroundingMultimodal CoT ReasoningMultimodal Understanding
AskPDF