KoViDoRe: A Benchmark for Korean Visual Document Retrieval

Yongbin Choi; Yongwoo Song; Mujeen Sung

cs.IRAug 21, 2026

KoViDoRe: A Benchmark for Korean Visual Document Retrieval

Authors: Yongbin Choi, Yongwoo Song, Mujeen Sung

Organizations: Kyung Hee University

Abstract

Recent advances in multimodal retrieval have improved the ability to retrieve information from visually rich documents such as PDFs and reports. However, existing benchmarks remain largely centered on English and provide limited coverage of Korean visual documents with complex structures. Furthermore, most existing Korean resources primarily evaluate single-page retrieval, failing to capture realistic scenarios that require evidence aggregation across multiple pages. To address these gaps, we introduce KoViDoRe, a benchmark for Korean visual document retrieval. The dataset is constructed from publicly available Korean documents with diverse layouts, including tables, figures, and multi-column structures. We develop a multi-stage data curation pipeline consisting of structured document parsing, synthetic query generation using both summary-based and context-based strategies, and relevance mapping with human verification. Using KoViDoRe, we evaluate a wide range of multimodal retrieval models and observe that current models struggle to effectively handle Korean visual document retrieval, particularly in settings involving structured content and diverse query types. Motivated by this finding, we further curate a large-scale training dataset, Ko-VDR Train Public, to support the development of retrieval models tailored to Korean visual documents. Together, KoViDoRe and Ko-VDR Train Public provide a unified benchmark and training resource for Korean visual document retrieval.

Figures & tables

Figure 1: Examples of queries in KoViDoRe and their corresponding relevant pages. Each query requires aggregating evidence from pages and diverse modalities, including text, tables, figures, and infographics.

Figure 2: Overview of the KoViDoRe data generation pipeline. Queries are generated through both summary-based and context-based strategies. Summary representations capture global document relationships, while context-based generation focuses on local content. The pipeline further includes filtering and manual verification to ensure query quality and reliable relevance mapping.

Subset	#Docs	#Pages	#Queries	#Qrels	Avg. Pages / Query
Cybersecurity	17	1,150	149	409	2.74
HR	9	2,109	221	726	3.30
Energy	11	1,993	173	525	3.03
Economic	20	1,477	163	413	2.55
Total	57	6,729	706	2,073	2.94

Table 1: Statistics of the KoViDoRe benchmark across domain-specific subsets. Avg. Pages / Query denotes the average number of relevant pages per query.

Figure 3: Distribution of query types across subsets. Note that query types are not mutually exclusive, accounting for the multi-faceted nature of complex information needs.

Figure 4: Distribution of query formats across subsets.

Model	Params	Cyber	Energy	Economic	HR	Avg.
Small Models ( $<$ 1B parameters)
openai/clip-vit-base-patch16 ♠	151M	4.1	0.8	0.0	0.6	1.4
vidore/colSmol-256M ⧫	256M	19.7	9.5	1.0	1.1	7.8
vidore/colSmol-500M ⧫	500M	26.2	9.9	0.6	0.9	9.4
jinaai/jina-clip-v2 ♠	865M	20.4	11.3	0.2	3.1	8.8
google/siglip-so400m-patch14-384 ♠	878M	15.3	5.3	1.3	1.1	5.8

Table 2: Performance comparison on KoViDoRe benchmark (nDCG@10, %). Bold indicates the best score; underline indicates the second-best score.

\spadesuit

: CLIP-based,

\blacklozenge

: late-interaction,

\clubsuit

: single-vector models. Cyber: Cybersecurity, HR: Human Resources.

Model	Params	Cyber	Energy	Economic	HR	Avg.
vidore/colSmol-500M	500M	26.2	9.9	0.6	0.9	9.4
+ Ko-VDR Train Public		39.4	35.0	14.4	18.7	26.9
vidore/colqwen2-v1.0	2.2B	53.3	42.0	8.0	14.7	29.5
+ Ko-VDR Train Public		75.6	67.6	18.3	49.6	52.8
jinaai/jina-embeddings-v4	3.8B	77.6	67.7	24.5	50.1	55.0
TomoroAI/tomoro-colqwen3-embed-4b	4.0B	55.3	31.0	9.1	10.1	26.4

Table 3: Comparison with representative retrieval baselines on KoViDoRe (nDCG@10, %). Bold indicates the best score; underline indicates the second-best score. Cyber: Cybersecurity, HR: Human Resources.

Figure 5: Similarity map on a document example.

Model	Cyber	Energy	Economic	HR	Avg.
vidore/colqwen2-v1.0	53.3	42.0	8.0	14.7	29.5
+ Private Only	70.3	58.8	18.7	37.7	46.4
+ Public Only	75.4	66.9	16.5	49.3	52.0
+ Private + Public	75.6	67.6	18.3	49.6	52.8

Table 4: Effect of training data composition on vidore/colqwen2-v1.0 (nDCG@10, %). Bold indicates the best score; underline indicates the second-best score. Cyber: Cybersecurity, HR: Human Resources.

Figure 6: nDCG@10 by the number of relevant pages per query. Performance tends to decrease as the number of relevant pages increases.

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 7: Distribution of queries by the number of relevant pages.

Category	Definition
open-ended	A query requiring synthesis and explanation of information. The answer must integrate multiple concepts into a coherent narrative rather than citing a single fact.
compare-contrast	A query requiring identification and articulation of similarities and/or differences between two or more entities, concepts, or topics.
enumerative	A query requesting a complete list of items that meet specific criteria.
numerical	A query expecting a numerical value, obtained either by direct extraction or calculation.
boolean	A query expecting a yes/no answer, potentially requiring reasoning over extracted information.
extractive	A query answerable by directly citing a specific fact or piece of information from the documents.

Appendix

Table 5: Query type categories and their definitions.

Category	Definition
question	A query presented in the form of a direct question, seeking specific information or clarification.
instruction	A query framed as a directive or command, requesting the model to perform a specific task or provide information in a particular manner.
keyword	A query consisting of noun phrases and keywords only, WITHOUT forming a complete sentence. No verbs, no question words, no sentence endings. Mimics how users type into search engines: fragmented, concise, noun-centric terms separated by spaces.

Appendix

Table 6: Query format categories and their definitions.

Panel of Figure 8 — Figure 8: Prompt for generating single-section summaries based on full-page context

Panel of Figure 9 — Figure 9: Prompt for cross-section summarization using aggregated section summaries

Panel of Figure 10 — Figure 10: Prompt for relevance mapping by identifying fully relevant and critically relevant pages

Figure 13: Example query-page pairs from the Economic and Energy subsets. Highlighted regions indicate the key evidence supporting each query.

Figure 14: Example query-page pairs from the HR and Cybersecurity subsets. Highlighted regions indicate the key evidence supporting each query.

Figure 15: Additional query-to-document similarity heatmap for the fine-tuned colqwen2-v1.0 model on a Korean document example. The model assigns high similarity to image patches corresponding to the term “실증/상용화,” highlighting its focus on query-relevant textual evidence.

Title	Provider	Pages	License
Cybersecurity
갠드크랩 랜섬웨어 악성코드 분석 기술 보고서	한국인터넷진흥원	92	No Restriction
사이버위협 동향보고서 (Windows 취약점 동향 및 업데이트 정책 등)	한국인터넷진흥원	104	No Restriction
사이버위협 동향보고서 (동형암호 기반 데이터 결합 및 분석 등)	한국인터넷진흥원	100	No Restriction
사이버 위협 동향보고서 (피싱 메일 공격 사례 등)	한국인터넷진흥원	96	No Restriction
사이버 위협 동향보고서 (기업 보안관리자의 크리덴셜 스터핑(Credential Stuffing) 공격 대응방안 등)	한국인터넷진흥원	104	No Restriction

Appendix

Table 7: Document metadata for KoViDoRe across all subsets.

Title	Provider	Pages	License
에너지총조사	기후에너지환경부	774	No Restrictions
주요업무계획	경기도 하남시	640	No Restrictions
지방공무원 인사실무	행정안전부	521	No Restrictions
항만편람	해양수산부	515	No Restrictions
국가연구개발사업 상위평가보고서	과학기술정보통신부	479	No Restrictions
연구보고서 현황	한국방송통신전파진흥원	425	No Restrictions

Appendix

Table 8: Document metadata for Ko-VDR Train Public.

Visual Document Retrieval

Visual document retrieval finds relevant pages or regions in image-based documents using visual and multimodal representations.

See all 66 papers

Aug 2, 2026

KoVRE: Training an Efficient Embedding Model for Korean Visual Document Retrieval

Visual Document Retrieval (VDR) directly matches text queries against document images, preserving visual and structural information that may be lost during text extraction. However, existing VDR models and training resources remain predominantly English-centric, while many high-performing systems rely on massive backbones or storage-intensive multi-vector representations. To address these limitations, we introduce KoVRE: Korean Visual Document Retrieval Embedding, a single-vector retriever for Korean visual documents, alongside a comprehensive training recipe. We train the model on 708,729 Korean and English query-page pairs using positive-aware hard-negative mining and conduct controlled analyses of training-data composition, hard-negative treatment, and reranker-based knowledge distillation. Across Korean visual document retrieval benchmarks, our 2B model substantially improves over the base backbone model, outperforming both its 8B single-vector counterpart and a strong multi-vector baseline. These results demonstrate that targeted bilingual supervision and our carefully designed training strategies can produce a highly effective Korean VDR model across diverse document domains, without requiring a scaled-up backbone or multi-vector representations.

Yongbin Choi, Gyuho Shim, Youngjoon Jang

Kyung Hee University · Korea University

Multimodal Embedding Visual Document Retrieval

Ask PDF

May 28, 2026

DocRetriever: A Plug-and-Play Framework for Multimodal Document Retrieval with Comprehensive Benchmark

Multimodal documents contain diverse elements, such as tables, figures, and layouts, which can complicate retrieval tasks. While current approaches typically combine dense visual embedding models with supervised rerankers to achieve high-precision retrieval, they face inherent limitations. First, the coarse-grained nature of dense embeddings tends to obfuscate explicit semantics, failing to leverage structurally salient information. Second, supervised reranking models suffer from generalization bottlenecks, as their performance heavily relies on domain-specific training data. Furthermore, existing benchmarks often lack diverse assessment dimensions and comprehensive relevance annotations, limiting reliable evaluation. To address these challenges, we propose DocRetriever, a plug-and-play framework. It enhances visual retrieval via a layout-aware sparse embedding technique, enabling effective hybrid encoding without the overhead of optical character recognition (OCR). We also introduce a generalizable reranker that leverages reasoning-augmented demonstrations and optimized sampling to improve accuracy in few-shot settings. Finally, we construct a new benchmark, MultiDocR, to enable more rigorous evaluation. Experiments across diverse benchmarks validate DocRetriever's superiority over state-of-the-art methods.

Ruofan Hu, Menghui Zhu, Jieming Zhu +8

Zhejiang University · Hangzhou, Zhejiang, China · Huawei Technologies Co., Ltd +2

Multimodal IR LLM Reranking Information Retrieval Visual Document Retrieval Hybrid Retrieval

Ask PDF

Feb 23, 2026

Unlocking Multimodal Document Intelligence: From Current Triumphs to Future Frontiers of Visual Document Retrieval

With the rapid proliferation of multimodal information, Visual Document Retrieval (VDR) has emerged as a critical frontier in bridging the gap between unstructured visually rich data and precise information acquisition. Unlike traditional natural image retrieval, visual documents exhibit unique characteristics defined by dense textual content, intricate layouts, and fine-grained semantic dependencies. This paper surveys the VDR landscape as a retrieval problem in its own right, rather than as the front end of a generation pipeline, and does so specifically through the lens of the Multimodal Large Language Model (MLLM) era. We begin by examining the benchmark landscape, including the recent turn toward reasoning-intensive evaluation, and then dive into the methodological evolution along two orthogonal axes: what a retriever is, spanning multimodal embedding models and reranker models, and how it is deployed, from single-stage retrieval through Retrieval-Augmented Generation (RAG) to Agentic systems. Cutting across both, we analyse the representation--efficiency trade-off that late interaction imposes. We ground these categories in leaderboard evidence on where the empirical frontier actually lies, and close by identifying persistent challenges and outlining promising future directions for multimodal document intelligence.

Yibo Yan, Jiahao Huo, Guanbo Feng +12

Hong Kong University of Science and Technology (Guangzhou) · Alibaba Cloud Computing · Hong Kong University of Science and Technology +1

Multimodal RAG Multimodal IR Multimodal Embedding Visual Document Retrieval

Ask PDF

Abstract

Visual Document Retrieval

Explore similar work

KoVRE: Training an Efficient Embedding Model for Korean Visual Document Retrieval

DocRetriever: A Plug-and-Play Framework for Multimodal Document Retrieval with Comprehensive Benchmark

Unlocking Multimodal Document Intelligence: From Current Triumphs to Future Frontiers of Visual Document Retrieval