Recent advances in multimodal retrieval have improved the ability to retrieve information from visually rich documents such as PDFs and reports. However, existing benchmarks remain largely centered on English and provide limited coverage of Korean visual documents with complex structures. Furthermore, most existing Korean resources primarily evaluate single-page retrieval, failing to capture realistic scenarios that require evidence aggregation across multiple pages. To address these gaps, we introduce KoViDoRe, a benchmark for Korean visual document retrieval. The dataset is constructed from publicly available Korean documents with diverse layouts, including tables, figures, and multi-column structures. We develop a multi-stage data curation pipeline consisting of structured document parsing, synthetic query generation using both summary-based and context-based strategies, and relevance mapping with human verification. Using KoViDoRe, we evaluate a wide range of multimodal retrieval models and observe that current models struggle to effectively handle Korean visual document retrieval, particularly in settings involving structured content and diverse query types. Motivated by this finding, we further curate a large-scale training dataset, Ko-VDR Train Public, to support the development of retrieval models tailored to Korean visual documents. Together, KoViDoRe and Ko-VDR Train Public provide a unified benchmark and training resource for Korean visual document retrieval.
Figures & tables
Figure 1: Examples of queries in KoViDoRe and their corresponding relevant pages. Each query requires aggregating evidence from pages and diverse modalities, including text, tables, figures, and infographics.
Figure 2: Overview of the KoViDoRe data generation pipeline. Queries are generated through both summary-based and context-based strategies. Summary representations capture global document relationships, while context-based generation focuses on local content. The pipeline further includes filtering and manual verification to ensure query quality and reliable relevance mapping.
Subset
#Docs
#Pages
#Queries
#Qrels
Avg. Pages / Query
Cybersecurity
17
1,150
149
409
2.74
HR
9
2,109
221
726
3.30
Energy
11
1,993
173
525
3.03
Economic
20
1,477
163
413
2.55
Total
57
6,729
706
2,073
2.94
Table 1: Statistics of the KoViDoRe benchmark across domain-specific subsets. Avg. Pages / Query denotes the average number of relevant pages per query.
Figure 3: Distribution of query types across subsets. Note that query types are not mutually exclusive, accounting for the multi-faceted nature of complex information needs.
Figure 4: Distribution of query formats across subsets.
Model
Params
Cyber
Energy
Economic
HR
Avg.
Small Models ( < 1B parameters)
openai/clip-vit-base-patch16 ♠
151M
4.1
0.8
0.0
0.6
1.4
vidore/colSmol-256M ⧫
256M
19.7
9.5
1.0
1.1
7.8
vidore/colSmol-500M ⧫
500M
26.2
9.9
0.6
0.9
9.4
jinaai/jina-clip-v2 ♠
865M
20.4
11.3
0.2
3.1
8.8
google/siglip-so400m-patch14-384 ♠
878M
15.3
5.3
1.3
1.1
5.8
Table 2: Performance comparison on KoViDoRe benchmark (nDCG@10, %). Bold indicates the best score; underline indicates the second-best score. ♠ : CLIP-based, ⧫ : late-interaction, ♣ : single-vector models. Cyber: Cybersecurity, HR: Human Resources.
Model
Params
Cyber
Energy
Economic
HR
Avg.
vidore/colSmol-500M
500M
26.2
9.9
0.6
0.9
9.4
+ Ko-VDR Train Public
39.4
35.0
14.4
18.7
26.9
vidore/colqwen2-v1.0
2.2B
53.3
42.0
8.0
14.7
29.5
+ Ko-VDR Train Public
75.6
67.6
18.3
49.6
52.8
jinaai/jina-embeddings-v4
3.8B
77.6
67.7
24.5
50.1
55.0
TomoroAI/tomoro-colqwen3-embed-4b
4.0B
55.3
31.0
9.1
10.1
26.4
Table 3: Comparison with representative retrieval baselines on KoViDoRe (nDCG@10, %). Bold indicates the best score; underline indicates the second-best score. Cyber: Cybersecurity, HR: Human Resources.
Figure 5: Similarity map on a document example.
Model
Cyber
Energy
Economic
HR
Avg.
vidore/colqwen2-v1.0
53.3
42.0
8.0
14.7
29.5
+ Private Only
70.3
58.8
18.7
37.7
46.4
+ Public Only
75.4
66.9
16.5
49.3
52.0
+ Private + Public
75.6
67.6
18.3
49.6
52.8
Table 4: Effect of training data composition on vidore/colqwen2-v1.0 (nDCG@10, %). Bold indicates the best score; underline indicates the second-best score. Cyber: Cybersecurity, HR: Human Resources.
Figure 6: nDCG@10 by the number of relevant pages per query. Performance tends to decrease as the number of relevant pages increases.
Appendix figures & tables11 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 7: Distribution of queries by the number of relevant pages.
Category
Definition
open-ended
A query requiring synthesis and explanation of information. The answer must integrate multiple concepts into a coherent narrative rather than citing a single fact.
compare-contrast
A query requiring identification and articulation of similarities and/or differences between two or more entities, concepts, or topics.
enumerative
A query requesting a complete list of items that meet specific criteria.
numerical
A query expecting a numerical value, obtained either by direct extraction or calculation.
boolean
A query expecting a yes/no answer, potentially requiring reasoning over extracted information.
extractive
A query answerable by directly citing a specific fact or piece of information from the documents.
Appendix
Table 5: Query type categories and their definitions.
Category
Definition
question
A query presented in the form of a direct question, seeking specific information or clarification.
instruction
A query framed as a directive or command, requesting the model to perform a specific task or provide information in a particular manner.
keyword
A query consisting of noun phrases and keywords only, WITHOUT forming a complete sentence. No verbs, no question words, no sentence endings. Mimics how users type into search engines: fragmented, concise, noun-centric terms separated by spaces.
Appendix
Table 6: Query format categories and their definitions.
Figure 8: Prompt for generating single-section summaries based on full-page context
Figure 9: Prompt for cross-section summarization using aggregated section summaries
Figure 10: Prompt for relevance mapping by identifying fully relevant and critically relevant pages
Figure 13: Example query-page pairs from the Economic and Energy subsets. Highlighted regions indicate the key evidence supporting each query.
Figure 14: Example query-page pairs from the HR and Cybersecurity subsets. Highlighted regions indicate the key evidence supporting each query.
Figure 15: Additional query-to-document similarity heatmap for the fine-tuned colqwen2-v1.0 model on a Korean document example. The model assigns high similarity to image patches corresponding to the term “실증/상용화,” highlighting its focus on query-relevant textual evidence.
Title
Provider
Pages
License
Cybersecurity
갠드크랩 랜섬웨어 악성코드 분석 기술 보고서
한국인터넷진흥원
92
No Restriction
사이버위협 동향보고서 (Windows 취약점 동향 및 업데이트 정책 등)
한국인터넷진흥원
104
No Restriction
사이버위협 동향보고서 (동형암호 기반 데이터 결합 및 분석 등)
한국인터넷진흥원
100
No Restriction
사이버 위협 동향보고서 (피싱 메일 공격 사례 등)
한국인터넷진흥원
96
No Restriction
사이버 위협 동향보고서 (기업 보안관리자의 크리덴셜 스터핑(Credential Stuffing) 공격 대응방안 등)
한국인터넷진흥원
104
No Restriction
Appendix
Table 7: Document metadata for KoViDoRe across all subsets.
Title
Provider
Pages
License
에너지총조사
기후에너지환경부
774
No Restrictions
주요업무계획
경기도 하남시
640
No Restrictions
지방공무원 인사실무
행정안전부
521
No Restrictions
항만편람
해양수산부
515
No Restrictions
국가연구개발사업 상위평가보고서
과학기술정보통신부
479
No Restrictions
연구보고서 현황
한국방송통신전파진흥원
425
No Restrictions
Appendix
Table 8: Document metadata for Ko-VDR Train Public.