Vision-Language Model (VLM) based retrievers have advanced visual document retrieval (VDR) to impressive quality. They require the same multi-billion parameter encoder for both document indexing and query encoding, incurring high latency and GPU dependence even for plain-text queries. We observe that this design is unnecessarily symmetric: documents are visually complex and demand strong visual understanding, whereas queries are just short text strings. NanoVDR exploits this query--document asymmetry by decoupling the two encoding paths: a frozen 2B VLM teacher indexes documents offline, while a distilled text-only student as small as 69M parameters encodes queries at inference. The key design choice is the distillation objective. Through systematic comparison of six objectives across three backbones and 22 ViDoRe benchmark datasets, we find that pointwise cosine alignment on query text consistently outperforms ranking-based and contrastive alternatives, while requiring only pre-cached teacher query embeddings and no document processing during training. Furthermore, we identify cross-lingual transfer as the primary performance bottleneck, and resolve it cheaply by augmenting training data with machine-translated queries. The resulting NanoVDR-S-Multi (DistilBERT, 69M) retains 95.1% of teacher quality and outperforms DSE-Qwen2 (2B) on v2 and v3 with 32× fewer parameters and 50× lower CPU query latency, at a total training cost under 13 GPU-hours.
Figures & tables
Figure 1: Motivation and deployment advantage of NanoVDR. (a) Symmetric vs. asymmetric retrieval: Current VDR systems (top) use the same heavy VLM encoder (2B) for both offline document indexing and online query encoding ( > 2,000 ms per query). NanoVDR (bottom) decouples the two: the frozen VLM teacher encodes documents offline, while a distilled text-only student (70M) encodes queries online in ∼ 50 ms on CPU. (b) Performance vs. latency: On the ViDoRe benchmark (mean NDCG@5 across v1/v2/v3), NanoVDR models achieve near-teacher accuracy (dashed line) at 50–143 × lower CPU latency. Bubble size is proportional to model parameter count; the star ( ★ ) marks NanoVDR-S-Multi, the multilingual-augmented variant (§ 3.3 ).
Figure 2: Query-centric distillation training of NanoVDR. Left: The frozen VLM teacher pre-caches training query embeddings via text-only inference. Right: The student text encoder is trained to minimize Lalign=1−cos(vtQ,vsQ) , requiring no document images or negative sampling.
Table 1: Main results (NDCG@5 × 100) on the ViDoRe benchmark. Each benchmark version is averaged over the indicated number of datasets. Params = total model parameters; for NanoVDR this includes the backbone + MLP projector. Scoring = retrieval scoring method: MaxSim (token-level late interaction) or Cosine (single-vector dot product). † : Qwen3-VL-Embedding-2B Li et al. (2026) , our frozen teacher used for offline document indexing. Best per group in bold . Subscripts on NanoVDR rows indicate teacher retention (%, student/teacher). ‡ : SERVAL Nguyen et al. (2025) generates a textual description of every page offline with a VLM and indexes those descriptions with a standard text encoder. Its ViDoRe v2 score is recomputed from the per-dataset results reported in that paper; see Appendix J .
Model
Params
Dim
Tok/Doc
Size
Scoring
Encode
Ret/1K
Ret/10K
Index/1M
Multi-vector VLMs (MaxSim scoring)
Tomoro-8B
8,768M
320
1,280
35.1 GB
MaxSim
8,225 ms
706 ms
7,124 ms
819 GB
ColNomic-7B
7,829M
128
1,000
31.3 GB
MaxSim
7,471 ms
234 ms
2,385 ms
256 GB
ColPali
2,964M
128
1,030
11.9 GB
MaxSim
7,284 ms
269 ms
2,786 ms
264 GB
ColModernVBert
259M
128
1,000
1.0 GB
MaxSim
183 ms
242 ms
2,553 ms
256 GB
Single-vector models (Cosine scoring)
Table 2: Deployment cost comparison (query-time). All latency measured on a single CPU thread (batch size 1). Params = total parameters. Dim = embedding dimension per vector. Tok/Doc = tokens stored per document page (1 = single-vector). Size = query encoder checkpoint (float32); for NanoVDR this is the student only. Encode = single-query encoding latency. Ret/ N = scoring latency against N candidate documents. Index/1M = index storage for 1M documents (float32 for single-vector, float16 for multi-vector). NanoVDR shares the teacher’s (Qwen3-VL-Embedding-2B) pre-computed document index. The teacher itself is excluded as it is used only for offline GPU-based document indexing.
λa
λr
v1 (10)
v2 (4)
v3 (8)
1
0
82.2
61.4
44.1
1
0.5
81.6
59.8
42.8
1
1
81.5
59.1
42.5
0.5
1
81.5
58.6
42.1
0
1
81.1
57.4
41.6
InfoNCE
71.5
39.8
30.0
Table 3: Loss ablation (NDCG@5 × 100, averaged over 3 backbones). λa , λr = weights for Lalign and Lrank . All trained under identical settings (20 epochs, lr=2e-4).
Language
Train%
#Queries
Teacher
Student
Ret.
English
68.7
6,237
64.0
60.3
94.3%
French
7.6
3,074
56.2
51.7
92.1%
Italian
7.6
2,419
49.0
44.1
90.0%
Spanish
8.1
2,694
51.4
46.1
89.7%
German
8.0
2,694
49.3
42.3
85.7%
Portuguese
0.0
2,419
48.7
36.8
75.6%
Table 4: Distillation retention by query language (NanoVDR-S). Train% = proportion of 711K training pairs in each language. Teacher and Student are mean NDCG@5 ( × 100). Ret. = Student / Teacher. All 22 ViDoRe datasets pooled (19,537 unique queries after exact-duplicate removal).
Language
#Queries
Teacher
Original
+Aug
Δ
Ret.
English
6,237
64.0
60.3
60.3
+0.0
94.3%
French
3,074
56.2
51.7
53.2
+1.5
94.7%
Italian
2,419
49.0
44.1
45.7
+1.6
93.3%
Spanish
2,694
51.4
46.1
47.8
+1.8
93.1%
German
2,694
49.3
42.3
45.4
+3.1
92.0%
Portuguese
2,419
48.7
36.8
46.1
+9.3
94.6%
Table 5: Effect of multilingual query augmentation by language (NDCG@5 × 100). Original = NanoVDR-S trained on 711K pairs; +Aug = retrained on 1.49M pairs with translated queries. Δ = +Aug − Original. Ret. = +Aug / Teacher. Same 19,537 queries as Table 4 .
Appendix figures & tables14 assets
Supplementary material from the paper’s appendix.
Appendix
v1
v2
v3
Datasets
10
4
8
Query languages
2
4
6
Query source
VQA + LLM
LLM + human
LLM + expert
SOTA NDCG@5
> 90
59–65
57–59
Appendix
Table 6: Cross-version comparison of the ViDoRe benchmark.
Dataset
Ver.
Document Domain
Doc
Query
Source
DocVQA
v1
Industrial documents
EN
EN
H
ArXivQA
v1
Scientific papers
EN
EN
H
InfoVQA
v1
Infographics
EN
EN
H
TatDQA
v1
Financial tables
EN
EN
H
TabFQuAD
v1
Tables in French PDFs
FR
FR
H
SyntheticDocQA-AI
v1
AI-related documents
EN
EN
L
Appendix
Table 7: All 22 ViDoRe evaluation datasets used in this work. Doc = document corpus language. Query = languages in which queries are provided. Source: H = human-authored (from VQA benchmarks or expert annotation); L = LLM-generated; L+H = LLM-generated with human review. Six languages = EN/FR/ES/DE/IT/PT.
Source
Subset
Train
%
VisRAG-Synthetic
—
233,817
32.9
ColPali
—
109,044
15.3
VisRAG-InDomain
—
94,016
13.2
VDR-Multilingual
Spanish
57,491
8.1
German
56,994
8.0
French
54,079
7.6
Appendix
Table 8: Training data composition after deduplication. VDR-Multilingual is split by language.
English
French
Model
ArXivQA
TatDQA
InfoVQA
DocVQA
Syn-AI
Syn-En.
Syn-Gov.
Syn-Hlt.
Shift.
TabFQ.
Avg
Tomoro-8B
91.3
81.5
94.6
66.2
98.9
96.2
96.9
98.7
87.6
94.3
90.6
Tomoro-4B
90.9
80.8
94.2
66.0
98.5
95.9
95.7
98.9
87.4
94.0
90.2
ColNomic-7B
87.9
80.5
91.9
60.9
100.0
96.8
95.8
99.1
88.7
96.0
89.8
ColPali
82.7
70.3
85.4
58.9
96.7
94.2
96.0
96.9
75.2
85.9
84.2
ColModernVBert
72.7
71.3
83.5
47.2
96.7
89.0
93.0
94.7
50.1
68.9
76.7
Appendix
Table 9: Per-dataset NDCG@5 ( × 100) on ViDoRe v1. † : Qwen3-VL-Embedding-2B (our teacher). Best overall in bold , best among NanoVDR variants underlined .
Model
ESG
BioMed
Econ
ESG-H
Avg
Tomoro-8B
60.4
65.4
59.5
74.7
65.0
Tomoro-4B
62.6
65.6
56.0
76.8
65.2
ColNomic-7B
53.6
63.2
55.6
69.4
60.4
ColPali
54.9
55.9
49.4
58.7
54.7
ColModernVBert
30.5
26.6
27.2
49.3
33.4
DSE-Qwen2
54.9
55.9
52.0
60.0
55.7
Appendix
Table 10: Per-dataset NDCG@5 ( × 100) on ViDoRe v2. † : Qwen3-VL-Embedding-2B (our teacher). Best overall in bold , best among NanoVDR variants underlined .
Model
Fin-EN
Fin-FR
CS
HR
Energy
Indust.
Pharma
Physics
Avg
Tomoro-8B
62.8
45.9
72.4
61.5
65.8
52.5
64.3
46.7
59.0
Tomoro-4B
61.3
42.9
72.5
57.3
63.3
52.7
64.0
46.6
57.6
ColNomic-7B
55.2
42.8
74.1
57.0
62.4
49.4
61.1
45.0
55.9
ColPali
34.9
21.8
63.6
43.4
45.7
34.6
52.7
39.5
42.0
ColModernVBert
18.3
11.6
25.9
12.9
22.9
11.4
19.4
17.0
17.4
DSE-Qwen2
40.1
22.4
60.8
40.8
45.0
30.4
51.8
39.0
41.3
Appendix
Table 11: Per-dataset NDCG@5 ( × 100) on ViDoRe v3. † : Qwen3-VL-Embedding-2B (our teacher). Best overall in bold , best among NanoVDR variants underlined .
ViDoRe v1
ViDoRe v2
ViDoRe v3
Config
λa
λr
S
M
L
S
M
L
S
M
L
Align
1
0
82.2
82.1
82.4
60.5
62.2
61.5
43.5
44.7
44.2
Align-dom
1
0.5
81.4
81.6
81.8
59.5
59.7
60.2
41.9
43.5
43.1
Combined
1
1
81.2
81.4
82.0
58.6
59.4
59.3
41.4
42.9
43.0
Rank-dom
0.5
1
81.1
81.6
81.8
57.3
58.6
59.9
40.9
42.6
42.8
Rank
0
1
80.3
81.1
81.9
56.5
57.0
58.9
40.0
41.8
42.9
Appendix
Table 12: Loss ablation (NDCG@5 × 100) per variant. S = NanoVDR-S, M = NanoVDR-M, L = NanoVDR-L.
Requirement
Align-only
With ranking
Teacher query encoding (text)
✓
✓
Teacher doc encoding (images)
—
✓
Query encoding time
0.8 GPU-h
0.8 GPU-h
Doc encoding time
—
24 GPU-h
Total pre-cache cost
0.8 GPU-h
24.8 GPU-h
Embedding storage
2.9 GB
5.8 GB
Appendix
Table 13: Pre-caching and training cost comparison for align-only ( λr=0 ) vs. ranking-based ( λr>0 ) distillation on 711K training pairs with a 2B VLM teacher.
Figure 3: Data efficiency of NanoVDR-S. NDCG@5 vs. fraction of 711K training pairs, with teacher upper bounds (dashed). Percentages indicate retention (student/teacher). Diminishing returns are pronounced after 25% of training data.
Dataset
Ver.
EN
FR
ES
DE
IT
PT
ArXivQA
v1
94.9
—
—
—
—
—
TatDQA
v1
93.1
—
—
—
—
—
InfoVQA
v1
97.9
—
—
—
—
—
DocVQA
v1
92.8
—
—
—
—
—
SyntheticDocQA-AI
v1
97.8
—
—
—
—
—
SyntheticDocQA-En.
v1
99.3
—
—
—
—
—
Appendix
Table 14: Per-language retention (NanoVDR-S NDCG@5/ Teacher NDCG@5 × 100) across all 22 ViDoRe datasets. “—” = language not present in that dataset. V1 datasets are single-language (8 English, 2 French).
Teacher
N
cˉ
r with Ret.
+0.607
− 0.566
+0.094
Appendix
Table 15: Pearson correlations between corpus-level features and English-only retention (NanoVDR-S, 19 datasets). Teacher = mean teacher NDCG@5; N = corpus size; cˉ = mean student–teacher query cosine similarity.
Language
Existing
Translated
Total
Model
Portuguese
0
200,000
200,000
en-ROMANCE
Italian
53,787
146,213
200,000
en-it
French
54,079
145,921
200,000
en-fr
German
56,994
143,006
200,000
en-de
Spanish
57,491
142,509
200,000
en-es
Total
777,649
Appendix
Table 16: Translation statistics per target language. Existing = count already in the 711K training set. Translated = newly machine-translated queries. Target = 200K per language.
SERVAL reports
This paper
ViDoRe v2
in Nguyen et al. (2025)
32B+7B
ColNomic
ColNomic
ESG
SRSM
59.5
57.3
53.6
BioMed
SMBTIM
63.0
64.2
63.2
Econ
SEMEM
55.5
56.7
55.6
ESG-H
RERB
70.7
73.9
69.4
Average
62.2
63.0
60.4
Appendix
Table 17: Recomputing SERVAL’s ViDoRe v2 score. Scores are NDCG@5 × 100, taken from Table 2 of Nguyen et al. (2025) ; the short names are that paper’s own abbreviations, resolved to ViDoRe datasets through the Hugging Face repository redirects. ColNomic-7B is the one model both papers evaluate, and its two columns bound how far the protocols differ.
Dataset
n
min
mean
median
p95
p99
max
> 512
ViDoRe v1
ArxivQA
500
9
25.3
23
42
58
69
0
DocVQA
500
6
11.8
11
18
23
28
0
InfoVQA
500
6
15.7
15
25
32
42
0
TabFQuAD
280
12
31.5
31
45
52
59
0
TAT-DQA
1,663
7
16.9
16
26
32
48
0
Appendix
Table 18: DistilBERT-tokenized query lengths across the 22 ViDoRe datasets. The longest single query in the entire benchmark (ViDoRe v3 Computer Science, 182 tokens) uses 35.5% of the 512-token budget; 99% of queries fit within 78 tokens (15.2%).
Multi-vector retrievers built on vision-language models lead visual document retrieval (VDR), but they run a multi-billion-parameter query encoder on every search. Distilling this encoder into a small student that queries the teacher's existing index would remove the bottleneck. The standard recipe, however, matches the teacher's MaxSim scores and so requires encoding and caching every training page, which can reach terabytes of page tokens. NanoVDR avoids pages entirely by training on the teacher's query embeddings alone, but only for single-vector retrievers. We present ColNanoVDR, to our knowledge the first framework to bring this document-free distillation to multi-vector VDR. Its objective, OTW (Optimal Transport with Learned Weights), aligns the student's query tokens with the teacher's by entropic optimal transport, with a learned weight for each student token, and needs no correspondence between the two tokenizations. We prove that the resulting alignment cost bounds the MaxSim score difference on every page. Distilled from five state-of-the-art teachers, the 149M text-only students retain about 95% of their teachers' NDCG@5 on ViDoRe v1-v3 while encoding queries up to 26x faster. Under identical training, OTW matches score distillation while encoding no page and reading 12.6x less cached teacher data.
Zhuchenyang Liu, Ziyi Wang, Yao Zhang +1
Aalto University, Finland · Independent Researcher, Netherlands
Visual document retrieval (VDR) is dominated by multi-billion-parameter models that are slow to index at full corpus scale and expensive to serve. Prior compression routes either train a smaller multi-vector encoder from scratch or distil only the query side; neither yields a compact single-vector retriever end-to-end. We present DistilVDR, a 524M end-to-end VDR system distilled bilaterally from a single 8B vision-language teacher under a pointwise cosine alignment loss. All supervision comes from the frozen teacher's embedding space, which was itself trained with relevance supervision, so the student objective needs no relevance labels, negative sampling, or contrastive term. We match VDR's text-query and image-document input asymmetry with an asymmetric encoder-only student that concentrates visual capacity on the document side and keeps the query side at 70M parameters. We release two variants that share the same encoders and training and differ only in the document encoder's visual-tile budget: DistilVDR-HiRes attains 61.74 average NDCG@5 on ViDoRe v1+v2+v3 (86.9% of the 8B teacher) and leads every reproduced sub-1B baseline on the high-resolution-sensitive v3 benchmark, while DistilVDR-Fast attains 59.98 at a 3 times smaller visual-token budget. Both variants store one million documents in a 15.6 times smaller index than the strongest sub-1B multi-vector baseline and index the corpus an order of magnitude faster. The code is available at https://github.com/Ryenhails/NanoVDR.
Zhuchenyang Liu, Ziyi Wang, Yao Zhang +1
Aalto University, Finland · Independent Researcher, Netherlands
Visual Document Retrieval (VDR) models mostly rely on late interaction architectures, in which documents are represented by a set of local patch embeddings and then matched against query tokens. While efficient, this architecture prioritizes local similarity over global layout structure of documents to estimate relevancy between documents and query. In practice, this leads to errors as relevance originates from layout structure of documents with heterogeneous layouts combining figures, tables, and text. We make document layout learnable without changing inference. We propose a multimodal encoder that augments local patch representations with a global layout embedding, trained via textual descriptions encoding document layout information. Across four ViDoRe-v2 datasets, our model improves over the strongest architecturally comparable ColPali/ColQwen baseline by +2.4 nDCG@5 and +2.3 MAP@5, with statistically significant per-dataset gains over ColQwen.
Pascal Tilli, Mohsen Mesgar
University of Stuttgart, Stuttgart, Germany · Bosch Center for Artificial Intelligence, Renningen, Germany