Visual-language models (VLMs) excel at data mappings, but real-world document heterogeneity and unstructuredness disrupt the consistency of cross-modal embeddings. Recent late-interaction methods enhance image-text alignment through multi-vector representations, yet traditional training with limited samples and static strategies cannot adapt to the model's dynamic evolution, causing cross-modal retrieval confusion. To overcome this, we introduce Evo-Retriever, a retrieval framework featuring an LLM-guided curriculum evolution built upon a novel Viewpoint-Pathway collaboration. First, we employ multi-view image alignment to enhance fine-grained matching via multi-scale and multi-directional perspectives. Then, a bidirectional contrastive learning strategy generates "hard queries" and establishes complementary learning paths for visual and textual disambiguation to rebalance supervision. Finally, the model-state summary from the above collaboration is fed into an LLM meta-controller, which adaptively adjusts the training curriculum using expert knowledge to promote the model's evolution. On ViDoRe V2 and MMEB (VisDoc), Evo-Retriever achieves state-of-the-art performance, with nDCG@5 scores of 65.2% and 77.1%.
Figures & tables
Figure 1: Challenges in complex visual document retrieval: (a) Insufficient Spatial Awareness : Models must extract and integrate dispersed information (e.g., combining "Plastic 12%" and "Metal 8%") from dynamic layouts. (b) Intrinsic Vulnerability to Textual Confusion : Models must identify differentiated needs in obfuscated queries (e.g., franchisee info and corporate chart). (c) Stagnation from a Static Curriculum : The learning effectiveness of the initial negative sample set (e.g., containing only "salary") gradually diminishes under a fixed curriculum.
Figure 2: The motivation behind LLM-guided evolutionary curriculum. (a) A static threshold is effective in the early training stage, providing informative gradient signals. (b) As the model converges, the same threshold captures only trivial negative samples, leading to diminished gradients. (c) Our LLM-guided evolutionary curriculum adjusts the mining interval to continually discover challenging negatives, ensuring sustained optimization.
Figure 3: Evo-Retriever enhances representation via multi-view (Evo. 1) and bidirectional contrast (Evo. 2). Guided by expert knowledge, this Viewpoint-Pathway collaboration dynamically mines hard samples (Evo. 3), enabling continual evolution of the model.
Figure 4: Overview of the proposed LLM-EC.
Model
Size (B)
Avg.
MIT
Economics
ESG Restaurant
ESG Restaurant
Biomedical
Macro
Human
Synthetic
Multilingual
Multilingual
English
Multilingual
llama-nemoretriever-3b-v1
4.4
63.5
64.3
55.9
75.4
58.6
llama-nemoretriever-1b-v1
2.4
62.1
62.9
53.2
76.4
55.9
colnomic-embed-multimodal-7b
7.0
60.8
63.4
56.2
68.7
54.8
colqwen2.5-v0.2
3.0
59.3
59.2
53.3
66.4
58.3
Table 1: Performance comparison on the ViDoRe V2 benchmark. All scores are nDCG@5 (%). Bold and underlined numbers denote the 1st and 2nd scores, respectively. Our models achieve SOTA performance, especially on challenging multilingual subsets.
Model
Size (B)
Avg.
ViDoRe V1
ViDoRe V2
VisRAG
VisDoc-OOD
gme-Qwen2-VL-7B-Instruct
8.3
75.18
89.44
55.61
84.99
44.40
Seed-1.6-embedding
-
73.44
85.53
56.57
84.74
43.14
gme-Qwen2-VL-2B-Instruct
2.2
72.71
86.15
53.96
82.52
43.12
colpali-v1.3
2.9
70.97
83.60
51.98
81.13
43.12
Ops-MM-embedding-v1-7B
8.3
70.34
80.05
59.59
79.32
43.34
Ops-MM-embedding-v1-2B
2.2
66.96
76.39
53.18
77.64
41.17
Table 2: Performance comparison on the MMEB (VisDoc) benchmark. All scores are nDCG@linear 5 (%). Bold and underlined numbers denote the 1st and 2nd scores, respectively. Our models demonstrate superior generalization and robustness across a wide range of document understanding tasks.
ID
Configuration
nDCG@5
(a)
Baseline ( Net0 )
61.17
— Multi-view Alignment —
(b)
Downsample-only
59.75
(c)
Stitched-only
58.59
(d)
+ MVA ( Net1 )
62.25
— Bidirectional Learning —
Table 3: Ablation study on ViDoRe V2 (3B model). Bold numbers denote the best scores.
Factor
Setting
nDCG@5
1. Exploration Phase
Enabled (default, 60 steps)
63.05
Disabled
61.86
2. Difficulty Granularity
10 Intervals (Coarse)
62.89
16 Intervals (Default)
63.05
Table 4: Analysis of key design choices for LLM-EC, showing deviations from the default setting. Bold numbers denote the best scores.
Figure 5: Impact of Temperature τ on the Normalized Gradient Profile. The plot shows the normalized gradient, σ(Δs/τ) , as a function of the similarity gap Δs . A lower τ (e.g., our setting of 0.02, red line) creates a much steeper and narrower transition zone where gradients become substantial. This highlights the need for a curriculum that can precisely target negatives within this narrow “sweet spot” to ensure efficient learning.
Action ID
Difficulty Range
Low-Signal Zone
A
[0.70,0.85]
B
[0.70,0.90]
C
[0.70,0.92]
D
[0.75,0.90]
Effective-Learning Zone
Table 5: The action space of M=16 overlapping sampling policies.
τ
nDCG@5
1.0
57.37
0.02 (Ours)
63.05
Table 6: Ablation on the temperature τ in the margin loss on ViDoRe V2 (3B model). All scores are nDCG@5 (%). Bold numbers denote the best score.
Partitioning Strategy
nDCG@5
Uniform + Overlap
61.89
Non-uniform + Overlap (Ours)
63.05
Table 7: Effect of interval density on ViDoRe V2 (3B model). All scores are nDCG@5 (%). Bold marks the best result.
Transition steps
Lock-in steps
nDCG@5
50
50
62.18
200 (Ours)
200 (Ours)
63.05
Table 8: Ablation on Transition and Lock-in phase lengths for LLM-EC (3B, ViDoRe V2). All scores are nDCG@5 (%). Bold numbers denote the best score.
Configuration
nDCG@5
baseline (Net0)
64.03
Fixed Window 80–98%
64.32
Linear Curriculum
64.29
Rule-based Oracle
64.36
LLM-EC (Ours)
65.20
Table 9: Curriculum ablation on Qwen3-VL-4B-Instruct. All scores are nDCG@5 (%). Bold numbers denote the best score.
Scheduler
Default
Perturbed
Drop
Rule-based Oracle
62.81
61.75
−1.06
LLM-EC (Ours)
63.05
62.40
−0.65
Table 10: Robustness to threshold perturbations on ViDoRe V2 (Evo-Retriever-3B). In the Lock-in phase, thresholds in the Upgrade and Downgrade Conditions are perturbed: Lend<0.3→0.36 , relative loss reduction for upgrade 50%→60% , and relative loss increase for downgrade 30%→25.5% . Drop = Default − Perturbed. All scores are nDCG@5 (%).
Figure 6: Curriculum trajectories during training for Evo-Retriever-3B. Thick lines indicate the selected difficulty interval, while thin lines show the hard-negative loss. The rule-based oracle follows monotonic threshold-triggered jumps, whereas LLM-EC adapts difficulty according to training dynamics and performs rollback when instability is detected.