Current state-of-the-art multi-vector models are obtained through a small Knowledge Distillation (KD) training step on top of strong single-vector models, leveraging the large-scale pre-training of these models. In this paper, we study the pre-training of multi-vector models and show that large-scale multi-vector pre-training yields much stronger multi-vector models. Notably, a fully ColBERT-pre-trained model, ColBERT-Zero, trained only on public data, outperforms GTE-ModernColBERT as well as its base model, GTE-ModernBERT, which leverages closed and much stronger data, setting new state-of-the-art for model this size. We also find that, although performing only a small KD step is not enough to achieve results close to full pre-training, adding a supervised step beforehand allows to achieve much closer performance while skipping the most costly unsupervised phase. Finally, we find that aligning the fine-tuning and pre-training setups is crucial when repurposing existing models. To enable exploration of our results, we release various checkpoints as well as code used to train them.
Figures & tables
Figure 1: An illustration of the three training phases. Unsupervised contrastive pre-training is a very large scale training relying exclusively on large batch sizes to use in-batch negatives. Supervised contrastive fine-tuning is a refinement step that leverages mined hard negative to provide a stronger signal. The knowledge distillation step use a strong teacher to scores various documents and use KL divergence to make the student distribution fits the teachers’.
Figure 2: The different training pipelines compared in this work. KD only is the most common setup, while ColBERT-Zero is the most expensive. Supervised + KD is an in-between much cheaper than the full pre-training.
Phase
LR sweep range
Batch size
Temp.
Query length
Doc. length
GPU hours
Unsupervised
[3\text{\times}{10}^{-3},1\text{\times}{10}^{-5}]
16384
0.2
39
187
368
Supervised
[2\text{\times}{10}^{-5},8\text{\times}{10}^{-8}]
64
0.2
39
519
32
KD
[1\text{\times}{10}^{-3},1\text{\times}{10}^{-7}]
128⋆
n.a.
39
519
8
Table 1: Training hyperparameters of the 3 different phases of learning. The learning rates are swept logarithmically in the given ranges, testing 10 values for each phase. The query and document lengths include the tokens of the prompts (see Section 3.2 ). GPU hours are measured on a cluster of GH200 GPUs, where nodes have 4 GPU each; unsupervised is done on 8 nodes, while supervised and KD on 1 node.
Model
Avg
FiQA
NFCorpus
TREC-COVID
Touche
ArguAna
Quora
SCIDOCS
SciFact
NQ
ClimateFEVER
HotpotQA
DBPedia
CQADupstack
FEVER
MSMARCO
Baselines
ModernBERT-embed-unsupervised
47.05
42.53
35.33
68.44
18.58
48.82
88.63
19.83
72.30
46.32
22.97
60.00
37.97
42.40
67.39
34.23
ModernBERT-embed-supervised
52.89
40.59
33.40
84.15
31.91
48.96
88.85
18.59
69.63
62.15
35.67
67.11
41.50
42.08
87.35
41.47
GTE-ModernColBERT
54.67
45.28
37.93
83.59
31.23
48.51
86.61
19.06
76.34
61.80
30.62
77.32
48.03
41.00
87.44
45.32
gte-modernbert
55.33
48.81
36.44
81.95
21.68
72.68
88.55
21.29
77.40
57.62
37.74
69.47
41.79
42.63
91.03
40.90
KD from dense supervised
Table 2: Retrieval performance (nDCG@10) across BEIR benchmark datasets.
Model
Avg
FiQA
NFCorpus
TREC-COVID
Touche
ArguAna
Quora
SCIDOCS
SciFact
NQ
ClimateFEVER
HotpotQA
DBPedia
CQADupstack
FEVER
MSMARCO
ModernBERT-embed-unsupervised with prompts
47.05
42.53
35.33
68.44
18.58
48.82
88.63
19.83
72.30
46.32
22.97
60.00
37.97
42.40
67.39
34.23
Supervised + KD with prompts
55.12
41.50
36.51
77.46
33.77
52.45
86.26
18.66
74.90
62.24
37.27
80.07
48.27
41.60
89.71
46.17
Supervised + KD without prompts
54.44
41.30
36.00
79.23
33.52
52.27
87.52
18.28
74.90
61.35
33.25
77.64
47.40
41.18
86.57
46.15
ColBERT-Zero-unsupervised with prompts
51.44
45.38
36.88
67.82
22.59
51.53
87.78
22.30
76.76
58.80
24.24
68.29
43.16
45.76
81.58
38.78
Supervised + KD with prompts
55.43
42.62
37.28
78.69
36.13
53.07
85.24
19.88
76.50
61.66
35.72
79.41
47.48
41.34
90.59
45.80
Supervised + KD without prompts
54.17
42.99
37.49
71.95
27.90
49.29
88.26
20.29
76.85
62.20
32.77
77.91
47.43
44.67
89.36
43.14
Table 3: Retrieval performance (nDCG@10) across BEIR benchmark datasets when performing supervised + KD phases on top of unsupervised models pre-trained with or without prompts and the impact of adding the prompts during fine-tuning.
Appendix figures & tables2 assets
Supplementary material from the paper’s appendix.
Appendix
Settings
Datasets
Method
+Prompt
+Length
Avg
FiQA
NFC
COV
Tou
Arg
Quo
SciD
SciF
NQ
Clim
Hot
DBP
CQA
FEV
MSM
KD from dense supervised (with prompts)
Distilled
✗
✗
52.58
41.58
36.33
78.69
33.30
50.16
82.59
18.01
75.15
60.30
25.00
75.00
45.92
39.39
82.27
45.06
✓
✗
53.83
41.74
36.87
79.65
34.52
51.06
87.31
18.41
76.06
61.24
27.56
75.38
47.48
40.43
84.24
45.47
✗
✓
53.28
41.99
36.41
78.79
36.76
47.77
87.51
18.18
75.21
60.56
25.63
76.34
45.83
39.61
83.42
45.16
✓
✓
54.09
42.51
37.01
79.52
34.58
51.75
87.67
18.15
75.04
61.45
28.31
76.70
47.54
40.68
84.82
45.57
Appendix
Table 4: Retrieval performance on the BEIR benchmark studying the impact of adding prompts and increasing query/document lengths.
Method / Training
Baseline
+Prompt
+Length
+Both
KD from dense supervised (with prompts)
Distilled
52.58
+1.25 (53.83)
+0.70 (53.28)
+1.51 (54.09)
Supervised + KD from dense unsupervised (with prompts)
Supervised
51.33
–0.25 (51.08)
–0.25 (51.08)
–0.61 (50.72)
Distilled
54.44
+0.51 (54.95)
+0.38 (54.82)
+0.68 (55.12)
Supervised + KD from ColBERT unsupervised (with prompts)
Appendix
Table 5: Average BEIR nDCG@10 with delta ( Δ ) for adding prompts and increasing lengths vs. baseline (no prompt, no length)
Reproducibility must validate architectural robustness, not just numerical accuracy. We evaluate ColBERT-v2 and ConstBERT across five dimensions, finding that while ConstBERT reproduces within 0.05% MRR@10 on MS-MARCO, both models show a drop of 86-97% on long, narrative queries (TREC ToT 2025). Ablations prove this failure is architectural: performance plateaus at 20 words because the MaxSim operator's uniform token weighting cannot distinguish signal from filler noise. Furthermore, undocumented backend parameters create an 8-point gap due to ConstBERT's sparse centroid coverage, and fine-tuning with 3x more data actually degrades performance by up to 29%. We conclude that architectural constraints in multi-vector retrieval cannot be overcome by adaptation alone. Code: https://github.com/utshabkg/multi-vector-reproducibility.
We previously reported a ModernBERT encoder trained from scratch for Turkish (MoganBERT-TR) and a single-vector embedding model built on top of it (MoganBERT-embed). This work introduces the third model in that lineage: MoganColBERT-TR, a multi-vector retrieval model that, instead of compressing a query or a document into a single vector, represents it at the token level through a 768->128 projection and scores it with MaxSim late interaction. The model is not trained from scratch: the embedding model's encoder is taken as the starting point and adapted to the ColBERT objective with a single-epoch distillation phase. Training data is produced from two sources - title-to-passage pairs carved out of our own pretraining corpus in the character domain and at sentence boundaries, and two Turkish question-based retrieval sets - and is distilled from the soft scores of a cross-encoder teacher (bge-reranker-v2-m3) over one positive and seven mined negatives. We show that in hard negative mining, rank-based skipping alone is insufficient and must be combined with a group mask and a cosine ceiling. Evaluation is carried out with the official pipeline of TurkColBERT, a benchmark built for Turkish late-interaction retrieval (PLAID index, exact MaxSim), on five Turkish BEIR datasets; none of them appears in our training pool, so all five results are clean zero-shot. With 148.9M parameters, MoganColBERT-TR reaches an overall score of 37.36 (35.53 nDCG@100, 31.81 nDCG@10) averaged over the five datasets and finishes second among the five models compared: it outperforms the twice-as-large ColmmBERT-base-TR on four of five datasets and by +3.05 overall, and the benchmark's largest model by +12.30. The gap to the leading model (mLateOn) is concentrated on ArguAna-TR, the dataset with by far the longest queries.
Furkan Yilmaz, Habibe Aleyna Tasdemir, Muhammed Faruk Gozay
State-of-the-art retrieval models increasingly rely on closed training data, creating a reproducibility gap. We present an open end-to-end recipe for training retrieval models and study how English supervision transfers to multilingual retrieval through translate-train. We first reconstruct and curate 665M English contrastive pre-training pairs from 1.4B pairs across 34 public sources and build 1.88M supervised fine-tuning pairs with mined hard negatives. Training yields two 149M-parameter models: DenseOn, a single-vector dense model, and LateOn, a ColBERT-style late-interaction model. They achieve 56.20 and 57.22 average nDCG@10 on BEIR, respectively, setting new state-of-the-art results for this size class. We then translate the validated English data into eight languages, yielding 2.8B pairs with cross-lingual samples, and train mDenseOn and mLateOn, two 307M-parameter models built on mmBERT-base. Despite sharing their backbone, data, and objectives, their representations behave differently: the dense model is strong on English and translated languages but degrades outside translate-train support, whereas the late-interaction model generalizes better to unseen languages and scripts. This suggests that token-level matching turns translate-train from a target-language expansion strategy into a multilingual generalization recipe. We publicly release the models, datasets, and training code.
Raphaël Sourty, Antoine Chaffin, Paulo Roberto Moura Junior +1