Current state-of-the-art multi-vector models are obtained through a small Knowledge Distillation (KD) training step on top of strong single-vector models, leveraging the large-scale pre-training of these models. In this paper, we study the pre-training of multi-vector models and show that large-scale multi-vector pre-training yields much stronger multi-vector models. Notably, a fully ColBERT-pre-trained model, ColBERT-Zero, trained only on public data, outperforms GTE-ModernColBERT as well as its base model, GTE-ModernBERT, which leverages closed and much stronger data, setting new state-of-the-art for model this size. We also find that, although performing only a small KD step is not enough to achieve results close to full pre-training, adding a supervised step beforehand allows to achieve much closer performance while skipping the most costly unsupervised phase. Finally, we find that aligning the fine-tuning and pre-training setups is crucial when repurposing existing models. To enable exploration of our results, we release various checkpoints as well as code used to train them.
Figures & tables
Figure 1: An illustration of the three training phases. Unsupervised contrastive pre-training is a very large scale training relying exclusively on large batch sizes to use in-batch negatives. Supervised contrastive fine-tuning is a refinement step that leverages mined hard negative to provide a stronger signal. The knowledge distillation step use a strong teacher to scores various documents and use KL divergence to make the student distribution fits the teachers’.
Figure 2: The different training pipelines compared in this work. KD only is the most common setup, while ColBERT-Zero is the most expensive. Supervised + KD is an in-between much cheaper than the full pre-training.
Phase
LR sweep range
Batch size
Temp.
Query length
Doc. length
GPU hours
Unsupervised
[3\text{\times}{10}^{-3},1\text{\times}{10}^{-5}]
16384
0.2
39
187
368
Supervised
[2\text{\times}{10}^{-5},8\text{\times}{10}^{-8}]
64
0.2
39
519
32
KD
[1\text{\times}{10}^{-3},1\text{\times}{10}^{-7}]
128⋆
n.a.
39
519
8
Table 1: Training hyperparameters of the 3 different phases of learning. The learning rates are swept logarithmically in the given ranges, testing 10 values for each phase. The query and document lengths include the tokens of the prompts (see Section 3.2 ). GPU hours are measured on a cluster of GH200 GPUs, where nodes have 4 GPU each; unsupervised is done on 8 nodes, while supervised and KD on 1 node.
Model
Avg
FiQA
NFCorpus
TREC-COVID
Touche
ArguAna
Quora
SCIDOCS
SciFact
NQ
ClimateFEVER
HotpotQA
DBPedia
CQADupstack
FEVER
MSMARCO
Baselines
ModernBERT-embed-unsupervised
47.05
42.53
35.33
68.44
18.58
48.82
88.63
19.83
72.30
46.32
22.97
60.00
37.97
42.40
67.39
34.23
ModernBERT-embed-supervised
52.89
40.59
33.40
84.15
31.91
48.96
88.85
18.59
69.63
62.15
35.67
67.11
41.50
42.08
87.35
41.47
GTE-ModernColBERT
54.67
45.28
37.93
83.59
31.23
48.51
86.61
19.06
76.34
61.80
30.62
77.32
48.03
41.00
87.44
45.32
gte-modernbert
55.33
48.81
36.44
81.95
21.68
72.68
88.55
21.29
77.40
57.62
37.74
69.47
41.79
42.63
91.03
40.90
KD from dense supervised
Table 2: Retrieval performance (nDCG@10) across BEIR benchmark datasets.
Model
Avg
FiQA
NFCorpus
TREC-COVID
Touche
ArguAna
Quora
SCIDOCS
SciFact
NQ
ClimateFEVER
HotpotQA
DBPedia
CQADupstack
FEVER
MSMARCO
ModernBERT-embed-unsupervised with prompts
47.05
42.53
35.33
68.44
18.58
48.82
88.63
19.83
72.30
46.32
22.97
60.00
37.97
42.40
67.39
34.23
Supervised + KD with prompts
55.12
41.50
36.51
77.46
33.77
52.45
86.26
18.66
74.90
62.24
37.27
80.07
48.27
41.60
89.71
46.17
Supervised + KD without prompts
54.44
41.30
36.00
79.23
33.52
52.27
87.52
18.28
74.90
61.35
33.25
77.64
47.40
41.18
86.57
46.15
ColBERT-Zero-unsupervised with prompts
51.44
45.38
36.88
67.82
22.59
51.53
87.78
22.30
76.76
58.80
24.24
68.29
43.16
45.76
81.58
38.78
Supervised + KD with prompts
55.43
42.62
37.28
78.69
36.13
53.07
85.24
19.88
76.50
61.66
35.72
79.41
47.48
41.34
90.59
45.80
Supervised + KD without prompts
54.17
42.99
37.49
71.95
27.90
49.29
88.26
20.29
76.85
62.20
32.77
77.91
47.43
44.67
89.36
43.14
Table 3: Retrieval performance (nDCG@10) across BEIR benchmark datasets when performing supervised + KD phases on top of unsupervised models pre-trained with or without prompts and the impact of adding the prompts during fine-tuning.
Appendix figures & tables2 assets
Supplementary material from the paper’s appendix.
Appendix
Settings
Datasets
Method
+Prompt
+Length
Avg
FiQA
NFC
COV
Tou
Arg
Quo
SciD
SciF
NQ
Clim
Hot
DBP
CQA
FEV
MSM
KD from dense supervised (with prompts)
Distilled
✗
✗
52.58
41.58
36.33
78.69
33.30
50.16
82.59
18.01
75.15
60.30
25.00
75.00
45.92
39.39
82.27
45.06
✓
✗
53.83
41.74
36.87
79.65
34.52
51.06
87.31
18.41
76.06
61.24
27.56
75.38
47.48
40.43
84.24
45.47
✗
✓
53.28
41.99
36.41
78.79
36.76
47.77
87.51
18.18
75.21
60.56
25.63
76.34
45.83
39.61
83.42
45.16
✓
✓
54.09
42.51
37.01
79.52
34.58
51.75
87.67
18.15
75.04
61.45
28.31
76.70
47.54
40.68
84.82
45.57
Appendix
Table 4: Retrieval performance on the BEIR benchmark studying the impact of adding prompts and increasing query/document lengths.
Method / Training
Baseline
+Prompt
+Length
+Both
KD from dense supervised (with prompts)
Distilled
52.58
+1.25 (53.83)
+0.70 (53.28)
+1.51 (54.09)
Supervised + KD from dense unsupervised (with prompts)
Supervised
51.33
–0.25 (51.08)
–0.25 (51.08)
–0.61 (50.72)
Distilled
54.44
+0.51 (54.95)
+0.38 (54.82)
+0.68 (55.12)
Supervised + KD from ColBERT unsupervised (with prompts)
Appendix
Table 5: Average BEIR nDCG@10 with delta ( Δ ) for adding prompts and increasing lengths vs. baseline (no prompt, no length)