Backward-compatible retrieval enables a new encoder to query the gallery that an old model has already encoded, so that a cross-modal retrieval system can be upgraded without re-encoding billions of items. Existing compatible methods either fit the old query and stay near the accuracy of the old model, or run the old and the new encoder for every query. We propose VIBRA, which matches relevance on the old gallery, corrects the errors of a single model with the barycenter of several views, which is itself a query in the old space, and trains the new encoder to inherit this consensus, so that models absent at deployment still shape every query. We show that the barycenter gains margin over the average of its views exactly when their hardest irrelevant items differ, and we characterize the share of this gain that one encoder can reach. On five benchmarks with five upgrades among six CLIP, MetaCLIP and SigLIP2 backbones, VIBRA is the best compatible method in all 50 settings, improving Recall@1 by up to 5.6 points over the strongest baseline and by up to 18.0 points over the old model, while adding less than 1 ms to the latency of the new encoder.
Figures & tables
Figure 1: Four ways to upgrade the query encoder. Only VIBRA (d) keeps the gallery and one encoder per query while improving accuracy.
Figure 2: Overview of VIBRA. (a) Training workflow. Sec. 3.2 fits the stage 1 query q0 with the relevance loss LRC ; Sec. 3.3 pools the views qv offline into the teacher T with weights cv ; Sec. 3.4 trains a low rank update ΔW so that qθ matches T , reaching the share ΠLG of the gain G . (b) Retrieval example, in which an image query and a text query each run one encoder and retrieve the correct item from the old gallery.
Figure 3: Consensus correction on the sphere. Bc projects the mean of the views (hollow point) onto the sphere and ranks y+ first inside the green cone. (a) Views share their hardest irrelevant caption, so D=0 and Bc fails. (b) Their hardest irrelevant captions differ, so D>0 and Bc succeeds while every view fails.
Image → text
Text → image
Method
Flickr30k
COCO
NoCaps
CxC
XM3600
Flickr30k
COCO
NoCaps
CxC
XM3600
CLIP ViT-B/32 → CLIP ViT-L/14
Old → old
40.7
28.6
71.2
51.6
53.3
21.8
14.5
45.3
32.3
38.7
New → new
48.7
34.4
73.4
58.0
54.1
28.2
18.7
47.8
38.2
42.5
Drift-Adapter [ 41 ]
42.4
29.1
67.4
52.4
45.9
21.7
14.1
43.9
31.9
38.6
XBT [ 14 ]
42.5 †
30.7 †
75.0 †
50.8 ‡
50.8 ‡
22.4 †
15.6 †
48.0 †
33.9 ‡
41.6 ‡
Table 1: Recall@1 (%) against the old gallery. Gray: references. † published; ‡ our reproduction of XBT. Bold : best compatible method.
Figure 4: Recall@1 against query latency at batch size 1. Each marker is one new backbone, and the dashed line is the frontier of the baselines. Points toward the upper left are better.
L/14
B/16
SO400M
MC L/14
MC → SO
Queries with D>0 (%)
91.2
90.1
93.0
92.0
88.4
Errors corrected, D>0 (%)
11.3
10.9
13.4
12.5
10.0
Errors corrected, D=0 (%)
4.2
4.2
4.5
4.2
4.0
Net R@1 gain (points)
4.7
4.6
5.2
5.3
1.2
Table 2: Consensus correction split by the condition of Proposition 1 (image → text, one positive caption per image). Errors corrected: share of the errors of the new view that the barycenter corrects. Columns name the new model for CLIP ViT-B/32; MC → SO: MetaCLIP ViT-B/32 → SigLIP2 SO400M/14.
L/14
B/16
SO400M
MC L/14
MC → SO
Mean
Procrustes map
48.8
48.6
46.8
50.9
56.3
50.3
+ Relevance matching
52.4
51.2
57.6
53.3
63.8
55.7
+ Consensus teacher
54.8
52.6
58.6
55.2
64.1
57.1
VIBRA (all three)
55.8
53.2
58.8
55.8
64.2
57.6
Table 3: Ablation of the components (image → text R@1, %); each row adds one component. Columns as in Table 2 .
Figure 5: Sensitivity of image → text R@1 to one hyperparameter at a time.
Appendix figures & tables9 assets
Supplementary material from the paper’s appendix.
Appendix
Method
Flickr30k
COCO
NoCaps
CxC
XM3600
R@1
R@5
R@10
R@1
R@5
R@10
R@1
R@5
R@10
R@1
R@5
R@10
R@1
R@5
R@10
CLIP ViT-B/32 → CLIP ViT-L/14
Old → old
40.7
64.7
73.7
28.6
50.4
59.9
71.2
91.9
96.3
51.6
77.4
85.5
53.3
79.1
86.5
New → new
48.7
73.0
81.2
34.4
56.0
65.4
73.4
93.5
97.4
58.0
81.7
88.4
54.1
78.8
86.9
Drift-Adapter [ 41 ]
42.4
66.8
76.0
29.1
50.1
59.3
67.4
90.1
95.4
52.4
76.8
84.8
45.9
72.6
81.8
XBT [ 14 ]
42.5 †
66.0 †
74.9 †
30.7 †
52.6 †
62.3 †
75.0 †
93.3 †
97.3 †
50.8 ‡
76.0 ‡
85.0 ‡
50.8 ‡
77.2 ‡
85.6 ‡
Appendix
Table S1 : Image → text retrieval against the old gallery (R@1 / R@5 / R@10, %). The old gallery is never reencoded. Every method except SLERP encodes the query with one network; SLERP runs the old and the new encoder. Gray rows are references: Old → old keeps the old model, New → new reencodes the whole gallery. † published number. ‡ our reproduction of XBT at the scale of its paper, used where the paper reports no number. Bold marks the best compatible method.
Method
Flickr30k
COCO
NoCaps
CxC
XM3600
R@1
R@5
R@10
R@1
R@5
R@10
R@1
R@5
R@10
R@1
R@5
R@10
R@1
R@5
R@10
CLIP ViT-B/32 → CLIP ViT-L/14
Old → old
21.8
41.6
51.1
14.5
30.2
38.9
45.3
75.0
84.6
32.3
58.8
70.0
38.7
66.5
76.5
New → new
28.2
49.6
59.0
18.7
35.9
44.4
47.8
76.6
85.2
38.2
63.9
74.2
42.5
69.2
77.9
Drift-Adapter [ 41 ]
21.7
41.6
51.1
14.1
29.9
38.5
43.9
74.3
83.8
31.9
58.3
69.5
38.6
66.6
76.4
XBT [ 14 ]
22.4 †
42.7 †
52.4 †
15.6 †
32.3 †
41.3 †
48.0 †
79.0 †
88.2 †
33.9 ‡
60.6 ‡
71.8 ‡
41.6 ‡
70.7 ‡
80.0 ‡
Appendix
Table S2 : Text → image retrieval against the old gallery (R@1 / R@5 / R@10, %). The old gallery is never reencoded. Every method except SLERP encodes the query with one network; SLERP runs the old and the new encoder. Gray rows are references: Old → old keeps the old model, New → new reencodes the whole gallery. † published number. ‡ our reproduction of XBT at the scale of its paper, used where the paper reports no number. Bold marks the best compatible method.
Pair
Old → old
New → new
Drift-Adapter
XBT
SLERP
LST
λ -orth
VIBRA
I → T
CLIP ViT-B/32 → CLIP ViT-L/14
9.0 / 8603
16.6 / 711
16.6 / 710
16.8 / 688
25.2 / 659
16.5 / 709
16.5 / 709
16.6 / 709
CLIP ViT-B/32 → CLIP ViT-B/16
9.0 / 8600
8.8 / 2707
9.0 / 2702
9.1 / 2560
17.6 / 2083
8.9 / 2693
8.9 / 2693
8.9 / 2693
CLIP ViT-B/32 → SigLIP2 SO400M/14
8.9 / 8588
19.0 / 549
18.6 / 547
18.1 / 590
27.1 / 517
18.8 / 546
18.7 / 546
18.9 / 546
CLIP ViT-B/32 → MetaCLIP ViT-L/14
8.8 / 8601
16.2 / 712
16.2 / 711
16.4 / 687
24.4 / 660
16.2 / 710
16.3 / 710
16.2 / 710
MetaCLIP ViT-B/32 → SigLIP2 SO400M/14
8.9 / 8587
19.1 / 549
19.0 / 547
18.3 / 590
27.4 / 517
19.0 / 546
18.9 / 546
18.9 / 546
T → I
CLIP ViT-B/32 → CLIP ViT-L/14
9.1 / 26898
9.2 / 19041
9.5 / 17226
9.6 / 18954
18.4 / 11298
9.5 / 19018
9.4 / 19099
10.0 / 18844
Appendix
Table S3 : Query latency on the Large COCO galleries: ms per query at batch size 1 / queries per second at batch size 256. One A100-SXM4-40GB, bf16, exact top 10 search over the old gallery (175,680 captions for image → text, 35,136 images for text → image); preprocessing and tokenization excluded.
Pair
λ=0
0.25
0.5
0.75
1
λ∗
B/32 → L/14
48.9 / 33.2
50.8 / 33.9
52.4 / 34.4
53.1 / 34.2
51.7 / 32.2
53.0 / 34.0 (0.82)
B/32 → B/16
47.1 / 32.9
49.7 / 33.7
51.2 / 33.9
51.9 / 33.4
50.6 / 30.9
51.9 / 33.1 (0.81)
B/32 → SO400M
56.0 / 37.2
56.9 / 37.8
57.6 / 38.5
57.8 / 39.0
55.9 / 38.8
57.5 / 39.1 (0.84)
B/32 → MetaCLIP L/14
51.9 / 36.1
52.5 / 36.6
53.3 / 37.1
53.0 / 37.0
51.6 / 36.5
53.0 / 37.1 (0.83)
MetaCLIP B/32 → SO400M
62.6 / 38.7
63.4 / 39.3
63.8 / 40.0
64.0 / 40.4
62.9 / 40.0
63.9 / 40.4 (0.84)
Appendix
Table S4: Mixing weight λ of the stage 1 target (stage 1 query q0 ; mean R@1 over the five sets, image → text / text → image, %). λ=0 uses the label only and λ=1 the ranking of the new model only; λ∗ is the minimum variance weight estimated on COCO DEV.
Pair
β=0
0.25
0.5
0.75
1
B/32 → L/14
48.8 / 30.3
52.1 / 33.1
53.2 / 34.5
52.4 / 34.4
50.2 / 32.9
B/32 → B/16
48.6 / 28.8
51.0 / 32.0
52.1 / 33.7
51.2 / 33.9
49.6 / 33.1
B/32 → SO400M
46.8 / 32.8
55.3 / 36.7
59.0 / 38.6
57.6 / 38.5
52.9 / 36.8
B/32 → MetaCLIP L/14
50.9 / 35.0
53.9 / 36.8
54.5 / 37.5
53.3 / 37.1
50.9 / 35.5
MetaCLIP B/32 → SO400M
56.3 / 37.2
61.4 / 39.4
63.7 / 40.1
63.8 / 40.0
62.3 / 38.7
Appendix
Table S5: Weight β of the stage 1 head in q0=unit(βf(z)+(1−β)unit(P⊤z)) (mean R@1 over the five sets, image → text / text → image, %). β=0 is the Procrustes map alone, β=1 the head alone; the deployed value is β=0.75 .
Pair
New view
Barycenter
Ceiling
Rescued
Needing D
All views wrong
Shared
B/32 → L/14
23.0
27.7
38.0
8.0
4.7
1.2
8.7
B/32 → B/16
22.9
27.5
37.1
7.6
4.4
1.1
9.9
B/32 → SO400M
24.3
29.5
44.3
9.4
5.7
1.7
7.0
B/32 → MetaCLIP L/14
23.0
28.3
40.8
9.0
5.2
1.4
8.0
MetaCLIP B/32 → SO400M
32.3
33.6
40.6
6.0
3.4
1.0
11.6
Appendix
Table S6: Consensus correction on the test sets, extending Table 2 of the main paper (image → text, one designated positive caption per image, other positives removed from the negatives; mean over the five sets, %). Rescued: the barycenter ranks the positive first while the new view does not. Needing D : rescued although the weighted average of the view margins is not positive. All views wrong: rescued although every view ranks the positive lower. Shared: all three views have the same hardest irrelevant item.
Stage 1
Single view
Teacher
Student
Pair
Procrustes
Head
q0
Old
H/14
2 views
3 views
2 views
3 views, μ=0
VIBRA
B/32 → L/14
48.8
50.2
52.4
49.1
50.2
56.0
56.6
54.8
54.8
55.8
B/32 → B/16
48.6
49.6
51.2
49.1
50.2
54.3
55.8
52.7
52.6
53.2
B/32 → SO400M
46.8
52.9
57.6
49.1
50.2
58.8
59.7
58.8
58.6
58.8
B/32 → MetaCLIP L/14
50.9
50.9
53.3
49.1
50.2
56.8
57.4
55.5
55.2
55.8
MetaCLIP B/32 → SO400M
56.3
62.3
63.8
54.0
54.5
64.5
64.9
64.5
64.1
64.2
Appendix
Table S7 : Full ablation, extending Table 3 of the main paper. Contribution of each component (image → text R@1, mean over the five datasets, %; students with seed 0). Stage 1: the Procrustes map alone ( β=0 ), the head alone ( β=1 ) and q0 . Single views and teachers are queries in the old space; the teacher of two views drops the auxiliary encoder. Students: trained on the teacher of two views without the auxiliary term, on the teacher of three views without the auxiliary term ( μ=0 ), and VIBRA.
Method
B/32 → L/14
B/32 → B/16
B/32 → SO400M
B/32 → MetaCLIP L/14
MetaCLIP B/32 → SO400M
XBT [ 14 ] , paper scale
10.03
4.18
18.94
10.03
18.93
VIBRA (ours)
0.44
0.22
0.71
0.44
0.71
Full backfill, per 1M images
0.39
0.10
0.51
0.39
0.51
Full backfill, per 1B images
388
101
506
389
506
Gallery where backfill costs as much as VIBRA
1.1M
2.2M
1.4M
1.1M
1.4M
Appendix
Table S8: Training cost in A100 hours (bf16, preprocessing and tokenization excluded, one configuration and one seed), counting the encoding of the training data with every encoder a method needs and the optimization, and the cost of a full backfill with the new model.
Method
Encoders
Image → text
Text → image
All
Drift-Adapter [ 41 ]
1
14.0
0.4
6.8
LST [ 20 ]
1
4.2
−7.5
−2.0
λ -orth [ 29 ]
1
−17.6
3.0
−6.7
XBT [ 14 ]
1
−7.0
6.2
0.0
SLERP [ 30 ]
2
54.4
19.9
36.1
VIBRA (ours)
1
68.4
43.3
55.0
Appendix
Table S9: Update gain [ 36 ] (%), the share of the R@1 gain of reencoding that a method obtains without reencoding, over all pairs and datasets.
DINFO (Department of Information Engineering), University of Florence, Italy · MICC (Media Integration and Communication Center) · University of Trento, Italy
Media Integration and Communication Center (MICC) Dipartimento di Ingegneria dell’Informazione University of Florence · Multimedia & Human Understanding (MHUG) Department of Information Engineering and Computer Science University of Trento