VIBRA: Backward-Compatible Cross-Modal Retrieval with Consensus Barycenters
Organizations: University of Washington
Abstract
Backward-compatible retrieval enables a new encoder to query the gallery that an old model has already encoded, so that a cross-modal retrieval system can be upgraded without re-encoding billions of items. Existing compatible methods either fit the old query and stay near the accuracy of the old model, or run the old and the new encoder for every query. We propose VIBRA, which matches relevance on the old gallery, corrects the errors of a single model with the barycenter of several views, which is itself a query in the old space, and trains the new encoder to inherit this consensus, so that models absent at deployment still shape every query. We show that the barycenter gains margin over the average of its views exactly when their hardest irrelevant items differ, and we characterize the share of this gain that one encoder can reach. On five benchmarks with five upgrades among six CLIP, MetaCLIP and SigLIP2 backbones, VIBRA is the best compatible method in all 50 settings, improving Recall@1 by up to 5.6 points over the strongest baseline and by up to 18.0 points over the old model, while adding less than 1 ms to the latency of the new encoder.
Figures & tables
| Image text | Text image | |||||||||
| Method | Flickr30k | COCO | NoCaps | CxC | XM3600 | Flickr30k | COCO | NoCaps | CxC | XM3600 |
| CLIP ViT-B/32 CLIP ViT-L/14 | ||||||||||
| Old old | 40.7 | 28.6 | 71.2 | 51.6 | 53.3 | 21.8 | 14.5 | 45.3 | 32.3 | 38.7 |
| New new | 48.7 | 34.4 | 73.4 | 58.0 | 54.1 | 28.2 | 18.7 | 47.8 | 38.2 | 42.5 |
| Drift-Adapter [ 41 ] | 42.4 | 29.1 | 67.4 | 52.4 | 45.9 | 21.7 | 14.1 | 43.9 | 31.9 | 38.6 |
| XBT [ 14 ] | 42.5 † | 30.7 † | 75.0 † | 50.8 ‡ | 50.8 ‡ | 22.4 † | 15.6 † | 48.0 † | 33.9 ‡ | 41.6 ‡ |
| L/14 | B/16 | SO400M | MC L/14 | MC SO | |
|---|---|---|---|---|---|
| Queries with (%) | 91.2 | 90.1 | 93.0 | 92.0 | 88.4 |
| Errors corrected, (%) | 11.3 | 10.9 | 13.4 | 12.5 | 10.0 |
| Errors corrected, (%) | 4.2 | 4.2 | 4.5 | 4.2 | 4.0 |
| Net R@1 gain (points) | 4.7 | 4.6 | 5.2 | 5.3 | 1.2 |
| L/14 | B/16 | SO400M | MC L/14 | MC SO | Mean | |
|---|---|---|---|---|---|---|
| Procrustes map | 48.8 | 48.6 | 46.8 | 50.9 | 56.3 | 50.3 |
| + Relevance matching | 52.4 | 51.2 | 57.6 | 53.3 | 63.8 | 55.7 |
| + Consensus teacher | 54.8 | 52.6 | 58.6 | 55.2 | 64.1 | 57.1 |
| VIBRA (all three) | 55.8 | 53.2 | 58.8 | 55.8 | 64.2 | 57.6 |
Appendix figures & tables9 assets
Supplementary material from the paper’s appendix.
Appendix
| Method | Flickr30k | COCO | NoCaps | CxC | XM3600 | ||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| R@1 | R@5 | R@10 | R@1 | R@5 | R@10 | R@1 | R@5 | R@10 | R@1 | R@5 | R@10 | R@1 | R@5 | R@10 | |
| CLIP ViT-B/32 CLIP ViT-L/14 | |||||||||||||||
| Old old | 40.7 | 64.7 | 73.7 | 28.6 | 50.4 | 59.9 | 71.2 | 91.9 | 96.3 | 51.6 | 77.4 | 85.5 | 53.3 | 79.1 | 86.5 |
| New new | 48.7 | 73.0 | 81.2 | 34.4 | 56.0 | 65.4 | 73.4 | 93.5 | 97.4 | 58.0 | 81.7 | 88.4 | 54.1 | 78.8 | 86.9 |
| Drift-Adapter [ 41 ] | 42.4 | 66.8 | 76.0 | 29.1 | 50.1 | 59.3 | 67.4 | 90.1 | 95.4 | 52.4 | 76.8 | 84.8 | 45.9 | 72.6 | 81.8 |
| XBT [ 14 ] | 42.5 † | 66.0 † | 74.9 † | 30.7 † | 52.6 † | 62.3 † | 75.0 † | 93.3 † | 97.3 † | 50.8 ‡ | 76.0 ‡ | 85.0 ‡ | 50.8 ‡ | 77.2 ‡ | 85.6 ‡ |
| Method | Flickr30k | COCO | NoCaps | CxC | XM3600 | ||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| R@1 | R@5 | R@10 | R@1 | R@5 | R@10 | R@1 | R@5 | R@10 | R@1 | R@5 | R@10 | R@1 | R@5 | R@10 | |
| CLIP ViT-B/32 CLIP ViT-L/14 | |||||||||||||||
| Old old | 21.8 | 41.6 | 51.1 | 14.5 | 30.2 | 38.9 | 45.3 | 75.0 | 84.6 | 32.3 | 58.8 | 70.0 | 38.7 | 66.5 | 76.5 |
| New new | 28.2 | 49.6 | 59.0 | 18.7 | 35.9 | 44.4 | 47.8 | 76.6 | 85.2 | 38.2 | 63.9 | 74.2 | 42.5 | 69.2 | 77.9 |
| Drift-Adapter [ 41 ] | 21.7 | 41.6 | 51.1 | 14.1 | 29.9 | 38.5 | 43.9 | 74.3 | 83.8 | 31.9 | 58.3 | 69.5 | 38.6 | 66.6 | 76.4 |
| XBT [ 14 ] | 22.4 † | 42.7 † | 52.4 † | 15.6 † | 32.3 † | 41.3 † | 48.0 † | 79.0 † | 88.2 † | 33.9 ‡ | 60.6 ‡ | 71.8 ‡ | 41.6 ‡ | 70.7 ‡ | 80.0 ‡ |
| Pair | Old old | New new | Drift-Adapter | XBT | SLERP | LST | -orth | VIBRA | |
|---|---|---|---|---|---|---|---|---|---|
| I T | CLIP ViT-B/32 CLIP ViT-L/14 | 9.0 / 8603 | 16.6 / 711 | 16.6 / 710 | 16.8 / 688 | 25.2 / 659 | 16.5 / 709 | 16.5 / 709 | 16.6 / 709 |
| CLIP ViT-B/32 CLIP ViT-B/16 | 9.0 / 8600 | 8.8 / 2707 | 9.0 / 2702 | 9.1 / 2560 | 17.6 / 2083 | 8.9 / 2693 | 8.9 / 2693 | 8.9 / 2693 | |
| CLIP ViT-B/32 SigLIP2 SO400M/14 | 8.9 / 8588 | 19.0 / 549 | 18.6 / 547 | 18.1 / 590 | 27.1 / 517 | 18.8 / 546 | 18.7 / 546 | 18.9 / 546 | |
| CLIP ViT-B/32 MetaCLIP ViT-L/14 | 8.8 / 8601 | 16.2 / 712 | 16.2 / 711 | 16.4 / 687 | 24.4 / 660 | 16.2 / 710 | 16.3 / 710 | 16.2 / 710 | |
| MetaCLIP ViT-B/32 SigLIP2 SO400M/14 | 8.9 / 8587 | 19.1 / 549 | 19.0 / 547 | 18.3 / 590 | 27.4 / 517 | 19.0 / 546 | 18.9 / 546 | 18.9 / 546 | |
| T I | CLIP ViT-B/32 CLIP ViT-L/14 | 9.1 / 26898 | 9.2 / 19041 | 9.5 / 17226 | 9.6 / 18954 | 18.4 / 11298 | 9.5 / 19018 | 9.4 / 19099 | 10.0 / 18844 |
| Pair | ||||||
|---|---|---|---|---|---|---|
| B/32 L/14 | 48.9 / 33.2 | 50.8 / 33.9 | 52.4 / 34.4 | 53.1 / 34.2 | 51.7 / 32.2 | 53.0 / 34.0 (0.82) |
| B/32 B/16 | 47.1 / 32.9 | 49.7 / 33.7 | 51.2 / 33.9 | 51.9 / 33.4 | 50.6 / 30.9 | 51.9 / 33.1 (0.81) |
| B/32 SO400M | 56.0 / 37.2 | 56.9 / 37.8 | 57.6 / 38.5 | 57.8 / 39.0 | 55.9 / 38.8 | 57.5 / 39.1 (0.84) |
| B/32 MetaCLIP L/14 | 51.9 / 36.1 | 52.5 / 36.6 | 53.3 / 37.1 | 53.0 / 37.0 | 51.6 / 36.5 | 53.0 / 37.1 (0.83) |
| MetaCLIP B/32 SO400M | 62.6 / 38.7 | 63.4 / 39.3 | 63.8 / 40.0 | 64.0 / 40.4 | 62.9 / 40.0 | 63.9 / 40.4 (0.84) |
| Pair | |||||
|---|---|---|---|---|---|
| B/32 L/14 | 48.8 / 30.3 | 52.1 / 33.1 | 53.2 / 34.5 | 52.4 / 34.4 | 50.2 / 32.9 |
| B/32 B/16 | 48.6 / 28.8 | 51.0 / 32.0 | 52.1 / 33.7 | 51.2 / 33.9 | 49.6 / 33.1 |
| B/32 SO400M | 46.8 / 32.8 | 55.3 / 36.7 | 59.0 / 38.6 | 57.6 / 38.5 | 52.9 / 36.8 |
| B/32 MetaCLIP L/14 | 50.9 / 35.0 | 53.9 / 36.8 | 54.5 / 37.5 | 53.3 / 37.1 | 50.9 / 35.5 |
| MetaCLIP B/32 SO400M | 56.3 / 37.2 | 61.4 / 39.4 | 63.7 / 40.1 | 63.8 / 40.0 | 62.3 / 38.7 |
| Pair | New view | Barycenter | Ceiling | Rescued | Needing | All views wrong | Shared |
|---|---|---|---|---|---|---|---|
| B/32 L/14 | 23.0 | 27.7 | 38.0 | 8.0 | 4.7 | 1.2 | 8.7 |
| B/32 B/16 | 22.9 | 27.5 | 37.1 | 7.6 | 4.4 | 1.1 | 9.9 |
| B/32 SO400M | 24.3 | 29.5 | 44.3 | 9.4 | 5.7 | 1.7 | 7.0 |
| B/32 MetaCLIP L/14 | 23.0 | 28.3 | 40.8 | 9.0 | 5.2 | 1.4 | 8.0 |
| MetaCLIP B/32 SO400M | 32.3 | 33.6 | 40.6 | 6.0 | 3.4 | 1.0 | 11.6 |
| Stage 1 | Single view | Teacher | Student | |||||||
|---|---|---|---|---|---|---|---|---|---|---|
| Pair | Procrustes | Head | Old | H/14 | 2 views | 3 views | 2 views | 3 views, | VIBRA | |
| B/32 L/14 | 48.8 | 50.2 | 52.4 | 49.1 | 50.2 | 56.0 | 56.6 | 54.8 | 54.8 | 55.8 |
| B/32 B/16 | 48.6 | 49.6 | 51.2 | 49.1 | 50.2 | 54.3 | 55.8 | 52.7 | 52.6 | 53.2 |
| B/32 SO400M | 46.8 | 52.9 | 57.6 | 49.1 | 50.2 | 58.8 | 59.7 | 58.8 | 58.6 | 58.8 |
| B/32 MetaCLIP L/14 | 50.9 | 50.9 | 53.3 | 49.1 | 50.2 | 56.8 | 57.4 | 55.5 | 55.2 | 55.8 |
| MetaCLIP B/32 SO400M | 56.3 | 62.3 | 63.8 | 54.0 | 54.5 | 64.5 | 64.9 | 64.5 | 64.1 | 64.2 |
| Method | B/32 L/14 | B/32 B/16 | B/32 SO400M | B/32 MetaCLIP L/14 | MetaCLIP B/32 SO400M |
|---|---|---|---|---|---|
| XBT [ 14 ] , paper scale | 10.03 | 4.18 | 18.94 | 10.03 | 18.93 |
| VIBRA (ours) | 0.44 | 0.22 | 0.71 | 0.44 | 0.71 |
| Full backfill, per 1M images | 0.39 | 0.10 | 0.51 | 0.39 | 0.51 |
| Full backfill, per 1B images | 388 | 101 | 506 | 389 | 506 |
| Gallery where backfill costs as much as VIBRA | 1.1M | 2.2M | 1.4M | 1.1M | 1.4M |
| Method | Encoders | Image text | Text image | All |
|---|---|---|---|---|
| Drift-Adapter [ 41 ] | 1 | 14.0 | 0.4 | 6.8 |
| LST [ 20 ] | 1 | 4.2 | ||
| -orth [ 29 ] | 1 | 3.0 | ||
| XBT [ 14 ] | 1 | 6.2 | 0.0 | |
| SLERP [ 30 ] | 2 | 54.4 | 19.9 | 36.1 |
| VIBRA (ours) | 1 | 68.4 | 43.3 | 55.0 |
Cross-Modal Retrieval
Cross-modal retrieval matches queries and items across different data types, such as text, images, video, audio, and sensor signals.
See all 199 papers