cs.CVNov 17, 2026

VIBRA: Backward-Compatible Cross-Modal Retrieval with Consensus Barycenters

Authors: Shubing Yang, Xutong Liu, Dongfang Zhao

Organizations: University of Washington

Abstract

Backward-compatible retrieval enables a new encoder to query the gallery that an old model has already encoded, so that a cross-modal retrieval system can be upgraded without re-encoding billions of items. Existing compatible methods either fit the old query and stay near the accuracy of the old model, or run the old and the new encoder for every query. We propose VIBRA, which matches relevance on the old gallery, corrects the errors of a single model with the barycenter of several views, which is itself a query in the old space, and trains the new encoder to inherit this consensus, so that models absent at deployment still shape every query. We show that the barycenter gains margin over the average of its views exactly when their hardest irrelevant items differ, and we characterize the share of this gain that one encoder can reach. On five benchmarks with five upgrades among six CLIP, MetaCLIP and SigLIP2 backbones, VIBRA is the best compatible method in all 50 settings, improving Recall@1 by up to 5.6 points over the strongest baseline and by up to 18.0 points over the old model, while adding less than 1 ms to the latency of the new encoder.

Figures & tables

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. Spherical Interpolation for Backward-Compatible Multimodal Representations

    Sep 30, 2026Simone Ricci, Niccolò Biondi, Federico PerniciCross-Modal AlignmentCross-Modal Retrieval

  2. Multi-modal Knowledge Preserving Adapter for Embedding Backward Compatibility

    Sep 15, 2026Jaeseok Byun, Gukyeong Kwon, Han-Kai Hsu +4Multimodal Large Language ModelsAdapter Tuning

  3. Hyperspherical Simplex Representations for Backward-Compatible Retrieval

    Oct 21, 2026Simone Ricci, Niccolò Biondi, Federico Pernici +1Representation LearningHyperspherical Representation Learning