Spherical Interpolation for Backward-Compatible Multimodal Representations

Simone Ricci; Niccolò Biondi; Federico Pernici

cs.CVSep 30, 2026

Spherical Interpolation for Backward-Compatible Multimodal Representations

Authors: Simone Ricci, Niccolò Biondi, Federico Pernici

Organizations: DINFO (Department of Information Engineering), University of Florence, Italy · MICC (Media Integration and Communication Center) · University of Trento, Italy

Abstract

Contrastive vision-language models map visual and textual representations into a shared normalized embedding space, making cosine similarity the natural metric for cross-modal retrieval. A practical challenge arises during model upgrades: independently trained models generally produce incompatible representation spaces, so replacing a deployed model typically requires recomputing embeddings for the entire gallery, which is prohibitively expensive at scale. Orthogonal post-hoc alignment can partially mitigate this problem by mapping new-model queries into the old-model gallery space. However, because independently trained models can differ in fine-grained representation structure, the orthogonal alignment remains approximate, leaving a residual angular discrepancy between the old-model query and the aligned new-model query. We study whether interpolation along the spherical geodesic between these two normalized query representations can improve retrieval without re-indexing the gallery. We characterize when this path contains an interior query direction closer to an idealized retrieval-optimal direction than either endpoint, and connect this characterization to Recall@ $K$ through a local margin-based certification result. Experiments across multiple benchmarks and model families show that post-alignment spherical interpolation improves over orthogonal alignment alone, recovering backward-compatibility in most evaluated settings. Consistent with our geometric characterization, per-query oracle analysis shows that retrieval-favorable interior points occur frequently in practice. Code is available at https://github.com/miccunifi/SLERP_backward_compatibility .

Figures & tables

Panel of Figure 1 — Figure 1 : Residual discrepancy after orthogonal alignment from SigLIP2 to CLIP ViT-B/32. We show two complementary views: the distribution of residual angular mismatch between paired embeddings from the two models, and a 2D t-SNE projection of the corresponding feature spaces. The remaining angular mismatch after alignment suggests that the orthogonal map leaves a residual cross-model gap, motivating a post-alignment interpolation strategy.

Panel of Figure 2 — Figure 2 : Geometry of SLERP in the interpolation plane. The old query $u$ and aligned new query $v$ define a minor geodesic arc. Let $p$ be the projection of the retrieval-optimal direction $q^{\ast}$ onto this plane, and let $\rho=\|p\|$ . The effect of interpolation is determined by the normalized in-plane projection $p/\rho$ when $\rho>0$ . A strict interior improvement occurs exactly when $p/\rho$ lies in the relative interior of the arc; when $\rho=0$ , the angular gap is constant along the arc.

Panel of Figure 3 — Figure 3 : Text-to-image Recall@1 along the SLERP interpolation path for Flickr30k (left) and COCO (right). The image gallery is encoded by the old model, while text queries interpolate between the old-model query ( $\alpha=0$ ) and the SVD-aligned new-model query ( $\alpha=1$ ). Markers indicate the CC3M-selected $\hat{\alpha}$ and the dataset-specific oracle $\alpha^{\star}$ ; dashed lines report the old-model, new-model, and SVD performance.

Panel of Figure 4 — Figure 4 : Flip-rate trade-off for T2I Recall@1. Bars show positive and negative flip rates relative to the old-to-old evaluation. The blue curve reports $\Delta\mathrm{R@1}$ . Here $\alpha=0$ corresponds to old-model queries and $\alpha=1$ to SVD-aligned new-model queries. Red and blue markers denote the dataset-specific oracle $\alpha^{\star}$ and the CC3M-selected weight $\hat{\alpha}$ , respectively.

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 5 — Figure 5 : Empirical comparison of SLERP and NLERP for text-only image-to-text retrieval after SVD alignment. The columns correspond to the two source-target model pairs: CLIP ViT-L/14 $\rightarrow$ CLIP ViT-B/32 on the left and SigLIP2 $\rightarrow$ CLIP ViT-B/32 on the right. The rows correspond to the two retrieval datasets: Flickr30k on the top row and COCO on the bottom row. Across Flickr30k and COCO, and for both source-model pairs, the two interpolation schemes trace nearly identical retrieval curves and attain their best performance in the same interior region of the interpolation path. The small discrepancies arise because a uniform grid in the NLERP weight does not correspond to a uniform grid in angular position along the shared geodesic arc. This supports using SLERP as a geometrically canonical parameterization rather than as a different family of query representations.

Panel of Figure 6 — Figure 6 : Image-to-text Recall@1 along the SLERP path for CLIP ViT-H/14 $\rightarrow$ CLIP ViT-B/32. The text gallery is encoded by the old model, while image queries interpolate between the old-model query and the SVD-aligned new-model query. The case $\alpha=0$ corresponds to old-model queries, while $\alpha=1$ corresponds to SVD-aligned new-model queries. Markers show the CC3M-selected $\hat{\alpha}$ and the dataset-specific test oracle $\alpha^{\star}$ ; dashed lines show old-model, SVD, and new-model reference performance.

Panel of Figure 7 — Figure 7 : Image-to-text Recall@1 along the SLERP path for SigLIP1 $\rightarrow$ CLIP ViT-H/14. The text gallery is encoded by the old model, while image queries interpolate between the old-model query and the SVD-aligned new-model query. The case $\alpha=0$ corresponds to old-model queries, while $\alpha=1$ corresponds to SVD-aligned new-model queries. Markers show the CC3M-selected $\hat{\alpha}$ and the dataset-specific test oracle $\alpha^{\star}$ ; dashed lines show old-model, SVD, and new-model reference performance.

Figure 8 : Recall@1 along the SLERP path under different target-set fractions. Solid curves report the average over three random seeds; shaded regions denote $\pm 1\sigma$ across seeds.

Panel of Figure 9 — Figure 9 : I2T Recall@1 flip-rate trade-off on Flickr30k and COCO. Bars show positive and negative flip rates relative to the old-to-old evaluation. The blue curve reports $\Delta\mathrm{R@1}=\mathrm{PFR}-\mathrm{NFR}$ . Here $\alpha=0$ corresponds to old-model queries and $\alpha=1$ to SVD-aligned new-model queries. Red and blue markers denote the dataset-specific oracle $\alpha^{\star}$ and the CC3M-selected weight $\hat{\alpha}$ , respectively.

Panel of Figure 10 — Figure 10 : NoCaps Recall@1 flip-rate trade-off. We report both T2I and I2T flip rates for the same old–new pairs used in the main paper. Bars show positive and negative flip rates relative to the old-to-old evaluation, and the blue curve reports $\Delta\mathrm{R@1}=\mathrm{PFR}-\mathrm{NFR}$ . Here $\alpha=0$ corresponds to old-model queries and $\alpha=1$ to SVD-aligned new-model queries. Red and blue markers denote the dataset-specific oracle $\alpha^{\star}$ and the CC3M-selected weight $\hat{\alpha}$ , respectively.

Panel of Figure 11 — Figure 11 : Per-query oracle analysis on Flickr30k image-to-text retrieval for CLIP ViT-L/14 $\rightarrow$ CLIP ViT-B/32 with text-only alignment. (a) Distribution of per-query oracle interpolation weights, where each query is assigned the weight attaining the best rank of its highest-ranked relevant item, with ties broken by the highest relevant-item similarity and then by proximity to $\alpha=0.5$ . Queries are divided into those for which Recall@1 is achievable at some interpolation weight (blue, $n=18{,}344$ ) and those for which it is not (gray, $n=12{,}670$ ); shaded columns mark the two endpoints of the interpolation path. (b) Recall@1 obtained with a single fixed interpolation weight applied to all queries, maximized at the dataset-level optimum $\alpha^{\star}=0.5$ . (c) Cosine similarity between the interpolated query and its highest-scoring relevant item at each weight, averaged over Recall@1-achievable queries. Red dashed and cyan dotted lines mark the dataset oracle $\alpha^{\star}=0.5$ and the CC3M-selected weight $\hat{\alpha}=0.7$ , respectively. Because most successful queries attain rank one over many weights, the oracle distribution follows the relevant-item similarity and peaks at $\alpha=0.3$ , rather than at the Recall@1-optimal fixed weight $\alpha^{\star}$ .

Method	R@1
$\blacktriangleright$ Old model	40.62
SVD	42.89
SLERP ( $\hat{\alpha}=0.7$ )	48.00
SLERP ( $\alpha^{\star}=0.5$ )	48.61
$\blacktriangleright$ New model	48.72
Endpoints only (per-query oracle)	54.61

Appendix

Table 9 : Flickr30k I2T Recall@1 (%) for CLIP ViT-L/14

\rightarrow

ViT-B/32 with text-only alignment.

Method	I2T	T2I	$\Delta_{\mathrm{SVD}}$	$\Delta_{\mathrm{old}}$	Comp.
$\blacktriangleright$ Old-model query	53.36	34.26	+1.06	0.00	–
Alignment only
SVD (Procrustes)	51.72	33.77	0.00	$-1.06$	45/90
Affine	31.69	23.10	$-15.35$	$-16.42$	11/90
Ridge	33.96	28.89	$-11.32$	$-12.38$	14/90
One-sided CCA	33.96	28.89	$-11.32$	$-12.38$	14/90

Appendix

Table 10 : Aggregate Recall@1 ablation over

45

configurations (

90

compatibility evaluations) reported in Tabs. 4 , 5 , and 6 .

\Delta_{\mathrm{SVD}}

and

\Delta_{\mathrm{old}}

are averaged over both retrieval directions; Comp. counts evaluations satisfying Eq. 15 .

			Encoder compute (GFLOPs)		Latency (ms)
Serving configuration	Gallery	Query encoders	I2T	T2I	I2T	T2I
Old model	$\phi_{\mathrm{old}}$	$\phi_{\mathrm{old}}$	8.82	5.96	19.93	19.95
Full re-indexing	$\phi_{\mathrm{new}}$	$\phi_{\mathrm{new}}$	162.03	13.30	25.59	20.40
SLERP, one GPU	$\phi_{\mathrm{old}}$	$\phi_{\mathrm{old}}+\phi_{\mathrm{new}}$	170.85	19.26	31.04	25.76
SLERP, two GPUs	$\phi_{\mathrm{old}}$	$\phi_{\mathrm{old}}+\phi_{\mathrm{new}}$	170.85	19.26	25.87	20.70
SLERP, cached old query	$\phi_{\mathrm{old}}$	$\phi_{\mathrm{new}}$	162.03	13.30	25.87	20.68

Appendix

Table 11 : Per-query serving cost for CLIP ViT-L/14

\rightarrow

CLIP ViT-B/32 with a gallery of

10^{7}

items on one NVIDIA A100. Encoder compute is taken from the OpenCLIP profiles and excludes the Procrustes projection (

\approx 0.79

MFLOPs) and the search. Latency is the sum of independently measured median (p50) encoder and IVF-PQ search components; in the two-GPU setting, the old and new encoders are executed concurrently on separate devices.

Figure 12 : Image-to-text Recall@1 during partial backfilling of the text gallery on Flickr30k, with text-only Procrustes support. The partial-backfill baseline issues SVD-aligned new-model queries against a mixed gallery in which an increasing fraction of old-model caption embeddings is replaced by new-model embeddings. At

0\%

, the baseline coincides with the SVD query on the fixed old-model gallery (square), and at

100\%

with full re-indexing (star). SVD+SLERP uses the CC3M-selected

\hat{\alpha}

and leaves the gallery unchanged. Shaded regions mark backfilling fractions at which the baseline falls below the old-model system and thus violates Eq. 15 .

Explore similar work

Multimodal Representation Alignment for Cross-modal Information Retrieval
Jun 10, 2025Fan Xu, Luis A. LeivaMultimodal Alignment Multimodal Retrieval
aDepartment of Computer Science, University of Luxembourg, 6, avenue de la Fonte, Esch-sur-Alzette, L-4364, Luxembourg
RE-TRIANGLE: Does TRIANGLE Enable Multimodal Alignment Beyond Cosine Similarity in Retrieval?
May 22, 2026Arijit Ghosh, Aritra Bandyopadhyay, Chiranjeev Bindra +1Multimodal Alignment Multimodal Retrieval
University of Amsterdam Amsterdam, The Netherlands
Query-Conditioned Spherical Centroid Aggregation for Multimodal Retrieval
Sep 14, 2026Ambuj Mehrish, Anindya Nag, Sebastiano VasconMultimodal Retrieval Center
Department of Environmental Science, Informatics and Statistics Ca’ Foscari University of Venice, Italy

Jun 10, 2025, cs.IRJ/K move · Enter open · S save

Multimodal Representation Alignment for Cross-modal Information Retrieval

Fan Xu, Luis A. Leiva

aDepartment of Computer Science, University of Luxembourg, 6, avenue de la Fonte, Esch-sur-Alzette, L-4364, Luxembourg

Multimodal Alignment Multimodal Retrieval Cross-Modal Contrastive Loss

Open paper AskPDF arXiv

Different machine learning models can represent the same underlying concept in different ways. This variability is particularly valuable for in-the-wild multimodal retrieval, where the objective is to identify the corresponding representation in one modality given another modality as input. This challenge can be effectively framed as a representation alignment problem. For example, given a sentence encoded by a language model, retrieve the most semantically aligned image based on representations produced by an image encoder, or vice versa. To gain insights into the performance impact of different metrics, embedding spaces, and representation alignment for retrieval tasks, we first empirically investigate the geometric relationships between visual and textual embeddings derived from both vision-language models and combined unimodal models. We then align these representations using four standard similarity metrics as well as two learned ones, implemented via neural networks of different architectures with varying losses across multiple benchmarks. Our experimental findings indicate that cosine similarity consistently outperforms all the investigated metrics in representation alignment tasks, and that Wasserstein distance provides a complementary perspective on cross-modal distributional differences. We also observe that our proposed custom contrastive loss is advantageous over the MSE loss for aligning image and text representations, for both multilayer perceptrons and transformer-based models. Taken together, our findings offer novel insights and practical considerations for researchers working in multimodal information retrieval, particularly in real-world, cross-modal applications. Our code is publicly available.

Abstract

Explore similar work

Multimodal Representation Alignment for Cross-modal Information Retrieval

RE-TRIANGLE: Does TRIANGLE Enable Multimodal Alignment Beyond Cosine Similarity in Retrieval?

Query-Conditioned Spherical Centroid Aggregation for Multimodal Retrieval