Hyperspherical Simplex Representations for Backward-Compatible Retrieval

Simone Ricci; Niccolò Biondi; Federico Pernici; Alberto Del Bimbo

cs.CVOct 21, 2026

Hyperspherical Simplex Representations for Backward-Compatible Retrieval

Authors: Simone Ricci, Niccolò Biondi, Federico Pernici, Alberto Del Bimbo

Organizations: Media Integration and Communication Center (MICC) Dipartimento di Ingegneria dell’Informazione University of Florence · Multimedia & Human Understanding (MHUG) Department of Information Engineering and Computer Science University of Trento

Abstract

Training modern AI models has become increasingly expensive, and updating deployed models can alter their internal feature representations, affecting applications built upon them. In retrieval systems, such updates often require complete gallery re-indexing, a computationally expensive process at scale. Existing backward-compatibility methods enable direct comparison of representations from updated and earlier models, avoiding re-indexing. However, they typically require compatibility-specific training, learned mapping functions, or architectural modifications, and often depend explicitly on the previous model. We propose hyperspherical simplex representations derived from softmax outputs and logits for backward-compatible retrieval across independently trained models. They leverage the simplex structure induced by softmax cross-entropy training with one-hot labels as a common geometric reference across model updates. Since this reference depends on the number of classes, adding new ones expands the simplex, misaligning existing class prototypes with their earlier counterparts. We therefore derive a deterministic linear transformation to restore their alignment. Our theoretical analysis of the formal backward-compatibility definition identifies two geometric conditions under which its inequalities hold in expectation, assuming von Mises--Fisher class distributions with increasing concentration across updates. Aligned regular-simplex representations satisfy these conditions, providing a theoretical basis for the proposed representations under the stated assumptions. Without using additional compatibility-specific losses, learned mappings, or architectural modifications, the proposed representations achieve competitive results on standard compatibility benchmarks, including expanded class sets, more expressive architectures, and independently trained models from public repositories.

Figures & tables

Figure 1: Softmax outputs lie in the probability simplex $\Delta^{C^{k}-1}$ (colored area), centered at $\mathbf{o}^{k}$ , and are driven during training toward its vertices $\mathbf{e}_{c}$ , which correspond to the one-hot target labels. The projection $\mathbf{P}_{k,k}$ shifts the simplex center to the origin, mapping softmax outputs onto a regular simplex whose vertices define the class prototypes $\mathbf{v}^{k}_{c}$ (colored vectors). The features $\mathbf{h}^{k}$ of Eq. 4 are then obtained by $\ell_{2}$ -normalization onto the hypersphere, where each class is modeled by a vMF distribution with mean direction along the corresponding class prototype.

Figure 2: Softmax outputs and logits obtained from a ResNet-18 trained on three CIFAR-10 classes. While softmax outputs concentrate near the vertices of the probability simplex (colored area), logits exhibit a similar simplex geometry (gray area) centered at the origin with a broader within-class spread.

Figure 3: Plots of training loss and Neural Collapse (NC) hypotheses for softmax outputs (blue) and logits (orange) on CIFAR-100 with ResNet-18. The terminal phase of training is shown with a vertical red line.

Figure 4: Deterministic linear transformation across model updates of the class prototypes in Eq. 2 to which the hyperspherical simplex representations of Eq. 4 and Eq. 5 align. From left to right: Step 1, a 1-simplex with two classes (red and blue); Step 2, a 2-simplex with a new class (green); Step 3, a 3-simplex with a new class (yellow). Dashed arrows at steps 2 and 3 show how the class prototypes rotate to form the new simplex. The black dashed circle on the right shows the angular misalignment

\theta_{c}^{k\to t}

of the class-3 prototype between steps 2 and 3. Solid black arrows indicate the linear transformation

\mathbf{P}_{t,k}

, which aligns the updated prototypes of the previously existing classes with their counterparts at the earlier step.

Method	2 steps			5 steps			20 steps			50 steps
Method	$AC$	$AA$	$ACA$	$AC$	$AA$	$ACA$	$AC$	$AA$	$ACA$	$AC$	$AA$	$ACA$
Baseline	0	29.63	0	0	13.60	0	0	04.22	0	0	02.29	0
BCT	1	40.64	35.51	0	31.03	0	0	22.66	0	0.01	15.78	0.84
RACT	1	44.19	45.09	0.60	30.43	16.53	0.05	21.61	2.42	0	16.71	0
AdvBCT	0	35.32	0	0.40	26.13	11.67	0.02	19.79	0.19	0	14.70	0.03
LCE	1	43.48	40.71	0.10	32.63	04.63	0	20.95	0.25	0	13.81	0.03

Table 1: Comparative analysis of compatibility performance under the “Extended Classes” benchmark on CIFAR-100 for multiple updates. Dark and light blue indicate respectively the highest and the second-highest value for each metric.

Method	ImageNet-1K						Google Landmark v2
	2 steps			5 steps			2 steps			5 steps
	$AC$	$AA$	$ACA$	$AC$	$AA$	$ACA$	$AC$	$AA$	$ACA$	$AC$	$AA$	$ACA$
Baseline	0	37.52	0	0	16.18	0	0	07.66	0	0	07.87	0
BCT	1	48.81	45.86	0.20	34.96	5.28	1	09.37	08.22	0	08.46	0
RACT	1	55.17	51.16	0.10	40.94	6.20	0	11.01	0	0	09.99	0
AdvBCT	0	53.07	0	0	43.91	0	1	11.12	09.67	0	10.73	0

Table 2: Comparative analysis of compatibility performance under the “Extended Classes” benchmark on ImageNet-1K and Google Landmarks v2 for multiple updates. Dark and light blue indicate respectively the highest and the second-highest value for each metric.

Top- $k$	PSP			LSP
with $k=$	$AC$	$AA$	$ACA$	$AC$	$AA$	$ACA$
256	1	35.20	31.09	0.7	37.96	24.85
384	1	35.20	31.09	0.7	38.00	24.87
512	1	35.20	31.09	0.5	38.03	14.24
768	1	35.20	31.09	0.5	38.01	14.28
896	1	35.20	31.09	0.5	38.03	14.28

Table 3: Comparison of PSP and LSP across different top-

k

values in terms of

AC

AA

, and

ACA

, in the 5-step ImageNet-1K setting of the extended-classes benchmark.

Method	Ext. Backbone						Ext. Classes & Backbone
	CIFAR-100			ImageNet-1K			CIFAR-100			ImageNet-1K
	$AC$	$AA$	$ACA$	$AC$	$AA$	$ACA$	$AC$	$AA$	$ACA$	$AC$	$AA$	$ACA$
Baseline	0	35.13	0	0	40.58	0	0	29.88	0	0	35.88	0
BCT	1	53.57	54.06	1	61.80	61.43	1	43.61	40.79	1	50.64	42.98
RACT	1	53.80	54.62	1	62.15	62.07	1	43.66	41.01	1	52.74	48.92
AdvBCT	0	49.56	0	1	59.22	57.73	1	39.06	35.05	1	50.37	45.42

Table 4: Comparative analysis of compatibility performance under the “Extended Backbone” and “Extended Classes and Backbone” benchmarks on CIFAR-100 and ImageNet-1K. Dark and light blue indicate respectively the highest and the second-highest value for each metric.

Method	AlexNet $\xrightarrow{}$ R-50 $\xrightarrow{}$ RegNet $\xrightarrow{}$ R-152 $\xrightarrow{}$ MaxViT						ViT-B-32 $\xrightarrow{}$ ViT-B-16 $\xrightarrow{}$ ViT-L-16
	ImageNet-1K			Places365			ImageNet-1K			Places365
	$AC$	$AA$	$ACA$	$AC$	$AA$	$ACA$	$AC$	$AA$	$ACA$	$AC$	$AA$	$ACA$
Baseline	0	21.63	0	0	8.38	0	0	38.63	0	0	15.10	0
PSP	1	74.34	76.12	0.9	15.92	14.08	1	78.94	80.53	1	18.31	17.60
LSP	0.7	60.68	37.61	0.2	21.36	4.06	0.67	71.82	45.91	0.33	25.71	7.99

Table 5: Compatibility between pretrained models on ImageNet-1K and Places365: ( a ) AlexNet

\xrightarrow{}

ResNet-50 (R-50)

\xrightarrow{}

RegNetX_3.2GF (RegNet)

\xrightarrow{}

ResNet-152 (R-152)

\xrightarrow{}

MaxViT_T and ( b ) ViT-B-32

\xrightarrow{}

ViT-B-16

\xrightarrow{}

ViT-L-16. Models are pretrained on ImageNet-1K and downloaded from Marcel & Rodriguez (2010) . Dark blue numbers indicate the highest values, while light blue the second-highest values.

Panel of Figure 5 — Figure 5: Compatibility between independently trained models, all pretrained on ImageNet-1K. Compatibility matrices for five steps with increasingly expressive models (AlexNet, ResNet-50, RegNetX_3.2GF, ResNet-152, and MaxViT_T). The matrices display retrieval performance on the ImageNet-1K test set. Entries that do not satisfy the compatibility criterion in Eq. 8 are highlighted with a light-red background. Model representations from: (a) encoder output (Baseline); (b) softmax outputs (PSP); (c) logits (LSP).

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 6 — Figure 6: Plots of training loss and Neural Collapse hypotheses for feature representations derived from softmax outputs (blue) and logits (orange) on CIFAR-100 and Tiny-ImageNet-200 with ResNet-50 and DenseNet. The terminal phase of training is shown with a vertical red line.

Panel of Figure 7 — Figure 7: Expected cosine distances for aligned regular-simplex representations in the (a) same-class and (b) different-class cases, corresponding to Eq. 25a and Eq. 25b , respectively. The curves are obtained from the closed-form expected-distance expressions in Eq. 27a and Eq. 27b , under the assumptions of Theorem 3.6 , with the updated class distributions having higher concentration than the old ones. Across all considered feature-space dimensions $d$ , the expected distance between features from the updated and old models is lower than the corresponding old-model distance for same-class pairs and higher for different-class pairs, illustrating that the compatibility inequalities are satisfied in expectation.

Panel of Figure 8 — Figure 8: Per-class dispersion $\delta_{c}$ of the $\ell_{2}$ -normalized logits (a) and softmax outputs (b) on CIFAR-10 for the old model (solid bars) and the new model (hatched bars). In each subfigure, the left panel shows the extended-classes update from 5 to 10 classes, where the last 5 classes are unknown to the old model, and the right panel shows the update between independently trained models, with ResNet-18 as the old model and ResNet-50 as the new model, both pretrained on ImageNet-1K. Lower values of $\delta_{c}$ indicate a tighter concentration around the class mean direction.

Method	$\mathbf{P}_{k,k}$	$\mathbf{P}_{t,k}$	$\gamma_{\mathrm{same}}$	$\gamma_{\mathrm{diff}}$	Compatible pairs
$\ell_{2}$ -Softmax	–	–	$-0.0111\pm 0.0030$	$-0.0055\pm 0.0009$	0/9
PSP w/o $\mathbf{P}_{t,k}$	✓	–	$-0.0568\pm 0.0033$	$-0.0142\pm 0.0008$	0/9
PSP w/o $\mathbf{P}_{k,k}$	–	✓ ∗	$+0.0120\pm 0.0019$	$-0.0133\pm 0.0010$	0/9
PSP	✓	✓	$+0.0155\pm 0.0025$	$+0.0039\pm 0.0006$	9/9

Appendix

Table 6: Analysis of the centering projection

\mathbf{P}_{k,k}

and the deterministic transformation

\mathbf{P}_{t,k}

on CIFAR-10 with ResNet-18 under class expansion (

5\to 10

classes). Average margins of Eq. 28 (mean

\pm

std over 9 model pairs; positive values indicate that the inequality is satisfied on average) and number of compatible pairs, i.e., model pairs for which both margins are positive. ∗ Without the centering projection, i.e.,

[\mathbf{I}_{C^{k}}\,|\,\mathbf{0}]

in place of

\mathbf{P}_{t,k}

. Dark and light blue indicate respectively the highest and the second-highest value for each column.

Panel of Figure 9 — Figure 9: Compatibility matrices showing CMC@ $1$ retrieval metric values at each model update step for the 5-step scenario on the CIFAR-100 dataset. Each matrix entry failing to satisfy the compatibility criterion defined in Eq. 8 is highlighted with a light-red background.

Panel of Figure 10 — Figure 10: Open-set scenario. Compatibility matrices for five steps with increasingly expressive models (AlexNet, ResNet-50, RegNetX_3.2GF, ResNet-152, and MaxViT_T), all pretrained on ImageNet-1K. The matrices display retrieval performance on the Places365 dataset. Entries that do not satisfy the compatibility criterion in Eq. 8 are highlighted with a light-red background. Model representations obtained from: (a) encoder output (Baseline); (b) softmax outputs (PSP); (c) logits (LSP).

Panel of Figure 11 — Figure 11: Closed-set scenario. Compatibility matrices for three steps with advanced Vision Transformer (ViT) model architectures (ViT-B/32, ViT-B/16, and ViT-L/16), all pretrained on ImageNet-1K. The matrices show closed-set retrieval performance on the ImageNet-1K test set. Entries that do not satisfy the compatibility criterion in Eq. 8 are highlighted with a light-red background. Model representations obtained from: (a) encoder output (Baseline); (b) softmax outputs (PSP); (c) logits (LSP).

Panel of Figure 12 — Figure 12: Open-set scenario. Compatibility matrices for three steps with advanced Vision Transformer (ViT) model architectures (ViT-B/32, ViT-B/16, and ViT-L/16), all pretrained on ImageNet-1K. The matrices show open-set retrieval performance on the Places365 test set. Entries that do not satisfy the compatibility criterion in Eq. 8 are highlighted with a light-red background. Model representations obtained from: (a) encoder output (Baseline); (b) softmax outputs (PSP); (c) logits (LSP).

Table 7: Compatibility results for self-supervised and vision-language pretrained encoders in a linear-probe setting on ImageNet-1K. For each checkpoint, a linear probe is trained on top of frozen backbone features. Dark blue numbers indicate the highest values, while light blue numbers indicate the second-highest values.

Panel of Figure 13 — Figure 13: Compatibility matrices for three steps with self-supervised Vision Transformer checkpoints from the DINO family (DINO-ViT-S, DINOv2-ViT-S/14, and DINOv3-ViT-L/16) obtained from Hugging Face Hub ( Wolf et al., 2019 ) . For each checkpoint, we train a linear probe on ImageNet-1K with the backbone frozen, and construct PSP/LSP features from the resulting classifier outputs. The matrices show closed-set retrieval performance on the ImageNet-1K test set. Entries that do not satisfy the compatibility criterion in Eq. 8 are highlighted with a light-red background. Model representations obtained from: (a) encoder output (Baseline); (b) softmax outputs (PSP); (c) logits (LSP).

Panel of Figure 14 — Figure 14: Compatibility matrices for three steps with vision-language CLIP family (ViT-B/32, ViT-B/16, and ViT-L/14) pretrained by OpenAI. For each checkpoint, we train a linear probe on ImageNet-1K with the backbone frozen, and construct PSP/LSP features from the resulting classifier outputs. The matrices show closed-set retrieval performance on the ImageNet-1K test set. Entries that do not satisfy the compatibility criterion in Eq. 8 are highlighted with a light-red background. Model representations obtained from: (a) encoder output (Baseline); (b) softmax outputs (PSP); (c) logits (LSP).

Representation Learning

Representation learning develops data transformations that expose useful structure for prediction, transfer, generation, and control.

See all 797 papers

Jun 10, 2026

A Stationary (and Therefore Compatible) Representation is All You Need

Learning compatible representations aims to learn feature representations that can be used interchangeably over time whenever a model undergoes updates. In this paper, we demonstrate that stationary representations learned by d-Simplex fixed classifiers imply compatibility as in its formal definition. This result establishes a foundation for future works and can be directly exploited in practical learning scenarios. We address the challenge of learning compatibility using

d

-Simplex fixed classifiers when the model is sequentially fine-tuned. Learning according to a d-Simplex fixed classifier with the cross-entropy loss aligns feature distributions at the first-order statistics. Consequently, it may not fully capture higher-order dependencies in the representation between model updates. To address this issue, we demonstrate that training the model using a

d

-Simplex fixed classifier through a convex combination of the cross-entropy loss and a contrastive loss not only captures higher-order dependencies, but is also equivalent to learning with the cross-entropy under the compatibility constraints. We confirm our findings with extensive experiments also considering a new scenario where a pre-trained model is sequentially fine-tuned and occasionally replaced with an improved model. We show that stationary representations enable uninterrupted retrieval services (without reprocessing gallery images) while improving performance during model updates and replacements, achieving state-of-the-art. Code at https://github.com/miccunifi/iamcl2r.

Niccolò Biondi, Federico Pernici, Simone Ricci +1

Media Integration and Communication Center (MICC), Dipartimento di Ingegneria dell’Informazione, Università degli Studi di Firenze, 50139, Firenze, Italy

Shared Representation Learning Representation Learning Image Retrieval

Ask PDF

Sep 30, 2026

Spherical Interpolation for Backward-Compatible Multimodal Representations

Contrastive vision-language models map visual and textual representations into a shared normalized embedding space, making cosine similarity the natural metric for cross-modal retrieval. A practical challenge arises during model upgrades: independently trained models generally produce incompatible representation spaces, so replacing a deployed model typically requires recomputing embeddings for the entire gallery, which is prohibitively expensive at scale. Orthogonal post-hoc alignment can partially mitigate this problem by mapping new-model queries into the old-model gallery space. However, because independently trained models can differ in fine-grained representation structure, the orthogonal alignment remains approximate, leaving a residual angular discrepancy between the old-model query and the aligned new-model query. We study whether interpolation along the spherical geodesic between these two normalized query representations can improve retrieval without re-indexing the gallery. We characterize when this path contains an interior query direction closer to an idealized retrieval-optimal direction than either endpoint, and connect this characterization to Recall@

K

through a local margin-based certification result. Experiments across multiple benchmarks and model families show that post-alignment spherical interpolation improves over orthogonal alignment alone, recovering backward-compatibility in most evaluated settings. Consistent with our geometric characterization, per-query oracle analysis shows that retrieval-favorable interior points occur frequently in practice. Code is available at https://github.com/miccunifi/SLERP_backward_compatibility .

Simone Ricci, Niccolò Biondi, Federico Pernici

DINFO (Department of Information Engineering), University of Florence, Italy · MICC (Media Integration and Communication Center) · University of Trento, Italy

Cross-Modal Alignment Cross-Modal Retrieval

Ask PDF

Oct 23, 2026

Closing the Modality Gap via Simplex-Constrained Representations

While multimodal contrastive models like CLIP and CLAP successfully learn shared representations, their embeddings typically exhibit a persistent modality gap. We propose addressing this separation through a purely geometric approach: changing the embedding domain from the unit sphere,

S^{d-1}

, to the probability simplex,

Δ^{d-1}

. Although both manifolds share the same intrinsic dimension, the simplex acts as a structural prior; its nonnegative, unit-mass constraints serve as a cross-modal calibration that forces both modalities to ground themselves in the same coordinate system. By attaching lightweight softmax adapters and using Total Variation similarity, we reduce the centroid

\ell_2

gap by 97--99% across five multimodal retrieval benchmarks. This confirms that simplicial geometry induces genuine semantic alignment, matching or exceeding the retrieval performance of standard unconstrained models.

Shubham Gupta, Siva Reddy, Perouz Taslakian +2

Mila – Québec AI Institute · ServiceNow Research · Université Laval +1

Modality Gap in VLMs Cross-Modal Alignment Representation Learning Multimodal Contrastive Learning Cross-Modal Retrieval

Ask PDF

Abstract

Representation Learning

Explore similar work

A Stationary (and Therefore Compatible) Representation is All You Need

Spherical Interpolation for Backward-Compatible Multimodal Representations

Closing the Modality Gap via Simplex-Constrained Representations