Anchor Divergence for Semantic Geometry in Contrastive Learning

Akash Kannan; Kiho Park; Victor Veitch

cs.AIOct 2, 2026

Anchor Divergence for Semantic Geometry in Contrastive Learning

Authors: Akash Kannan, Kiho Park, Victor Veitch

Organizations: University of Chicago

Abstract

This paper concerns how semantic context determines geometry in learned vector representations. Similarity is typically measured using cosine similarity, which provides a single fixed geometry. Semantic similarity, however, is inherently context dependent: two images may be similar because they depict the same object, share a visual style, or are relevant to the same clinical finding. We show that contrastive representations naturally encompass a family of geometries that can be specialized to particular semantic structure. The key idea is to use an interplay between contrastive learning, exponential families, and information geometry to establish a correspondence between probability distributions over "anchors" and Bregman geometries on the representation space. We use this correspondence to define "Anchor Divergences", a method for specifying context-specific semantic geometries on fixed representations. Under this correspondence, modeling the anchor distribution models the geometry itself. Experiments on retrieval show that anchor divergences provide an effective and efficient way to specify context-specific semantic similarity.

Figures & tables

Figure 1: Anchor divergences induce context-specific neighborhoods for the same query. Cosine similarity gives a single fixed ranking. In contrast, changing the semantic anchors changes which aspect of the query defines its neighborhood. For example, breed anchors retrieve other Huskies while allowing background and clothing to vary; background and clothing anchors behave analogously. The underlying image representations remain fixed throughout.

Anchor model	Empirical	Gaussian	Gaussian Mixture
CGF $\kappa(\eta)$	$\displaystyle\log\frac{1}{\|\Gamma\|}\sum_{\gamma\in\Gamma}\exp(\eta^{\top}\gamma)$	$\displaystyle\hat{\mu}^{\top}\eta+\frac{1}{2}\eta^{\top}\hat{\Sigma}\eta$	$\displaystyle\log\sum_{m=1}^{M}\hat{\pi}_{m}\exp\left(\hat{\mu}_{m}^{\top}\eta+\frac{1}{2}\eta^{\top}\hat{\Sigma}_{m}\eta\right)$

Table 1: Modeling the anchor distribution induces different semantic geometries. Each column specifies an anchor-distribution model and its CGF. The empirical model places uniform mass on the observed anchors; for the Gaussian model,

\hat{\mu}

and

\hat{\Sigma}

are the sample mean and covariance; for the Gaussian mixture,

\{(\hat{\pi}_{m},\hat{\mu}_{m},\hat{\Sigma}_{m})\}_{m=1}^{M}

are fit by expectation–maximization.

Figure 2: Anchor divergences can emphasize relevant semantics over nuisance variation. The query X-ray shows the target pathology, pleural effusion, as well as a support device. Cosine similarity retrieves an image sharing device-related structure but not the target pathology, whereas both softmax and (Gaussian model) parametric anchor divergences retrieve images with pleural effusion.

Pathology	Softmax $-$ Cosine	Softmax $-$ ITI
Enlarged Cardiomediastinum	$-0.0002\pm 0.0003$	$-0.0045\pm 0.0010$
Cardiomegaly	$0.0279\pm 0.0034$	$0.0069\pm 0.0091$
Lung Opacity	$0.0090\pm 0.0032$	$0.0137\pm 0.0035$
Lung Lesion	$0.0001\pm 0.0005$	$0.0051\pm 0.0010$
Edema	$0.0316\pm 0.0044$	$0.0286\pm 0.0072$
Consolidation	$0.0062\pm 0.0014$	$0.0073\pm 0.0023$

Table 2: Pathology-specific anchors specialize clinical retrieval. Condition-wise AUPRC differences between softmax anchor divergence and each reference geometry; positive values favor anchor divergence. Results are averaged over 10 independently generated anchor sets and 10 CheXzero checkpoints, with 95% intervals accounting for bootstrap and anchor-generation variability. Values

>0.005

are bold, values in

[-0.005,0.005]

are gray, and values

<-0.005

are bold red.

Figure 3: Anchor distributions select the semantic coordinate. Left: Mixing object and style anchor distributions on ImageNet-R shifts retrieval from preserving style to preserving object identity as $p$ increases. Right: For a video-game-style space shuttle query, retrieved neighbors similarly shift from style-matched to object-matched images.

Figure 4: Parametric modeling is useful with limited anchors. Left: On ImageNet, parametric anchor divergences are strongest in the red-shaded limited-anchor regime, while softmax improves with coverage and becomes strongest in the blue high-coverage regime. Right: With full dataset-matched anchor sets on ImageNet-R, CUB-200, and Sketchy, softmax gives the strongest retrieval performance, while parametric anchor divergences retain much of its gain over cosine similarity.

Figure 5: Posterior anchor modeling adapts broad geometries using few target anchors. Dotted curves use only target-specific ImageNet anchors, while solid curves use posterior anchor models adapted from WordNet with the same target anchors. Posterior models are strongest at small and moderate target-anchor budgets, with the advantage narrowing as coverage grows.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Pathology	ITI $-$ Cosine	Softmax $-$ Cosine	Gaussian $-$ Cosine	Latent G. $-$ Cosine	Latent S. $-$ Cosine
Enlarged Cardiomediastinum	$0.0042\pm 0.0009$	$-0.0002\pm 0.0003$	$-0.0002\pm 0.0003$	$0.0000\pm 0.0002$	$-0.0004\pm 0.0003$
Cardiomegaly	$0.0211\pm 0.0083$	$0.0279\pm 0.0034$	$0.0327\pm 0.0030$	$0.0152\pm 0.0015$	$0.0194\pm 0.0020$
Lung Opacity	$-0.0047\pm 0.0050$	$0.0090\pm 0.0032$	$0.0092\pm 0.0031$	$0.0037\pm 0.0016$	$0.0083\pm 0.0019$
Lung Lesion	$-0.0050\pm 0.0011$	$0.0001\pm 0.0005$	$-0.0000\pm 0.0004$	$0.0003\pm 0.0003$	$0.0013\pm 0.0004$
Edema	$0.0030\pm 0.0064$	$0.0316\pm 0.0044$	$0.0309\pm 0.0045$	$0.0127\pm 0.0024$	$0.0194\pm 0.0027$
Consolidation	$-0.0011\pm 0.0020$	$0.0062\pm 0.0014$	$0.0056\pm 0.0012$	$0.0026\pm 0.0006$	$0.0020\pm 0.0007$

Appendix

Table 3: Parametric CheXpert anchor divergences. Entries are pathology-wise mean AUPRC differences relative to cosine similarity, with 95% confidence interval. Results average over independently generated anchor sets and CheXzero checkpoints, with intervals accounting for bootstrap and anchor-generation variability. Softmax and Gaussian anchor divergences give the largest average gains, while latent models also improve over cosine on average.

Figure 6: Parametric structure is useful when anchor coverage is limited. AUPRC as the number of class-name anchors varies on ImageNet-R (left) and CUB-200 (right). Parametric anchor divergences are strongest across much of the limited- and moderate-anchor regime, while softmax improves with anchor coverage and is strongest when the full class-anchor set is available.

Figure 7: Mixture complexity behaves differently across anchor sets. On ImageNet, increasing the number of GMM components generally improves retrieval, consistent with a heterogeneous class-name anchor distribution. On CUB-200, gains are smaller and less consistent, suggesting that simpler structured geometries may already capture much of the relevant anchor variation.

Method	$\mathbf{2^{10}}$	$\mathbf{2^{12}}$	$\mathbf{2^{14}}$	$\mathbf{2^{16}}$	$\mathbf{2^{17}}$
Cosine	0.152	0.152	0.152	0.151	0.152
Softmax	0.216	0.281	0.538	1.622	3.066
Gaussian	0.364	0.363	0.367	0.370	0.375

Appendix

Table 4: Runtime scaling with anchor-set size. Wall-clock retrieval time (in seconds) on a single NVIDIA H100 NVL GPU for 10,000 queries and 50,000 gallery images. Runtimes include fitting, preprocessing, and retrieval and are averaged over 10 runs following 3 warm-up runs.

Panel of Figure 8 — Figure 8: Qualitative examples of anchor-controlled retrieval. Blue boxes show object-anchor retrieval, which prioritizes object identity across styles; red boxes show style-anchor retrieval, which prioritizes style across object categories. Rows compare some of the different anchor divergences.

Neural Representation Geometry

Neural representation geometry studies how learned features are arranged and how that structure relates to model behavior and performance.

See all 253 papers

The Loss Is Not Enough: Sampling Conditions and Inductive Bias in Contrastive Representation Learning
Jun 2, 2026Justinas Zaliaduonis, Patrick Putzky, Till Richter +1Contrastive Learning Self-Supervised Learning
Technical University of Munich, Munich, Germany · Stanford University, Stanford, USA · Merantix Momentum GmbH, Berlin, Germany +2
The Geometric Mechanics of Contrastive Representation Learning: Alignment Potentials, Entropic Dispersion, and Cross-modal Divergence
Jan 27, 2026Yichao Cai, Zhen Zhang, Yuhang Liu +1Cross-Modal Learning Contrastive Learning
Australian Institute for Machine Learning, Adelaide University · Responsible AI Research Centre
A Unified Geometric Framework for Weighted Contrastive Learning
May 13, 2026Raphael Vock, Edouard Duchesnay, Benoit DufumierContrastive Learning Neural Representation Geometry
GAIA Lab, NeuroSpin, CEA, CNRS · Université Paris-Saclay, Gif-sur-Yvette, France

Jun 2, 2026J/K move · Enter open · S save

The Loss Is Not Enough: Sampling Conditions and Inductive Bias in Contrastive Representation Learning

Justinas Zaliaduonis, Patrick Putzky, Till Richter, Sergios Gatidis

Technical University of Munich, Munich, Germany · Stanford University, Stanford, USA · Merantix Momentum GmbH, Berlin, Germany+2

Contrastive Learning Self-Supervised Learning Inductive Bias

Open paper AskPDF arXiv

Contrastive learning has become a leading paradigm for self-supervised representation learning, yet the conditions under which it recovers meaningful latent geometry remain incompletely understood. We develop a measure-theoretic framework formalizing the diversity condition, a support requirement on positive-pair sampling that is necessary for isometric latent recovery. We show that the standard full-support von Mises-Fisher setting implies the satisfaction of the diversity condition and as a consequence global contrastive loss minimizers recover latent geometry up to orthogonal transformation, while restricted conditionals can make non-orthogonal maps attain strictly lower asymptotic contrastive loss. We introduce a support-corrected Information Noise Contrastive Estimation (InfoNCE) variant as a theoretical fix: this correction makes orthogonal latent space recovery achievable but does not uniquely select it. Experiments on synthetic benchmarks validate the identifiability predictions, and CIFAR-10 experiments are consistent with the qualitative prediction that architectural inductive bias becomes more important when sampling diversity is limited. Together, our results clarify how sampling mechanisms and encoder inductive bias interact in contrastive representation learning.

Abstract

Neural Representation Geometry

Explore similar work

The Loss Is Not Enough: Sampling Conditions and Inductive Bias in Contrastive Representation Learning

The Geometric Mechanics of Contrastive Representation Learning: Alignment Potentials, Entropic Dispersion, and Cross-modal Divergence

A Unified Geometric Framework for Weighted Contrastive Learning