Anchor Divergence for Semantic Geometry in Contrastive Learning

Akash Kannan; Kiho Park; Victor Veitch

cs.AIOct 2, 2026

Anchor Divergence for Semantic Geometry in Contrastive Learning

Authors: Akash Kannan, Kiho Park, Victor Veitch

Organizations: University of Chicago

Abstract

This paper concerns how semantic context determines geometry in learned vector representations. Similarity is typically measured using cosine similarity, which provides a single fixed geometry. Semantic similarity, however, is inherently context dependent: two images may be similar because they depict the same object, share a visual style, or are relevant to the same clinical finding. We show that contrastive representations naturally encompass a family of geometries that can be specialized to particular semantic structure. The key idea is to use an interplay between contrastive learning, exponential families, and information geometry to establish a correspondence between probability distributions over "anchors" and Bregman geometries on the representation space. We use this correspondence to define "Anchor Divergences", a method for specifying context-specific semantic geometries on fixed representations. Under this correspondence, modeling the anchor distribution models the geometry itself. Experiments on retrieval show that anchor divergences provide an effective and efficient way to specify context-specific semantic similarity.

Figures & tables

Figure 1: Anchor divergences induce context-specific neighborhoods for the same query. Cosine similarity gives a single fixed ranking. In contrast, changing the semantic anchors changes which aspect of the query defines its neighborhood. For example, breed anchors retrieve other Huskies while allowing background and clothing to vary; background and clothing anchors behave analogously. The underlying image representations remain fixed throughout.

Anchor model	Empirical	Gaussian	Gaussian Mixture
CGF $\kappa(\eta)$	$\displaystyle\log\frac{1}{\|\Gamma\|}\sum_{\gamma\in\Gamma}\exp(\eta^{\top}\gamma)$	$\displaystyle\hat{\mu}^{\top}\eta+\frac{1}{2}\eta^{\top}\hat{\Sigma}\eta$	$\displaystyle\log\sum_{m=1}^{M}\hat{\pi}_{m}\exp\left(\hat{\mu}_{m}^{\top}\eta+\frac{1}{2}\eta^{\top}\hat{\Sigma}_{m}\eta\right)$

Table 1: Modeling the anchor distribution induces different semantic geometries. Each column specifies an anchor-distribution model and its CGF. The empirical model places uniform mass on the observed anchors; for the Gaussian model,

\hat{\mu}

and

\hat{\Sigma}

are the sample mean and covariance; for the Gaussian mixture,

\{(\hat{\pi}_{m},\hat{\mu}_{m},\hat{\Sigma}_{m})\}_{m=1}^{M}

are fit by expectation–maximization.

Figure 2: Anchor divergences can emphasize relevant semantics over nuisance variation. The query X-ray shows the target pathology, pleural effusion, as well as a support device. Cosine similarity retrieves an image sharing device-related structure but not the target pathology, whereas both softmax and (Gaussian model) parametric anchor divergences retrieve images with pleural effusion.

Pathology	Softmax $-$ Cosine	Softmax $-$ ITI
Enlarged Cardiomediastinum	$-0.0002\pm 0.0003$	$-0.0045\pm 0.0010$
Cardiomegaly	$0.0279\pm 0.0034$	$0.0069\pm 0.0091$
Lung Opacity	$0.0090\pm 0.0032$	$0.0137\pm 0.0035$
Lung Lesion	$0.0001\pm 0.0005$	$0.0051\pm 0.0010$
Edema	$0.0316\pm 0.0044$	$0.0286\pm 0.0072$
Consolidation	$0.0062\pm 0.0014$	$0.0073\pm 0.0023$

Table 2: Pathology-specific anchors specialize clinical retrieval. Condition-wise AUPRC differences between softmax anchor divergence and each reference geometry; positive values favor anchor divergence. Results are averaged over 10 independently generated anchor sets and 10 CheXzero checkpoints, with 95% intervals accounting for bootstrap and anchor-generation variability. Values

>0.005

are bold, values in

[-0.005,0.005]

are gray, and values

<-0.005

are bold red.

Figure 3: Anchor distributions select the semantic coordinate. Left: Mixing object and style anchor distributions on ImageNet-R shifts retrieval from preserving style to preserving object identity as $p$ increases. Right: For a video-game-style space shuttle query, retrieved neighbors similarly shift from style-matched to object-matched images.

Figure 4: Parametric modeling is useful with limited anchors. Left: On ImageNet, parametric anchor divergences are strongest in the red-shaded limited-anchor regime, while softmax improves with coverage and becomes strongest in the blue high-coverage regime. Right: With full dataset-matched anchor sets on ImageNet-R, CUB-200, and Sketchy, softmax gives the strongest retrieval performance, while parametric anchor divergences retain much of its gain over cosine similarity.

Figure 5: Posterior anchor modeling adapts broad geometries using few target anchors. Dotted curves use only target-specific ImageNet anchors, while solid curves use posterior anchor models adapted from WordNet with the same target anchors. Posterior models are strongest at small and moderate target-anchor budgets, with the advantage narrowing as coverage grows.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Pathology	ITI $-$ Cosine	Softmax $-$ Cosine	Gaussian $-$ Cosine	Latent G. $-$ Cosine	Latent S. $-$ Cosine
Enlarged Cardiomediastinum	$0.0042\pm 0.0009$	$-0.0002\pm 0.0003$	$-0.0002\pm 0.0003$	$0.0000\pm 0.0002$	$-0.0004\pm 0.0003$
Cardiomegaly	$0.0211\pm 0.0083$	$0.0279\pm 0.0034$	$0.0327\pm 0.0030$	$0.0152\pm 0.0015$	$0.0194\pm 0.0020$
Lung Opacity	$-0.0047\pm 0.0050$	$0.0090\pm 0.0032$	$0.0092\pm 0.0031$	$0.0037\pm 0.0016$	$0.0083\pm 0.0019$
Lung Lesion	$-0.0050\pm 0.0011$	$0.0001\pm 0.0005$	$-0.0000\pm 0.0004$	$0.0003\pm 0.0003$	$0.0013\pm 0.0004$
Edema	$0.0030\pm 0.0064$	$0.0316\pm 0.0044$	$0.0309\pm 0.0045$	$0.0127\pm 0.0024$	$0.0194\pm 0.0027$
Consolidation	$-0.0011\pm 0.0020$	$0.0062\pm 0.0014$	$0.0056\pm 0.0012$	$0.0026\pm 0.0006$	$0.0020\pm 0.0007$

Appendix

Table 3: Parametric CheXpert anchor divergences. Entries are pathology-wise mean AUPRC differences relative to cosine similarity, with 95% confidence interval. Results average over independently generated anchor sets and CheXzero checkpoints, with intervals accounting for bootstrap and anchor-generation variability. Softmax and Gaussian anchor divergences give the largest average gains, while latent models also improve over cosine on average.

Figure 6: Parametric structure is useful when anchor coverage is limited. AUPRC as the number of class-name anchors varies on ImageNet-R (left) and CUB-200 (right). Parametric anchor divergences are strongest across much of the limited- and moderate-anchor regime, while softmax improves with anchor coverage and is strongest when the full class-anchor set is available.

Figure 7: Mixture complexity behaves differently across anchor sets. On ImageNet, increasing the number of GMM components generally improves retrieval, consistent with a heterogeneous class-name anchor distribution. On CUB-200, gains are smaller and less consistent, suggesting that simpler structured geometries may already capture much of the relevant anchor variation.

Method	$\mathbf{2^{10}}$	$\mathbf{2^{12}}$	$\mathbf{2^{14}}$	$\mathbf{2^{16}}$	$\mathbf{2^{17}}$
Cosine	0.152	0.152	0.152	0.151	0.152
Softmax	0.216	0.281	0.538	1.622	3.066
Gaussian	0.364	0.363	0.367	0.370	0.375

Appendix

Table 4: Runtime scaling with anchor-set size. Wall-clock retrieval time (in seconds) on a single NVIDIA H100 NVL GPU for 10,000 queries and 50,000 gallery images. Runtimes include fitting, preprocessing, and retrieval and are averaged over 10 runs following 3 warm-up runs.

Panel of Figure 8 — Figure 8: Qualitative examples of anchor-controlled retrieval. Blue boxes show object-anchor retrieval, which prioritizes object identity across styles; red boxes show style-anchor retrieval, which prioritizes style across object categories. Rows compare some of the different anchor divergences.

Neural Representation Geometry

Neural representation geometry studies how learned features are arranged and how that structure relates to model behavior and performance.

See all 253 papers

Jun 2, 2026

The Loss Is Not Enough: Sampling Conditions and Inductive Bias in Contrastive Representation Learning

Contrastive learning has become a leading paradigm for self-supervised representation learning, yet the conditions under which it recovers meaningful latent geometry remain incompletely understood. We develop a measure-theoretic framework formalizing the diversity condition, a support requirement on positive-pair sampling that is necessary for isometric latent recovery. We show that the standard full-support von Mises-Fisher setting implies the satisfaction of the diversity condition and as a consequence global contrastive loss minimizers recover latent geometry up to orthogonal transformation, while restricted conditionals can make non-orthogonal maps attain strictly lower asymptotic contrastive loss. We introduce a support-corrected Information Noise Contrastive Estimation (InfoNCE) variant as a theoretical fix: this correction makes orthogonal latent space recovery achievable but does not uniquely select it. Experiments on synthetic benchmarks validate the identifiability predictions, and CIFAR-10 experiments are consistent with the qualitative prediction that architectural inductive bias becomes more important when sampling diversity is limited. Together, our results clarify how sampling mechanisms and encoder inductive bias interact in contrastive representation learning.

Justinas Zaliaduonis, Patrick Putzky, Till Richter +1

Technical University of Munich, Munich, Germany · Stanford University, Stanford, USA · Merantix Momentum GmbH, Berlin, Germany +2

Contrastive Learning Self-Supervised Learning Inductive Bias

Ask PDF

Jan 27, 2026

The Geometric Mechanics of Contrastive Representation Learning: Alignment Potentials, Entropic Dispersion, and Cross-modal Divergence

While InfoNCE underlies modern contrastive learning, its geometric mechanisms remain under-characterized beyond the canonical alignment--uniformity decomposition. We develop a measure-theoretic framework in which representation measures evolve on a fixed embedding manifold. In the large-batch limit, we prove value and gradient consistency, linking the stochastic objective to explicit deterministic energy landscapes and revealing a geometric bifurcation between unimodal and symmetric multimodal regimes. In the unimodal case, the intrinsic energy is strictly convex and admits a unique Gibbs equilibrium, showing that entropy acts as a tie-breaker within the aligned basin. In the multimodal case, the intrinsic geometry becomes cross-coupled and contains a persistent negative symmetric divergence term: each modality's marginal reshapes the effective landscape of the other, allowing strong pairwise alignment to coexist with a persistent modality gap. Controlled synthetic experiments and analyses of pretrained CLIP representations support these predictions. Overall, our results shift the analytical lens from pointwise discrimination to population geometry, showing that pairwise alignment alone is insufficient to control cross-modal marginal structure.

Yichao Cai, Zhen Zhang, Yuhang Liu +1

Australian Institute for Machine Learning, Adelaide University · Responsible AI Research Centre

Cross-Modal Learning Contrastive Learning Neural Representation Geometry Cross-Modal Alignment Multimodal Contrastive Learning

Ask PDF

May 13, 2026

A Unified Geometric Framework for Weighted Contrastive Learning

Contrastive learning (CL) aims to preserve relational structure between samples by learning representations that reflect a similarity graph. Yet, the geometry of the resulting embeddings remains poorly understood. Here we show that weighted InfoNCE objectives can be interpreted as Distance Geometry Problems, where the weighting scheme specifies the target geometry to be realized by the representation. This viewpoint yields exact characterizations of the optimal embeddings for several supervised and weakly supervised objectives. In supervised classification, both SupCon and Soft SupCon (a dense relaxation of it where pairs from distinct classes have small non-zero similarity) collapse samples within each class to a single prototype. However, while balanced SupCon recovers the classical regular simplex geometry, class imbalance breaks this symmetry: SupCon induces non-uniform inter-class similarities depending on class sizes, whereas Soft SupCon preserves a regular simplex geometry regardless of class imbalance. In continuous-label settings, our framework reveals a different failure mode: y-Aware CL generally cannot attain its entropic optimum unless the labels lie on a hypersphere, exposing a mismatch between Euclidean label weights and spherical latent similarity. By contrast, geometrically consistent choices such as Euclidean-Euclidean weighting or X-CLR admit unique optimal embeddings. Our results show that the choice of weighting scheme determines whether contrastive learning is geometrically realizable, degenerate, or inconsistent, providing a principled framework for designing contrastive objectives.

Raphael Vock, Edouard Duchesnay, Benoit Dufumier

GAIA Lab, NeuroSpin, CEA, CNRS · Université Paris-Saclay, Gif-sur-Yvette, France

Contrastive Learning Neural Representation Geometry Supervised Contrastive Learning

Ask PDF

Abstract

Neural Representation Geometry

Explore similar work

The Loss Is Not Enough: Sampling Conditions and Inductive Bias in Contrastive Representation Learning

The Geometric Mechanics of Contrastive Representation Learning: Alignment Potentials, Entropic Dispersion, and Cross-modal Divergence

A Unified Geometric Framework for Weighted Contrastive Learning