Geometry-Aware Adaptation for Pretrained Models

Nicholas Roberts; Xintong Li; Dyah Adila; Sonia Cromp; Tzu-Heng Huang; Jitian Zhao; Frederic Sala

cs.LGJul 23, 2023

Geometry-Aware Adaptation for Pretrained Models

Authors: Nicholas Roberts, Xintong Li, Dyah Adila, Sonia Cromp, Tzu-Heng Huang, Jitian Zhao, Frederic Sala

Organizations: University of Wisconsin-Madison

Abstract

Machine learning models -- including prominent zero-shot models -- are often trained on datasets whose labels are only a small proportion of a larger label space. Such spaces are commonly equipped with a metric that relates the labels via distances between them. We propose a simple approach to exploit this information to adapt the trained model to reliably predict new classes -- or, in the case of zero-shot prediction, to improve its performance -- without any additional training. Our technique is a drop-in replacement of the standard prediction rule, swapping argmax with the Fréchet mean. We provide a comprehensive theoretical analysis for this approach, studying (i) learning-theoretic results trading off label space diameter, sample complexity, and model dimension, (ii) characterizations of the full range of scenarios in which it is possible to predict any unobserved class, and (iii) an optimal active learning-like next class selection procedure to obtain optimal training classes for when it is not possible to predict the entire range of unobserved classes. Empirically, using easily-available external metrics, our proposed approach, Loki, gains up to 29.7% relative improvement over SimCLR on ImageNet and scales to hundreds of thousands of classes. When no such metric is available, Loki can use self-derived metrics from class embeddings and obtains a 10.5% improvement on pretrained zero-shot models such as CLIP.

Figures & tables

Figure 1: Classification regions in the probability simplex of 3-class classifiers faced with a 100-class problem. The probability simplex using $\argmax$ prediction can only output one of three classes. Loki uses the entire probability vector to navigate the class metric space, leading to more prediction regions. (Left) regions from $\argmax$ prediction. (Centers, Right) classification regions from Loki .

Model	Metric Space	$\argmax$	Loki	Relative Improvement
CLIP-RN50 [ 30 ]	Internal	0.2922	0.2613	10.57%
CLIP-ViT-L-14 [ 30 ]	Internal	0.1588	0.1562	1.63%
ALIGN [ 15 ]	Internal	0.1475	0.1430	3.02%
CLIP-RN50 [ 30 ]	$K_{100}$	0.5941 ∗	0.5941 ∗	0.0% ∗
CLIP-RN50 [ 30 ]	Default tree	7.3528	7.1888	2.23%
CLIP-RN50 [ 30 ]	WordNet tree	24.3017	19.5549	19.53%

Table 1: CIFAR-100. Improving CLIP predictions using Loki . Results are reported as

\mathbb{E}[d^{2}(y,\hat{y})]

in the respective metric space. CLIP-like zero-shot models can be improved using Loki even without access to an external metric, and internal class embedding distances are used. When an external metric is available, Loki outperforms CLIP using the default CIFAR-100 hierarchy and WordNet.

$K$	5-NN	5-NN + Loki
100	1.68666	1.42591
250	1.52374	1.47801
500	1.64074	1.45921

Table 2: PubMed. Loki improves baseline for all settings of

K

. The metric space is Euclidean distances applied to SimCSE embeddings.

Panel of Figure 2 — Figure 2: (Top left) ImageNet. Mean squared distances under uniform class sampling and under the Gibbs distribution— Loki improves upon the baseline SimCLR one-vs-rest classifier. (Top right) Synthetic. Active class selection consistently leads to larger loci compared to uniform sampling. (Bottom) Active selection on ImageNet. Active class selection improves performance on ImageNet.

Panel of Figure 3 — Figure 3: CLIP on CIFAR-100 with no external metric. (Left) Reliability diagrams across a range of Softmax temperatures, highlighting the CLIP default temperature, the optimal temperature for Loki , and the minimizer of the Expected Calibration Error (ECE). All three are well-calibrated. (Center) Tradeoff between optimizing for ECE and the expected squared distance. As with the reliability diagrams, the CLIP default temperature, the Loki -optimal temperature, and the ECE-optimal temperature are similar. (Right) Tradeoff between optimizing for zero-one error and the expected squared distance. Temperature can be tuned to improve accuracy when using Loki .

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 4 — Figure 4: CLIP on CIFAR-100 with the WordNet hierarchy. (Left) Reliability diagrams across a range of Softmax temperatures, highlighting the CLIP default temperature, the optimal temperature for Loki , and the minimizer of the Expected Calibration Error (ECE). All three are well-calibrated. (Center) Tradeoff between optimizing for ECE and the expected squared distance. As with the reliability diagrams, the CLIP default temperature, the Loki -optimal temperature, and the ECE-optimal temperature are similar. (Right) Tradeoff between optimizing for zero-one error and the expected squared distance. Depending on the metric space, temperature scaling can improve accuracy .

${\color[rgb]{1,0,0}\beta}=$	$K=250$	$K=500$	$K=750$
0.5	61.28	46.57	36.31
1	52.98	41.06	33.14
2 (Loki)	46.78	37.76	29.88
4	47.99	39.58	34.65
8	65.29	58.42	54.90

Appendix

Table 4: Expected squared distances of SimCLR+Loki alternatives on ImageNet. We ablate over the choice of distance exponent in Loki (where

{\color[rgb]{1,0,0}\beta}=2

, corresponding to the Fréchet mean), including the Fréchet median (

{\color[rgb]{1,0,0}\beta}=1

). That is, we tune

{\color[rgb]{1,0,0}\beta}:\hat{y}\in\text{arg min}_{y\in\mathcal{Y}}\sum_{i=1}^{K}\mathbf{P}_{\lambda_{i}|x}d^{{\color[rgb]{1,0,0}\beta}}(y,\lambda_{i})

and find that

{\color[rgb]{1,0,0}\beta}=2

gives the lowest loss for each

K

Metric space $G$	${\color[rgb]{1,0,0}\text{diam}(G)^{2}}$	$\mathbb{E}[d^{2}(y,\hat{y})]/{\color[rgb]{1,0,0}\text{diam}(G)^{2}}$
Complete graph	1	0.5941 (0-1 error)
Default tree	16	0.4493
WordNet tree	169	0.1157
CLIP features	0.9726	0.2686

Appendix

Table 5: Comparison across metric spaces for CLIP on CIFAR-100 by normalizing by the squared diameter of the metric space:

\mathbb{E}[d^{2}(y,\hat{y})]/{\color[rgb]{1,0,0}\text{diam}(G)^{2}}

Figure 5: Classification regions in the probability simplex of 3-class classifiers faced with a 100-class problem where the classes are related by random trees graphs.

Figure 6: Classification regions in the probability simplex of 3-class classifiers faced with a 100-class problem where the classes are related by random phylogenetic trees graphs.

Figure 7: Classification regions in the probability simplex of 3-class classifiers faced with a 100-class problem where the classes are related by random small-world graphs generated by a Watts–Strogatz model.

Figure 8: Classification regions in the probability simplex of 3-class classifiers faced with a 100-class problem where the classes are related by random Erdős-Rényi graphs.

Figure 9: Classification regions in the probability simplex of 3-class classifiers faced with a 100-class problem where the classes are related by random Barabási–Albert graphs.

Explore similar work

Label Shift Aware Adaptation for Online Zero-shot Learning with Contrastive Language-Image Pre-Training (CLIP)
Jun 13, 2026Pengxiao Han, Changkun Ye, Yanshuo Wang +5Domain Adaptation Zero-Shot
Australian National University, Canberra, ACT, Australia · China North Vehicle Research Institute, Beijing, China · The Hong Kong Polytechnic University, Hong Kong SAR, China +3
Zero-Shot Neural Network Evaluation with Sample-Wise Activation Patterns
May 8, 2026Yameng Peng, Andy Song, HaythamM. Fayek +2Zero-Shot Convolutional Neural Networks
School of Computing Technologies, RMIT University, Australia · Department of Electronic Engineering and Information Science, University of Science and Technology of China
Multi-Label Test-Time Adaptation with Bayesian Conditional Priors
Jun 11, 2026Qiru Li, Ao Zhou, Zhiwei Jiang +4Vision-Language Model Adaptation Multi-Label Classification
State Key Laboratory of Novel Software Technology, Nanjing University, Nanjing 210023, China

Jun 13, 2026, cs.CVJ/K move · Enter open · S save

Label Shift Aware Adaptation for Online Zero-shot Learning with Contrastive Language-Image Pre-Training (CLIP)

Pengxiao Han, Changkun Ye, Yanshuo Wang, Jinguang Tong+4

Australian National University, Canberra, ACT, Australia · China North Vehicle Research Institute, Beijing, China · The Hong Kong Polytechnic University, Hong Kong SAR, China+3

Domain Adaptation Zero-Shot

Open paper AskPDF arXiv

Vision-language models like Contrastive Language-Image Pre-Training (CLIP) have been extensively studied in data-scarce scenarios. A particularly challenging and realistic task in this area is online zero-shot learning with CLIP, where unknown test samples are predicted sequentially in random order by CLIP while keeping the feature extraction and model parameters fixed during the sequential inference phase. Most existing approaches in this setting address the problem by adapting representations online using incoming test samples, while neglecting the distribution of the data on which CLIP was initially trained. This mismatch can lead to degraded performance when the label distribution in the test data differs from that of the training domain. To address this gap, we propose Label Shift Aware (LSA), which formulates the online zero-shot classification task as a domain adaptation problem. Specifically, LSA adapts the predictions computed by CLIP, which was trained on an unknown source distribution, to a target distribution using only unlabeled test data, and applies label shift correction to mitigate the mismatch between the source and target domains. The extensive experiments across multiple datasets demonstrate that the proposed LSA consistently outperforms state-of-the-art online zero-shot learning methods based on CLIP.

Abstract

Explore similar work

Label Shift Aware Adaptation for Online Zero-shot Learning with Contrastive Language-Image Pre-Training (CLIP)

Zero-Shot Neural Network Evaluation with Sample-Wise Activation Patterns

Multi-Label Test-Time Adaptation with Bayesian Conditional Priors