GeoTTER: Leveraging Local Geometry of Optimal Transport for Zero-Shot Classification

Wei-Yang Alex Lee; Rudrasis Chakraborty; Vishnu Lokhande

cs.LGSep 11, 2026

GeoTTER: Leveraging Local Geometry of Optimal Transport for Zero-Shot Classification

Authors: Wei-Yang Alex Lee, Rudrasis Chakraborty, Vishnu Lokhande

Organizations: SUNY Buffalo · Lawrence Livermore National Lab

Abstract

We present GeoTTER, a novel framework that redefines optimal transport in the realm of zero-shot classification. Conventional methods often suffer from miscalibration and a lack of adaptability, as they rely on fixed cost matrices derived solely from pre-trained model embeddings. In contrast, GeoTTER addresses these limitations by incorporating two key techniques. First, to alleviate high-frequency label jaggedness (sample-level manifold jitter that assigns neighboring embeddings to different classes), GeoTTER integrates local geometric structure into the optimal transport formulation via graph-Laplacian smoothing, a technique grounded in spectral graph theory that enforces neighborhood consistency. Second, to correct coherent angular drift (a low-frequency orientation bias in which large groups of samples share the same angular offset from their true label prototypes), we fuse clustering-guided cost components with a globally adjusted transport cost, achieving a multi-objective optimization that respects both global distribution constraints and latent data structure. With a median improvement of +6.82% compared to zero-shot and +2.13% compared to OTTER, GeoTTER shows robust improvements across a diverse set of benchmarks.

Figures & tables

Figure 1: Test-time adjustment (TTA) of a Frozen CLIP Encoder nder Label-Distribution Shift. The target domain is the four-class EUROSAT subset: forest, industrial, highway, annual-crop, whose classes occur in an approximately 1: 1: 1: 1 ratio, unlike the highly skewed class prior in CLIP’s Internet-scale pre-training data (estimated: forest $\gg$ industrial $\approx$ highway $\gg$ annual-crop). An unseen satellite image $x$ is first embedded by the frozen CLIP image encoder, yielding a zero-shot prediction $y^{\prime}$ that is biased toward the pre-training prior and therefore suffers an accuracy drop. A lightweight adapter, learned on the unlabeled target images at test time, transforms $y^{\prime}$ into an adjusted prediction $y^{\prime\prime}$ , effectively compensating for the prior mismatch and recovering performance, without updating CLIP’s weights or requiring any test-set labels.

N	128	256	512	1000	2000	4000	8000	12000	16000	22000	32000	50000	75000
EMD (ms, MB)	0.31 (0.00)	0.45 (0.00)	0.78 (0.00)	1.39 (0.00)	4.36 (0.00)	19.95 (0.00)	46.00 (0.00)	77.76 (0.00)	335.11 (0.00)	214.12 (0.00)	438.97 (0.00)	1317.09 (0.00)	4945.55 (0.00)
Sinkhorn (ms, MB)	1.89 (82.34)	1.92 (82.37)	1.98 (82.44)	2.05 (82.56)	2.04 (82.81)	1.90 (83.30)	3.19 (84.29)	2.04 (85.29)	2.06 (86.28)	2.05 (87.77)	2.42 (225.76)	2.55 (298.14)	3.39 (371.44)
Total OTTER (ms)	0.88	1.00	1.58	1.95	4.97	20.60	47.22	79.13	336.82	216.31	442.84	1322.84	4953.70
Total GeoTTER (ms)	203.85	211.25	209.97	210.00	213.76	216.43	223.97	247.72	254.15	265.45	289.69	336.15	381.81

Table 1: Complexity Analysis: Timing results vs. Number of samples (N)

Dataset	Standard Baselines			Other Baselines		Our Contribution
	ZS	PM ( Liusie et al.,, 2023 )	OTTER ( Shin et al.,, 2024 )	FPM	Sinkhorn	GeoTTER w/o Label Ratio	GeoTTER w/ Label Ratio	Multi- GeoTTER
CIFAR10	88.34	28.51	91.74	88.37	91.70	91.04	92.93	93.22
CIFAR100	63.82	60.97	67.94	65.16	68.12	63.92	68.88	67.73
Caltech101	79.75	50.66	88.73	83.57	57.81	79.55	89.01	88.31
Caltech256	79.80	8.74	86.99	82.48	70.81	78.30	89.62	89.23
Food101	85.59	79.88	89.87	86.38	90.01	85.10	90.46	89.46

Table 2: Comparison of classification accuracy (%) between GeoTTER variants and baselines using ViT-B/16 embeddings. Columns refer to methods defined in the paper: ZS (Zero-Shot baseline; Sec. 5.1), PM (Prior Matching; Sec. 3.2), OTTER (Sec. 3.3), FPM (Fenchel Prior Matching; Sec. 3.2 and Appendix), Sinkhorn (Sinkhorn-regularized OTTER; Sec. 3.3), GeoTTER w/o Label Ratio (Sec. 4.3), GeoTTER w/ Label Ratio (Secs. 4.1–4.2), and Multi-GeoTTER (multi-step propagation variant; Sec. 4.1). Note that PM, OTTER, FPM, Sinkhorn, GeoTTER w/ Label Ratio, and Multi-GeoTTER all uses label ratio in test time.

Symbol / Notation	Meaning in This Paper
$n$	Size of the test batch $X$ ; the number of unlabeled samples $x_{i}$ .
$m$	Number of classes (dimension of every probability vector).
$X=\{x_{i}\}_{i=1}^{n}$	Unlabeled test-time batch in the target domain.
$f$	Frozen backbone classifier that outputs logits for each $x_{i}$ .
$p_{i}=\mathrm{softmax}\!\bigl(f(x_{i})\bigr)$	Raw soft-max scores for $x_{i}$ in probability simplex $\Delta_{m-1}$ .
$\tilde{p}_{i}$	Calibrated probability vector obtained after Test-Time Adaptation (TTA).

Table 3: Glossary of symbols and definitions used in Sections 3–4.

Dataset	ZS	OTTER(EMD) ( Liusie et al.,, 2023 )	SK	GeoTTER
CIFAR10	88.92	89.68	89.65	91.05
CIFAR100	58.51	64.41	64.59	65.71
Caltech101	80.33	88.03	88.23	88.75
Caltech256	78.20	84.95	85.20	86.95
Food101	80.19	84.98	85.08	85.40
STL10	97.05	97.79	97.79	98.16

Table 4: Classification accuracy (%) on backbone ViT-B/32. The highest accuracy for each dataset is highlighted in bold.

Dataset	ZS	OTTER(EMD) ( Liusie et al.,, 2023 )	SK	GeoTTER
CIFAR10	95.00	95.96	96.06	96.93
CIFAR100	72.34	77.67	78.03	78.25
Caltech101	78.23	91.72	91.85	92.18
Caltech256	83.41	90.65	91.06	93.06
Food101	89.79	93.58	93.62	94.23
STL10	99.15	99.38	99.38	99.66

Table 5: Classification accuracy (%) on backbone ViT-L/14. The highest accuracy for each dataset is highlighted in bold.

Dataset	ZS	OTTER(EMD) ( Liusie et al.,, 2023 )	SK	GeoTTER
CIFAR10	66.51	76.78	76.99	78.57
CIFAR100	38.69	44.42	44.66	45.76
Caltech101	73.49	83.68	83.90	84.96
Caltech256	72.98	80.50	80.78	83.07
Food101	76.27	81.09	81.15	81.78
STL10	93.11	95.53	95.56	96.35

Table 6: Classification accuracy (%) on backbone RN50. The highest accuracy for each dataset is highlighted in bold.

Dataset	ZS	OTTER(EMD) ( Liusie et al.,, 2023 )	SK	GeoTTER
CIFAR10	79.08	81.11	81.04	83.38
CIFAR100	46.32	50.93	50.97	51.96
Caltech101	81.02	89.36	89.65	89.67
Caltech256	76.62	83.71	83.95	86.19
Food101	80.90	84.36	84.45	84.97
STL10	96.47	97.15	97.21	97.89

Table 7: Classification accuracy (%) on backbone RN101. The highest accuracy for each dataset is highlighted in bold.

Dataset	ZS	OTTER(EMD) ( Liusie et al.,, 2023 )	SK	GeoTTER
amazon	72.14	97.14	97.17	97.54
gender	51.08	75.89	75.93	76.14
civilcomments	56.13	82.80	83.49	85.90
hatexplain	27.14	32.26	32.26	32.63

Table 8: Classification accuracy (%) on backbone ADA. The highest accuracy for each dataset is highlighted in bold.

Dataset	ZS	OTTER(EMD) ( Liusie et al.,, 2023 )	SK	GeoTTER
amazon	74.02	92.38	92.44	94.18
gender	84.02	92.38	92.38	92.37
civilcomments	48.29	82.17	82.25	85.13
hatexplain	30.41	35.23	35.29	34.24

Table 9: Classification accuracy (%) on backbone BERT. The highest accuracy for each dataset is highlighted in bold.

Setting	CF	GS	Accuracy (%)
Baseline	no	no	88.61
+CF	yes	no	88.20
+GS	no	yes	89.62
+CF+GS	yes	yes	88.88

Table 10: ViT-B/16 on Caltech256: Ablation study on Clustering Fusion (CF) and Graph Smoothing (GS).

Setting	CF	GS	Accuracy (%)
Baseline	no	no	49.05
+CF	yes	no	50.73
+GS	no	yes	47.22
+CF+GS	yes	yes	49.60

Table 11: ViT-B/16 on EUROSAT: Ablation study on Clustering Fusion (CF) and Graph Smoothing (GS).

Setting	CF	GS	Accuracy (%)
Baseline	no	no	77.07
+CF	yes	no	77.23
+GS	no	yes	76.34
+CF+GS	yes	yes	77.05

Table 12: ViT-B/16 on Flowers102: Ablation study on Clustering Fusion (CF) and Graph Smoothing (GS).

Figure 2: Ablation Average ViTB16 on Caltech256, and its LODO

Figure 3: Ablation Average ViTB16 on EUROSAT, and its LODO

Figure 4: Ablation Average ViTB16 on Flowers102, and its LODO

Figure 5: Multi-GeoTTER with ViTB16 backbone on all datasets

Figure 6: Effect of Increasing CF and GS strength, under the varying strength gNoise on Flowers102

Figure 7: Effect of Increasing CF and GS strength, under the varying strength lNoise on Flowers102

Panel of Figure 8 — Figure 8: Noise on different ways of constructing gNoise and lNoise

Figure 9: Clustering-guided cost fusion on CIFAR-10. 2-D projection of CIFAR-10 embeddings: grey points are individual samples, and the black-rimmed circles mark the $K=10$ centroids obtained with K-Means that are later used to build the cluster-level cost term.

Figure 10: Toy illustration of Laplacian graph smoothing. Beginning with the $k$ -nearest-neighbor graph constructed on the same embeddings (left), a few iterations of the Laplacian update in Eq. (2) average each node over its neighborhood, eliminating high-frequency noise and producing a smoothly varying surface (right).

Dataset	Min Shot	Max Shot	Total Samples	Num Classes
CIFAR10	1000	1000	10000	10
CIFAR100	100	100	10000	100
Caltech101	16	785	7162	101
Caltech256	50	797	22897	257
Country211	100	100	21100	211
DTD	40	40	1880	47

Table 13: Dataset statistics including min shots, max shots, sample size, and class count.

Zero-Shot Image Classification

Zero-shot image classification uses semantic descriptions and pretrained visual representations to recognize image classes without labeled examples.

See all 49 papers

LAGO: Language-Guided Adaptive Object-Region Focus for Zero-Shot Visual-Text Alignment
May 4, 2026Junyi Hu, Qiji Zhou, Lei Zhang +1VLM Robustness Efficient VLM Inference
Beijing Jiaotong University · Westlake University · Rochester Institute of Technology
Geometry-Aware Adaptation for Pretrained Models
Jul 23, 2023Nicholas Roberts, Xintong Li, Dyah Adila +4Zero-Shot Learning VLM Adaptation
University of Wisconsin-Madison
Local-Global Prompt Learning via Sparse Optimal Transport
Mar 9, 2026Deniz Kizaroğlu, Ülku Tuncer Küçüktas, Emre Çakmakyurdu +1VLM Adaptation Visual Prompt Learning
Graduate School of Informatics, Middle East Technical University, Ankara, Turkey · Department of Electrical and Electronics Engineering, Gazi University, Ankara, Turkey

May 4, 2026J/K move · Enter open · S save

LAGO: Language-Guided Adaptive Object-Region Focus for Zero-Shot Visual-Text Alignment

Junyi Hu, Qiji Zhou, Lei Zhang, Yue Zhang

Beijing Jiaotong University · Westlake University · Rochester Institute of Technology

VLM Robustness Efficient VLM Inference Zero-Shot Image Classification Vision-Language Alignment

Open paper AskPDF arXiv

Zero-shot recognition aims to classify an image by selecting the most compatible label description from a set of candidate classes without any task-specific supervision. In fine-grained settings, however, the relevant evidence often lies in localized parts, attributes, or textures rather than in the full image, making whole-image alignment suboptimal. Recent localized visual-text alignment methods address this by comparing class descriptions with multiple image regions, but they typically rely on large sets of random or redundant crops, increasing inference cost and introducing many highly redundant or weakly relevant candidates. Moreover, introducing semantic guidance too early can create an error-amplifying feedback process in which inaccurate intermediate predictions bias later localization and reinforce subsequent mistakes; we refer to this failure mode as the prediction loop. We propose LAGO (LAnguage-Guided adaptive Object-region focus), a framework for efficient and robust zero-shot localized visual-text alignment. LAGO first performs class-agnostic object-centric candidate discovery to obtain a stable visual initialization, and then applies adaptive language-guided refinement with the strength of semantic guidance controlled by intermediate confidence. It further combines object-level, contextual, and full-image evidence through an effective object-context dual-channel aggregation strategy. Extensive experiments show that LAGO consistently achieves state-of-the-art performance on standard zero-shot benchmarks and challenging distribution-shift settings, while requiring substantially fewer candidate regions at inference time.

Abstract

Zero-Shot Image Classification

Explore similar work

LAGO: Language-Guided Adaptive Object-Region Focus for Zero-Shot Visual-Text Alignment

Geometry-Aware Adaptation for Pretrained Models

Local-Global Prompt Learning via Sparse Optimal Transport