CAST: Closed-form Analytic Semantic Transfer for Zero-Shot Classifier Extension

William Heyden; Habib Ullah; Muhammad Salman Siddiqui; Fadi Al Machot

cs.CVAug 13, 2026

CAST: Closed-form Analytic Semantic Transfer for Zero-Shot Classifier Extension

Authors: William Heyden, Habib Ullah, Muhammad Salman Siddiqui, Fadi Al Machot

Organizations: Faculty of Science and Technology (REALTEK) Norwegian University of Life Sciences NMBU 1430 Ås, Norway

Abstract

Large pre-trained models have become foundational components of modern machine learning systems. Yet adapting these models to novel categories typically requires examples from the target distribution. In many domains, however, such data are unavailable. Zero-shot learning (ZSL) permits recognition under these limitations through relying on auxiliary semantic information such as textual descriptions. We introduce CAST (Closed-form Analytic Semantic Transfer), a training-free, image-free framework for extending a pre-trained classifier to previously unseen classes through weight injection. We provide a theoretical foundation for CAST and derive a finite-sample error decomposition that identifies the semantic extrapolation residual $ρ_u$ . The residual is a computable, model-agnostic measure and provides a principled criterion for dataset curation and benchmark design. Experiments on standard zero-shot learning benchmarks demonstrate that CAST matches or exceeds existing image-free approaches and approaches the performance of few-shot adaptation methods, while requiring neither iterative optimization nor examples from the target distribution.

Figures & tables

Figure 1: Geometric overview of CAST. Left: an unseen class’s CLIP text embedding $z_{u}$ (e.g. Penguin ) decomposes into an in-span component and an out-of-span residual (Eq. ( 20 )). Middle: the Ridge estimator $\hat{P}$ (Eq. ( 12 )) is the closed-form coordinate change between the two spaces. Right: seen-class weights obey a predictable structure; the synthesized $\hat{w}_{u}$ differs from the true, unobserved $w_{u}^{*}$ by the intrinsic gap and the extrapolation penalty (Proposition 1 ).

Panel of Figure 2 — Figure 2: Empirical validation of A1 and A2 on AWA2. ( 2(a) ) Cosine alignment $\cos(w_{c},\mu_{c})$ across five architectures (NC3): architectures satisfying NC3 by design concentrate near $\cos=1$ . ( 2(b) ) Unseen accuracy vs. CLIP validation loss during training, rising monotonically as $\varepsilon\to 0$ (A2).

Figure 3: Unseen accuracy vs. mean semantic extrapolation residual $\bar{\rho}_{u}$ across AWA2, CUB, APY, and FLO. Higher $\bar{\rho}_{u}$ consistently predicts lower accuracy (A3), independently of architecture.

Figure 4: Geometry of $\hat{w}_{u}$ vs. the true $w_{u}^{*}$ in weight space (AWA2, t-SNE). Stars: $w_{u}^{*}$ from a classifier retrained per held-out class. Crosses: synthesised $\hat{w}_{u}$ . Segment colour: $\rho_{u}$ . Low- $\rho_{u}$ classes synthesise close to their true weight; Blue+whale, the highest- $\rho_{u}$ class, shows the largest gap.

Panel of Figure 5 — Figure 5: Effect of unseen-class description on $\rho_{u}$ and synthesis quality, on AWA2. ( 5(a) ) Accuracy vs. bridge quality across description templates (listed in the supplementary material). ( 5(b) ) Per-class accuracy vs. $\rho_{u}$ .

AWA2		CLIP
Classifier	Baseline	ViT-b32	ViT-b16	RN50	RN101
AlexNet	92.68	63.5 (92.4)	64.0 (92.1)	60.1 (92.6)	68.3 (92.5)
ConvNext B	96.13	72.1 (96.3)	78.5 (96.3)	74.9 (96.2)	75.7 (96.2)
DensNet161	95.49	72.0 (95.4)	68.9 (95.3)	64.8 (95.4)	73.2 (95.4)
EfficientNet-v2-M	89.49	72.5 (89.4)	71.1 (89.2)	71.5 (89.5)	72.1 (89.4)
GoogLeNet	89.17	67.9 (89.2)	68.3 (88.9)	61.5 (89.2)	68.3 (89.2)

Table 1: Zero-shot classification performance on unseen classes using our proposed CAST method, in the conventional zero-shot setting. Red and blue values indicates best and second best performance of classifier per CLIP model. Parenthetical values report top-1 accuracy for synthesized seen-class weights, reflecting bridge reconstruction quality.

	AWA2			CUB			APY			FLO
Method	S	U	H	S	U	H	S	U	H	S	U	H
CLIP Base	41.9	61.3	49.8	35.4	43.1	38.9	21.4	11.2	14.7	50.6	37.3	42.9
ConSE [ 28 ]	96.1	3.0	5.7	88.0	0.5	0.9	90.5	2.8	5.5	98.5	0.3	0.7
COSTA [ 26 ]	96.1	0.0	0.0	87.6	0.0	0.0	90.2	0.0	0.0	98.7	0.0	0.0
SMO [ 47 ]	92.4	31.8	47.3	52.3	39.2	44.8	80.9	17.7	29.0	94.7	18.8	31.4
WAvg [ 47 ]	92.4	5.5	10.4	52.3	1.9	3.7	77.9	7.4	13.5	94.5	7.4	7.1

Table 2: Generalized zero-shot learning results. The compared methods are image-free only, i.e. they are trained through gradient descent. Our model is the only training-free approach. CLIP linear prope is shown for reference. Best and second-best results per dataset are shown in red and blue , respectively.

Panel of Figure 6 — Figure 6: Imprinting oracle experiment on AWA2. ( 6(a) ) Accuracy vs. $k$ for $k$ -shot imprinting against CAST and the oracle ( $k=\infty$ ). ( 6(b) ) Per-class breakdown, tracking $\rho_{u}$ (Section 5.2 ).

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

CUB		CLIP
Classifier	Baseline	RN50	RN101	ViT-B-32	ViT-B-16
AlexNet	78.34	30.1 (76.3)	30.9 (74.9)	30.74 (74.2)	30.1 (74.6)
ConvNext	54.99	30.0 (54.1)	29.2 (54.1)	32.9 (51.5)	29.4 (54.0)
DenseNet	67.60	33.3 (66.9)	36.1 (66.4)	36.2 (66.0)	35.5 (65.8)
EfficientNet-v2-M	43.20	22.0 (43.3)	23.1 (42.9)	24.3 (42.5)	21.7 (43.1)
GoogLeNet	44.66	25.2 (43.5)	25.6 (42.9)	24.4 (42.5)	22.4 (42.6)

Appendix

Table 3: Zero-shot classification performance on unseen CUB classes, in the same format as Table 1 . The consistently lower absolute accuracy relative to AWA2 reflects CUB’s higher semantic extrapolation difficulty

\bar{\rho}

(Section 5.2 ), not a failure specific to any classifier or CLIP backbone.

Zero-Shot Learning

Zero-shot learning lets models recognize classes or solve tasks absent from task-specific training examples by using semantic descriptions or pretrained knowledge.

See all 184 papers

Geometry-Aware Adaptation for Pretrained Models
Jul 23, 2023Nicholas Roberts, Xintong Li, Dyah Adila +4Zero-Shot Learning VLM Adaptation
University of Wisconsin-Madison
When Semantics Saturate or Emerge: Adaptation-Conditional Semantic Utility in Source-Free Cross-Domain Few-Shot Learning
Aug 7, 2026Wei Liu, Xing Deng, Haijian ShaoSource-Free Cross-Domain Few-Shot Learning Few-Shot Domain Adaptation
College of Computer Science, Jiangsu University of Science and Technology, Zhenjiang 212003, China
Mutually Causal Semantic Distillation Network for Zero-Shot Learning
Mar 18, 2026Shiming Chen, Shuhuang Chen, Guo-Sen Xie +1Zero-Shot Learning Cross-Modal Representation Learning
Huazhong University of Science and Technology, Wuhan, China · National Anti-Counterfeit Engineering Research Center, Wuhan, Chna · Nanjing University of Science and Technology, Nanjing, China

Jul 23, 2023J/K move · Enter open · S save

Geometry-Aware Adaptation for Pretrained Models

Nicholas Roberts, Xintong Li, Dyah Adila, Sonia Cromp+3

University of Wisconsin-Madison

Zero-Shot Learning VLM Adaptation Classification Zero-Shot Image Classification

Open paper AskPDF arXiv

Machine learning models -- including prominent zero-shot models -- are often trained on datasets whose labels are only a small proportion of a larger label space. Such spaces are commonly equipped with a metric that relates the labels via distances between them. We propose a simple approach to exploit this information to adapt the trained model to reliably predict new classes -- or, in the case of zero-shot prediction, to improve its performance -- without any additional training. Our technique is a drop-in replacement of the standard prediction rule, swapping argmax with the Fréchet mean. We provide a comprehensive theoretical analysis for this approach, studying (i) learning-theoretic results trading off label space diameter, sample complexity, and model dimension, (ii) characterizations of the full range of scenarios in which it is possible to predict any unobserved class, and (iii) an optimal active learning-like next class selection procedure to obtain optimal training classes for when it is not possible to predict the entire range of unobserved classes. Empirically, using easily-available external metrics, our proposed approach, Loki, gains up to 29.7% relative improvement over SimCLR on ImageNet and scales to hundreds of thousands of classes. When no such metric is available, Loki can use self-derived metrics from class embeddings and obtains a 10.5% improvement on pretrained zero-shot models such as CLIP.

Abstract

Zero-Shot Learning

Explore similar work

Geometry-Aware Adaptation for Pretrained Models

When Semantics Saturate or Emerge: Adaptation-Conditional Semantic Utility in Source-Free Cross-Domain Few-Shot Learning

Mutually Causal Semantic Distillation Network for Zero-Shot Learning