TALON: Test-time Adaptive Learning for On-the-Fly Category Discovery

Yanan Wu; Yuhan Yan; Tailai Chen; Zhixiang Chi; ZiZhang Wu; Yi Jin; Yang Wang; Zhenbo Li

cs.CVMar 9, 2026

TALON: Test-time Adaptive Learning for On-the-Fly Category Discovery

Authors: Yanan Wu, Yuhan Yan, Tailai Chen, Zhixiang Chi, ZiZhang Wu, Yi Jin, Yang Wang, Zhenbo Li

Organizations: College of Information and Electrical Engineering, China Agricultural University, China · College of Science, China Agricultural University, China · Woof RoboT Co., Ltd., China · Department of Electrical and Computer Engineering, University of Toronto, Canada ·

Abstract

On-the-fly category discovery (OCD) aims to recognize known categories while simultaneously discovering novel ones from an unlabeled online stream, using a model trained only on labeled data. Existing approaches freeze the feature extractor trained offline and employ a hash-based framework that quantizes features into binary codes as class prototypes. However, discovering novel categories with a fixed knowledge base is counterintuitive, as the learning potential of incoming data is entirely neglected. In addition, feature quantization introduces information loss, diminishes representational expressiveness, and amplifies intra-class variance. It often results in category explosion, where a single class is fragmented into multiple pseudo-classes. To overcome these limitations, we propose a test-time adaptation framework that enables learning through discovery. It incorporates two complementary strategies: a semantic-aware prototype update and a stable test-time encoder update. The former dynamically refines class prototypes to enhance classification, whereas the latter integrates new information directly into the parameter space. Together, these components allow the model to continuously expand its knowledge base with newly encountered samples. Furthermore, we introduce a margin-aware logit calibration in the offline stage to enlarge inter-class margins and improve intra-class compactness, thereby reserving embedding space for future class discovery. Experiments on standard OCD benchmarks demonstrate that our method substantially outperforms existing hash-based state-of-the-art approaches, yielding notable improvements in novel-class accuracy and effectively mitigating category explosion. The code is publicly available at \textcolor{blue}{https://github.com/ynanwu/TALON}.

Figures & tables

Figure 1: At test time , existing methods ( left ) rely on static inference and fail to adapt to label space shifts, leading to inaccurate discovery and recognition. In contrast, our TALON ( right ) continually accumulates knowledge from unlabeled test data to overcome this challenge. Moreover, our hash-free framework ( right ) replaces heuristic hash-based ( left ) designs, achieving higher representational expressiveness and better discovery stability.

Figure 2: Overview of the proposed TALON framework. (a) During the offline stage, we introduce margin-aware logit calibration to enlarge inter-class margins and enhance intra-class compactness, reserving embedding space for future category discovery. (b) At test-time, we jointly update the encoder and class prototypes, enabling the model to learn through discovery rather than static inference.

	Method	CIFAR10 (%)			CIFAR100 (%)			ImageNet-100 (%)			CUB-200-2011 (%)			Stanford Cars (%)			Oxford Pets (%)			Food101 (%)
		All	Old	New	All	Old	New	All	Old	New	All	Old	New	All	Old	New	All	Old	New	All	Old	New
Greedy–Hungarian	SLC [ 17 ]	65.9	96.5	50.9	46.9	62.1	16.6	34.2	86.6	7.1	30.2	46.5	22.1	14.4	23.9	9.8	–	–	–	–	–	–
	MLDG [ 26 ]	71.6	97.5	58.6	58.4	69.0	37.3	33.6	74.4	13.1	34.2	57.9	22.4	28.0	49.1	17.7	–	–	–	–	–	–
	RankStat [ 16 ]	56.5	81.1	44.2	36.9	45.7	19.3	33.1	74.2	12.4	22.7	30.0	19.1	16.2	23.3	12.8	–	–	–	–	–	—
	WTA [ 21 ]	65.4	88.0	54.1	44.1	55.5	21.2	33.1	75.8	11.7	23.8	30.5	20.4	18.3	26.2	14.5	–	–	–	–	–	–
	SMILE [ 11 ]	78.2	99.3	67.6	61.3	70.7	42.5	39.9	87.1	16.2	41.1	67.6	27.8	33.4	58.4	21.3	54.1	66.1	47.8	34.4	64.0	19.4

Table 1: Comparison with other SOTA methods. “DiffGRE+S” and “DiffGRE+P” represent the integration of DiffGRE with SMILE and PHE, respectively. The best results are highlighted in bold , while the second-best results are underlined . Blue numbers indicate results not reported in the original papers but reproduced using the official code.

Method	CUB-200-2011 (%)			Stanford Cars (%)
Method	All	Old	New	All	Old	New
Baseline	44.5	57.6	37.9	47.8	66.7	38.6
Baseline+MLC	45.7	58.6	39.2	49.0	65.4	41.4
Baseline+TTA-P	45.4	56.4	39.9	48.7	63.0	41.8
Baseline+MLC+TTA-P	46.7	57.7	41.2	52.7	69.7	44.5
Baseline+TTA-M	44.5	56.4	38.5	47.7	66.3	38.8

Table 2: Ablation study of various components of our TALON on the CUB and Scars datasets. Here, MLC, TTA-P, and TTA-M denote the margin-aware logit calibration, prototype update, and model adaptation modules, respectively.

Method	CUB ( $C{=}200$ )				SCars ( $C{=}196$ )
Method	#Cls	All	Old	New	#Cls	All	Old	New
SMILE-64bit [ 11 ]	2910	22.6	45.3	11.2	4788	16.5	38.2	6.1
SMILE-32bit [ 11 ]	2146	27.3	52.0	15.0	2953	21.9	46.8	9.9
SMILE-16bit [ 11 ]	924	31.9	52.7	21.5	896	27.5	52.5	15.4
PHE-64bit [ 61 ]	493	38.1	60.1	27.2	917	32.1	66.9	15.3
PHE-32bit [ 61 ]	474	38.5	59.9	27.8	762	31.5	64.0	15.8

Table 3: Comparison of estimated category numbers and accuracies on the CUB-200-2011 and Stanford Cars datasets.

Figure 3: Angular analysis of the margin-aware logit calibra-tion on the Pets dataset. Left: angles between samples and their class prototypes; Right: angles between different class prototypes.

Method	Stanford Cars (%)			Oxford Pets (%)
Method	All	Old	New	All	Old	New
baseline+MLC	49.0	65.4	41.4	61.7	70.4	57.1
baseline+MLC+TENT [ 50 ]	48.1	66.4	39.2	58.4	66.1	54.3
baseline+MLC+OSTTA [ 9 ]	47.2	62.3	39.9	60.3	70.4	54.9
baseline+MLC+TTA-P-TENT	49.4	67.4	40.7	60.0	65.0	57.4
baseline+MLC+TTA-P-OSTTA	47.6	65.2	39.2	61.9	70.6	57.3

Table 4: Comparison with existing test-time adaptation methods on the Stanford Cars and Oxford Pets datasets.

Figure 4: Hyperparameter analysis on CUB-200-2011, Stanford Cars, and Oxford Pets datasets. Each column corresponds to one dataset, showing the effects of adaptation batch size, angular margin $m$ , and similarity threshold $\tau$ on accuracy ( All , Old , New ) and the number of newly discovered categories (NDC).

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Dataset	CIFAR10	CIFAR100	ImageNet-100
$\|Y_{Q}\|$	5	80	80
$\|Y_{S}\|$	10	100	100
$\|D_{S}\|$	12.5K	20.0K	31.9K
$\|D_{Q}\|$	37.5K	30.0K	95.3K

Appendix

Table 5: Statistics of coarse-grained datasets.

Dataset	CUB-200-2011	Stanford Cars	Oxford Pets	Food101
$\|Y_{Q}\|$	100	98	19	51
$\|Y_{S}\|$	200	196	38	101
$\|D_{S}\|$	1.5K	2.0K	0.9K	19.1K
$\|D_{Q}\|$	4.5K	6.1K	2.7K	56.6K

Appendix

Table 6: Statistics of fine-grained datasets.

Figure 5: Hyperparameter analysis on CUB-200-2011, Stanford Cars, and Oxford Pets datasets. Each column corresponds to one dataset, showing the effects of logit scale

s

and the smoothing constant

\kappa

on accuracy ( All , Old , New ) and the number of newly discovered categories (NDC).

	Method	Backbone	CIFAR10 (%)			CIFAR100 (%)			ImageNet-100 (%)			CUB-200-2011 (%)			Stanford Cars (%)			Oxford Pets (%)			Food101 (%)
			All	Old	New	All	Old	New	All	Old	New	All	Old	New	All	Old	New	All	Old	New	All	Old	New
Greedy–Hungarian	SMILE [ 11 ]	DINO	78.2	99.3	67.6	61.3	70.7	42.5	39.9	87.1	16.2	41.1	67.6	27.8	33.4	58.4	21.3	54.1	66.1	47.8	34.4	64.0	19.4
	SMILE [ 11 ]	CLIP	82.4	97.4	74.9	56.4	64.6	40.0	47.5	71.0	35.7	43.7	69.7	30.8	36.7	57.2	26.8	58.2	77.5	48.1	40.5	70.4	25.2
	PHE [ 61 ]	DINO	83.0	98.0	75.5	64.8	78.8	36.9	53.1	83.5	38.1	46.9	76.0	32.4	46.3	78.3	30.8	63.3	91.3	48.6	50.0	89.3	30.0
	PHE [ 61 ]	CLIP	79.3	97.0	70.4	66.1	80.3	37.5	52.9	87.8	35.5	44.2	70.3	31.1	46.4	78.1	31.1	64.1	86.2	52.4	47.8	88.4	27.0
	Ours	DINO	86.2	95.4	79.3	72.5	85.2	47.0	84.1	94.3	63.4	52.6	83.3	37.2	42.9	78.1	25.9	81.0	92.4	75.1	44.5	80.4	26.2

Appendix

Table 7: Comparison with State-of-the-Art Methods Using Different Backbones

Dataset	Train (ms)	Prototype computation (ms)	Test (ms)
CIFAR-10	746.506 $\pm$ 1.445	667.455 $\pm$ 41.925	1496.276 $\pm$ 37.384
CIFAR-100	747.667 $\pm$ 3.647	734.105 $\pm$ 55.240	1504.007 $\pm$ 65.634
ImageNet-100	746.663 $\pm$ 1.120	666.157 $\pm$ 3.393	2324.649 $\pm$ 535.563
CUB-200-2011	748.577 $\pm$ 1.020	642.528 $\pm$ 1.003	1753.667 $\pm$ 44.756
Stanford Cars	750.333 $\pm$ 3.086	653.369 $\pm$ 1.991	1736.116 $\pm$ 64.858
Oxford Pets	749.094 $\pm$ 3.262	641.062 $\pm$ 0.394	1472.476 $\pm$ 72.950

Appendix

Table 8: Three-stage time consumption proportion display table

Figure 6: Display of the proportion of time consumed in the three stages.

Method	CUB	SCars	Food	Pets
SMILE	4204.9	5846.7	27000.7	2646.5
PHE	3300.0	5313.6	22765.7	2287.9
Ours	1452.3	1823.2	12157.5	1074.5

Appendix

Table 9: Comparison of training times (in second)

Test-Time Adaptation

Test-time adaptation updates or recalibrates pretrained models at inference to handle distribution shifts, often without target labels.

See all 318 papers

Learning through Creation: A Hash-Free Framework for On-the-Fly Category Discovery
Mar 14, 2026Bohan Zhang, Weidong Tang, Zhixiang Chi +4Generalized Category Discovery
College of Information and Electrical Engineering, China Agricultural University, China · Department of Electrical and Computer Engineering, University of Toronto, Canada · School of Computer and Information Technology, Beijing Jiaotong University, China +1
PACO: Proxy-Task Alignment and Online Calibration for On-the-Fly Category Discovery
Apr 13, 2026Weidong Tang, Bohan Zhang, Zhixiang Chi +3Generalized Category Discovery Model Calibration
China Agricultural University, College of Information and Electrical Engineering, Beijing, China · Concordia University, Department of Computer Science and Software Engineering, Montreal, Canada
Test-Time Generalized Category Discovery
Sep 27, 2026Shambhavi Mishra, Omprakash Chakraborty, Julio Silva-Rodriguez +3Test-Time Adaptation of VLMs Generalized Category Discovery
ÉTS Montréal, Canada

Mar 14, 2026J/K move · Enter open · S save

Learning through Creation: A Hash-Free Framework for On-the-Fly Category Discovery

Bohan Zhang, Weidong Tang, Zhixiang Chi, Yi Jin+3

College of Information and Electrical Engineering, China Agricultural University, China · Department of Electrical and Computer Engineering, University of Toronto, Canada · School of Computer and Information Technology, Beijing Jiaotong University, China+1

Generalized Category Discovery

Open paper AskPDF arXiv

On-the-Fly Category Discovery (OCD) aims to recognize known classes while simultaneously discovering emerging novel categories during inference, using supervision only from known classes during offline training. Existing approaches rely either on fixed label supervision or on diffusion-based augmentations to enhance the backbone, yet none of them explicitly train the model to perform the discovery task required at test time. It is fundamentally unreasonable to expect a model optimized on limited labeled data to carry out a qualitatively different discovery objective during inference. This mismatch creates a clear optimization misalignment between the offline learning stage and the online discovery stage. In addition, prior methods often depend on hash-based encodings or severe feature compression, which further limits representational capacity. To address these issues, we propose Learning through Creation (LTC), a fully feature-based and hash-free framework that injects novel-category awareness directly into offline learning. At its core is a lightweight, online pseudo-unknown generator driven by kernel-energy minimization and entropy maximization (MKEE). Unlike previous methods that generate synthetic samples once before training, our generator evolves jointly with the model dynamics and synthesizes pseudo-novel instances on the fly at negligible cost. These samples are incorporated through a dual max-margin objective with adaptive thresholding, strengthening the model's ability to delineate and detect unknown regions through explicit creation. Extensive experiments across seven benchmarks show that LTC consistently outperforms prior work, achieving improvements ranging from 1.5 percent to 13.1 percent in all-class accuracy. The code is available at https://github.com/brandinzhang/LTC

Abstract

Test-Time Adaptation

Explore similar work

Learning through Creation: A Hash-Free Framework for On-the-Fly Category Discovery

PACO: Proxy-Task Alignment and Online Calibration for On-the-Fly Category Discovery

Test-Time Generalized Category Discovery