SmoothOperator: Enhancing Representations for Fine-grained Open-set Recognition via Modulated Label Smoothing

Thiru Thillai Nadarasar Bahavan; Yu Xia; Sachith Seneviratne; Saman Halgamuge

cs.CVOct 1, 2026

SmoothOperator: Enhancing Representations for Fine-grained Open-set Recognition via Modulated Label Smoothing

Authors: Thiru Thillai Nadarasar Bahavan, Yu Xia, Sachith Seneviratne, Saman Halgamuge

Organizations: The University of Melbourne Parkville, VIC, Australia

Abstract

Open Set Recognition (OSR) aims to enable models to accurately classify known classes while rejecting samples from unseen classes. A key challenge in OSR lies in the inability to model the unbounded distribution of unknown classes during training, often leading to the misclassification of samples from these classes. Rather than modeling unknowns, recent work shapes the feature space so that known classes are compact and well separated, and spherical representation learning methods have achieved strong results this way. Label smoothing has been identified as one of the key drivers of this success, yet it applies the same coefficient to every training sample, regardless of how well each sample is already embedded. We show that the spherical representation learning objectives used in OSR share a single alignment--uniformity structure in which labels enter only through the alignment term. Label smoothing therefore acts as an alignment dial, and a fixed coefficient sets this dial to the same value for every sample. We propose a plug-in, SmoothOperator (SmoothOP), which sets the smoothing coefficient of each sample from its \textbf{prominence}, an embedding-space signal measuring how clearly the sample's own class stands out against its strongest competing class. Our method integrates into four existing spherical representation learning methods at minimal training overhead. SmoothOP assigns strong smoothing to samples with high prominence, which reduces their alignment and relaxes their pull. On the Semantic Shift Benchmark, SmoothOP-augmented variants generally outperform their base objectives across datasets, degrees of semantic shift, and OSR post-processors, with gains of up to 4.7% in AUROC, OSCR, and closed-set accuracy.

Figures & tables

Figure 1: Overview of the Proposed Framework. A three-stage process in which an input sample passes through an augmentation pipeline before reaching the feature encoder. An Observer Classifier estimates prominence from gradient-detached features to modulate label smoothing in the representation learning objective.

Method	Target $S_{i\cdot}$	Alignment $\mathcal{L}^{(i)}_{\mathrm{align}}$	Uniformity $\mathcal{L}^{(i)}_{\mathrm{unif}}$
SupCON	$S_{ij}=\dfrac{s(\tilde{\mathbf{y}}_{i},\tilde{\mathbf{y}}_{j})}{\sum_{n\neq i}s(\tilde{\mathbf{y}}_{i},\tilde{\mathbf{y}}_{n})}$	$\dfrac{1}{2\tau}\sum\nolimits_{j\neq i}S_{ij}\,\lVert\mathbf{z}_{i}-\mathbf{z}_{j}\rVert_{2}^{2}$	$\log\sum\nolimits_{n\neq i}\exp\!\left(-\dfrac{\lVert\mathbf{z}_{i}-\mathbf{z}_{n}\rVert_{2}^{2}}{2\tau}\right)$
ConOSR	$S_{ij}=\dfrac{s(\tilde{\mathbf{y}}_{i},\tilde{\mathbf{y}}_{j})}{\sum_{n\neq i}s(\tilde{\mathbf{y}}_{i},\tilde{\mathbf{y}}_{n})}$	$\dfrac{1}{2\tau}\sum\nolimits_{j\neq i}S_{ij}\,\lVert\mathbf{z}_{i}-\mathbf{z}_{j}\rVert_{2}^{2}$	$\log\sum\nolimits_{n\neq i}\exp\!\left(-\dfrac{\lVert\mathbf{z}_{i}-\mathbf{z}_{n}\rVert_{2}^{2}}{2\tau}\right)$
SpHOR-Lite	$S_{ik}=\dfrac{\tilde{y}_{ik}}{\sum_{j=1}^{C}\tilde{y}_{ij}}$	$\dfrac{1}{2\tau}\sum\nolimits_{k=1}^{C}S_{ik}\,\lVert\mathbf{z}_{i}-\boldsymbol{\mu}_{k}\rVert_{2}^{2}$	$\log\sum\nolimits_{j=1}^{C}\exp\!\left(-\dfrac{\lVert\mathbf{z}_{i}-\boldsymbol{\mu}_{j}\rVert_{2}^{2}}{2\tau}\right)$
SpHOR	$S_{ik}=\dfrac{\tilde{y}_{ik}}{\sum_{j=1}^{C}\tilde{y}_{ij}}$	$\dfrac{1}{2\tau}\sum\nolimits_{k=1}^{C}S_{ik}\,\lVert\mathbf{z}_{i}-\boldsymbol{\mu}_{k}\rVert_{2}^{2}$	$\log\sum\nolimits_{j=1}^{C}\exp\!\left(-\dfrac{\lVert\mathbf{z}_{i}-\boldsymbol{\mu}_{j}\rVert_{2}^{2}}{2\tau}\right)$

Table 1: Alignment–uniformity decomposition of the target-dependent loss of each representation learning objective. The target distribution

S_{ab}

, the sole carrier of the label smoothing degree, appears only in the alignment term, while the uniformity term is independent of the targets. This is detailed in Appendix F . Here

s(\cdot,\cdot)

is cosine similarity and

\tilde{\mathbf{y}}_{i}

is the label-smoothed target

SupCON +SmoothOP: CUB $-0.5$ , Aircraft $-4$ , Cars $-3$
		OSCR						AUROC
		CUB		Aircraft		Cars		CUB		Aircraft		Cars
		E	H	E	H	E	H	E	H	E	H	E	H
MSP	CE	$58.64\,{\pm}\,0.71$	$55.75\,{\pm}\,0.44$	$69.63\,{\pm}\,0.72$	$67.97\,{\pm}\,0.45$	$83.28\,{\pm}\,0.36$	$76.48\,{\pm}\,0.25$	$73.20\,{\pm}\,1.32$	$68.42\,{\pm}\,0.24$	$76.94\,{\pm}\,0.55$	$75.13\,{\pm}\,1.05$	$87.54\,{\pm}\,0.33$	$79.53\,{\pm}\,0.38$
	SupCON(w/o LS)	$59.54\,{\pm}\,1.30$	$54.49\,{\pm}\,1.17$	$69.16\,{\pm}\,1.02$	$68.46\,{\pm}\,0.82$	$82.13\,{\pm}\,0.51$	$75.61\,{\pm}\,0.79$	$74.85\,{\pm}\,0.87$	$66.51\,{\pm}\,1.18$	$77.14\,{\pm}\,1.06$	$75.53\,{\pm}\,0.72$	$86.37\,{\pm}\,0.80$	$78.75\,{\pm}\,0.82$
MSP	Baseline	$61.42\,{\pm}\,0.87$	$56.64\,{\pm}\,0.84$	$70.87\,{\pm}\,1.75$	$66.93\,{\pm}\,2.34$	$83.61\,{\pm}\,0.74$	$75.36\,{\pm}\,1.46$	$76.65\,{\pm}\,0.45$	$68.76\,{\pm}\,0.29$	$80.28\,{\pm}\,0.60$	$74.82\,{\pm}\,1.34$	$88.49\,{\pm}\,0.13$	$78.70\,{\pm}\,1.07$

Table 2: Fine-grained OSR performance on the Semantic Shift Benchmark (SCARS, FGVC-Aircraft, and CUB): OSCR and AUROC Each dataset has an "Easy" and "Hard" unknown class split ( Vaze et al., 2022 ) . We designate OSCR as the primary metric. For each representation learning method (ConOSR ( Xu et al., 2023 ) , SpHOR ( Bahavan et al., 2026 ) , Supervised Contrastive Learning ( Khosla and others, 2020 ) ) and its augmented variant (+), we evaluate performance across four SOTA scoring functions/postprocessors. Our augmented variants consistently outperform baselines across various metrics and datasets. All methods utilize a ResNet-50 backbone pre-trained on the Places dataset. Gains are in blue. The results are given as the mean

\pm

sd over three training runs with fixed seeds.

	CUB			Aircraft			Cars
Method		+SmoothOP	$\Delta\%$		+SmoothOP	$\Delta\%$		+SmoothOP	$\Delta\%$
SupCON	$71.74\,{\pm}\,1.16$	$72.29\,{\pm}\,0.62$	$\mathbf{+0.77\%}$	$81.21\,{\pm}\,1.62$	$82.99\,{\pm}\,0.52$	$\mathbf{+2.19\%}$	$90.87\,{\pm}\,0.87$	$91.40\,{\pm}\,0.32$	$\mathbf{+0.58\%}$
ConOSR	$75.89\,{\pm}\,0.53$	$75.42\,{\pm}\,1.23$	$\mathbf{-0.62\%}$	$85.73\,{\pm}\,0.67$	$86.19\,{\pm}\,0.73$	$\mathbf{+0.54\%}$	$92.67\,{\pm}\,0.18$	$93.26\,{\pm}\,0.53$	$\mathbf{+0.64\%}$
SpHOR-Lite	$74.20\,{\pm}\,0.88$	$74.58\,{\pm}\,0.75$	$\mathbf{+0.51\%}^{\dagger}$	$83.53\,{\pm}\,0.76$	$83.67\,{\pm}\,0.66$	$\mathbf{+0.17\%}$	$91.80\,{\pm}\,0.62$	$91.86\,{\pm}\,0.45$	$\mathbf{+0.07\%}$
SpHOR	$77.05\,{\pm}\,0.84$	$77.29\,{\pm}\,0.94$	$\mathbf{+0.31\%}$	$85.83\,{\pm}\,0.46$	$85.91\,{\pm}\,0.24$	$\mathbf{+0.09\%}$	$92.93\,{\pm}\,0.69$	$93.32\,{\pm}\,0.51$	$\mathbf{+0.42\%}^{\dagger}$

Table 3: Fine-grained OSR performance on the Semantic Shift Benchmark (SCARS, FGVC-Aircraft, and CUB): Closed-set Accuracy Gains are in blue. The results are given as the mean

\pm

sd over three training runs with fixed seeds.

	SupCON		ConOSR		SpHOR-Lite		SpHOR
	Comp. $\downarrow$	Disp. $\uparrow$	Comp. $\downarrow$	Disp. $\uparrow$	Comp. $\downarrow$	Disp. $\uparrow$	Comp. $\downarrow$	Disp. $\uparrow$
CUB	$-0.0052$	$+0.0111$	$+0.0005$	$+0.0035$	$-0.0135$	$+0.0004$	$-0.0152$	$+0.0256$
Aircraft	$-0.0167$	$+0.0166$	$-0.0088$	$+0.0169$	$-0.0059$	$+0.0126$	$-0.0148$	$+0.0360$
Cars	$-0.0124$	$+0.0155$	$-0.0068$	$+0.0250$	$-0.0052$	$+0.0058$	$-0.0201$	$+0.0457$

Table 4: Feature geometry in terms of the gains in Intra-class compactness (Comp) and Inter-class dispersion (Disp). Each

\Delta

reflects the gain Comp/Disp when the baseline method was augmented with SmoothOP. Colored in blue means the intervention improved the feature geometry in that aspect. Results over 3 seeds.

Appendix figures & tables12 assets

Supplementary material from the paper’s appendix.

Appendix

Dataset	Total Images	Total Classes	Known Classes	Unknown Classes
CUB	11,788	200	100	100
Stanford Cars	16,185	196	98	98
FGVC-Aircraft	10,000	100	50	50

Appendix

Table 6: Statistics of datasets used in the Semantic Shift Benchmark (SSB). Known classes are used for training, while unknown classes are reserved for open-set evaluation.

Panel of Figure 2 — Figure 2: OSCR operating curves for SupCON against SSB with the MSP post-processor. Solid indicates with SmoothOP ; dashed is the baseline without the plug-in. The curves are vertically averaged over 3 seeds for a fixed FPR. The results are shown for both the Easy and Hard Splits. SmoothOP pushes out baselines almost consistently across all operating points.

Panel of Figure 3 — Figure 3: OSCR operating curves for ConOSR against SSB with the MSP post-processor. Solid indicates with SmoothOP ; dashed is the baseline without the plug-in. The curves are vertically averaged over 3 seeds for a fixed FPR. The results are shown for both the Easy and Hard Splits. SmoothOP pushes out baselines almost consistently across all operating points.

Panel of Figure 4 — Figure 4: OSCR operating curves for SpHOR-Lite against SSB with the MSP post-processor. Solid indicates with SmoothOP ; dashed is the baseline without the plug-in. The curves are vertically averaged over 3 seeds for a fixed FPR. The results are shown for both the Easy and Hard Splits. SmoothOP pushes out baselines almost consistently across all operating points.

Panel of Figure 5 — Figure 5: OSCR operating curves for SpHOR against SSB with the MSP post-processor. Solid indicates with SmoothOP ; dashed is the baseline without the plug-in. The curves are vertically averaged over 3 seeds for a fixed FPR. The results are shown for both the Easy and Hard Splits. SmoothOP pushes out baselines almost consistently across all operating points.

Dataset	$C$	Head params	% of backbone params	% of train step
CUB	100	204 900	0.80%	0.010%
FGVC-Aircraft	50	102 450	0.40%	0.005%
Stanford Cars	98	200 802	0.79%	0.010%

Appendix

Table 7: Performance cost of the Observer classifier on the ResNET-50 backbone. For heavier backbones, the cost will become even smaller.

easy
		OSCR, mean of $3$ seeds				treatment $-$ control
arm	data	Baseline	+SmoothOP	+Batch-mean	+Rev.As.	$\Delta^{Batch-mean}$	$\Delta^{reverse}$
SupCON	CUB	$59.67$	$60.83$	$60.46$	$60.58$	$+0.37$	$+0.25$
	Aircraft	$70.87$	$72.12$	$71.91$	$70.88$	$+0.22$	$+1.25$
	Cars	$82.40$	$82.92$	$82.97$	$82.69$	$-0.05$	$+0.23$
ConOSR	CUB	$64.05$	$63.67$	$63.81$	$64.20$	$-0.14$	$-0.52$

Appendix

Table 8: OSCR averaged over MSP, MLS, NNGuide-5 and PostMax across

3

seeds. The three regimes are Batch mean (+Batch-mean), Reversed Assignment (+Rev.As.) and Treatment (+SmoothOP). To isolate the effectiveness of the per-sample assignment, we calculate two gains by subtracting the Reversed Assignment/Batch mean regime from the SmoothOP regime, which highlights the gain in terms of the per-sample smoothing. If the gain is positive, it is marked in blue and is supportive of our hypothesis.

Figure 6: Evolution of the Label Smoothing coefficient

\varepsilon

across the training epochs across all dataset-method combinations, (a) SupCON, (b) ConOSR, (c) SpHOR-Lite, (d) SpHOR , and rows are the three datasets, each annotated with its inherited

\lambda

. Results averaged over 3 seeds. The line is the batch-mean of

\varepsilon

and the band is inter-decile (

p_{10}

–

p_{90}

) range across the batch . The dashed line is the

\alpha=0

baseline. The X-axis is the time mapped to the training epochs

		OSCR-Easy			OSCR-Hard
Objective	Dataset	coeff.	margin	$\Delta$	coeff.	margin	$\Delta$
ConOSR	CUB	$65.41$	$65.19$	$-0.23$	$59.72$	$59.86$	$+0.14$
ConOSR	Aircraft	$76.56$	$76.94$	$+0.38$	$71.70$	$72.09$	$+0.38$
ConOSR	Cars	$86.05$	$86.13$	$+0.08$	$78.05$	$78.17$	$+0.12$
SpHOR	CUB	$67.80$	$68.26$	$+0.45$	$61.98$	$62.20$	$+0.21$
SpHOR	Aircraft	$76.85$	$77.55$	$+0.70$	$71.50$	$72.04$	$+0.53$

Appendix

Table 9: Margin blending beats coefficient blending on

11

12

cells. OSCR under MSP at shared hyperparameters, mean over

3

seeds;

\Delta=\text{margin blending}-\text{coeff. blending}

, red where margin blending is ahead.

easy
		AS $\downarrow$			NS $\uparrow$
arm	data	w/o	$+$ SmoothOP	gain	w/o	$+$ SmoothOP	gain
SupCON	CUB	$\mathbf{0.8793}$	$0.8796$	$-0.0003$	$69.22$	$\mathbf{72.21}$	$+2.99$
	Aircraft	$\mathbf{0.9130}$	$0.9159$	$-0.0030$	$77.59$	$\mathbf{78.07}$	$+0.48$
	Cars	$\mathbf{0.8959}$	$0.8987$	$-0.0027$	$\mathbf{84.98}$	$84.14$	$-0.84$
ConOSR	CUB	$0.8771$	$\mathbf{0.8766}$	$+0.0005$	$\mathbf{68.94}$	$68.52$	$-0.42$

Appendix

Table 10: Angular Separability (AS, Eq. 28 , lower is better) and Norm Separability (NS, Eq. 29 , AUROC in

\%

, higher is better) on the test set, mean of

3

seeds, without and with

+

SmoothOP. Within each pair the better level is bold. Gain is the improvement in the metric’s own direction, w/o

-

+

SmoothOP for AS and

+

SmoothOP

-

w/o for NS, paired by seed; a positive gain is red.

Figure 7: SupCON and ConOSR: Hyperparameter Sweep for $\alpha$

Figure 8: SpHOR-Lite and SpHOR: Hyperparameter Sweep for $\alpha$

Explore similar work

Jun 25, 2026cs.CV

Geometric Gradient Rectification for Safe Open-Set Semi-Supervised Learning

Open-set semi-supervised learning aims to leverage unlabeled data that may contain out-of-distribution outliers while maintaining performance on in-distribution classes. Existing methods mainly follow two paradigms: filtering suspicious samples or incorporating unlabeled objectives with soft weighting. We argue that both face a common trade-off: aggressive filtering can discard informative but hard ID samples, whereas utilization can introduce auxiliary gradients that conflict with supervised learning when pseudo labels are wrong. We therefore shift the focus from sample selection to gradient-level control. We propose \textit{Geometric Gradient Rectification} (GGR), a plug-in framework that uses the supervised gradient as an anchor and projects conflicting auxiliary gradients onto an admissible region in gradient space. This makes the applied auxiliary update first-order non-opposing within the rectified coordinate block while preserving orthogonal components that may still carry useful representation signals. We further extend GGR with subspace-aware rectification to stabilize the anchor under noisy mini-batch gradients. Experiments on CIFAR and ImageNet benchmarks show that GGR improves representative OSSL baselines in most settings and yields gains in both closed-set generalization and open-set robustness. Code will be available at https://github.com/JiaheChen2002/GGR.

Jiahe Chen, Qian Shao, Qiyuan Chen +4

College of Computer Science and Technology, Zhejiang University · Shanghai Innovation Institute · The Hong Kong University of Science and Technology (Guangzhou) +3

Rectification Semi-Supervised Learning Geometric Machine Learning Gradient

Ask PDF

Apr 30, 2026cs.CV

SECOS: Semantic Capture for Rigorous Classification in Open-World Semi-Supervised Learning

In open-world semi-supervised learning (OWSSL), a model learns from labeled data and unlabeled data containing both known and novel classes. In practical OWSSL applications, models are expected to perform rigorous classification by directly selecting the most semantically relevant label from a candidate set for each sample. Existing OWSSL methods fail to achieve this because novel samples are trained without explicit supervision, and these methods lack mechanisms to extract latent semantic information, resulting in predicted labels that have no semantic correspondence to candidate textual labels. To address this, we introduce SEmantic Capture for Open-world Semi-supervised learning (SECOS), which directly predicts textual labels from the candidate set without post-processing, meeting the requirements of practical OWSSL applications. SECOS leverages external knowledge to extract and align semantic representations across modalities for both known and novel classes, providing explicit supervisory signals for training novel classes. Extensive experiments demonstrate that even when existing OWSSL methods are evaluated under the more lenient post-hoc matching setting, SECOS still surpasses them by up to 5.4% without such assistance, highlighting its superior effectiveness. Code is available at https://github.com/ganchi-huanggua/OSSL-Classification.

Hezhao Liu, Jiacheng Yang, Junlong Gao +4

Key Laboratory of Multimedia Trusted Perception and Efficient Computing, Ministry of Education of China, Xiamen University, Xiamen, China · College of Computer Science and Software Engineering, Shenzhen University, Shenzhen, China · School of Computer Science and Technology, Guangdong University of Technology, Guangzhou, China +1

Semi-Supervised Learning

Ask PDF

May 1, 2026cs.CV

DMDSC: A Dynamic-Margin Deep Simplex Classifier for Open-Set Recognition on Medical Image Datasets

Medical imaging datasets are often characterized by extreme class imbalances, where rare pathologies are significantly underrepresented compared to common conditions. This imbalance poses a dual challenge for Open-Set Recognition (OSR): models must maintain high classification accuracy on known classes while reliably rejecting unknown samples unseen during training in the clinical settings. While recently proposed Deep Simplex Classifier (DSC)~~\cite{cevikalp2024reaching} and UnCertainty-aware Deep Simplex Classifier (UCDSC)~~\cite{Aditya_2026_WACV} successfully leverage Neural Collapse to ensure maximal inter-class separation, they rely on a uniform margin that does not account for the varying densities of medical classes. In this paper, we propose DMDSC an enhanced framework featuring a dynamic margin approach. Our approach automatically adapts class-specific margins based on label frequency, enforcing a higher penalty and tighter feature clustering for rare pathologies to counteract the effects of data imbalance. Extensive experiments conducted on diverse medical benchmarks on BloodMNIST\cite{medmnistv2}, OCTMNIST\cite{medmnistv2}, DermaMNIST\cite{medmnistv2}, and BreaKHis~\cite{spanhol2015dataset} datasets, demonstrate that our framework outperforms state-of-the-art methods.

Vishal, Arnav Aditya, Nitin Kumar +1

Shiv Nadar Institution of Eminence, Delhi NCR, India

Medical Imaging Datasets Imbalanced Classification Open-Set Recognition Rare Disease

Ask PDF