Reframing Long-Tailed Learning via Loss Landscape Geometry

Shenghan Chen; Yiming Liu; Yanzhen Wang; Yujia Wang; Xiankai Lu

cs.CVMar 22, 2026

Reframing Long-Tailed Learning via Loss Landscape Geometry

Authors: Shenghan Chen, Yiming Liu, Yanzhen Wang, Yujia Wang, Xiankai Lu

Organizations: Shandong University · Zhejiang Sci-Tech University

Abstract

Balancing performance trade-off on long-tail (LT) data distributions remains a long-standing challenge. In this paper, we posit that this dilemma stems from a phenomenon called "tail performance degradation" (the model tends to severely overfit on head classes while quickly forgetting tail classes) and pose a solution from a loss landscape perspective. We observe that different classes possess divergent convergence points in the loss landscape. Besides, this divergence is aggravated when the model settles into sharp and non-robust minima, rather than a shared and flat solution that is beneficial for all classes. In light of this, we propose a continual learning inspired framework to prevent "tail performance degradation". To avoid inefficient per-class parameter preservation, a Grouped Knowledge Preservation module is proposed to memorize group-specific convergence parameters, promoting convergence towards a shared solution. Concurrently, our framework integrates a Grouped Sharpness Aware module to seek flatter minima by explicitly addressing the geometry of the loss landscape. Notably, our framework requires neither external training samples nor pre-trained models, facilitating the broad applicability. Extensive experiments on four benchmarks demonstrate significant performance gains over state-of-the-art methods. The code is available at:https://gkp-gsa.github.io/.

Figures & tables

Figure 1: “Tail performance degradation” from the loss landscape view. Starting from a randomly initialized point $\theta(t_{0})$ of LT model, (a) Training only on tail classes converges to $\theta(t_{1})$ in a flat region, while (b) standard training on the long-tailed dataset converges to $\theta(t_{2})$ in a sharp region. The optimization trajectory settles in $\theta(t_{2})$ , which causes tail performance degradation by diverging from the tail convergence point $\theta(t_{1})$ . In contrast, our optimization ( red line) steers the model towards a solution $\theta^{*}$ that remains closer to the tail-class minimum $\theta(t_{1})$ and resides in a flatter region ( $\theta_{1}$ and $\theta_{2}$ on the axes denote projection directions for 2D visualization [ 23 ] ).

Figure 2: (a) Standard training results in a sharp loss landscape for tail classes, where the corresponding feature quality peaks and then declines. (b) In contrast, our method flattens the landscape and preserves high feature quality for both head and tail classes.

Figure 3: Our framework consists of two key components: (a) The Grouped Sharpness Aware (GSA) module, which minimizes group-specific sharpness to find flat minima. (b) The Grouped Knowledge Preservation (GKP) module, which prevents tail performance degradation of other groups’ optimal parameters.

Figure 4: Investigation on the perturbation direction. We decompose the original gradient $\nabla_{\theta}\mathcal{L}_{\mathcal{D}_{g}}(\theta)$ ( green ) into two components: the head-dominated global gradient $\nabla_{\theta}\mathcal{L}_{D}(\theta)$ ( red ) and the beneficial, group-specific gradient $\hat{\nabla}_{\theta}\mathcal{L}_{\mathcal{D}_{g}}(\theta)$ ( blue ).


	CIFAR100-LT			CIFAR10-LT			Statistic( $r$ =100)
Method	r=100 $\uparrow$	r=50 $\uparrow$	r=10 $\uparrow$	r=100 $\uparrow$	r=50 $\uparrow$	r=10 $\uparrow$	Many $\uparrow$	Med. $\uparrow$	Few $\uparrow$
CE ( Baseline )	38.3	43.9	55.7	70.4	74.8	86.4	65.2	37.1	9.1
Focal Loss [ 32 ] ( ICCV’17 )	38.4	44.3	55.8	70.4	76.7	86.7	65.3	38.4	8.1
LDAM-DRW [ 4 ] ( NeurIPS’19 )	42.0	46.6	58.7	77.0	81.0	88.2	61.5	41.7	20.2
cRT [ 19 ] ( ICLR’20 )	42.3	46.8	58.1	75.7	80.4	88.3	64.0	44.8	18.1

Table 1: Results on CIFAR100-LT [ 4 ] and CIFAR10-LT datasets [ 4 ] . The imbalance ratio

r

is set to 100, 50 and 10. Additionally, we present the results for different groups (“Many”, “Med.” and “Few”) in CIFAR100-LT with

r

= 100. † denotes methods that utilize Large Language Models.


Method	ImageNet-LT	iNature2018
CE( Baseline )	41.6	61.0
Focal Loss [ 32 ] ( ICCV’17 )	-	61.1
cRT [ 19 ] ( ICLR’20 )	47.3	65.2
RIDE (3 experts) [ 56 ] ( ICLR’21 )	54.9	72.7
BCL [ 75 ] ( CVPR’22 )	56.0	71.8

Table 2: Top-1 accuracy of ResNet-50 on ImageNet-LT [ 37 ] and iNaturalist 2018 [ 53 ] . † denotes methods that utilize Large Language Models.


Method	Many $\uparrow$	Med. $\uparrow$	Few $\uparrow$	All $\uparrow$
CE ( Baseline )	-	-	-	44.4
FocalLoss [ 32 ] ( ICCV’17 )	64.3	37.1	8.2	43.7
$\tau$ -norm [ 19 ] ( ICLR’20 )	59.1	46.9	30.7	49.4
BalancedSoftmax [ 44 ]	62.2	48.8	29.8	51.4
( NeurIPS’20 )

Table 3: Top-1 accuracy of ResNeXt-50 on ImageNet-LT [ 37 ] .


Method	Many $\uparrow$	Med. $\uparrow$	Few $\uparrow$	All $\uparrow$
CE	65.2	37.1	9.1	38.4
+GKP	71.1	41.2	9.3	42.1
+GKP + GSA	71.8	42.1	9.7	42.8
BCL [ 75 ]	67.2	53.1	32.9	51.9
+GKP	67.4	53.8	33.2	52.4

Table 4: Ablation study on main components of our method.

Figure 5: Ablation Study on group numbers of our method. The shaded area indicates the fluctuation of accuracy.


Method	Many $\uparrow$	Med. $\uparrow$	Few $\uparrow$	All $\uparrow$
SAM	66.3	53.0	34.5	52.1
GSA-proj	64.7	43.8	28.1	46.4
GSA (Ours)	67.3 $\uparrow$ 1.0	54.9 $\uparrow$ 1.9	34.9 $\uparrow$ 0.4	53.2 $\uparrow$ 1.1

Table 5: Importance of Gradient Decomposition in GSA.

Figure 6: Gradient imbalance in long-tail learning. The bars denote the mean similarity between class-level and batch-level gradients in each batch. This experiment compares our framework against the Cross-Entropy (CE) baseline on the CIFAR10-LT dataset [ 4 ] . We show the result of the top two head classes and the last two tail classes.

Figure 7: Ablation study about group numbers on ImageNet-LT and iNaturalist 2018.

Hyperparameter ( $\lambda$ )	Overall Acc.
10	52.6
50	52.7
100	53.2
500	52.4
1000	52.1

Table 6: Hyperparameter ablation analysis for the knowledge preservation strength

\lambda

on CIFAR100-LT (

r=100

Figure 8: The impact of the perturbation scale.

Panel of Figure 9 — Figure 9: Eigenvalue density distributions for head and tail layers, arranged by CE (the baseline method with the naive Cross Entropy loss), SAM [ 10 ] , and our method.

CE	BCL	Ours
0.52h	1.05h	1.67h

Table 7: Analysis about training time cost.

Long-Tail Learning

Long-tail learning develops models that perform reliably on rare classes and events despite data distributions dominated by frequent examples.

See all 48 papers

From Perturbation Correction to Geometry-Aware Sampling: Sharpness-Guided Equilibrium Sampling for Balanced Flat Minima in Long-Tailed Learning
Jul 24, 2026Jiaxin Deng, Junbiao PangSharpness-Aware Minimization Long-Tail Learning
School of Information Science and Technology, Beijing University of Technology, Beijing, China
Beyond Static Costs: Learning-Dynamics Aware Loss Functions for Long-Tailed Classification
Jul 28, 2026Varad Shinde, Nikhil Kumar Shrey, Magesh Rajasekaran +5Class-Imbalanced Learning Long-Tailed Classification
Dynamic Distillation and Gradient Consistency for Robust Long-Tailed Incremental Learning
May 5, 2026Taigo Sakai, Kazuhiro HottaClass-Incremental Learning Long-Tail Learning
Meijo University, 1-501 Shiogamaguchi, Tempaku-ku, Nagoya 468-8502, Japan

Jul 24, 2026J/K move · Enter open · S save

From Perturbation Correction to Geometry-Aware Sampling: Sharpness-Guided Equilibrium Sampling for Balanced Flat Minima in Long-Tailed Learning

Jiaxin Deng, Junbiao Pang

School of Information Science and Technology, Beijing University of Technology, Beijing, China

Sharpness-Aware Minimization Long-Tail Learning Adaptive Sampling Flat Minima Loss Landscape Sharpness

Open paper AskPDF arXiv

Long-tailed learning couples two sources of poor generalization: head classes dominate training exposure, while under-represented classes often converge to sharper regions of the loss landscape. Conventional re-sampling addresses the former without considering geometry, whereas existing long-tailed sharpness-aware minimization (SAM) methods modify losses or perturbations only after biased mini-batches have been drawn. We introduce Sharpness-Guided Equilibrium Sampling (SGS), which treats the sampling distribution as an active control variable for optimization geometry. SGS dynamically adjusts subsequent mini-batches by increasing the sampling probability of less frequently sampled classes while suppressing classes with large SAM-induced loss changes, using only cumulative class counts and EMA sharpness estimates obtained from the standard SAM update, without class-wise perturbations or additional backward passes. We characterize this sampling process through a continuous-time stochastic differential equation and a sampling-dependent PAC-Bayes analysis, explaining how frequency-sharpness feedback can move training toward a more balanced flatness profile. On CIFAR-100 LT with an imbalance ratio of 100, SGS-SAM improves Focal-SAM by 10.85 points in tail accuracy and 3.56 points overall. On ImageNet-LT, it improves ImbSAM by 6.59 points on tail classes and 1.20 points overall. Its training time is only

1.02\times

that of vanilla SAM. Beyond these gains, SGS establishes a sampling-side route to loss-landscape control, suggesting that future long-tailed methods can jointly regulate data exposure and optimization geometry rather than treating either as fixed.

Abstract

Long-Tail Learning

Explore similar work

From Perturbation Correction to Geometry-Aware Sampling: Sharpness-Guided Equilibrium Sampling for Balanced Flat Minima in Long-Tailed Learning

Beyond Static Costs: Learning-Dynamics Aware Loss Functions for Long-Tailed Classification

Dynamic Distillation and Gradient Consistency for Robust Long-Tailed Incremental Learning