Mastering the Minority: An Uncertainty-guided Multi-Expert Framework for Challenging-tailed Sequence Learning
Authors: Ye Wang, Zixuan Wu, Lifeng Shen, Jiang Xie, Xiaoling Wang, Hong Yu, Guoyin Wang
Organizations: Key Laboratory of Cyberspace Big Data Intelligent Security, Ministry of Education; School of Artificial Intelligence, Chongqing University of Post and Telecommunications, Chongqing 400065, China. · School of Computer Science and Technology, East China Normal University, Shanghai, 200062, China. · Chongqing Key Laboratory of Brain-Inspired Cognitive Computing and Educational Rehabilitation for Children with Special Needs, Chongqing Normal University, Chongqing, 401331, China.
Imbalanced data distribution remains a critical challenge in sequential learning, leading models to easily recognize frequent categories while failing to detect minority classes adequately. The Mixture-of-Experts model offers a scalable solution, yet its application is often hindered by parameter inefficiency, poor expert specialization, and difficulty in resolving prediction conflicts. To Master the Minority classes effectively, we propose the Uncertainty-based Multi-Expert fusion network (UME) framework. UME is designed with three core innovations: First, we employ Ensemble LoRA for parameter-efficient modeling, significantly reducing the trainable parameter count. Second, we introduce Sequential Specialization guided by Dempster-Shafer Theory (DST), which ensures effective specialization on the challenging-tailed classes. Finally, an Uncertainty-Guided Fusion mechanism uses DST's certainty measures to dynamically weigh expert opinions, resolving conflicts by prioritizing the most confident expert for reliable final predictions. Extensive experiments across four public hierarchical text classification datasets demonstrate that UME achieves state-of-the-art performance. We achieve a performance gain of up to 17.97% over the best baseline on individual categories, while reducing trainable parameters by up to 10.32%. The findings highlight that uncertainty-guided expert coordination is a principled strategy for addressing challenging-tailed sequence learning. Our code is available at https://github.com/CQUPTWZX/Multi-experts.
Figures & tables
Fig. 1 : As the classification hierarchy becomes deeper, the number of samples per label decreases significantly, leading to a pronounced challenging-tailed problem and a sharp drop in the classification performance of minority classes.
Fig. 2 : The structure of our proposed model UME. The uncertainty of every single expert is attained for multi-expert learning. Multi-expert Joint Uncertainty is dynamically ensembled by the degree of uncertainty among experts.
Fig. 3 : Class distribution analysis across four benchmark datasets. The histograms use a logarithmic scale to visualize the severe long-tailed nature and extreme class imbalance ratios (IR) present in WOS, RCV1-V2, AAPD, and BGC.
Dataset
∣Y∣
Avg(yi)
Depth
# Train
# Dev
# Test
WOS
141
2.0
2
30,070
7,518
9,397
RCV1-V2
103
3.24
4
18,520
4,629
781,265
AAPD
54
2.41
2
43,872
10,968
1,000
BGC
146
3.01
4
58,800
14,700
18,394
TABLE I : The statistical details of datasets. ∣Y∣ represents the number of unique labels in each dataset. Avg(yi) the average value of the labels for each entry across the dataset. Depth indicates the depth of the hierarchical structure in the dataset. The last three columns of the table include the number of instances in the training, validation, and testing sets, respectively.
Model
WOS
RCV1-V2
AAPD
BGC
NYT
Micro-F1
Macro-F1
Micro-F1
Macro-F1
Micro-F1
Macro-F1
Micro-F1
Macro-F1
Micro-F1
Macro-F1
Hierarchy-Aware Models
HiAGM (ACL2020)
85.11 ± 1.4
80.02 ± 0.7
83.17 ± 1.8
62.35 ± 1.9
75.54 ± 2.1
58.86 ± 1.2
76.61 ± 0.9
57.27 ± 0.7
—
—
HTCInfoMax (NAACL2021)
84.86 ± 0.6
79.12 ± 0.2
83.26 ± 0.8
60.66 ± 1.5
77.42 ± 0.7
57.72 ± 0.7
74.26 ± 0.4
56.81 ± 1.0
—
—
HiMatch (ACL2021)
85.04 ± 0.9
80.37 ± 1.3
84.42 ± 0.9
63.99 ± 1.1
76.70 ± 0.3
59.86 ± 0.5
76.63 ± 0.8
57.87 ± 1.0
—
—
Pretrained Language Models
TABLE II : The comparison of different models on WOS, RCV1-V2, AAPD, BGC and NYT.
Parameters
Description
r
LoRA Rank. Default: 64
lr
Learning rate
batch
Batch size
epoch
Default: 10
early-stop
Epoch before early stop
update
Gradient accumulate steps
TABLE III : Parameter description and details
Category
Count
Ours
HiTIN
HJCL
HGCLR
HPT
g157
1991
55.6 5%
55.25%
51.28%
49.42%
50.77%
c16
1871
58.04 %
54.62%
57.96%
53.82%
55.41%
gwelf
1818
48.01%
51.65 %
49.87%
45.21%
45.21%
e311
1658
79.79 %
61.46%
58.74%
61.82%
59.18%
c331
1179
77.26%
67.77%
79.55 %
54.20%
74.36%
e143
1172
77.39 %
65.61%
66.20%
65.87%
65.31%
TABLE IV : The performance of different models on challenging-tailed categories in the RCV1-V2 dataset. We report results regarding Win-Lose numbers.
Datasets
Top N Tailed Labels
HGCLR
HPT
HiTIN
HJCL
HILL
Ours
WOS
N=8
28.47
27.78
37.83
41.67
42.08
46.60
N=16
48.67
49.04
53.16
57.67
51.78
62.66
N=32
61.49
65.36
61.43
71.62
68.44
74.70
RCV1-V2
N=8
18.92
16.88
20.24
22.40
18.38
26.93
N=16
41.83
41.37
47.21
43.22
46.72
50.45
N=32
56.67
54.77
57.98
57.20
61.46
63.69
TABLE V : The Macro-F1 performance on tailed labels for the Top N least frequent labels across five datasets.
Fig. 4 : The confusion matrix highlights the contrast between HGCLR and multi-experts in terms of prediction accuracy.
Fig. 5 : Two examples illustrate the effectiveness of uncertainty-based multi-expert fusion. Compared with baselines, our classification results are more reliable.
Fig. 6 : The classification error rate distribution across different Conflict Metric ( Cm ) intervals. N denotes the number of samples in each bin.
Fig. 7 : Bar charts show the different performances with varying numbers of experts among four datasets.
Experts ( M )
Tail Micro-F1
Avg. Conflict ( Clast )
Last Expert Util.
3
76.21
0.1724
20.26
4
75.32
0.1597
1.80
5
74.05
0.1175
1.28
TABLE VI : Analysis of expert capacity and utilization on the WOS dataset with expert numbers ( M ). The metrics include Tail Micro-F1 (computed on the bottom 20% least frequent classes), the Average Conflict of the final expert ( Clast ), and the proportion of samples routed to the final expert (defined as weight >0.5 ).
Fig. 8 : The participation percentage among experts in the head, medium, and tail categories on RCV1-V2.
Fig. 9 : Sensitivity analysis on WOS and RCV1-V2.
Ablation Study
WOS
RCV1-V2
AAPD
BGC
w/o Levidence
87.52
86.61
78.86
79.66
w/o ( Levidence&Lml )
86.98
86.37
78.15
79.27
w/o ( Levidence&Lml&LC^&Lcl )
85.91
85.72
77.28
78.51
\mbox−r.p. Direct Integration
87.18
85.73
78.53
79.51
\mbox−r.p. Gated MoE
86.74
86.12
77.91
79.08
UME
87.71
86.88
79.21
80.46
TABLE VII : Ablation study results for Micro-F1.
Fig. 10 : Compare model parameters, training time and Macro-F1 scores on the WOS dataset.
Model
WOS
RCV1-V2
Micro-F1
Macro-F1
Micro-F1
Macro-F1
GPT-4
59.74
37.80
56.12
36.54
LlaMa-3
61.91
27.33
54.67
28.50
UME (Ours)
87.71
82.29
86.88
69.70
TABLE VIII : Performance comparison between LLMs and our UME framework on WOS and RCV1-V2 datasets. The LLMs are evaluated using the zero-shot prompt templates shown.
Prompt Template
You are an expert hierarchical text classifier. Your task is to classify the given text into the correct category path from the predefined taxonomy. [Taxonomy Definition] The categories are defined in a hierarchy formatted as: Level1>Level2>Level3 - The symbol “ > ” represents the parent-child relationship. - You must output the full path from the root to the specific leaf category. [Allowed Category Paths] (Insert your full list of category paths here, e.g., Technology > AI > NLP) [Constraints] 1. Output ONLY the category path. No explanation, no preamble. 2. If the text fits multiple paths, output the most specific and relevant one. 3. Use the exact string format provided in the [Allowed Category Paths]. [Input Text] {{text_content}} [Prediction]
TABLE IX : Prompt template for hierarchical text classification.
Long-tailed classification poses a reliability challenge because models trained on imbalanced data are unevenly reliable across frequent and underrepresented classes. While existing methods address imbalance through re-balancing, adjustment, representation learning, or multi-expert modeling, they rarely estimate which expert should be trusted for each class. This paper proposes CLEAR (Class-wise reLiability-aware Expert Aggregation for long-tailed Recognition), a modular ensemble framework for long-tailed classification. CLEAR generates diverse experts through threshold-based structured sampling while preserving the full label space, then estimates a class-wise trust score for each expert using a smoothed class-wise precision formulation. During inference, expert predictions are combined through class-wise generalized product-of-experts aggregation, allowing different experts to be emphasized for different classes. Experiments on CIFAR-100-LT, ImageNet-LT, and Places-LT across multiple backbones show that CLEAR achieves competitive overall accuracy and particularly strong few-shot performance. These results support class-wise expert reliability as a useful design principle for long-tailed ensemble learning.
Gawon Lim
School of Information Sciences University of Illinois Urbana-Champaign
Multi-expert models have become the dominant paradigm for long-tailed learning, largely attributed to their presumed ability to benefit from expert diversity. However, we revisit this central assumption and reveal that diversity induced by logit adjustment or explicit regularizers does not guarantee better ensemble accuracy. Our work suggests that multi-expert models benefit more from variance reduction than diversity maximization. We introduce VICAL, a VIcinal Consistency ALignment framework that improves long-tailed recognition not by enforcing expert diversity, but by reducing prediction variance. Specifically, our approach comprises two key components: Self-Consistency Learning and Deep Ensemble Distillation. Self-Consistency Learning discourages reliance on unstable high-frequency information, smoothing the local loss landscape and mitigating overfitting, especially for tail classes. Deep Ensemble Distillation promotes cross-expert low-frequency semantic agreement using a low-resolution view, thereby sidestepping optimization conflicts with established knowledge. Extensive experiments on CIFAR-LT, ImageNet-LT, and iNaturalist 2018 show that VICAL consistently outperforms state-of-the-art methods, validating the effectiveness of our consistency-driven design. Our code is available at VICAL.
Jiangang Zhu, Zheng Wang, Bin Zhu +2
Institute of Trustworthy Embodied AI, Fudan University, Shanghai, China · Shanghai Key Laboratory of Multimodal Embodied AI, Shanghai, China · Zhejiang University of Technology, Hangzhou, China +2
Various algorithms have been proposed to address the challenges posed by class-imbalanced learning from real-world data with long-tailed distributions. While these algorithms reduce prediction bias through rebalancing techniques, they often introduce increased prediction variance as a trade-off. Several multi-expert learning algorithms aim to address this variance but involve complex procedures. We propose a new multi-expert learning algorithm, called the dual-axis multi-expert learning (DAMEL), which reduces both bias and variance of predictions by using multiple experts along both representation and time axes. Along the representation axis, DAMEL concatenates the representations of multiple experts and trains an auxiliary balanced classifier simultaneously with the concatenated representations. Along the time axis, DAMEL aggregates network weights across training epochs, employing these aggregated weights during testing. Experimental results demonstrate that DAMEL reduces both bias and variance of predictions, highlighting its effectiveness in class-imbalanced learning.
Hyuck Lee, Taemin Park, Heeyoung Kim
AI Research, Krafton, Teheran-ro 231, Centerfield EAST, Gangnam-gu, Seoul, 06142, Republic of Korea · Department of Industrial and Systems Engineering, Korea Advanced Institute of Science and Technology (KAIST), 291 Daehak-ro, Yuseong-gu, Daejeon, 34141, Republic of Korea