CAMO: A Class-Aware Minority-Optimized Ensemble for Robust Language Model Evaluation on Imbalanced Data
Authors: Mohamed Ehab, Ali Hamdi, Khaled Shaban
Organizations: Faculty of computer science, October University for Modern Science & Arts, Giza,Egypt. · Department of Computer Science and Engineering, Qatar University, Doha, Qatar.
Real-world categorization is severely hampered by class imbalance because traditional ensembles favor majority classes, which lowers minority performance and overall F1-score. We provide a unique ensemble technique for imbalanced problems called CAMO (Class-Aware Minority-Optimized).Through a hierarchical procedure that incorporates vote distributions, confidence calibration, and inter model uncertainty, CAMO dynamically boosts underrepresented classes while preserving and amplifying minority forecasts. We verify CAMO on two highly unbalanced, domain-specific benchmarks: the DIAR-AI/Emotion dataset and the ternary BEA 2025 dataset. We benchmark against seven proven ensemble algorithms using eight different language models (three LLMs and five SLMs) under zero-shot and fine-tuned settings .With refined models, CAMO consistently earns the greatest strict macro F1-score, setting a new benchmark. Its benefit works in concert with model adaptation, showing that the best ensemble choice depends on model properties .This proves that CAMO is a reliable, domain-neutral framework for unbalanced categorization.
Figures & tables
Figure 1: CAMO ensemble decision process architecture at a high level. In order to optimize for minority class preservation, the hierarchical pipeline moves from unanimity checks to minority-aware boosting, combining vote distribution, confidence calibration, and inter-model uncertainty.
BEA Baseline Ensemble Strategies
Method
Description
Majority Voting
Simple plurality-based aggregation
Confidence-Weighted
Softmax probability weighted voting
Class-Balanced
Inverse class frequency weighting
Dynamic Threshold
Adaptive decision boundaries
Uncertainty-Aware
Entropy-based model weighting
Table 2
Model
Ensemble Method
Maj. Vote
Conf. Weight
Dyn. Thr.
Class Bal.
Unc. Aware
Meta Ens.
MSA Base.
CAMO
Phi-3 (3.8B)
32.7 70.3 46.4 84.0
33.5 69.2 47.3 84.1
12.3 15.5 47.3 84.1
32.8 65.6 46.4 84.0
16.6 23.4 45.6 83.9
32.8 69.0 46.4 84.0
9.1 12.0 46.4 84.0
4.7 7.5 45.6 83.9
DeepSeek-R1 (1.5B)
17.9 18.0 43.0 51.4
15.9 16.3 28.9 29.0
16.5 16.6 30.9 31.3
17.2 17.0 43.0 51.5
15.4 14.7 46.1 61.1
17.5 17.3 42.9 51.5
17.5 17.1 43.9 53.3
13.4 12.0 49.4 64.0
Qwen3 (0.6B)
30.1 67.7 48.2 77.1
30.4 55.2 47.2 65.3
30.4 55.2 47.3 65.4
30.8 64.9 48.2 77.1
33.9 65.0 48.6 77.8
30.9 65.2 48.2 77.1
30.8 64.9 48.3 77.2
4.8 7.7 45.6 83.9
Llama-3.2 (1B)
26.9 42.9 46.1 76.8
23.0 31.3 40.8 58.7
22.3 30.8 40.7 61.3
20.5 27.4 46.1 76.8
19.7 28.1 45.3 80.6
25.4 38.1 46.1 76.8
18.6 26.9 44.2 79.4
8.9 12.3 44.9 81.6
Falcon3 (1B)
12.0 15.8 45.9 82.5
6.6 9.4 45.1 82.2
5.2 8.1 45.3 82.9
7.0 9.4 45.9 82.5
4.7 7.5 45.6 83.9
6.9 9.8 45.2 82.6
5.3 8.2 45.4 83.3
4.7 7.5 45.6 83.9
Table 2: Performance comparison of ensemble methods across zero-shot (non-fine-tuned) language models (percentages) on BEA DATASET . Each cell shows: Strict F1 / Strict Accuracy / Lenient F1 / Lenient Accuracy .
Model
Accuracy
Precision
Recall
F1
Llama 8B (CAMO)
92.75%
87.94%
90.15%
88.70%
Llama 1B (CAMO)
92.45%
87.02%
92.33%
88.84%
GPT-4o-DPO Siddiqui et al. (2025)
93.10%
90.80%
87.09%
87.90%
Classifier Q-¿RA Henrichsen and Krebs (2025)
58.4%
-
-
-
Table 3: OVERALL PERFORMANCE COMPARISON ON DIAR-AI EMOTION DATASET
Category
Llama-8B (CAMO)
Llama-1B (CAMO)
Kermani Kermani et al. (2025) (2025)
F1
Prec
Rec
F1
Prec
Rec
F1
Prec
Rec
sadness
0.96
0.97
0.96
0.97
0.97
0.96
0.95
0.95
0.94
joy
0.95
0.98
0.92
0.95
0.97
0.92
0.94
0.94
0.93
love
0.84
0.74
0.98
0.82
0.72
0.94
0.81
0.80
0.82
anger
0.92
0.95
0.89
0.93
0.93
0.93
0.89
0.88
0.91
fear
0.90
0.87
0.93
0.89
0.96
0.82
0.89
0.89
0.88
Table 4: Per-class performance comparison on Emotion Dataset (Fine-tuned). CAMO shows improved minority class (love, surprise) performance