Aggregation-Induced Calibration Failure in Sum-Based Label Expansion for Multi-Label Facial Expression Recognition: A Diagnostic Study
Organizations: Beijing Forestry University, Beijing, China
Abstract
Label-space expansion - predicting a larger joint space and summing back to the original labels - is a natural design for multi-label facial expression recognition, but its failure modes have not been characterised. We evaluate Condition-Aware Label Expansion (CALE) on a self-built dataset (30,240 training images, 3,460 multi-label test images), on seven backbones, and on the public RAF-DB and RAF-ML benchmarks. The central finding is that the aggregation rule, not the representation, decides whether the method works. With a fixed threshold of 0.5, the aggregate score spans 28x across five backbones at 10% of the training data (0.24 to 6.79) and collapses to a 1.9x range at full data, while threshold-free mAP stays within 1.28x over the same backbones, which localises the fault to the summation step. The mechanism is the uncalibrated aggregation rule, which adds the main-branch score to half the sum of the three non-main branches; we verify it by measuring main- and non-main-branch activations. Two plain seven-dimensional sigmoid baselines (BCE, ASL) reach mAP 0.5055 and 0.5175; ASL exceeds three of the five CALE configurations and falls within 0.03 of the best. On single-label FER the gain over a seven-dimensional baseline is nominally significant (paired t-test, p = 0.0436) but does not survive Benjamini-Hochberg correction (p_BH = 0.2055) or a Wilcoxon test (p = 0.1250). We conclude that sum-based label expansion requires threshold calibration at the deployment data scale, or a learned aggregation, before it can support multi-label decisions.
Figures & tables
| Emotion | Number of Samples |
|---|---|
| Angry | 4,320 |
| Disgust | 4,320 |
| Fear | 4,320 |
| Happy | 4,320 |
| Sad | 4,320 |
| Surprise | 4,320 |
| Condition | Training Set | Test Set |
|---|---|---|
| Weak light | 7,520 | 949 |
| Strong light | 9,565 | 1,003 |
| Low resolution | 5,897 | 678 |
| Standard | 7,258 | 830 |
| Total | 30,240 | 3,460 |
| Condition | Rule |
|---|---|
| Weak light | Lap. variance and brightness mean |
| Strong light | Lap. variance and brightness mean |
| Low resolution | Lap. variance |
| Standard | otherwise |
| Variant | LE | MS | SE | Val Accuracy | Val UAR | Val Macro-F1 |
|---|---|---|---|---|---|---|
| V0: Baseline | – | – | – | |||
| V1: +MS | – | ✓ | – | |||
| V2: +LE | ✓ | – | – | |||
| V3: +SE | – | – | ✓ | |||
| V4: +MS+SE | – | ✓ | ✓ | |||
| V5: +LE+MS | ✓ | ✓ | – |
| Variant | LE | MS | SE | Macro-F1 |
|---|---|---|---|---|
| V0 | – | – | – | 0.5607 |
| V1 | – | ✓ | – | 0.5923 |
| V2 | ✓ | – | – | 0.5761 |
| V3 | – | – | ✓ | 0.5643 |
| V4 | – | ✓ | ✓ | 0.5928 |
| V5 | ✓ | ✓ | – | 0.5939 |
| Comparison | mean | |||
|---|---|---|---|---|
| V2 (LE) vs V0 | ||||
| V3 (SE) vs V0 | ||||
| V4 (MS+SE) vs V0 | ||||
| V5 (LE+MS) vs V0 | — | |||
| V6 (LE+SE) vs V0 | — | |||
| V7 (Full) vs V0 |
| Backbone | Params (M) | Single-label Macro-F1 | Multi-label Macro-F1 | ||
|---|---|---|---|---|---|
| Baseline | +CALE | Baseline | +CALE | ||
| ResNet-18 | 11.2 | 0.7830 | 0.7951 | 0.5596 | 0.5959 |
| ResNet-34 | 21.8 | 0.7829 | 0.8016 | 0.5637 | 0.5965 |
| ResNet-50 | 25.6 | 0.7834 | 0.8011 | 0.5566 | 0.5956 |
| MobileNetV2 | 3.5 | 0.7726 | 0.7859 | 0.5321 | 0.5948 |
| DenseNet121 | 8.0 | 0.7756 | 0.7977 | 0.5379 | 0.5985 |
| Setting | Training Data | Test Macro-F1 |
|---|---|---|
| Baseline (raw) | 12K | 0.6769 |
| V1 diagnostic (raw) | 12K | 0.7555 |
| CALE (raw) | 12K | 0.5226 |
| Baseline (multi-condition) | 49K | 0.7603 |
| CALE (multi-condition) | 49K | 0.7475 |
| Setting | AvgLabels | Macro-F1 |
|---|---|---|
| Prior BCE baseline [ 4 ] | – | 0.5954 |
| Prior ASL [ 21 ] | – | 0.6481 |
| Prior ASL with soft labels [ 21 ] | – | 0.6500 |
| Baseline (raw) | 2.06 | 0.6835 |
| CALE (raw, Focal) | 5.61 | 0.5526 |
| Baseline (multi-cond.) | 2.08 | 0.7359 |
| Condition | Baseline | CALE |
|---|---|---|
| Standard | 0.7555 | 0.7437 |
| Weak light | 0.7514 | 0.7482 |
| Strong light | 0.7078 | 0.6925 |
| Low resolution | 0.7142 | 0.7190 |
| Emotion | Baseline | CALE | |
|---|---|---|---|
| Angry | 0.8070 | 0.7905 | |
| Disgust | 0.7044 | 0.7295 | |
| Fear | 0.8247 | 0.8010 | |
| Happy | 0.8458 | 0.8977 | |
| Sad | 0.7817 | 0.7925 | |
| Surprise | 0.7062 | 0.7233 |
| Backbone | Setting | Params (M) | Val F1 | Test F1 | AvgLabels |
|---|---|---|---|---|---|
| ResNet-18 | CALE, SOFT=0.5 | 11.2 | 0.4069 | 0.5200 | 5.86 |
| ResNet-18 | CALE, SOFT=0.0 | 11.2 | 0.3533 | 0.5139 | 5.98 |
| ResNet-34 | CALE, SOFT=0.5 | 21.8 | 0.4272 | 0.5238 | 3.08 |
| ResNet-34 | CALE, SOFT=0.0 | 21.8 | 0.3645 | 0.3686 | 1.72 |
| ResNet-50 | CALE, SOFT=0.5 | 25.6 | 0.4582 | 0.3359 | 0.96 |
| ResNet-50 | CALE, SOFT=0.0 | 25.6 | 0.3975 | 0.1051 | 0.24 |
| Backbone | Setting | Params (M) | Val F1 | Test F1 | AvgLabels |
|---|---|---|---|---|---|
| ResNet-18 | CALE, SOFT=0.5 | 11.2 | 0.7606 | 0.3153 | 0.66 |
| ResNet-18 | CALE, SOFT=0.0 | 11.2 | 0.7393 | 0.2439 | 0.47 |
| ResNet-34 | CALE, SOFT=0.5 | 21.8 | 0.7576 | 0.3845 | 0.87 |
| ResNet-34 | CALE, SOFT=0.0 | 21.8 | 0.7569 | 0.2772 | 0.53 |
| ResNet-50 | CALE, SOFT=0.5 | 25.6 | 0.7496 | 0.3302 | 0.77 |
| ResNet-50 | CALE, SOFT=0.0 | 25.6 | 0.7496 | 0.2636 | 0.54 |
| Backbone | ||||
|---|---|---|---|---|
| ResNet-18 | ||||
| ResNet-34 |
| Backbone | AvgLabels | mAP | Test [email protected] |
|---|---|---|---|
| ResNet-18 | |||
| ResNet-34 | |||
| ResNet-50 | |||
| MobileNetV2 | |||
| EfficientNet-B0 | |||
| Range ( ) |
| Emotion | ResNet-18 | ResNet-34 | MobileNetV2 | EffB0 |
|---|---|---|---|---|
| Angry | 0.347 | 0.322 | 0.363 | 0.330 |
| Disgust | 0.497 | 0.618 | 0.450 | 0.459 |
| Fear | 0.303 | 0.413 | 0.252 | 0.292 |
| Happy | 0.745 | 0.705 | 0.741 | 0.514 |
| Sad | 0.504 | 0.545 | 0.482 | 0.377 |
| Surprise | 0.478 | 0.539 | 0.640 | 0.374 |
| Method | AvgLabels | [email protected] | mAP |
|---|---|---|---|
| 7-dim sigmoid + BCE | |||
| 7-dim sigmoid + ASL | |||
| CALE (ResNet-18) | |||
| CALE (ResNet-34) | |||
| CALE (ResNet-50) | |||
| CALE (MobileNetV2) |
| Scale | Weak | Strong | Low-res | Standard |
|---|---|---|---|---|
| Training set | ||||
| 18.4 | 48.6 | 16.6 | 16.4 | |
| 21.7 | 39.4 | 18.2 | 20.8 | |
| 24.9 | 31.6 | 19.5 | 24.0 | |
| 28.0 | 25.6 | 20.9 | 25.5 | |
| 32.1 | 19.9 | 22.1 | 26.0 | |
Multi-Label Classification
Multi-label classification assigns several potentially related tags to each example, from medical findings and topics to objects and user interests.
See all 142 papers