Label-space expansion - predicting a larger joint space and summing back to the original labels - is a natural design for multi-label facial expression recognition, but its failure modes have not been characterised. We evaluate Condition-Aware Label Expansion (CALE) on a self-built dataset (30,240 training images, 3,460 multi-label test images), on seven backbones, and on the public RAF-DB and RAF-ML benchmarks. The central finding is that the aggregation rule, not the representation, decides whether the method works. With a fixed threshold of 0.5, the aggregate score spans 28x across five backbones at 10% of the training data (0.24 to 6.79) and collapses to a 1.9x range at full data, while threshold-free mAP stays within 1.28x over the same backbones, which localises the fault to the summation step. The mechanism is the uncalibrated aggregation rule, which adds the main-branch score to half the sum of the three non-main branches; we verify it by measuring main- and non-main-branch activations. Two plain seven-dimensional sigmoid baselines (BCE, ASL) reach mAP 0.5055 and 0.5175; ASL exceeds three of the five CALE configurations and falls within 0.03 of the best. On single-label FER the gain over a seven-dimensional baseline is nominally significant (paired t-test, p = 0.0436) but does not survive Benjamini-Hochberg correction (p_BH = 0.2055) or a Wilcoxon test (p = 0.1250). We conclude that sum-based label expansion requires threshold calibration at the deployment data scale, or a learned aggregation, before it can support multi-label decisions.
Figures & tables
Emotion
Number of Samples
Angry
4,320
Disgust
4,320
Fear
4,320
Happy
4,320
Sad
4,320
Surprise
4,320
TABLE I: Training Set Sample Distribution
Condition
Training Set
Test Set
Weak light
7,520
949
Strong light
9,565
1,003
Low resolution
5,897
678
Standard
7,258
830
Total
30,240
3,460
TABLE II: Condition Distribution of Training and Test Sets
Fig. 1: Overall framework of the CALE method. After MTCNN alignment, the image is processed by a shared backbone with shallow layers frozen. A global average pooling layer and an SE module recalibrate channel features, followed by flatten and dropout. The 28-dimensional output is organized as four condition-specific 7-dimensional branches. Each branch is sigmoid-activated, and the four branches are aggregated per emotion to produce seven scores. A fixed threshold yields the final multi-label prediction. This figure was drawn by the author with TikZ.
Condition
Rule
Weak light
Lap. variance ≥10 and brightness mean <80
Strong light
Lap. variance ≥10 and brightness mean >120
Low resolution
Lap. variance <10
Standard
otherwise
TABLE III: Condition Classification Thresholds
Fig. 2: Visualization of the 512-dimensional SE scale factors for a test image. Higher values indicate channels more strongly enhanced by the SE module.
Variant
LE
MS
SE
Val Accuracy
Val UAR
Val Macro-F1
V0: Baseline
–
–
–
0.7635±0.0107
0.7634±0.0106
0.7618±0.0118
V1: +MS
–
✓
–
0.7716±0.0102
0.7715±0.0098
0.7705±0.0098
V2: +LE
✓
–
–
0.7745±0.0078
0.7745±0.0077
0.7729±0.0082
V3: +SE
–
–
✓
0.7726±0.0086
0.7725±0.0084
0.7706±0.0085
V4: +MS+SE
–
✓
✓
0.7723±0.0093
0.7722±0.0090
0.7709±0.0088
V5: +LE+MS
✓
✓
–
0.7773±0.0131
0.7772±0.0129
0.7758±0.0128
TABLE IV: Main results on the self-built dataset (5-fold single-label cross-validation, mean ± std). LE: label expansion; MS: multi-strategy training; SE: squeeze-and-excitation.
Variant
LE
MS
SE
Macro-F1
V0
–
–
–
0.5607
V1
–
✓
–
0.5923
V2
✓
–
–
0.5761
V3
–
–
✓
0.5643
V4
–
✓
✓
0.5928
V5
✓
✓
–
0.5939
TABLE V: Multi-label test set results at AvgLabels≈2.5 .
Comparison
Δ mean
t
pt
pW
V2 (LE) vs V0
+0.0078
2.917
0.0436
0.1250
V3 (SE) vs V0
+0.0058
1.947
0.1233
0.1875
V4 (MS+SE) vs V0
+0.0055
2.046
0.1102
0.1875
V5 (LE+MS) vs V0
+0.0105
2.614
0.0589
—
V6 (LE+SE) vs V0
+0.0096
3.270
0.0308
—
V7 (Full) vs V0
+0.0096
1.685
0.1672
0.1875
TABLE VI: Paired tests across the 5 folds on Validation UAR. V0 is a reproduction with Val UAR =0.7667 . Columns “ t / pt ” are a paired t -test; column “ pW ” is a Wilcoxon signed-rank test. After Benjamini–Hochberg FDR correction at q=0.05 over the 5 computable comparisons, none of the paired t -test comparisons are significant ( pBH≥0.2055 ); the same holds for the Wilcoxon comparisons ( pBH≥0.2344 ). We therefore do not claim a statistically established single-label advantage.
Backbone
Params (M)
Single-label Macro-F1
Multi-label Macro-F1
Baseline
+CALE
Baseline
+CALE
ResNet-18
11.2
0.7830
0.7951
0.5596
0.5959
ResNet-34
21.8
0.7829
0.8016
0.5637
0.5965
ResNet-50
25.6
0.7834
0.8011
0.5566
0.5956
MobileNetV2
3.5
0.7726
0.7859
0.5321
0.5948
DenseNet121
8.0
0.7756
0.7977
0.5379
0.5985
TABLE VII: Cross-backbone evaluation. Single-label results use the 1/8 validation split; multi-label results use the independent test set at AvgLabels≈2.5 . The multi-label column uses a threshold scan that matches the target label density on the evaluation set; these numbers should be interpreted as an upper bound rather than as achievable performance (see Section V-A ).
Setting
Training Data
Test Macro-F1
Baseline (raw)
12K
0.6769
V1 diagnostic (raw)
12K
0.7555
CALE (raw)
12K
0.5226
Baseline (multi-condition)
49K
0.7603
CALE (multi-condition)
49K
0.7475
TABLE VIII: RAF-DB evaluation ( 1/8 split used for baseline; augmented training for multi-condition settings). The row “V1 diagnostic (raw)” corresponds to a 7-dimensional baseline trained with the multi-strategy training (MS) configuration on raw RAF-DB.
Setting
AvgLabels
Macro-F1
Prior BCE baseline [ 4 ]
–
0.5954
Prior ASL [ 21 ]
–
0.6481
Prior ASL with soft labels [ 21 ]
–
0.6500
Baseline (raw)
2.06
0.6835
CALE (raw, Focal)
5.61
0.5526
Baseline (multi-cond.)
2.08
0.7359
TABLE IX: RAF-ML multi-label evaluation under the fixed decision threshold 0.5 . All rows use the fixed decision threshold 0.5 ; threshold-calibrated variants are not reported because they require evaluation-set label density.
Fig. 3: Threshold sensitivity on the multi-label test set. (a) Macro-F1 vs threshold. (b) UAR vs threshold. The CALE curve spans a wider threshold range because the aggregation score is not normalized, not because the model predicts more labels correctly.
Condition
Baseline
CALE
Standard
0.7555
0.7437
Weak light
0.7514
0.7482
Strong light
0.7078
0.6925
Low resolution
0.7142
0.7190
TABLE X: Per-condition UAR on augmented RAF-DB test set.
Emotion
Baseline
CALE
Δ
Angry
0.8070
0.7905
−0.0165
Disgust
0.7044
0.7295
+0.0251
Fear
0.8247
0.8010
−0.0237
Happy
0.8458
0.8977
+0.0519
Sad
0.7817
0.7925
+0.0108
Surprise
0.7062
0.7233
+0.0171
TABLE XI: Per-emotion F1 on the RAF-DB test set ( 1/8 split). Δ is CALE minus baseline.
Backbone
Setting
Params (M)
Val F1
Test F1
AvgLabels
ResNet-18
CALE, SOFT=0.5
11.2
0.4069
0.5200
5.86
ResNet-18
CALE, SOFT=0.0
11.2
0.3533
0.5139
5.98
ResNet-34
CALE, SOFT=0.5
21.8
0.4272
0.5238
3.08
ResNet-34
CALE, SOFT=0.0
21.8
0.3645
0.3686
1.72
ResNet-50
CALE, SOFT=0.5
25.6
0.4582
0.3359
0.96
ResNet-50
CALE, SOFT=0.0
25.6
0.3975
0.1051
0.24
TABLE XII: Diagnostic experiment under the 10% subject-level training subset with a single fold. Val Macro-F1 uses the single-label argmax protocol on the in-distribution validation split; Test Macro-F1 and AvgLabels use the independent multi-label test set. Val/Test are reported for the best epoch. The AvgLabels column is measured at threshold 0.5 . A hyperparameter sensitivity study on EfficientNet-B0 (the configuration with the largest AvgLabels) shows that its AvgLabels is itself hyperparameter-dependent: it ranges from 1.70 (lr=1e-5, bs=64) to 7.00 (lr=5e-6, bs=128) across four configurations, with the original configuration (lr=1e-5, bs=128) yielding 6.79 . This suggests that the extreme value in the table is not a stable architectural property, and that the range reported here should be interpreted as an upper bound.
Backbone
Setting
Params (M)
Val F1
Test F1
AvgLabels
ResNet-18
CALE, SOFT=0.5
11.2
0.7606
0.3153
0.66
ResNet-18
CALE, SOFT=0.0
11.2
0.7393
0.2439
0.47
ResNet-34
CALE, SOFT=0.5
21.8
0.7576
0.3845
0.87
ResNet-34
CALE, SOFT=0.0
21.8
0.7569
0.2772
0.53
ResNet-50
CALE, SOFT=0.5
25.6
0.7496
0.3302
0.77
ResNet-50
CALE, SOFT=0.0
25.6
0.7496
0.2636
0.54
TABLE XIII: Diagnostic experiment under the full training set ( 30,240 images) with a single fold. Same metrics and protocol as Table XII . Under full data, all five backbones collapse to the under-prediction regime ( AvgLabels<1 ). The AvgLabels range narrows from 28× (Table XII ) to 1.9× (this table).
Backbone
10%
25%
50%
100%
ResNet-18
5.86
1.14
0.87
0.66
ResNet-34
3.08
0.74
0.95
0.87
TABLE XIV: AvgLabels at threshold 0.5 as a function of the training-data fraction for two backbones. The transition from over-prediction ( AvgLabels>2 ) to under-prediction ( AvgLabels<1 ) occurs between 10% and 25% ; from 25% onward the AvgLabels are stable.
TABLE XV: Threshold-free mean average precision (mAP) and AvgLabels at threshold 0.5 for the five backbones under the 10% data regime. All values use the unified protocol. mAP is stable across backbones ( 1.28× range) while AvgLabels varies by 28× , localizing the failure to the aggregation step.
Emotion
ResNet-18
ResNet-34
MobileNetV2
EffB0
Angry
0.347
0.322
0.363
0.330
Disgust
0.497
0.618
0.450
0.459
Fear
0.303
0.413
0.252
0.292
Happy
0.745
0.705
0.741
0.514
Sad
0.504
0.545
0.482
0.377
Surprise
0.478
0.539
0.640
0.374
TABLE XVI: Per-class average precision (AP) for four backbones under the 10% data regime, ResNet-18 + CALE SOFT=0.5. Rows are sorted by the ResNet-18 AP. The per-class ordering is not consistent across backbones (e.g., disgust ranks above surprise on ResNet-18 and ResNet-34, but below surprise on MobileNetV2 and EfficientNet-B0), which further supports the calibration hypothesis over a class-discriminability explanation.
TABLE XVII: Fair multi-label comparison under the 10% data regime. All entries are evaluated under the unified decision protocol (sigmoid output, threshold 0.5 ). [email protected] is coupled to the number of predicted labels and is not comparable across configurations; mAP is threshold-free and is the more meaningful comparison. All entries use the same ResNet-18 backbone and the same 10% subject-level training subset.
Scale
Weak
Strong
Low-res
Standard
Training set
0.8×
18.4
48.6
16.6
16.4
0.9×
21.7
39.4
18.2
20.8
1.0×
24.9
31.6
19.5
24.0
1.1×
28.0
25.6
20.9
25.5
1.2×
32.1
19.9
22.1
26.0
TABLE XVIII: Condition classifier sensitivity to threshold perturbation. The default thresholds ( 80 , 120 , 10 ) are scaled by the factor in the first column. Numbers are the percentage of images assigned to each condition.