Beyond the Fold: Quantifying Split-Level Noise and the Case for Leave-One-Dataset-Out AU Evaluation

Saurabh Hinduja; Gurmeet Kaur; Maneesh Bilalpur; Jeffrey Cohn; Shaun Canavan

cs.CVApr 2, 2026

Beyond the Fold: Quantifying Split-Level Noise and the Case for Leave-One-Dataset-Out AU Evaluation

Authors: Saurabh Hinduja, Gurmeet Kaur, Maneesh Bilalpur, Jeffrey Cohn, Shaun Canavan

Organizations: CGI Technologies and Solutions Inc · University of Pittsburgh · University of South Florida

Abstract

Subject-exclusive cross-validation is the standard evaluation protocol for facial Action Unit (AU) detection, yet reported improvements are often small. We show that cross-validation itself introduces measurable stochastic variance. On BP4D+, repeated 3-fold subject-exclusive splits produce an empirical noise floor of $\pm 0.065$ in average F1, with substantially larger variation for low-prevalence AUs. Operating-point metrics such as F1 fluctuate more than threshold-independent measures such as AUC, and model ranking can change under different fold assignments. We further evaluate cross-dataset robustness using a Leave-One-Dataset-Out (LODO) protocol across five AU datasets. LODO removes partition randomness and exposes domain-level instability that is not visible under single-dataset cross-validation. Together, these results suggest that gains often reported in cross-fold validation may fall within protocol variance. Leave-one-dataset-out cross-validation yields more stable and interpretable findings

Figures & tables

AU	Min	Max	Range
1	0.0653	0.1217	0.0563
2	0.0490	0.1016	0.0526
4	0.0276	0.0870	0.0594
6	0.4525	0.5398	0.0873
7	0.6025	0.7212	0.1187
10	0.6154	0.6859	0.0704

Table 1: AU Prevalence Variation Across 3-Fold Subject Splits

Panel of Figure 1 — Figure 1: 3-fold performance range per AU across backbones. Markers denote mean; vertical bars fold-to-fold min–max range.

AU	Mean	$\sigma$	95% Margin ( $\pm 1.96\sigma$ )	Range	Mean	$\sigma$	95% Margin ( $\pm 1.96\sigma$ )	Range
	F1				AUC
1	0.4540	0.0568	$\pm 0.1112$	0.3424–0.5280	0.8311	0.0193	$\pm 0.0379$	0.8006–0.8797
2	0.3662	0.0476	$\pm 0.0933$	0.2623–0.4282	0.7926	0.0243	$\pm 0.0477$	0.7470–0.8252
4	0.3294	0.0590	$\pm 0.1156$	0.2597–0.4278	0.8118	0.0297	$\pm 0.0583$	0.7705–0.8444
6	0.8600	0.0108	$\pm 0.0212$	0.8418–0.8753	0.9355	0.0055	$\pm 0.0107$	0.9227–0.9443
7	0.8862	0.0120	$\pm 0.0236$	0.8565–0.9057	0.9086	0.0054	$\pm 0.0106$	0.8982–0.9162

Table 2: Empirical noise floor per AU (ResNet50, 3-fold subject-exclusive): F1 and AUC.

AU	$\sigma_{F1}$	$\sigma_{AUC}$	Volatility Ratio $\rho$
1	0.0568	0.0193	2.93
2	0.0476	0.0243	1.96
4	0.0590	0.0297	1.99
6	0.0108	0.0055	1.98
7	0.0120	0.0054	2.24
10	0.0102	0.0120	0.85

Table 3: Per-AU volatility comparison between F1 and AUC for ResNet50 under 3-fold subject-exclusive cross-validation. Volatility is measured as the cross-split standard deviation

\sigma

of each metric; the volatility ratio

\rho=\sigma_{F1}/\sigma_{AUC}

indicates how much more (or less) F1 fluctuates than AUC.

Model	Year	BP4D+ (F1)
VisAULa [ 26 ]	2026	0.667
STIF-DA [ 2 ]	2025	0.654
FMAE-IAT [ 27 ]	2024	0.668
MDHR [ 28 ]	2024	0.666
AUFormer [ 30 ]	2024	0.662
MCM [ 15 ]	2024	0.66

Table 4: State-of-the-art AU detection performance on BP4D+. Entries list the reported mean F1 (average over AUs) for representative methods by year under within-dataset subject-exclusive evaluation; the full spread of results falls within the empirical split-level noise band of

\pm 0.065

average F1.

AU	BP4D	BP4D+	DISFA	GFT	UNBC
1	0.211	0.097	0.067	0.075	–
2	0.171	0.082	0.056	0.105	–
4	0.203	0.058	0.188	0.041	0.022
6	0.461	0.498	0.149	0.288	0.115
7	0.549	0.663	–	0.369	0.070
10	0.594	0.648	–	0.321	0.011

Table 5: AU prevalence across datasets. “–” indicates AU not annotated in the corresponding dataset.

Metric	Test Dataset	AU1	AU2	AU4	AU6	AU7	AU10	AU12	AU14	AU15	AU17	AU23	AU24
F1	BP4D	0.537	0.496	0.549	0.797	0.803	0.847	0.863	0.686	0.484	0.546	0.507	0.238
	BP4D+	0.368	0.334	0.343	0.848	0.852	0.885	0.876	0.669	0.374	0.438	0.509	0.328
	DISFA	0.391	0.296	0.514	0.588	–	–	0.760	–	0.370	0.325	–	–
	GFT	0.221	0.186	0.112	0.742	0.556	0.527	0.769	0.084	0.168	0.210	0.179	0.146
	UNBC	–	–	0.095	0.419	0.224	0.036	0.485	–	0.001	–	–	–
AUC	BP4D	0.817	0.775	0.827	0.893	0.839	0.869	0.937	0.717	0.828	0.779	0.803	0.831

Table 6: LODO performance of the enhanced model. Each Test dataset is held out once. Performance is grouped by metric, with the Metric column indicating whether the row reports F1-score or AUC. Results are based on a fixed classification threshold of 0.5.

Figure 2: Overview of the LODO training pipeline.

AU	Mean $\Delta$ F1	Mean $\Delta$ AUC	Mean F1 DS	Mean AUC DS
1	-0.172	-0.065	67%	33%
2	-0.134	+0.003	33%	0%
4	-0.198	-0.028	67%	0%
6	-0.172	-0.028	100%	60%
7	-0.386	-0.027	100%	67%
10	-0.472	-0.071	100%	50%

Table 7: Domain Sensitivity (DS) across LODO transfers. Mean

\Delta

is the signed average shift in metric when moving from pooled-source evaluation to the held-out target dataset (target minus source mean). DS measures the frequency of statistically significant (

p<0.05

) performance changes under subject-level bootstrapping. F1 exhibits significantly higher sensitivity to domain shifts than AUC.

Cross-Validation

See all 16 papers

UAU-Net: Uncertainty-aware Representation Learning and Evidential Classification for Facial Action Unit Detection
Apr 23, 2026Yuze Li, Zhilei LiuRepresentation Learning Evidential Deep Learning
School of Artificial Intelligence Tianjin University Tianjin, China
AUCH-Net: Action Unit-Based Consistency-Aware Hypergraph Network for Cross-Domain Few-Shot Facial Expression Recognition
Jul 23, 2026Xinhan Qiu, Yan Yan, Rui Zhu +2Few-Shot Learning Cross-Domain Transfer Learning
Key Laboratory of Multimedia Trusted Perception and Efficient Computing, Ministry of Education of China, School of Informatics, Xiamen University, Xiamen 361102, China · Fujian Key Laboratory of Pattern Recognition and Image Understanding, Xiamen University of Technology, Xiamen 361024, China · Bayes Business School, City St George’s, University of London, London, EC1Y 8TZ, UK +1
Bidirectional Learning of Facial Action Units and Expressions via Structured Semantic Mapping across Heterogeneous Datasets
Apr 12, 2026Jia Li, Yu Zhang, Yin Chen +5Facial Action Unit Detection Facial Expression Recognition
School of Computer Science and Information Engineering, Hefei University of Technology, Hefei 230601, China · School of Computer Science and Engineering, and the Key Laboratory of New Generation Artificial Intelligence Technology and Its Interdisciplinary Applications, Southeast University, Nanjing 210096, China · Key Laboratory of Intelligent Information Processing, Institute of Computing Technology, Chinese Academy of Sciences, Beijing 100190, China, and also with the University of Chinese Academy of Sciences, Beijing 100049, China

Apr 23, 2026J/K move · Enter open · S save

UAU-Net: Uncertainty-aware Representation Learning and Evidential Classification for Facial Action Unit Detection

Yuze Li, Zhilei Liu

School of Artificial Intelligence Tianjin University Tianjin, China

Representation Learning Evidential Deep Learning Facial Action Unit Detection Facial Expression Recognition

Open paper AskPDF arXiv

Facial action unit (AU) detection remains challenging because it involves heterogeneous, AU-specific uncertainties arising at both the representation and decision stages. Recent methods have improved discriminative feature learning, but they often treat the AU representations as deterministic, overlooking uncertainty caused by visual noise, subject-dependent appearance variations, and ambiguous inter-AU relationships, all of which can substantially degrade robustness. Meanwhile, conventional point-estimation classifiers often provide poorly calibrated confidence, producing overconfident predictions, especially under the severe label imbalance typical of AU datasets. We propose UAU-Net, an Uncertainty-aware AU detection framework that explicitly models uncertainty at both stages. At the representation stage, we introduce CV-AFE, a conditional VAE (CVAE)-based AU feature extraction module that learns probabilistic AU representations by jointly estimating feature means and variances across multiple spatio-temporal scales; conditioning on AU labels further enables CV-AFE to capture uncertainty associated with inter-AU dependencies. At the decision stage, we design AB-ENN, an Asymmetric Beta Evidential Neural Network for multi-label AU detection, which parameterizes predictive uncertainty with Beta distributions and mitigates overconfidence via an asymmetric loss tailored to highly imbalanced binary labels. Extensive experiments on BP4D and DISFA show that UAU-Net achieves strong AU detection performance, and further analyses indicate that modeling uncertainty in both representation learning and evidential prediction improves robustness and reliability.

Abstract

Cross-Validation

Explore similar work

UAU-Net: Uncertainty-aware Representation Learning and Evidential Classification for Facial Action Unit Detection

AUCH-Net: Action Unit-Based Consistency-Aware Hypergraph Network for Cross-Domain Few-Shot Facial Expression Recognition

Bidirectional Learning of Facial Action Units and Expressions via Structured Semantic Mapping across Heterogeneous Datasets