Whose Face Is It Anyway? A Multi-Model Audit of Facial Affect Recognition on Children, and Why the Gap Is the Head, Not the Features

Tobias Hallmen; Robin-Nico Kampa; Elisabeth André

cs.CVOct 6, 2026

Whose Face Is It Anyway? A Multi-Model Audit of Facial Affect Recognition on Children, and Why the Gap Is the Head, Not the Features

Authors: Tobias Hallmen, Robin-Nico Kampa, Elisabeth André

Organizations: Chair for Human-Centered Artificial Intelligence, University of Augsburg · Institute for Distributed Intelligent Systems, University of the Bundeswehr Munich

Abstract

Facial affect models are trained almost entirely on adults, yet are increasingly applied to children in education, health, and developmental research. We present a controlled, multi-model audit of five AffectNet-pretrained expression models (EmoNet, EmotiEffLib, DDAMFN++, OpenFace 3.0, LibreFace) on children, across four child image datasets, the AffectNet-8 validation set, and two spontaneous child video datasets, through one shared harness. Three findings emerge. First, the child gap is model-agnostic: every architecture degrades from posed to naturalistic faces and shares the fear $\rightarrow$ surprise confusion. Second, it is concentrated and corroborated across all five models: open-mouth faces (read as surprise, correlating with the AU26 jaw drop) and South-Asian children degrade systematically, with a smaller averted-gaze penalty, while closed-mouth faces, White and Black children, and direct gaze do not; the bias tracks expression morphology and specific populations, not skin tone. Third, the gap is diagnosable: a linear probe on frozen features reaches 0.75-0.91 on unseen children versus 0.48-0.66 zero-shot, so it lies largely in the classifier head, not the representation, whereas dimensional valence/arousal regression degrades sharply under domain shift. Building on this, recalibrating only the head on a little target data recovers $+0.13$ to $+0.28$ on the two largest child sets across all five models at negligible adult cost, though the gain is in-distribution and does not transfer across child collections. We will release the harness, per-sample predictions, and analysis code; the child face data stays license-locked and is never redistributed.

Figures & tables

Figure 1 : The child-affect gap is model-agnostic, demographically concentrated and AU-grounded, and lives in the classifier head, not the representation. (a) Every model degrades from posed (RaFD) to naturalistic (CAFE) faces. (b) Open-mouth faces are misread (the AU26 jaw drop

\to

surprise), all five models. (c) A linear probe on frozen features (green) far exceeds zero-shot (grey): the gap is in the head, not the representation.

Model	Backbone	Emotion train	RaFD?	Child?
EmoNet	FAN enc.	AffN	no	no
EmotiEffLib	VGGFace2	AffN (+V/A)	no	no
DDAMFN++	MS-Celeb	AffN-8	no	no
OpenFace 3.0	VGGFace2	AffN	no	no
LibreFace	MAE	AffN+RAF-DB	no	no

Table 1 : Training-data provenance of the five audited models (weights as used; AffN

{=}

AffectNet). Columns list backbone, emotion training corpus, and whether RaFD or any curated child data appears in the documented training provenance (none for all).

Study	Domain	#Models	Subgroup slices	AU mech.	Head/feat.	Fix
Gender Shades [ 3 ]	adult (gender classif.)	3 comm.	skin type, gender	$\times$	$\times$	benchmark
Xu et al. [ 37 ]	FER	1	gender, race, age	$\times$	$\times$	disentangle
Domnich & Anbarjafari [ 7 ]	FER	5	gender	$\times$	$\times$	$\times$
Kim et al. [ 19 ]	adult FER	4 comm.	age	$\times$	$\times$	$\times$
Gaya-Morey et al. [ 12 ]	FER (age groups)	3	age	$\times$	$\times$	age-bias mitig.
Virgolin et al. [ 34 ]	child FER	1	$\times$	$\times$	$\times$	contrastive aug.

Table 2 : This audit versus prior facial-affect bias and child-FER work. No prior study grounds the error in action units or separates a classifier- head failure from a representation failure; we add both to a multi-model, demographically sliced child evaluation with a deployable fix. #Models counts distinct architectures or commercial (comm.) systems, not training/setup variants of one backbone.

	RaFD	ChildEFES	NIMH	CAFE	mean
Model	(80)	(158)	(533)	(1192)
EmoNet	.963 (.96)	.759 (.73)	.666 (.46)	.634 (.54)	.756 (.68)
EmotiEffLib	.950 (.95)	.703 (.64)	.690 (.47)	.637 (.63)	.745 (.67)
DDAMFN++	.963 (.96)	.741 (.70)	.578 (.40)	.636 (.55)	.730 (.66)
OpenFace 3.0	.887 (.88)	.601 (.58)	.480 (.36)	.492 (.42)	.615 (.56)
LibreFace	.662 (.57)	.525 (.39)	.675 (.56)	.575 (.50)	.609 (.51)

Table 3 : Child image datasets: accuracy, with macro-F1 in parentheses (8-class, mapped; bold marks the best accuracy per column). “mean” is the unweighted mean over the four datasets (which have very different

N

; under sample-weighting the top two means swap). Type, left to right: posed studio frontal

\rightarrow

posed/induced

\rightarrow

posed

\pm

gaze

\rightarrow

varied open/closed mouth. Every model degrades with naturalness; macro-F1 falls further on the harder sets (NIMH, CAFE), so accuracy understates the gap. No model escapes it.

Model	Acc-8	Acc-7	Val. CCC	Aro. CCC
EmotiEffLib	.675	.675	.581	.585
DDAMFN++	.643	.654	–	–
EmoNet	.594	.591	.684	.628
OpenFace 3.0	.562	.536	–	–
LibreFace	.436	.498	–	–

Table 4: Standard AffectNet-8 validation (raw 4000, single-crop, no TTA), the fair cross-model bar. Acc-8/Acc-7 are 8- and 7-class accuracy; Val./Aro. CCC are valence/arousal concordance (dash where a model has no V/A head).

Attribute	Hard subgroup	control	#mdl $\downarrow$	sig.
Mouth (CAFE)	open .46	.69	5/5	$p{<}.001$
Race (CAFE)	S. Asian .53	W.61/B.60	5/5	$p{=}.02$
Gaze (NIMH)	averted	direct	5/5	dir.

Table 5 : Demographic slices, pooled over the five models. Columns: hard subgroup accuracy versus its control, “#mdl

\downarrow

” (models scoring below their own overall accuracy on the subgroup), and significance.

Figure 2 : CAFE confusion averaged (row-normalized) over all five models. The fear $\rightarrow$ surprise confusion is shared across models (5-model mean $0.56$ vs $0.38$ correct); other errors such as anger $\rightarrow$ disgust are heterogeneous across models ( $0.00$ – $0.63$ ) and shown here only as the 5-model average ( $0.34$ ). (CAFE has no contempt.)

Figure 3 : Bias and fix are one head miscalibration (EmotiEffLib, held-out CAFE; cf. Tab. 7 ). The adult-only head over-predicts surprise , leaking neutral, happy, and fear faces into it (left, the AU26 jaw-drop boundary); the child-recalibrated head ( $w{=}10$ ) collapses that mass onto the diagonal (CAFE $0.61\to 0.87$ , backbone frozen; right).

Model	zero-shot	linear probe	$\Delta$
EmotiEffLib	.638	.910	$+.272$
EmoNet	.657	.878	$+.221$
OpenFace 3.0	.484	.859	$+.375$
DDAMFN++	.638	.843	$+.205$
LibreFace	.593	.747	$+.154$

Table 6: Linear probe on frozen features, CAFE, participant-disjoint (108 train / 46 test subjects), all five models. Columns: zero-shot accuracy, accuracy of a linear head probe on the frozen features, and their difference

\Delta

model	head	adults	CAFE	ChildEFES	NIMH	RaFD
EmotiEffLib	adult-only	.683	.606	.667	.704	.958
EmotiEffLib	$w{=}10$	.678	.865	.789	.858	.958
EmoNet	adult-only	.610	.657	.737	.747	.958
EmoNet	$w{=}10$	.595	.865	.702	.914	.917
DDAMFN++	adult-only	.639	.590	.649	.599	.750
DDAMFN++	$w{=}10$	.632	.859	.754	.784	1.00

Table 7 : Frozen-backbone head recalibration across all five models: a linear expression head retrained on pooled AffectNet-train + child data (child up-weight

w

), evaluated on held-out adults (AffectNet-val) and held-out, subject-disjoint children. Each model has two rows: the adult-only baseline (head retrained without child data) and the recalibrated head at

w{=}10

. Bold marks the recalibrated (

w{=}10

) result where it improves on the adult-only baseline.

Figure 4 : Data efficiency of the head recalibration on CAFE: held-out child accuracy versus the number of target subjects used to refit the linear head (8k-frame AffectNet anchor, child up-weight

w{=}10

; mean

\pm

std over 4 subject draws). Most of the gain arrives by 10–25 children.

Panel of Figure 5 — Figure 5 : The head recalibration buys child accuracy essentially for free. As the child up-weight $w$ rises, held-out children improve sharply on all five backbones 5(a) while held-out adults stay nearly flat 5(b) (at most $-0.015$ ): the adult cost is negligible relative to the child gains.

Explore similar work

Jul 14, 2026cs.CV

AffectFlow-DINO: Uncertainty-Aware Multi-Task Affect Estimation via Conditional Rectified Flow

We present \textbf{AffectFlow-DINO}, a multi-task learning system for the 11th ABAW challenge that extends a standard deterministic architecture with a conditional rectified-flow head to model the inherent ambiguity of in-the-wild facial behavior. Instead of predicting a single affect estimate, the model learns a conditional generative distribution, enabling uncertainty-aware one-to-many predictions through Monte Carlo sampling. The system jointly estimates continuous valence-arousal, classifies eight facial expressions, and detects twelve Action Units from static face images. Built on a frozen DINOv3 ViT-S/16 backbone, extensive ablation studies show that rectified-flow decoding consistently improves deterministic prediction, particularly for valence-arousal estimation (CCC-V

+0.058

). We further show that post-hoc threshold calibration effectively recovers performance on severely imbalanced rare classes (e.g., Fear:

3.8\% \rightarrow 33.1\%

) without retraining. Combined with backbone fine-tuning and flow retuning, the final model achieves

\mathbf{P_{MTL}=1.177}

, substantially outperforming the official challenge baseline of

P_{MTL}=0.45

Salah Eddine Bekhouche, Abdellah Zakaria Sellam, Fadi Dornaika +1

University of the Basque Country (UPV/EHU), Spain · Department of Innovation Engineering, University of Salento & Institute of Applied Sciences and Intelligent Systems (CNR), Italy · Faculty of Data Science and Computing, Universiti Malaysia Kelantan, Malaysia

Facial Expression Recognition Valence-Arousal Estimation Rectified Flow Dino In-The-Wild Videos

Ask PDF

May 7, 2026cs.CV

Multimodal Emotion Recognition via Causal-Diffusion Bridge (Affect-Diff)

Multimodal emotion recognition on CMU-MOSEI faces an extreme imbalance as Happy accounts for 65.9% of samples while three Ekman categories collectively represent under 7%, causing standard fusion models to maximize accuracy by ignoring minority emotions entirely. We present Affect-Diff, a Causal-Diffusion Bridge that addresses this through three jointly trained mechanisms: a NOTEARS-learned causal graph that re-weights modality contributions before fusion, a beta-VAE bottleneck for regularized latent compression, and a stop-gradiented 1D DDPM prior that structures the latent space against majority-class collapse. On 3,292 aligned CMU-MOSEI samples, Affect-Diff achieves validation balanced accuracy 0.384, an 18% relative improvement over the strongest baseline (TETFN: 0.324), while all evaluated baselines produce zero F1 on Fear, Disgust, and Surprise. Ablation studies confirm independent, non-redundant contributions from the diffusion prior (-24% without it) and causal graph (-13%). Notably, only the deterministic-encoder variant detects all six emotion classes, revealing KL regularization strength as a direct lever for minority-class sensitivity.

Ankit Sanjyal

Department of Computer Science – Data Science Fordham University New York, USA

Emotion Recognition Minority Impact Analysis

Ask PDF

Jun 19, 2026cs.CV

Chehre: An Emoji-Prompted Dataset to Explore Perceptual Flexibility in Video Language Models

Do people perceive the same facial expression in the same way? Should we expect vision models to be flexible in how they perceive facial expressions? Facial expressions are nonverbal social signals used in human interaction, but facial expression recognition datasets often focus on a single deterministic annotation per sample. We introduce Chehre, an emoji-prompted video dataset with a wide range of dynamic facial expressions for exploring perceptual variation. In Chehre, 203 participants were prompted to express and record 40 facial emojis. Later, their facial motions were transferred onto synthetic faces to preserve privacy. A separate group annotated the videos, resulting in 2,111 videos annotated by 1,242 perceivers, with ~30 annotators per video. Chehre enables us to define a new task: "distributional expression recognition", which tests whether a model can reproduce the variation observed across annotator responses. We test a selection of video language models on our task. Interestingly, we find that persona prompting can act as a controllable way to shift model perception while helping models better capture the variation observed across human annotators. The dataset and code are available at https://chehre-dataset.github.io/.

Bita Azari, Zoe Stanley, Avneet Batra +4

Simon Fraser University, Canada

Facial Expression Recognition Facial Images Video Dataset Expression Nonverbal Cues

Ask PDF