Facial Action Unit Detection

Momentum

0 papers in the last four weeks, with none the four weeks before. 0.0% of all new papers.

Jul 13Week of Sep 28

Latest papers 12

Jul 18, 2026cs.CV

Spatiotemporal Facial Action Unit Detection using Twin Cycle Autoencoders for Driver Monitoring

Driver monitoring systems (DMS) increasingly rely on facial cues to infer drowsiness, distraction, and cognitive load in real time. Facial Action Units (AUs), grounded in the Facial Action Coding System (FACS), provide an objective and interpretable representation of such states, but their automatic detection in the driving context is complicated by low and variable illumination, partial occlusion, head-pose variation, and the subtlety and short duration of relevant AU activations. Existing AU detectors largely treat spatial appearance and temporal dynamics separately, limiting their ability to exploit self-supervisory signal from abundant unlabeled driving video. We propose the Twin Cycle Autoencoder (TCA), a spatiotemporal architecture composed of two coupled cycle-consistent autoencoder branches: a Spatial Cycle Autoencoder that disentangles AU-relevant appearance from identity through image-level cycle consistency, and a Temporal Cycle Autoencoder that enforces forward-backward consistency over latent AU trajectories to capture onset-apex-offset dynamics. The two branches are coupled through a cross-branch latent alignment loss and fused via an attention module before multi-label AU classification. We evaluate TCA on the DISFA and BP4D benchmarks and on an in-cabin naturalistic driving dataset, and observe consistent improvements over CNN-RNN, 3D-CNN, and graph-based AU baselines, particularly for low-intensity and rapidly transitioning AUs relevant to fatigue (AU45, AU43) and yawning (AU26). We further show the model sustains real-time throughput on an embedded Jetson Xavier NX platform, supporting its use in production-grade advanced driver assistance systems (ADAS).

Sai Sidharth D

Autoencoders Facial Action Unit Detection Facial Expression Recognition Driver Monitoring

Ask PDF

Jul 17, 2026cs.CV

AffectFuse: Cross-Task Feature Fusion with Temporal Modeling for Multi-Task Affective Behavior Analysis

Affective behavior recognition in the wild requires joint prediction of continuous valence-arousal, categorical facial expression, and multi-label action units from unconstrained face images. We present our system for the Multi-Task Learning (MTL) track of the 11th Affective Behavior Analysis in-the-wild (ABAW) competition on s-Aff-Wild2, the static selected-frame version of Aff-Wild2. The method focuses on post-encoder adaptation: frozen AffectNet-supervised backbones provide multi-resolution features, while task-specific temporal heads and cross-task fusion modules select the useful signals for each target. For action-unit recognition, we adapt MAE-Face with Low-Rank Adaptation (LoRA) and use DISFA through per-unit expert routing rather than direct sequential transfer. Ablations over backbone, temporal, fusion, and AU-adaptation choices define the final configuration. The final system obtains P = 1.7302 on the official validation split, showing that post-encoder adaptation and task-wise modeling choices provide a strong MTL pipeline without training a new large-scale face foundation model.

Dipit Saha, Mohammad Raihan Rashid, Shah Mohammad Abdul Mannan +2

Bangladesh University of Engineering and Technology

Multi-Task Learning Facial Action Unit Detection Valence-Arousal Modeling Facial Expression Recognition Feature Fusion Emotion Recognition

Ask PDF

Jul 15, 2026cs.CV

Task-Specific Feature Fusion Method for Multi-Task Affective Behavior Analysis

The 11th Affective Behavior Analysis in-the-wild (ABAW11) Multi-Task Learning Challenge requires a unified system to predict valence-arousal, categorical expressions, and facial action units from the official s-Aff-Wild2 images. Although these tasks are naturally related through facial behavior, our validation experiments show that they benefit from different visual features, temporal processing strategies, fusion mechanisms, and calibration procedures. In this paper, we study task-adaptive feature fusion for ABAW11 multi-task affective behavior analysis. We first adapt two pretrained visual backbones, DINOv2 ViT-L and DINOv3 ConvNeXt-base, on an external expression-oriented facial image set and then freeze them to extract complementary frame-level features from the official ABAW11 data. On top of these frozen features, we systematically compare frame-level prediction heads, temporal convolutional heads, post-hoc temporal smoothing, LightGBM models, feature concatenation, gated fusion, residual fusion, late logit fusion, threshold calibration, and shared MTL structures. The final system selects task-specific fusion and prediction strategies rather than forcing all tasks to share a single architecture. On the ABAW11 validation set, the selected system achieves an EXPR macro-F1 of 0.4222, an AU macro-F1 of 0.5402, and a mean VA CCC of 0.6717, resulting in an overall validation score of 1.6341. The results suggest that task-adaptive fusion of frozen visual features is a simple and effective strategy for ABAW-style multi-task affective behavior analysis.

Jiajun Sun, Zhe Gao

Shanghai Normal University, Shanghai, China

Multi-Task Learning Facial Action Unit Detection Valence-Arousal Modeling Facial Expression Recognition Feature Fusion Emotion Recognition

Ask PDF

Jul 14, 2026cs.CV

AffectFlow-DINO: Uncertainty-Aware Multi-Task Affect Estimation via Conditional Rectified Flow

We present \textbf{AffectFlow-DINO}, a multi-task learning system for the 11th ABAW challenge that extends a standard deterministic architecture with a conditional rectified-flow head to model the inherent ambiguity of in-the-wild facial behavior. Instead of predicting a single affect estimate, the model learns a conditional generative distribution, enabling uncertainty-aware one-to-many predictions through Monte Carlo sampling. The system jointly estimates continuous valence-arousal, classifies eight facial expressions, and detects twelve Action Units from static face images. Built on a frozen DINOv3 ViT-S/16 backbone, extensive ablation studies show that rectified-flow decoding consistently improves deterministic prediction, particularly for valence-arousal estimation (CCC-V

+0.058

). We further show that post-hoc threshold calibration effectively recovers performance on severely imbalanced rare classes (e.g., Fear:

3.8\% \rightarrow 33.1\%

) without retraining. Combined with backbone fine-tuning and flow retuning, the final model achieves

\mathbf{P_{MTL}=1.177}

, substantially outperforming the official challenge baseline of

P_{MTL}=0.45

Salah Eddine Bekhouche, Abdellah Zakaria Sellam, Fadi Dornaika +1

University of the Basque Country (UPV/EHU), Spain · Department of Innovation Engineering, University of Salento & Institute of Applied Sciences and Intelligent Systems (CNR), Italy · Faculty of Data Science and Computing, Universiti Malaysia Kelantan, Malaysia

Multi-Task Learning Facial Action Unit Detection Valence-Arousal Modeling Facial Expression Recognition Flow-Based Generative Modeling Emotion Recognition+1

Ask PDF

Jul 13, 2026cs.CV

Causal Supervision of Attention for Affective Behaviour Analysis

The \textit{11th Affective Behaviour Analysis in-the-wild Competition} includes the Multi-Task Learning Challenge, where participants develop a unified framework for Valence-Arousal Estimation, Expression Recognition, and Action Unit Detection. The challenge lies in learning emotion-related representations that generalize across subjects while remaining robust to spurious factors such as identity, illumination, pose, and demographic variation. To aggregate features extracted by a pre-trained backbone into a compact representation for prediction, attention mechanisms selectively weight the most informative facial regions. However, these attention weights can still capture dataset-specific correlations rather than genuine affective cues. To address this limitation, we propose an attention pooling framework that combines causal supervision with cross-covariance regularization of attention components, encouraging subject-invariant attention and non-redundant representations that improve generalization. Our method achieves

CCC_{VA}=0.5123

for VA estimation on the official validation set, together with

F_{EX}=0.3116

and

F_{AU}=0.3974

for expression recognition and action unit detection, respectively, resulting in an overall

P

score (the sum of the individual task metrics) of

1.2214

Nemanja Rašajski, Konstantinos Makantasis, Antonios Liapis +1

Institute of Digital Games, University of Malta, Malta · AI Department, University of Malta, Malta

Causal Representation Learning Facial Action Unit Detection Valence-Arousal Modeling Facial Expression Recognition Attention Mechanisms Emotion Recognition+1

Ask PDF

Jun 13, 2026cs.CV

Facial Affect Analysis for Service-Oriented Systems: Advances, Challenges, and Future Visions

Facial Affect Analysis (FAA) is evolving from a stand-alone recognition task into a reusable perception capability for Service-Oriented Software Ecosystems (SoSE). This paper preserves the FAA methodological core while reframing recent advances through systems-engineering requirements for composable and dependable services. We review representative progress in static and dynamic expression analysis, action-unit and micro-expression modeling, and modern CNN, Transformer, graph, and hybrid architectures, then interpret these advances by their operational fit in edge, cloud, and hybrid service pipelines. The synthesis emphasizes SoSE concerns that determine deployability: service contracts for uncertainty-aware outputs, latency and availability envelopes, lifecycle monitoring and recalibration, governance-aware integration, and interoperability across independently evolving components. Our analysis shows that benchmark gains alone are insufficient for SoSE readiness; robustness under shift, intervention stability, fairness, privacy posture, and runtime guarantees are equally critical. We conclude with a roadmap for treating FAA as an operational service component with explicit interfaces, measurable quality attributes, and accountable lifecycle management.

Spyridon Georgiou, Aggelos Psiris, Thomas Lagkas +4

Department of Informatics and Telematics, Harokopio University of Athens, Athens, Greece · Department of Informatics, Faculty of Science, Democritus University of Thrace, Kavala, Greece · Department of Networks and Digital Media, Kingston University, Kingston upon Thames, United Kingdom +1

Software Engineering Facial Action Unit Detection Facial Expression Recognition

Ask PDF

Jun 3, 2026cs.CV

Horse Eye Blink Detection and Classification for Equine Affective State Assessment

Automated detection of equine facial action units (AUs) is a promising yet under-explored avenue for pain and affective state assessment in horses. Half and full-blink movements are recognised indicators of pain and stress, but as micro-expressions, their subtle, fine-grained nature makes them easily missed by the naked eye and only discernible through frame-by-frame video inspection, making reliable automated detection from video a particularly demanding task. We develop and evaluate three methods for automated blink classification from horse videos: a frame-based YOLOv12 detector, an optical flow magnitude thresholding approach, and a fine-tuned VideoMAE model, tested on a publicly available dataset. We achieve a macro-F1 score of 0.898 when doing blink classification and 0.926 on binary blink detection. Our results highlight both the potential and the inherent challenges of fine-grained AU detection for equine welfare monitoring.

João Alves, Signe Møller-Skuldbøl, Pia Haubro Andersen +1

Visual Analysis and Perception Lab, Aalborg University · Department of Animal Biosciences, Swedish University of Agricultural Sciences

Facial Action Unit Detection Facial Expression Recognition Affective Computing

Ask PDF

May 25, 2026cs.CV

Generating 3D models from sketches of human faces using a combined approach of Convolutional Neural Networks, Procedural Modeling, and Contour Mapping

Generating 3D models from face sketches is an active topic of research in Computer Graphics due to its potential to tremendously facilitate the modeling of faces for both professional 3D arists and novices. Motivated by the observation that facial expressions are responsible for significantly altering and shaping the contours in our faces, we combine both expression detection and 3D model generation in our approach. The result is a novel approach to generating 3D models from sketches which relies on three components: Convolutional Neural Networks, a parametric 3D face model (Valley Girl), and Active Snake Contours. For the first time in the literature, CNNs are trained (using our own generated dataset) to detect the expression in the given sketch through detecting the active FACS Action Units. The expression is then duplicated on Valley Girl to obtain a 3D model with a similar expression. Active Snake Contours are then used to find the transforms needed to close the gaps between that model and the given sketch.

Nancy Iskander

Graduate Department of Computer Science University of Toronto

Facial Action Unit Detection 3D Face Reconstruction

Ask PDF

May 24, 2026cs.CV

From Affect to Complex Behavior: Advancing Multimodal Human-Centered AI at the 10th ABAW Workshop & Competition

The 10th Affective & Behavior Analysis in-the-Wild (ABAW) Workshop and Competition, held at CVPR 2026, continues to advance research on modelling, analysis, understanding of human affect and behavior in real-world, unconstrained environments. The workshop maintains its dual structure, comprising both a competition and a paper track. The ABAW Competition introduces a diverse set of challenges targeting key aspects of affective and behavioral understanding, including continuous affect (valence-arousal) estimation, discrete affect (expression and action unit) recognition, as well as more complex behavior analysis tasks, such as emotional mimicry intensity estimation, ambivalence/hesitancy recognition and fine-grained violence detection. These challenges are built upon large-scale in-the-wild datasets, providing comprehensive benchmarks for state-of-the-art approaches. In parallel, the paper track presents a wide range of contributions spanning pose, motion & behavior estimation, affect modelling & multimodal learning, benchmarks, datasets & evaluation protocols, fairness, robustness & deployment. Overall, the 10th ABAW Workshop and Competition continues to serve as a key platform for benchmarking, collaboration and innovation, shaping the development of next-generation multimodal, human-centered AI systems.

Dimitrios Kollias, Panagiotis Tzirakis, Alan Cowen +10

Queen Mary University of London (QMUL), UK · Hume AI, USA · Google Deepmind, USA +7

Facial Action Unit Detection Valence-Arousal Modeling Multimodal Emotion Recognition Facial Expression Recognition Emotion Recognition Affective Computing

Ask PDF

Apr 23, 2026cs.CV

UAU-Net: Uncertainty-aware Representation Learning and Evidential Classification for Facial Action Unit Detection

Facial action unit (AU) detection remains challenging because it involves heterogeneous, AU-specific uncertainties arising at both the representation and decision stages. Recent methods have improved discriminative feature learning, but they often treat the AU representations as deterministic, overlooking uncertainty caused by visual noise, subject-dependent appearance variations, and ambiguous inter-AU relationships, all of which can substantially degrade robustness. Meanwhile, conventional point-estimation classifiers often provide poorly calibrated confidence, producing overconfident predictions, especially under the severe label imbalance typical of AU datasets. We propose UAU-Net, an Uncertainty-aware AU detection framework that explicitly models uncertainty at both stages. At the representation stage, we introduce CV-AFE, a conditional VAE (CVAE)-based AU feature extraction module that learns probabilistic AU representations by jointly estimating feature means and variances across multiple spatio-temporal scales; conditioning on AU labels further enables CV-AFE to capture uncertainty associated with inter-AU dependencies. At the decision stage, we design AB-ENN, an Asymmetric Beta Evidential Neural Network for multi-label AU detection, which parameterizes predictive uncertainty with Beta distributions and mitigates overconfidence via an asymmetric loss tailored to highly imbalanced binary labels. Extensive experiments on BP4D and DISFA show that UAU-Net achieves strong AU detection performance, and further analyses indicate that modeling uncertainty in both representation learning and evidential prediction improves robustness and reliability.

Yuze Li, Zhilei Liu

School of Artificial Intelligence Tianjin University Tianjin, China

Representation Learning Evidential Deep Learning Facial Action Unit Detection Facial Expression Recognition

Ask PDF

Apr 20, 2026cs.CV

MEDN: Motion-Emotion Feature Decoupling Network for Micro-Expression Recognition

Unlike macro-expression, micro-expression does not follow a strictly consistent mapping rule between emotions and Action Units (AUs). As a result, some micro-expressions share identical AUs yet represent completely opposite emotional categories, making them highly visually similar. Existing microexpression recognition (MER) methods mostly rely on explicit facial motion cues (e.g., optical flow, frame differences, AU features) while ignoring implicit emotion information. To tackle this issue, this paper presents a Motion Emotion Feature Decoupling Network (MEDN) for MER. We design a dual-branch framework to separately extract motion and emotion features. In the motion branch, an AU-detection task restricts features to the explicit motion domain, and orthogonal loss is adopted to reduce motion emotion feature coupling. For implicit emotion modeling, we propose a Sparse Emotion Vision Transformer (SEVit) that sparsifies spatial tokens to highlight local temporal variations with multi-scale sparsity rates. A Collaborative Fusion Module (CoFM) is further developed to fuse disentangled motion and emotion features adaptively. Extensive experiments on three benchmark datasets validate that MEDN effectively decouples motion and emotion features and achieves superior recognition performance, offering a new perspective for enhancing recognition accuracy and generalization.

Chenxing Hu, Kun Xie, Qiguang Miao +3

School of Computer Science and Technology, Xidian University, Xi’an, Shaanxi 710071, China · Xi’an Key Laboratory of Big Data and Intelligent Vision, Xi’an, Shaanxi 710071, China · Key Laboratory of Collaborative Intelligence Systems, Ministry of Education, Xidian University, Xi’an 710071, China +1

Disentangled Representation Learning Vision Transformer Facial Action Unit Detection Facial Expression Recognition

Ask PDF

Mar 9, 2026cs.CV

AULLM++: Structured-Token-Conditioned Large Language Models for Micro-Expression Action Unit Detection

Micro-expression Action Unit (AU) detection identifies localized AUs from subtle facial muscle activations, providing a foundation for decoding affective cues. Previous methods face three key limitations: (1) heavy reliance on low-density visual information, rendering discriminative evidence vulnerable to background noise; (2) coarse-grained feature processing that misaligns with the demand for fine-grained representations; and (3) neglect of inter-AU correlations, restricting the parsing of complex expression patterns. We propose AULLM++, a reasoning-oriented framework leveraging Large Language Models (LLMs), which injects visual features into textual prompts as actionable semantic premises to guide inference. It formulates AU prediction into three stages: evidence construction, structure modeling, and deduction-based prediction. Specifically, a Multi-Granularity Evidence-Enhanced Fusion Projector (MGE-EFP) fuses mid-level texture cues with high-level semantics, distilling them into a compact Content Token (CT). Furthermore, inspired by micro- and macro-expression AU correspondence, we encode AU relationships as a sparse structural prior and learn interaction strengths via a Relation-Aware AU Graph Neural Network (R-AUGNN), producing an Instruction Token (IT). We then fuse CT and IT into a structured textual prompt and introduce Counterfactual Consistency Regularization (CCR) to construct counterfactual samples, enhancing the model's generalization. Extensive experiments demonstrate AULLM++ achieves state-of-the-art performance on standard benchmarks and exhibits superior cross-domain generalization.

Zhishu Liu, Kaishen Yuan, Bo Zhao +2

Great Bay University, Dongguan, China · The Hong Kong University of Science and Technology (Guangzhou), Guangzhou, China · Shenzhen International Graduate School, Tsinghua University, Shenzhen, China

Facial Action Unit Detection Facial Expression Recognition Counterfactual Learning

Ask PDF

Momentum

Related topics

Latest papers 12