cs.CVMar 29, 2026

Towards Emotion Recognition with 3D Pointclouds Obtained from Facial Expression Images

Authors: Laura Rayón Ropero, Jasper De Laet, Filip Lemic, Pau Sabater Nácher, Nabeel Nisar Bhat, Sergi Abadal, Jeroen Famaey, Eduard Alarcón, 

Organizations: imec, Belgium · ICREA, Spain

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Facial Emotion Recognition is a critical research area within Affective Computing due to its wide-ranging applications in Human Computer Interaction, mental health assessment and fatigue monitoring. Current FER methods predominantly rely on Deep Learning techniques trained on 2D image data, which pose significant privacy concerns and are unsuitable for continuous, real-time monitoring. As an alternative, we propose High-Frequency Wireless Sensing (HFWS) as an enabler of continuous, privacy-aware FER, through the generation of detailed 3D facial pointclouds via on-person sensors embedded in wearables. We present arguments supporting the privacy advantages of HFWS over traditional 2D imaging, particularly under increasingly stringent data protection regulations. A major barrier to adopting HFWS for FER is the scarcity of labeled 3D FER datasets. Towards addressing this issue, we introduce a FLAME-based method to generate 3D facial pointclouds from existing public 2D datasets. Using this approach, we create AffectNet3D, a 3D version of the AffectNet database. To evaluate the quality and usability of the generated data, we design a pointcloud refinement pipeline focused on isolating the facial region, and train the popular PointNet++ model on the refined pointclouds. Fine-tuning the model on a small subset of the unseen 3D FER dataset BU-3DFE yields a classification accuracy exceeding 70%, comparable to oracle-level performance. To further investigate the potential of HFWS-based FER for continuous monitoring, we simulate wearable sensing conditions by masking portions of the generated pointclouds. Experimental results show that models trained on AffectNet3D and fine-tuned with just 25% of BU-3DFE outperform those trained solely on BU-3DFE. These findings highlight the viability of our pipeline and support the feasibility of continuous, privacy-aware FER via wearable HFWS systems.

Figures & tables

Figure 1
Paradigm / approach familyPrimary privacy mechanismData representation (stored/processed)Formal privacy guarantee?Typical deployment burdenCompatibility with our pipeline
Standard 2D FER (RGB)None (full texture exposure)Raw RGB facial imagesNoModerate (standard training/inference)Not applicable (different privacy objective)
Landmark-based FERRepresentation minimization (sparse geometry cues)2D landmarks / keypointsNo (biometric leakage possible)LowCompatible (alternative or auxiliary input)
AU-based FERRepresentation minimization (semantic/physiological coding)Action Unit vectors / intensitiesNo (depends on pipeline)Low–Moderate (AU extraction + training)Compatible (can be fused or used as auxiliary)
Federated learning FERDistributed training (data stays local)Local training + model updates (gradients/weights)No (unless combined with DP/secure aggregation)High (protocol, comms, orchestration)Orthogonal; can be combined with ours
Differential privacy (DP) FERNoise injection during trainingNoisy gradients/updates; privacy budget ε\varepsilonYes (under DP assumptions)High (privacy budget selection, tuning, utility trade-off)Orthogonal; can be combined with ours
Geometry-only 3D FER (this work)Representation minimization (no texture)Texture-free 3D pointcloud / mesh geometryNoModerate (3D processing + FER model)Core approach (representation layer)
TABLE I: Privacy–utility positioning of representative FER paradigms. This table provides a high-level design-space positioning rather than a performance leaderboard; reported utility trends depend on dataset, protocol, and (for DP) privacy budget.
Fig. 3: System design of the envisioned privacy-aware 3D FER system trained on in-the-wild 2D images
Figure 4
Fig. 6: High-level design of the PointNet++ network, extracted from [ 51 ]
Figure 6
Experiment/MetricUsed part of BU-3DFE [%]Training accuracy [%]Validation accuracy [%]Inference accuracy [%]Training time [min]
BASELINE: Training and Inference on BU-3DFE 
E1.a: Raw BU-3DFE (full)8096.50-74.02198
E1.b: Raw BU-3DFE (512)8098.25-78.90162
E2.a: Raw BU-3DFE (full)2567.3450.1245.5710
E2.b: Raw BU-3DFE (512)2566.1362.9849.128
E3.a: Raw BU-3DFE (full)1084.3850.0037.344.5
TABLE II: Summarized classification performance observed in different experiments
Fig. 9: Confusion matrix for experiment E6.b
Fig. 10: Classification accuracy under reduced facial coverage for example masking strategies (utility under constrained observability)
Fig. 11: Illustrative example of THz array configurations and beam coverage for facial surface 3D imaging
Fig. 12: Classification accuracy for different number of downsampled points and corresponding bandwidths required for 3D imaging
Figure 12
Figure 13
Figure 14
Figure 15
Figure 16
Figure 17
Figure 18
Figure 19
Figure 20
More in this topic

Facial Expression Recognition

Facial Expression Recognition studies how computers infer emotion and facial actions from images and video.

See all 79 papers

Explore similar work

CardsList
Oct 8, 2026

TAP3D: Thermal-Assisted 3D Human Point Clouds

Human body point clouds are a versatile representation for AI-enabled human sensing. However, existing methods using LiDAR, radar, and depth cameras suffer from inherent drawbacks in high cost, sparse reconstruction, and privacy concerns, etc. In this paper, we exploit low-cost thermal arrays and present TAP3D, the first system to reconstruct 3D human point clouds from body heat signatures, offering significant advantages in cost, density, human sensitivity, and privacy. To overcome major challenges in depth estimation, thermal interference, and multi-person separation, we propose a novel physics-informed design, which integrates a forward thermal physics model with two distinct modules: multi-primitive estimation for self-supervised joint recovery of depth and other thermal properties, and geometric perspective fusion for suppressing interference and disentangling multiple people. We implement TAP3D using a single commodity thermal array sensor and build a large-scale dataset (160K samples, 8 environments, 11 users) for evaluation. TAP3D achieves remarkable accuracy for dense point cloud generation, enabling downstream tasks like fall detection (91.46%), indoor tracking (21.86 cm MAE), and human mesh recovery (4.87 cm error). By transforming body heat into point clouds for the first time, TAP3D pioneers a new paradigm for privacy-first, fully passive human sensing for many applications. TAP3D is open-sourced at https://github.com/aiot-lab/TAP3D.

Xie Zhang, Chengxiao Li, Xuan Liu +1

The University of Hong Kong

3D Human ReconstructionPoint Cloud GenerationThermal Imaging
AskPDF
Sep 12, 2026

Event-Level Emotion Recognition in the Wild Using Deep Facial Expression Analysis

Facial emotion recognition (FER) in real-world environments remains challenging due to unconstrained imaging conditions, including multiple faces, occlusions, pose variations, and complex lighting. Most existing studies focus on individual facial emotion classification and do not address the analysis of collective emotional states at the event level. This paper proposes an end-to-end pipeline for event-level emotion recognition from photographs. The approach detects faces in each image, classifies facial expressions using a deep convolutional neural network, and aggregates face-level emotion probabilities to estimate the overall emotional distribution of a public event. A comparative evaluation of several CNN architectures on the FER- 2013 and RAF-DB datasets demonstrates that transfer learning with EfficientNet-B2 trained on RAF-DB is more suitable for real-world RGB data. The proposed method is evaluated on a real-world event dataset containing 1658 images. Experimental results show stable emotion distributions across event subsets, confirming the effectiveness of event-level aggregation for emotion analysis in the wild.

Aleksandr Semerikov, Pakizar Shamoi

School of Information Technology and Engineering Kazakh-British Technical University Almaty, Kazakhstan

Emotion RecognitionFacial Expression Recognition
AskPDF
May 17, 2026

On Applicability of Synthetic Datasets for Facial Expression Recognition

Facial Expression Recognition faces two core challenges. The first is class imbalance in public datasets, which skews the learning process and weakens generalization. The second is related to privacy and data collection constraints, which limit the sharing of facial images and restrict the creation of large, balanced datasets. To address these issues, we examine three complementary strategies for constructing privacy-preserving FER datasets in the standard seven discrete facial expression classes setting. Our strategies are: (i) pseudo-labeling large unlabeled face collections with a teacher model under a confidence-thresholding scheme, (ii) prompt-driven synthesis using diffusion models conditioned on demographic attributes, and (iii) task-aware GAN-based expression editing that modifies facial expression while preserving identity and realism. For training and evaluation, we employed widely adopted datasets, including AffectNet, RAF-DB, and FER2013. We utilized the synthetic datasets DigiFace, DCFace, and EmoNet-Face BIG as unlabeled sources for pseudo-labeling. Additionally, we utilized the FFHQ dataset as the source for generative synthesis. The main experiments are conducted using a classic CNN backbone, IR50, and we also explore a more complex architecture, POSTERv1, to assess its feasibility and robustness. Using cross-dataset evaluations, we analyze the trade-offs each strategy presents in curated datasets. The findings demonstrate how synthetic data can effectively substitute or be combined with real datasets to mitigate imbalance and privacy limitations. Code and generated datasets:https://www.github.com/AliAZ98/SyntFER

Ali Azmoudeh, Erdi Sarıtaş, Ömer Yıldırım +1

Department of Computer Engineering, Istanbul Technical University, Istanbul, Türkiye · Department of Informatics, University of Zurich, Zurich, Switzerland · Division of Engineering, NYU Abu Dhabi, Abu Dhabi, UAE

Class-Imbalanced LearningSynthetic Data GenerationPrivacy-Preserving MLFacial Expression RecognitionSynthetic Data Evaluation
AskPDF