Affective Computing

Ask about this topic

Overview

Affective computing studies how machines can infer and work with emotion, mood, engagement, and related states from speech, facial and bodily behavior, text, and physiological signals. Common approaches include supervised classifiers, multimodal fusion, temporal models that track changing affect, and appraisal-based or language-model methods for interpreting context. Applications span healthcare and wearables, education, human-robot interaction, and emotionally responsive digital tools, with evaluation increasingly extending from controlled datasets to naturalistic conversations and daily-life settings. A central challenge is that emotional cues are ambiguous and vary across people and cultures, so accuracy must be balanced with uncertainty, fairness, privacy, and respect for the limits of what a system can infer about an individual.

Latest papers

CardsList

141 papers14 papers in the last four weeks, down 18% on the four weeks before. 0.1% of all new papers.

Sep 1, 2026

An Intelligent Decision Support System for Emotion Monitoring using Microscopic Fixational Dynamics

The rising prevalence of psychological disorders necessitates effective emotion monitoring, yet current methods relying on facial or physiological signals often suffer from intrusiveness and privacy issues. This paper proposes an intelligent decision support system and pervasive edge-computing framework that leverages smart glasses and a companion smartphone to infer emotional states from microscopic visual fixation patterns. Moving beyond traditional macroscopic gaze metrics, the proposed system extracts and decomposes three distinct neurophysiological micro-movements: microsaccades, ocular drifts, and ocular microtremors. We introduce an interpretable hybrid artificial intelligence pipeline combining a multi-head attention mechanism, extreme gradient boosting, and a support vector machine to extract deep temporal features, quantify their physiological importance, and perform efficient on-device classification. Through an extensive evaluation involving 60 volunteers, we rigorously validate the framework under a strict leave-one-subject-out cross-validation protocol across both controlled and naturalistic mobile scenarios. Ablation studies unequivocally demonstrate that these fixational micro-movements are substantially more discriminative for emotion inference than traditional macroscopic features. Furthermore, aligned with contemporary affective science, the system incorporates a few-shot personalization mechanism to bridge universal physiological baselines with individual emotional heterogeneity, achieving a highly robust personalized F1-score of 83.6%. This work establishes a physiologically interpretable, unobtrusive, and deployable paradigm for continuous real-time emotion monitoring.

Xiangyu Shen, Feiyang Deng, Zijian Dai +4

College of Advanced Interdisciplinary Studies, Central South University of Forestry and Technology, Changsha, 410004, Hunan, China · Chow Yei Ching School of Graduate Studies, City University of Hong Kong, Hong Kong, 999077, Hong Kong · College of Computer and Mathematics, Central South University of Forestry and Technology, Changsha, 410004, Hunan, China +1

Eye TrackingEmotion RecognitionAffective Computing
AskPDF
Aug 28, 2026

PAMoR: Parameterized Affective Motion Generation in Real Time for Humanoid Robots

People read a humanoid robot's motion in social settings not only for the action performed but for the affect conveyed. Motion carrying that affect has so far been generated for human avatars, where style is taken from a reference clip or an emotion word, neither of which can be quantitatively parameterized. We present PAMoR, which turns affect into a measured control parameter: a valence-arousal (V-A) coordinate computed natively on robot kinematics. It is obtained in closed form from postural expansion and movement energy, and these measurements serve directly as generation conditions, with no human annotation. An action prior and two affect priors, trained in a shared latent space, are composed at each denoising step: the action prior fixes what is performed, the affect priors modulate how. Whole-body motion rolls out autoregressively on a 29-DoF Unitree G1 in real time, with action and affect both editable. Generated motion tracks the commanded V-A over its full range while text-to-motion fidelity still matches text-only baselines. In a perceptual study, raters identify the commanded emotion on 0.38 of trials, above both baselines and approaching the 0.44 reported for acted human bodies.

Yan Pan, Lingfan Bao, Tianhu Peng +1

Department of Computer Science, University College London, London, UK

Robot Motion GenerationValence-Arousal ModelingHumanoid RoboticsAffective Computing
AskPDF
Aug 24, 2026

AffectOmni: RL-Verifiable People-Centric Grounded Affective Reasoning for Social and Art-Related Scenes

Multimodal large language models (MLLMs) achieve strong performance on VQA and scene understanding, yet affective reasoning remains vulnerable to shortcut behavior. Models may predict correct answers while neglecting people-centric cues such as micro expressions and body language, which weakens traceability and external verification. Prior reinforcement learning approaches mainly reward context or logical coherence without explicitly enforcing attention to human evidence. In addition, LLM as a Judge scoring often suffers from score clustering, which reduces reward discriminability. We propose AffectOmni, a GRPO trained framework for verifiable affective reasoning. AffectOmni introduces People Focus and Temporal Order rewards to encourage people-centric evidence selection and temporally structured reasoning, and it adopts within-group comparative scoring to produce more stable and discriminative reward signals. For verification, a Thinking Summarizer converts free form rationales into executable evidence instructions, which are grounded into pixel level evidence regions via SAM3 to provide an externally auditable interface outside the training loop. Experiments on IntentBench, Daily Omni, and WorldSense show consistent improvements over open source 7B scale baselines, including gains of 4.66% on emotion recognition and +14.29% on temporally sensitive tasks. Code is available at https://github.com/eliot127825-rgb/AffectOmni_nobody.

Yibo Wang, Rui Yang, Jisheng Dang +7

Lanzhou University, Lanzhou, China · Hainan University, Haikou, China · School of Computing, National University of Singapore, Singapore

VLM ReasoningAffective ComputingMultimodal Emotion RecognitionRL for VLMsReinforcement Learning with Verifiable RewardsVisually Grounded Reasoning
AskPDF
Aug 21, 2026

EmotionDialogCN: A Spontaneous Multimodal Dataset for Mandarin Emotional Dialogue

Face-to-face audiovisual interaction is central to human communication, conveying rich emotional and social cues. However, existing multimodal dialogue datasets remain limited by inadequate emotion annotations, poor emotional diversity, and small scale. We introduce EmotionDialogCN, a large-scale audiovisual-emotional dataset designed to capture authentic face-to-face communication. It contains 21,880 dialogue sessions performed by 119 professional actors across 20 everyday scenarios, covering 18 emotion categories with over 400 hours of recordings, the largest and most comprehensive dataset of its kind. A novel data collection framework minimizes equipment interference, enabling natural and nuanced emotional expressions. EmotionDialogCN achieves an emotion distribution deviation of 0.64 from real human emotion statistics (versus 5.65 for prior datasets) and consistent subject framing (52-59% frame occupancy). Together, these properties translate into stable unimodal and multimodal performance across acoustic, lexical, and visual modalities, with fusion results further underscoring strong multimodal alignment and cross-modal complementarity.

Yi Zheng, Yifan Xu, Yan Zhou +8

Affective ComputingEmotion Recognition in ConversationsMultimodal Emotion Recognition
AskPDF
Aug 17, 2026

Closing the Affective Loop: Multimodal Speaker-Listener Emotion-Dynamics-Aware Empathetic Social Robots

Empathetic social robots should respond not only to what users say, but also to how their emotions dynamically evolve during interaction. However, existing empathetic dialogue systems are often text-centered and primarily model empathy as a one-way mapping from the user's emotion to the system response, limiting their ability to capture embodied speaker--listener affective exchange. We present AffectLoop, a multimodal speaker-listener emotion-dynamics-aware spoken dialogue system implemented on the Misty II robot. The system tracks the speaker's verbal and facial affective dynamics, estimates the robot listener's own verbal and behavioral affective state, and conditions LLM-based response generation on both affective streams. The robot then generates a short spoken empathetic response together with emotionally congruent embodied behavior, forming a closed speaker--listener affective loop. We evaluate the system in a pilot within-subject study with five participants, comparing it with an otherwise identical utterance-conditioned baseline that omits the speaker- and listener-affective-state inputs. The proposed system received higher overall impression ratings, especially for empathetic response and user satisfaction. Post-hoc log analysis further showed higher speaker-listener affective alignment and stronger valence-based distress recovery. These preliminary results suggest that explicitly modeling both speaker emotional dynamics and listener affective state can improve embodied empathetic interaction.

Zi Haur Pang, Casey Kennington, Tatsuya Kawahara

Graduate School of Informatics, Kyoto University, Japan · Computer Science Department, Boise State University, United States

Emotion-Aware Dialogue GenerationSpoken Dialogue SystemsSocial RoboticsAffective Computing
AskPDF
Aug 14, 2026

ProFocus: Interpreting Affective Experience in Artistic Images with Progressive Visual Focusing

Interpreting the emotional responses triggered by images is central to achieving emotional intelligence. Compared with natural images, visual art is intentionally created to elicit emotional responses from its viewers through abstract concepts and visual metaphors, making affective interpretation particularly challenging. However, most existing methods rely on general-purpose visual embeddings (e.g., CLIP), failing to capture the nuanced cues underlying artistic emotion. To address this gap, we propose ProFocus, a novel framework that models affective experience in artistic images via progressive visual focusing. The key idea is to model visual representation learning inspired by a hierarchical cognitive theory of human aesthetic appreciation. Technically, ProFocus contains two core components: a Hierarchical Art Critic (HAC) and a Progressive Hint Fusion (PHF) module. HAC leverages multimodal large language models to generate structured linguistic priors at three cognitive levels--atmospheric style, narrative subjects, and concrete details--thereby translating artistic perception into coherent semantic guidance. Building upon these priors, PHF departs from conventional cross-modal fusion by sequentially injecting the hierarchical hints into visual features, enabling a progressive focusing process that mirrors human perception. This design allows the model to capture subtle affective cues and produce more faithful explanations. Extensive experiments on the ArtEmis v1.0 and v2.0 datasets demonstrate that ProFocus consistently outperforms state-of-the-art methods in both emotion recognition and affective explanation. Project page: https://github.com/Zhang-Zhiyan/ProFocus.

Zhiyan Zhang, Zicheng Yan, Jianqi Chen +3

University of Science and Technology of China Hefei, Anhui, China · Anhui University Hefei, Anhui, China

Emotion RecognitionAffective ComputingVisual Representation Learning
AskPDF
Aug 7, 2026

PsychoAgent: An Affect-Sensitive Cognitive Architecture for Conflict-Aware Memory in LLM Agents

Human-like cognition does not select past experience by topical similarity alone: affective significance and unresolved conflict also shape what becomes accessible. We present PsychoAgent, a cognitive architecture for LLM agents that separates factual and affective memory and integrates both through a conflict-aware executive controller. Affective memories are first filtered by semantic relevance and then re-ranked by salience, preserving topical fit while allowing emotionally important traces to enter the prompt. Across three controlled conflict scenarios, the full architecture retrieved more conflict-critical memories than semantic-affective and single-memory RAG baselines (0.933 vs. 0.500 and 0.667), with a small semantic-similarity cost. Five blinded raters evaluated 27 outputs. After within-rater standardization, the full architecture had the highest overall mean (+0.22 SD), but corrected pairwise differences were not significant. A three-day illustrative trace further shows persistent affect, offline memory recombination, and selective memory reweighting. The findings support affect-sensitive retrieval as an inspectable mechanism for modeling human-like conflict effects in LLM agents.

Mohammad Amanlou, Parham Abed Azad, Farbod Davoodi +3

School of Electrical and Computer Engineering, College of Engineering, University of Tehran, Tehran, Iran. · Department of Computer Engineering, Sharif University of Technology, Tehran, Iran. · Missouri University of Science and Technology, Rolla, MO, USA. +1

Cognitive Architectures for AI AgentsLLM Agent MemoryLLM MemoryAffective ComputingMemory-Augmented Language Model Agents
AskPDF
Aug 5, 2026

NTDH: Complex Reasoning for Comprehensive Affective Analysis

Comprehensive affective analysis is challenging for two reasons: it spans heterogeneous prediction tasks with continuous, ordinal, and multi-label outputs, and affective meaning is context-dependent, requiring conflicting cues to be reconciled rather than mapped directly to labels. Existing methods learn this mapping directly and do not model the reconciliation explicitly. We recast the task as a complex-reasoning problem, which yields one output interface across heterogeneous label spaces and a trajectory over which a verifiable reward can be optimised; to our knowledge, this is the first such treatment covering both sentiment and emotion. The obstacle is on the data side: affective reasoning traces must be synthesised, and generic synthesis is misaligned with the targets, tolerances, and phenomena of affect, and discards or leaks its failure cases. We propose NTDH, which addresses these four failures. Naturalisation sets the training answer to the gold label, so it is correct by construction. A Tolerance-aware gate checks each answer against the task's own scoring margin. Domain-aware strategies refine the reasoning using ideas from affective science. Directional Hints report only the type and direction of an error, without exposing the target. We train Qwen3-8B with SFT and then GRPO under the same tolerance used for verification (up to a more permissive construction gate on the multi-label subtask), and a component ablation quantifies the data-quality effect of each part. Using 16,302 training records, about 14x fewer than comparable instruction-tuned systems, the final policy improves over its SFT checkpoint on five of six official-test metrics and achieves the strongest EI-reg result among the compared systems, at a Pearson correlation of 0.862.

Tianlei Zhu, Zhiwei Liu, Yuyan Wang +2

Columbia University, New York, NY, USA · Department of Computer Science, The University of Manchester, Manchester, United Kingdom

CoT ReasoningAffective Computing
AskPDF
Jul 30, 2026

Learning to Predict Performance-induced Emotion Differences in Classical Piano Music

Music is often used as a medium for communicating emotion, with performers shaping perceived affect through interpretation. This study addresses the challenge of identifying and predicting subtle changes in perceived emotion that are exclusively due to differences in performance. We focus on classical solo piano music, using a set of 6 commercial recordings of Bach's Well-Tempered Clavier Book I, annotated in terms of valence and arousal. By encoding the recordings through performance-specific features only, we isolate performance information from aspects of the composition itself, which tend to dominate the overall perceived emotional category. A preliminary analysis validates that these features vary meaningfully across performers. We then propose a relative regression framework, Delta-VA, to predict deviations in valence-arousal relative to an ``average'' performance, thereby focusing on the changes in emotion brought about by a specific way of playing a piece. In addition to the standard R2R^2 regression score, we introduce geometric evaluation metrics to assess the preservation of pairwise differences between performances. Results indicate high directional consistency with the ground truth, but also a compression in prediction magnitude, indicating that the model tends to underestimate expressive performance effects.

Joann Ching, Gerhard Widmer

Institute of Computational Perception, Johannes Kepler University Linz, Austria · LIT AI Lab, Linz Institute of Technology, Austria

Valence-Arousal ModelingEmotion RecognitionAffective Computing
AskPDF
Jul 28, 2026

Knowledge-Guided Multimodal Reasoning over Interacting Streams for Video-Level Ambivalence and Hesitancy Recognition

Ambivalence and hesitancy (A/H) are conflicting affective states that precede the delay or abandonment of health behaviour change. Recognition of A/H at the video level is difficult, since the signal arises from disagreement across and within facial, vocal, linguistic, and bodily modalities, and manifests differently across individuals. The proposed PRISM-AH (Predictive Reasoning over Interacting Streams for Multimodal Ambivalence/Hesitancy Recognition), is a framework that treats A/H as a multimodal conflict that unfolds over time. Frozen vision, audio, and text encoders are aligned into short time windows and passed to a lightweight streaming model that scores cross-modal dissonance, predicts each next window to expose a hesitation surprise signal, discovers behaviour prototypes, and is conditioned on participant metadata. Dense window-level annotations supervise the model as an auxiliary objective, and the decision threshold is calibrated for macro F1. A knowledge-guided large language model then reasons over structured evidence using the expert cue taxonomy of the dataset, and its verdict is fused late only when validation performance improves. On the labelled public test partition of 525 videos, PRISM-AH attains a macro F1 of 0.6133, compared to the reported zero-shot baseline of 0.2827. The reasoning gain is validated to transfer from validation to the larger test partition.

Podakanti Satyajith Chary, Barath Parthiban, Pranesh Velmurugan +2

Department of Engineering Science, IIT Hyderabad, India · Department of Biomedical Engineering, IIT Hyderabad, India

Multimodal ReasoningMultimodal Emotion RecognitionAmbivalence and Hesitancy RecognitionAffective ComputingTemporal Video Understanding
AskPDF
Jul 25, 2026

Ordered Network Analysis of Epistemic Emotions during Collaborative Problem Solving

Investigating how affective states such as confusion and frustration persist and transition during co-situated collaborative problem solving (CPS) is important for understanding the dynamics of epistemic emotions. However, the accurate identification of affective states remain challenging as there is no gold-standard truth in this space. Here, we analyze affective states collected through retrospective cued-recall during an in-person CPS task. Using ordered network analysis (ONA), we examine (1) the overall ordered structure of affective states and how this structure differs across self-caught and probe-caught reporting methods, and (2) what aspects of this ordered structure are emphasized differently in slower and faster groups. We find that ONA reveals differences in persistence and transition patterns that are not apparent from descriptive summaries alone. In particular, we observe a stable epistemic core linking curiosity, optimism, and confusion, with different reporting methods emphasizing different connections among states. An analysis between faster and slower groups show that roles of confusion and disengagement also shift significantly during collaboration, particularly in their relationship to conflict. We interpret our findings in the context of collaboration and discuss their implications in developing AI systems that support CPS.

Sifatul Anindho, Videep Venkatesha, Jaclyn Ocumpaugh +1

Colorado State University, Fort Collins CO 80523, USA · University of Houston, Houston TX 77104, USA

Affective Computing
AskPDF
Jul 23, 2026

Explainable graph attention network for stress recognition (StressGAT) via differential action units

Stress is a dynamic process characterized by significant individual variability in facial expression. Traditional architectures, such as Recurrent Neural Networks (RNNs) and Convolutional Neural Networks (CNNs), often overlook person-specific baselines or lack the representational capacity to model the non-linear temporal progression of distress due to sequential bottlenecks and rigid grid-based constraints. Furthermore, many deep learning models lack the interpretability required for clinical deployment. This study introduces StressGAT, a Graph Attention Network that leverages the relational inductive bias of graph modeling to capture complex facial dynamics that indicate acute stress. By using Differential Action Units, the framework normalizes individual responses relative to neutral baselines to achieve personalized recognition. The proposed model achieves 88.62% accuracy on a diverse stress-induction cohort (58 participants) using a subject-independent, Leave-One-Subject-Out (LOSO) cross-validation protocol. Beyond predictive accuracy, the architecture integrates a Multiple Instance Learning (MIL) attention mechanism to identify peak stress intervals and reveal distinct expressivity phenotypes. By simultaneously optimizing for accuracy and interpretability, this framework provides a robust, explainable solution for personalized affective monitoring.

Thomas Kassiotis, Stefanos Gkikas, Nikolaos Smyrnis +1

Department of Electronic Engineering Hellenic Mediterranean University Chania, Greece · Honda Research Institute Japan Wako City, Japan · University Mental Health, Neurosciences and Precision Medicine, Research Institute “COSTAS STEFANIS” 2nd Department of Psychiatry, Medical School, National and Kapodistrian University of Athens Athens, Greece +1

Graph Attention NetworksFacial Expression RecognitionAffective Computing
AskPDF
Jul 21, 2026

Associative Emotional Learning in Convolutional Neural Networks

Associative emotional learning enables organisms to adaptively link pleasant or unpleasant outcomes to the presence of predictive stimuli. Whereas computational models such as the Rescorla-Wagner model have shed light on this important function, the limitations of these models are also known, especially when they are applied to neural data. The advent of deep neural networks has opened another avenue for modeling associative emotional learning. In this work we proposed a deep neural network model of visual valence processing, consisting of a visual module that encodes complex natural scenes and a module that recognizes their emotional significance in terms of valence, a key dimension of emotion, and tested a novel Pavlovian learning paradigm on the model. The results showed that with learning, the model reproduced several observations from human associative learning studies, including association formation and generalization, and that the neural representations of the conditioned and the unconditioned stimuli became increasingly aligned both at the single unit and at the neural population level. Comparison between the model and human experimental data provided further validation of our approach. This study thus suggests that deep neural network models, when combined with appropriate learning algorithms, can be used to model behavioral and neural signatures of associative emotion/valence learning.

Seowung Leem, Andreas Keil, Mingzhou Ding +1

J. Crayton Pruitt Family Dept. of Biomedical Engineering, University of Florida, Gainesville, FL 32611, USA · Center for the Study of Emotion and Attention, University of Florida, Gainesville, Florida, USA · Department of Electrical and Computer Engineering, University of Florida, Gainesville, FL 32611, USA +3

Convolutional Neural NetworksAssociative MemoryAffective Computing
AskPDF
Jul 16, 2026

Team RAS in 11th ABAW Competition: Multimodal Ambivalence Recognition Approach

Automatic recognition of ambivalence and hesitancy is challenging because these states may be expressed through inconsistent linguistic, acoustic, facial, and contextual patterns, while top-performing systems often rely on computationally expensive ensembles. We present a single text-centered multimodal approach for video-level ambivalence and hesitancy recognition for the 11th Affective & Behavior Analysis in-the-Wild (ABAW) Challenge. The proposed approach combines linguistic, acoustic, facial, and scene features using text-centered multimodal fusion model. Text Residual Fusion treats text as the anchor modality and applies gated residual adjustments based on the other modalities. Experiments on the Behavioural Ambivalence/Hesitancy (BAH) corpus confirm that text is the strongest unimodal modality. The Text Residual Fusion model achieves an average Macro F1-score (MF1) of 75.14% across the Development and Public Test subsets. On the Private Test subset, it reaches an MF1 of 78.24%, outperforming the text model by 4.03%. These results demonstrate that complementary multimodal information can improve recognition performance without requiring a large model ensemble.

Elena Ryumina, Maxim Markitantov, Alexandr Axyonov +4

St. Petersburg Federal Research Center of the Russian Academy of Sciences · St. Petersburg Federal Research Center of the Russian Academy of Sciences (SPC RAS), St. Petersburg, Russia · HSE University, St. Petersburg, Russia +1

Multimodal FusionAmbivalence and Hesitancy RecognitionAffective Computing
AskPDF
Jul 15, 2026

Audio-Text Cross-Attention with Psycholinguistic Support Features for Ambivalence/Hesitancy Recognition

We present a frame-independent audio-text system for the 3rd Ambivalence/Hesitancy Video Recognition Challenge at the 11th Affective & Behavior Analysis in-the-Wild (ABAW) Workshop. Videos are divided into overlapping 5-s windows aligned with transcript timestamps. Each window combines prosodic audio descriptors, emotion-oriented RoBERTa embeddings, and 74 psycholinguistic features representing uncertainty, hedging, and attitudinal conflict. Temporal cross-attention fuses audio and text, while the support features condition gated Multiple Instance Learning (MIL) pooling. A five-seed ensemble achieves an average precision of 0.875 and a macro-F1 of 0.722 on the 525-video labeled public-test split. Notably, our submission ranked third overall on the official challenge leaderboard, with a macro-F1 of 0.7455. Source code is available at https://github.com/Liga-de-IA-PUCPR/abaw-11-ah-challenge/.

Luiz F. B. F. Martins, Rodrigo W. Pisaia, Matheus M. Girardi +5

Pontifical Catholic University of Paraná, Curitiba, Brazil

Ambivalence and Hesitancy RecognitionAffective Computing
AskPDF
Jul 14, 2026

AffectFlow-DINO: Uncertainty-Aware Multi-Task Affect Estimation via Conditional Rectified Flow

We present AffectFlow-DINO, a multi-task learning system for the 11th ABAW challenge that extends a standard deterministic architecture with a conditional rectified-flow head to model the inherent ambiguity of in-the-wild facial behavior. Instead of predicting a single affect estimate, the model learns a conditional generative distribution, enabling uncertainty-aware one-to-many predictions through Monte Carlo sampling. The system jointly estimates continuous valence-arousal, classifies eight facial expressions, and detects twelve Action Units from static face images. Built on a frozen DINOv3 ViT-S/16 backbone, extensive ablation studies show that rectified-flow decoding consistently improves deterministic prediction, particularly for valence-arousal estimation (CCC-V +0.058+0.058). We further show that post-hoc threshold calibration effectively recovers performance on severely imbalanced rare classes (e.g., Fear: 3.8%→33.1%3.8\% \rightarrow 33.1\%) without retraining. Combined with backbone fine-tuning and flow retuning, the final model achieves PMTL=1.177\mathbf{P_{MTL}=1.177}, substantially outperforming the official challenge baseline of PMTL=0.45P_{MTL}=0.45.

Salah Eddine Bekhouche, Abdellah Zakaria Sellam, Fadi Dornaika +1

University of the Basque Country (UPV/EHU), Spain · Department of Innovation Engineering, University of Salento & Institute of Applied Sciences and Intelligent Systems (CNR), Italy · Faculty of Data Science and Computing, Universiti Malaysia Kelantan, Malaysia

Multi-Task LearningFacial Action Unit DetectionValence-Arousal ModelingFacial Expression RecognitionFlow-Based Generative ModelingEmotion Recognition+1
AskPDF
Jul 13, 2026

A Calibrated Multimodal Ensemble for Ambivalence/Hesitancy Recognition: System Description and Private-Test Submission Strategy

Ambivalence and hesitancy (A/H) undermine digital behaviour-change interventions, and recognizing them automatically from video is the goal of the ABAW A/H challenge on the BAH dataset. We describe our system for the 11th edition of the challenge: a calibrated, equal-weight ensemble of three fusion models over frozen face, audio, text, and pose embeddings, which reaches 0.7358 macro-F1 on the public test set. This year's private test, released on a disjoint set of 30 new participants, is scored on five allowed submissions; we report the configuration and rationale of each of our five submissions, and, where already available, the private-test score obtained. Our first submission, an exact replica of the calibrated ensemble tuned only on public validation, scored 0.7361 macro-F1 on the private test, matching our public-test estimate almost exactly and confirming the pipeline generalizes to unseen participants without leakage.

Josep Cabacas-Maso, Ismael Benito-Altamirano, Carles Ventura

eHealth Center, Faculty of Computer Science, Multimedia and Telecommunicactions, Universitat Oberta de Catalunya, 08016 Barcelona, Spain · MIND/IN2UB, Department of Electronic and Biomedical Engineeering, Universitat de Barcelona, 08028 Barcelona, Spain · Institute of Semiconductor Technology (IHT) & Laboratory for Emerging Nanometrology (LENA), Technische Universität Braunschweig, Braunschweig, Germany

Ambivalence and Hesitancy RecognitionAffective Computing
AskPDF
Jul 13, 2026

Causal Supervision of Attention for Affective Behaviour Analysis

The 11th Affective Behaviour Analysis in-the-wild Competition includes the Multi-Task Learning Challenge, where participants develop a unified framework for Valence-Arousal Estimation, Expression Recognition, and Action Unit Detection. The challenge lies in learning emotion-related representations that generalize across subjects while remaining robust to spurious factors such as identity, illumination, pose, and demographic variation. To aggregate features extracted by a pre-trained backbone into a compact representation for prediction, attention mechanisms selectively weight the most informative facial regions. However, these attention weights can still capture dataset-specific correlations rather than genuine affective cues. To address this limitation, we propose an attention pooling framework that combines causal supervision with cross-covariance regularization of attention components, encouraging subject-invariant attention and non-redundant representations that improve generalization. Our method achieves CCCVA=0.5123CCC_{VA}=0.5123 for VA estimation on the official validation set, together with FEX=0.3116F_{EX}=0.3116 and FAU=0.3974F_{AU}=0.3974 for expression recognition and action unit detection, respectively, resulting in an overall PP score (the sum of the individual task metrics) of 1.22141.2214.

Nemanja Rašajski, Konstantinos Makantasis, Antonios Liapis +1

Institute of Digital Games, University of Malta, Malta · AI Department, University of Malta, Malta

Causal Representation LearningFacial Action Unit DetectionValence-Arousal ModelingFacial Expression RecognitionAttention MechanismsEmotion Recognition+1
AskPDF
Jul 13, 2026

Simple Features and Honest Calibration for Ambivalence and Hesitancy Recognition in Video

We address ambivalence and hesitancy (A/H) recognition in the ABAW 2026 BAH Challenge: given a short interview video, predict whether the person shows signs of A/H. Our system combines affect-specialised text, audio, and visual representations with a small set of readable linguistic hesitation cues, fused by a reliability gate we call Affective Marker Fusion (AMF), and finished with a simple AP-weighted ensemble at a fixed decision threshold. We also introduce ASR-erased time: speech recognisers delete fillers and hesitation pauses from the transcript, but the chunk timestamps keep the time those events took, and sixteen features built from these gaps form the strongest and most independent non-verbal channel we measured (AP 0.7180.718, correlation 0.110.11--0.360.36 with all other members). Across controlled experiments we find three things: cross-modal conflict design does not reliably help on BAH; language is by far the strongest channel while affect-specialised audio is a useful second; and calibration matters more than architecture. Fitting ensemble weights and a threshold on the small validation split overfits: it scores 0.7410.741 macro-F1 on validation but only 0.6900.690 on the untouched test set. AP-weighting at a fixed threshold instead reaches 0.731\mathbf{0.731} on test.

Vikas Kumar, Aditya Mishra, Haroon R. Lone

Indian Institute of Science Education and Research Bhopal, India

Multimodal Emotion RecognitionAmbivalence and Hesitancy RecognitionAffective Computing
AskPDF
Jul 12, 2026

Personalized Emotional Intelligence in Generative AI through Symbolic Affective Reasoning

Emotional intelligence enables humans to recognize emotions, infer their causes, reason about interventions, and modify their environment to achieve desired affective states. Despite recent advances in artificial intelligence (AI), current models remain largely limited to generating realistic content or performing semantic reasoning, with little capacity for understanding, predicting, and personalizing human emotional responses. Here we introduce Emotion-augmented geneRatiOn System (EROS), a hybrid AI framework that integrates symbolic reasoning with deep learning to enable personalized emotion augmentation through visual content. Leveraging large-scale image-emotion datasets, EROS discovers generalizable affective rules, identifies emotion-relevant image regions, and predicts context-aware visual modifications that preserve scene semantics while steering emotional responses toward desired targets. To account for individual variability, EROS incorporates an expandable memory bank that supports inference-time personalization without model fine-tuning, yielding interpretable emotional profiles and rapid adaptation to new users. Across extensive human psychophysics experiments, EROS elicits target emotional responses more effectively than state-of-the-art large multimodal models while adapting to individual affective preferences. Beyond affective computing, EROS provides a foundation for AI systems that can understand, reason about, and augment human cognitive states, with potential applications in mental health, adaptive media, education, and human-computer interaction.

Qing Lin, Mengmi Zhang

College of Computing and Data Science, Nanyang Technological University, Singapore

LLM PersonalizationNeuro-Symbolic AIAffective Computing
AskPDF
Jul 11, 2026

Benchmarking Dynamic Affective Reasoning: A Viewer-Centric Video Emotion Dataset

Video emotion analysis is typically framed as a static classification problem, treating each clip as an independent labeled unit. However, such a formulation overlooks a key psychological fact: emotions change as a result of cumulative reactions to consecutive causal events. To bridge this gap, we introduce Dynamic Affective Reasoning, the first large-scale benchmark for viewer-centric affect transitions and causal reasoning over consecutive video events. DAR contains 15,087 videos and 36,908 event-aligned affective segments annotated with 27 emotion categories. Unlike existing video-based emotion datasets, DAR presents a new viewer-centric perspective on fine-grained emotional expressions and transitions, and provides dense, temporally grounded, and causally explicit reasoning chains. Based on DAR, we formally define three challenging tasks: affective segmentation, fine-grained emotion classification, and affective reasoning. Complementing this benchmark, we propose DAR-R1, a two-stage framework that combines supervised fine-tuning with Group Relative Policy Optimization. Experiments across 10+ MLLMs show that DAR-R1 sets a new state-of-the-art for dynamic affective reasoning, in terms of both emotional localization and affective reasoning. Project page: https://github.com/Zhang-Zhiyan/DAR.

Zhiyan Zhang, Peipei Song, Jinpeng Hu +3

University of Science and Technology of China, Hefei, China · Hefei University of Technology, Hefei, China

Video ReasoningCausal Reasoning in VideosEmotion RecognitionAffective Computing
AskPDF
Jul 8, 2026

Video2Reaction: Mapping Video to Audience Reaction Distribution in the Wild

Understanding and forecasting audience reactions to video content are crucial for improving content creation, recommendation systems, and media analysis. To enable audience reaction prediction and other content engagement applications, we introduce Video2Reaction\textbf{Video2Reaction}, a multimodal dataset that maps short movie segments to a distribution of induced emotions\textit{induced emotions} of viewers in the wild, as expressed through social media. Video2Reaction\textbf{Video2Reaction} spans more than 10,000 videos and serves as a reliable benchmark as well as a training resource for audience reaction prediction. To enable cost-effective continuous annotations as reactions may change over time, we develop a two-stage multi-agent pipeline using only open-source LLMs, achieving 86% correctness under blind human verification despite the inherently noisy and subjective nature of the task. We establish the first benchmark for video-to-reaction-distribution prediction in the wild and show that pretrained foundation video models fail in zero-shot settings, while finetuning transforms them into state-of-the-art predictors capable of modeling both full reaction distributions and dominant responses from video alone. However, the task remains challenging: even the strongest methods achieve only 77% Top-3 F1 in dominant reaction prediction (LLaVA-Next), highlighting a substantial gap in modeling collective audience reaction. \modification{Dataset and code are available at our project page: https://information-fusion-lab-umass.github.io/video2reaction-bench.github.io

Trang Nguyen, Sidong Zhang, Shiv Shankar +4

University of Massachusetts Amherst, 130 Governors Drive, Amherst, MA, USA · Dolby Laboratories, 1275 Market Street, San Francisco, CA, USA

Multimodal Emotion RecognitionAffective Computing
AskPDF
Jul 3, 2026

Where do LLMs Fall Short in CBT-Guided Affective Reasoning?

Cognitive Behavioral Therapy (CBT) provides a structured framework for understanding a user's mental state by examining the interaction between cognitive and behavioral factors. However, out-of-the-box LLMs respond fluently and empathetically, yet collapse into validation & reflection, regardless of what the user actually needs. They know theoretical CBT (scoring up to 96% accuracy on licensing exam questions) but fail to apply it effectively. We explore this gap with a knowledge-guided framework that treats CBT dialogue as controlled affective reasoning: user narratives are decomposed into Beck's Cognitive Conceptualization structure, grounded in clinical SNOMED CT concepts validated via Natural Language Inference, and a Multiple Chain-of-Thought (MCoT) strategy selection between Validation & Reflection, Socratic Questioning, or Alternative Perspectives. To measure whether such guidance actually changes behavior, we introduce the Protocol Leverage Force (F), a behavior-level metric that captures how far an intervention shifts a model away from its default response. Across three open-weight LLMs and 14 RealCBT-derived case studies, evaluated with human experts, valence-arousal trajectories, and linguistic entrainment, F shows that simply introducing protocol definitions via single chain-of-thought prompting fails to change LLM behavior, while MCoT on these definitions guides strategy selection better. Still, the effect stays within 1% (approx. 1.2-1.3%), and all models remain biased toward Validation & Reflection. These results show CBT knowledge alone does not ensure effective application, giving the affective-computing community instrumentation to measure where LLMs fall short.

Vaishnavi Sinha, Pooja Guttal, Pranay Deep Reddy Katike +5

Department of Computer Science and Electrical Engineering, UMBC · Department of Electrical Engineering and Computer Science, UC Irvine · Department of Psychology, UMBC +1

Mental HealthClinical Language Model EvaluationAffective ComputingCognitive Behavioral Therapy
AskPDF
Jul 1, 2026

Automatic Detection of Stress from Speech in the Trier Social Stress Test

Automatically detecting stress in speech provides an unobtrusive way to gain insights relevant to behavioral research or clinical assessment. This study investigates the automatic differentiation between a stressful and non-stressful situation, and the prediction of physiological and affective stress responses. Speech data was collected from 50 participants who either completed the Trier Social Stress Test (TSST) or a non-stressful control condition. With a processing pipeline that included speaker diarization and machine learning models, we achieved stress detection performance significantly above a mean baseline. Moreover, relevant physiological and affective stress responses were partially predictable from acoustic-prosodic features. Feature-importance analyses identified the most informative predictors contributing to model performance. The findings demonstrate that speech can serve as a meaningful and unobtrusive indicator of multiple dimensions of the human stress response.

Hanna Drimalla, Wieland R. Cremer, Christine Kraus +1

Human-Centered Artificial Intelligence Group, Faculty of Technology, Bielefeld University, Bielefeld, Germany. · Department of Cognitive Psychology, Faculty of Psychology, Ruhr University Bochum, Bochum, Germany.

Speech Emotion RecognitionAffective Computing
AskPDF
Jun 30, 2026

Who Determines the Meaning of an Emotion? Affective Sovereignty as an Epistemic Consequence of Measurement Limits

Emotion-sensing AI is rapidly becoming embedded in vehicles, home appliances, dialogue agents, and social infrastructure, giving rise to a sphere in which emotion is no longer confined to individual experience but is instead observed and computed at a societal scale, a domain we term the Affectosphere. Yet a central normative question in this domain has remained underexplored: who has the final authority to determine the meaning of one's own emotion? This study addresses the question from the epistemological side of measurement's structural limits. We define a meaning distribution as the distribution of labels assigned by annotators drawn from a population under a fixed annotation protocol, and decompose its uncertainty into reducible and irreducible components. We then demonstrate that, while emotion AI can assign high-confidence point labels and discriminate real differences at an aggregate level, the irreducible component of the meaning distribution for individual instances cannot be estimated with adequate coverage under realistic annotator counts, a systematic divergence we term the epistemic gap. The key finding is that high device confidence does not constitute evidence that irrecoverable meaning has been recovered. From this epistemic gap, together with an explicitly stated normative premise, namely that the output of a system which cannot recover a quantity in principle must not be treated as its authoritative determination, we derive the norm that the final interpretive authority over the meaning of one's emotion is procedurally reserved for the experiencing subject, the norm of affective sovereignty. These results suggest that the design, evaluation, and regulation of emotion AI should place explicit allocation of interpretive authority, rather than accuracy maximisation, at their core.

Keito Inoshita

Faculty of Business and Commerce, Kansai University, 3-3-35 Yamate-cho, Suita, 564-8680, Osaka, Japan.

Human Oversight of AIEmotion RecognitionAffective ComputingEpistemic Uncertainty
AskPDF
Jun 27, 2026

A Comparative Study on Affective Cues in Text Embeddings Across Psychological Emotion Theories

Text encoders are known for their utility in natural language processing, as they are able to efficiently compress inputs into dense vectors while preserving semantics. These models have been applied to affective computing, in particular to help with solving sentiment analysis and emotion recognition tasks. Nevertheless, it remains unclear to what extent the latent representations produced by modern text encoders capture well-defined psychological theories of affect. In this work, we investigate the affective capabilities of twelve recently released text encoders by probing their generated embeddings as input features for solving regression and classification tasks across three established emotion frameworks, using both word- and sentence-level data. Additionally, we apply a semantic data-leakage prevention technique to improve robustness in word-level evaluations. Our main findings show that the latent manifolds of the latest instruction-aware open-weight encoders enclose an equal or even a larger amount of affective information in comparison with proprietary counterparts when evaluated at word level. In contrast, embeddings of task-tuned and proprietary encoders reach the highest scores on sentence-level affective classification. Furthermore, a qualitative analysis of latent representations and their encoded affective cues is provided.

Fabio Ciani, Harald Schweiger, Emilia Parada-Cabaleiro +1

Department of Electronics, Information, and Bioengineering, Polytechnic University of Milan · Multimedia Mining and Search Group, Institute of Computational Perception, Johannes Kepler University Linz · Department of Music Pedagogy, Nuremberg University of Music +1

Text EmbeddingsText Embedding EvaluationText Embedding ModelsEmotion RecognitionAffective Computing
AskPDF
Jun 27, 2026

Generative Learning as a Tool to Improve Perception of Emotional Body Motion Expressions

Emotional body motion expressions are an essential element of non-verbal communication. Effectively conveying these expressions through technology is of utmost importance, for example, with virtual reality avatars and in social robotics. Recent advances in generative models have opened new opportunities for advancing research on emotional body motion learning. However, generating accurate emotional expression representations is challenging, given the subtlety of emotional cues, individual variability, and cultural differences. We investigate whether a generative model can implicitly learn emotional body motions directly from culturally grounded motion-capture data, without explicit emotion-motion guidance. Using a dataset of emotional performances by 49 Japanese actors, we trained a Transformer-based generative model to generate expressive motions conditioned on 13 discrete emotion labels. We evaluate the generated motions from two perspectives: (1) an LSTM-based classifier to assess recognizability by machine observers, achieving a recognition accuracy of 22.80%, and (2) a human perception study with Japanese raters to assess alignment with human affective interpretations, yielding a recognition accuracy of 24.91%. Beyond these, we evaluate the utility of generative modeling for three practical tasks: augmenting emotion recognition models, extracting representative emotion-specific motion patterns, and synthesizing smooth transitions between emotion intensities. Our findings highlight the potential of implicit, data-driven generative modeling to enhance affective computing applications and our understanding of emotion expressions.

Huakun Liu, Miao Cheng, Xin Wei +6

Nara Institute of Science and Technology · Tohoku University

Human Motion GenerationConditional Generative ModelingEmotion RecognitionAffective Computing
AskPDF
Jun 25, 2026

State-specific respiratory signatures for affective and stress recognition: Interpretable respiratory markers, autocorrelation lags, and compact CNN models

Respiratory activity is a direct and interpretable physiological channel for wearable stress and affective-state recognition, yet many studies emphasize classification accuracy without identifying which respiratory properties separate different states. Using the chest respiratory channel of the WESAD dataset, we analyze 60 s windows under leave-one-subject-out validation and combine two complementary branches: compact raw-signal one-dimensional convolutional neural networks (1D-CNNs) and physically grouped handcrafted respiratory signatures. The primary task is stress versus non-stress detection, while baseline, stress, amusement, and meditation are additionally analyzed in a one-vs-rest setting to reveal state-specific respiratory markers. The feature space is organized into respiratory timing, breath-to-breath variability, waveform statistics, spectral/time-frequency descriptors, and autocorrelation/nonlinear predictability descriptors, with the raw 60 s signal treated as a sixth representation for the CNN branch. We introduce autocorrelation transition lags (Zpm/Zmp) as interpretable markers of respiratory correlation scale and separately evaluate exploratory forecast-error-growth/Lyapunov-like descriptors. In the final CNN refit setting, the raw-signal model achieved the strongest stress-vs-rest performance (accuracy 96.72%, macro-F1 95.30%, MCC 90.61%). In contrast, compact feature models were stronger for baseline (MCC 65.34%), amusement (MCC 35.69%), and especially meditation (MCC 88.65%). Strict nested reanalysis of the auxiliary Top-20 stress search yielded mean MCC=82.29% versus 85.45% originally, indicating modest selection optimism. These results show that CNNs are most useful for the practical stress detector, whereas interpretable respiratory signatures provide stronger and more physiologically transparent state-specific markers for several non-stress conditions.

Andrei Velichko, Mehmet Tahir Huyut

Institute of Physics and Technology, Petrozavodsk State University, Petrozavodsk, Russia · Department of Biostatistics and Medical Informatics, Faculty of Medicine, Erzincan Binali Yildirim University, Erzincan, Turkey

Wearable SensingConvolutional Neural NetworksAffective Computing
AskPDF
Jun 15, 2026

Data-Driven Decoding of Russell's Circumplex Model of Affect

Affective computing increasingly relies on deep learning to represent emotions, yet latent spaces often remain opaque, high-dimensional black boxes. This paper investigates whether Transformers' embeddings recover the geometric regularities of Russell's circumplex model. We unify two complementary experiments testing the hypothesis that, after training models on text and speech, their resulting latent spaces encode a topology consistent with valence-arousal and reproduce human-like neighborhood relations. Specifically, we evaluate deep representations extracted from Transformer-based text (RoBERTa) and speech (wav2vec 2.0) encoders, along with a multimodal Transformer fusion architecture, across naturalistic datasets like MSP-Podcast and controlled LLM-generated stimuli. Our analysis reveals that multimodal fusion of text and audio yields perfect topological alignment with Russell's primary emotion ordering. Furthermore, in a zero-shot setting using generic text embeddings, projected fine-grained emotion terms fall close to their established human-mapped coordinates. Our contribution is a novel, data-driven framework for validating emotion models, demonstrating that Russell's circumplex structure is intrinsically encoded in the embeddings of these modalities rather than being solely an artifact of human labeling, thereby bridging the gap between psychological theory and representation learning.

Amdjed Belaref, Samir Sadok, Zineb Noumir +1

Alten, France · CentraleSup´elec IETR UMR CNRS 6164, France

Multimodal EmbeddingRepresentation Geometry in Language ModelsAffective Computing
AskPDF
Jun 15, 2026

From Affect Prediction to Affect Forecasting: Evidence for Distinct Information Sources in Longitudinal Text

Modeling dimensional affect in longitudinal text requires distinguishing current affect estimation from future affective change forecasting. Existing approaches often treat each text as an independent observation and apply similar assumptions to both tasks, without testing whether they rely on different information sources. This paper investigates that distinction using longitudinal self-reported ecological essays and feeling-word entries. We propose the Trait--State Affective Prediction (TSAP) framework and its temporal extension E-TSAP for per-text valence and arousal prediction, evaluated on a held-out prediction test set of 1,737 entries from 91 users. We further propose the Affective Change Forecaster Hybrid (ACF-Hybrid) for next-step affective change forecasting, evaluated on a held-out forecasting test set of 46 users. For prediction, E-TSAP achieves composite Pearson correlations of 0.670 for valence and 0.449 for arousal. For forecasting, textual representations perform worse than compact numeric trajectory baselines: the text-inclusive model achieves only r=0.316 for valence and r=0.284 for arousal, whereas a simple prior-state baseline reaches r=0.615 and r=0.670, respectively. ACF-Hybrid, using dimension-specific numeric trajectory features, achieves r=0.659 for valence and r=0.658r=0.658 for arousal. These results show that textual semantics support current affect prediction, whereas future affective change is better captured through prior numeric trajectory dynamics.

Sadia Noor, Seemab Latif, Raja Khurram Shahzad +1

School of Electrical Engineering and Computer Science (SEECS), National University of Sciences and Technology (NUST), Islamabad, Pakistan · Department of Communication, Quality Management and Information Systems, Mid Sweden University, Ostersund, Sweden

Time Series ForecastingValence-Arousal ModelingAffective Computing
AskPDF