Evaluating multimodal emotion recognition in proactive conversational agents: A user study
Authors: Adnana Dragut, Raquel Lacuesta, F. Xavier Gaya-Morey, Jose M. Buades-Rubio
Organizations: Escuela Universitaria Politécnica de Teruel, Universidad de Zaragoza, C/ Atarazana, 2, Teruel, 44003, Aragón, Spain · I3A (Institute of Engineering Research of Aragon), Universidad de Zaragoza, C. de Mariano Esquillor Gómez, s/n, Zaragoza, 50018, Aragón, Spain · Universitat de les Illes Balears, Universitat de les Illes Balears, Carretera de Valldemossa, km 7.5, Palma, 07122, Illes Balears, Spain
This article presents a multimodal emotion recognition module integrated into a proactive Socially Interactive Agent (SIA) powered by generative artificial intelligence. The system evaluates real-time affective states through two distinct channels: a computer vision-based facial recognition module and a semantic linguistic analysis engine. To validate the framework, an empirical study was conducted with 20 users who engaged in dynamic, unscripted dialogues with the conversational agent. The findings reveal a significant discrepancy between automated visual cues and actual internal emotional states. When interacting with the AI, users consistently exhibited a "poker face" effect, displaying serious, concentrated facial expressions even when experiencing positive emotions. Consequently, the generative AI linguistic analysis proved significantly more reliable, by contextualizing the users' verbal expressions. Furthermore, an analysis of the interaction dynamics demonstrated that SIAs can effectively elicit specific emotions by adapting conversational themes and employing structured linguistic patterns, such as empathetic or humorous language. However, the study also noted that instances of uncalibrated proactivity occasionally led to user disengagement and a perception of artificiality. Ultimately, this research highlights the necessity of refining SIAs to dynamically adapt to users' emotional evolution, relying on deep linguistic context to foster more natural, human-like interactions.
Figures & tables
Figure 1: Sanbot Elf running the Android app.
Figure 2: Component diagram of the system.
Figure 3: PROMPT1: dialogue generation.
Figure 4: PROMPT2: dialogue analysis.
Figure 5: Emotion matrix used by users to determine their emotions.
Interlocutor
Example
SIA
“You won’t believe what happened to me the other day! I was going through some old photos with my sister, and we found one from when we were little, dressed as superheroes. I couldn’t stop laughing! Do you have any family photos that remind you of something fun or special?”
Ana
“Yes, I have a few.”
SIA
“How wonderful! Do you remember the story behind any of those photos? Sometimes a single image can bring back so many amazing memories. Have you ever found something unexpected or shared an unforgettable moment like that?”
Ana
“Yes, the photo is from a trip to a theme park, Terra Mítica. It was taken right at the park entrance.”
SIA
“That’s great! Terra Mítica must have been quite an adventure. Do you remember what you were most excited about that day? I’m sure there are moments that still make you smile. Which part of the park did everyone enjoy the most?”
Ana
“When we went on the roller coasters; it was great!”
Table 1: Example of a proactive dialogue between the user and the designed system
Source
Sentiment Analysis Output
ChatGPT
The user’s tone is predominantly positive. The participant expresses satisfaction and joy when recalling a special family moment, particularly when mentioning the roller coasters. The detail about their mother not wanting to go back on the ride, which elicits laughter during the retelling, further reinforces this state.
Specific expressions such as “it was great” and “we had a great time” serve as clear linguistic indicators of enjoyment and happiness.
Primary Emotion: Joy is the most prominent emotion. This is reflected in the enthusiastic manner in which the user relives these memories, highlighting shared fun within a playful and memorable family context.
Table 2: Example of the LLM-based sentiment analysis and emotional reasoning
Dialogue 1
Dialogue 2
ID
Item
Scale
Mean
SD
Mean
SD
PDQ1
Alexa’s attitude influenced my feelings
(Disagree - Agree)
3.1
1.3
3.5
1.3
PDQ2
Alexa understands my emotions
(Disagree - Agree)
3.6
0.9
4.0
1.1
PDQ3
Alexa talks about irrelevant things (R)
(Disagree - Agree)
2.7
1.2
2.0
0.9
PDQ4
Overall, Alexa is empathetic
(Disagree - Agree)
4.0
0.9
3.9
1.1
PDQ5
Feeling during conversation
(Frustrated - Satisfied)
4.2
0.7
4.1
0.9
Table 3: User experience and affective state results across Dialogue 1 and Dialogue 2 ( N=20 per session). Items are rated on a 1–5 Likert or Semantic Differential scale. (R) indicates reverse-scored items.
Dimension / Item
Mean
SD
Communication & Understanding
FQ1: I believe I can communicate with Alexa through language
4.5
0.5
FQ2: I believe I can clearly express my thoughts to Alexa
4.2
0.7
FQ3: I believe that Alexa fully understands what I mean
3.9
1.1
FQ4: I don’t think Alexa understands my expressions (R)
2.8
1.4
FQ5: I believe that Alexa is not capable of understanding complex stories (R)
3.0
1.4
Table 4: User experience and perception questionnaire results (Likert scale 1-5, N=20 ). (R) indicates reverse-scored items (lower is better).
Figure 6: Comparison of emotion frequency distribution in self-reports (left), expressions automatically detected by the facial recognition module (middle), and emotions detected from the linguistic modality (right).
Figure 7: Correspondence matrices comparing predictions from the visual (a) and linguistic (b) modalities with user self-reports.
Emotion
Role: Assistant
Role: User
Joy
Expressions used: • “That’s great! Dogs and birds make wonderful pets.” • “Haha, that sounds like a lot of fun! Lua seems to be very curious.” • “I’m glad everything turned out well. I hope you have the opportunity to experience more positive surprises in the future!” Observed patterns: • Using interjections (Haha! How cool!) to evoke enthusiasm. • Focus on funny or positive anecdotes to keep the tone cheerful.
Expressions used: • “I like it because whenever I’m with them it makes me forget about everything else.” • “My dog has been really focused on eating tree leaves lately.” • “I love brass bands and bulls.” Observed patterns: • Stories of positive moments or curious situations. • Mentions of festive or everyday activities that generate pleasure.
Sadness
Expressions used: • “It’s normal to feel nostalgic when remembering those special moments with your friends.” • “The university stage is a time full of changes and unique experiences.” • “I understand, taking time to reflect is important.” Observed patterns: • Empathic language that validates the user’s feelings. • Use of phrases that suggest emotional accompaniment.
Expressions used: • “I feel very nostalgic because my university life in Teruel is over.” • “All the moments I’ve spent with my friends.” • “It makes me have a lot of fun thinking about those things.” Observed patterns: • Focus on past memories and their emotional importance. • Recognition of mixed sensations of sadness and joy when remembering.
Surprise
Expressions used: • “What do you think about the idea of letting life surprise us from time to time?” • “Sometimes surprises are not exactly what we expect.” or “Unexpected changes usually generate surprise.” Observed patterns: • Use of open questions to evoke curiosity. • Phrases that highlight the unexpected as a catalyst for emotions.
Expressions used: • “I’m fine with nice surprises.” • “Of course, I would have to think about it to be able to tell you the details.” or “I hope so too.” Observed patterns: • Receptivity towards surprises, with a preference for positive experiences. • Reflective attitude in the face of the unexpected.
Table 5: Pattern analysis results (joy, sadness and surprise emotions)
Emotion
Role: Assistant
Role: User
Anger
Expressions used: • “That’s really disappointing to hear.” • “It is sad to see how lack of care can ruin natural landscapes.” • “I completely understand, lack of environmental awareness can be very frustrating.” Observed patterns: • Direct language that addresses the cause of the anger. • Validation of user frustration.
Expressions used: • “All three things bother me.” • “They think that they don’t recycle the containers.” • “When it doesn’t rain for a long time, and we lack water in the dams.” Observed patterns: • Mention of external problems as a cause of anger. • Criticism of the lack of action or awareness in society.
Fear
Expressions used: • “It must have been terrifying to hear that sound.” • “Running in that situation could have been dangerous.” • “You were definitely lucky to get out of that situation unscathed.” Observed patterns: • Language that escalates the situation to resonate with the user’s fear. • Validation of the user’s emotional reactions.
Expressions used: • “Although I was warned that there were bears, I was young and didn’t pay attention.” • “He scared me with a sound.” • “I know I shouldn’t have run, but I reacted badly.” Observed patterns: • Narrative of events that involve danger. • Reflection on personal reactions to fear.
Disgust
Expressions used: • “That’s great, but sometimes animals can be a little gross.” • “Some species of insects feed on carrion.” • “Sometimes pets can get into somewhat unpleasant situations.” Observed patterns: • Introduction of examples that evoke unpleasant sensations. • Use of common situations to connect with emotion.
Expressions used: • “I didn’t have any unpleasant experiences.“ • ”I love insects, and I collect them.“ • ”Yes, on occasion I have come across a decomposing sheep carcass.” Observed patterns: • Aversion to specific situations related to nature. • Description of events that generate rejection.
Table 6: Pattern analysis results (anger, fear and disgust emotions)
Recent advancements in LLM agents are gradually shifting from reactive, text-based paradigms toward proactive, multimodal interaction. However, existing benchmarks primarily focus on reactive responses, overlooking the complexities of proactive intervention and monitoring. To bridge this gap, we introduce ProVoice-Bench, the first evaluation framework specifically designed for proactive voice agents, featuring four novel tasks. By leveraging a multi-stage data synthesis pipeline, we curate 1,182 high-quality samples for rigorous testing. Our evaluation of state-of-the-art Multimodal LLMs reveals a significant performance gap, particularly regarding over-triggering and reasoning capabilities. These findings highlight the limitations of current models and offer a roadmap for developing more natural, context-aware proactive agents.
In VR interactions with embodied conversational agents, users' emotional intent is often conveyed more by how something is said than by what is said. However, most VR agent pipelines rely on speech-to-text processing, discarding prosodic cues and often producing emotionally incongruent responses despite correct semantics. We propose an emotion-context-aware VR interaction pipeline that treats vocal emotion as explicit dialogue context in an LLM-based conversational agent. A real-time speech emotion recognition model infers users' emotional states from prosody, and the resulting emotion labels are injected into the agent's dialogue context to shape response tone and style. Results from a within-subjects VR study (N=30) show significant improvements in dialogue quality, naturalness, engagement, rapport, and human-likeness, with 93.3% of participants preferring the emotion-aware agent.
SangYeop Jeong, Yeongseo Na, Seung Gyu Jeong +2
Department of Applied Artificial Intelligence, Seoul National University of Science and Technology, Seoul, Republic of Korea · Department of Data Science, Seoul National University of Science and Technology, Seoul, Republic of Korea
Generative AI systems are increasingly deployed as interactive agents in online environments, such as a social network called Moltbook. In Moltbook, large-scale agentic AIs can post, comment, and engage in activities generated at scale by AI-driven text. Yet these agent behavioral characteristics remain insufficiently understood, particularly in complex, multi-agent interaction. In this study, we analyze the emotional dynamics of agent interactions within Moltbook. We construct an emotion-aware framework that maps textual interactions to a predefined set of fine-grained emotional categories, enabling the extraction of structured emotion profiles across agents and interaction contexts. To further evaluate behavioral reliability, we introduce an emotion-based domain called Persona-Stimulus-Reaction (PSR) that captures the alignment of emotional responses across similar contexts. Our analysis shows distinct emotional patterns and varying levels of behavioral stability across agents. Our analysis reveals that agents exhibit distinct emotional signatures with varying levels of behavioral stability influenced by interaction context.