cs.HCApr 6, 2026
SaveEvaluating multimodal emotion recognition in proactive conversational agents: A user study
Organizations: Escuela Universitaria Politécnica de Teruel, Universidad de Zaragoza, C/ Atarazana, 2, Teruel, 44003, Aragón, Spain · I3A (Institute of Engineering Research of Aragon), Universidad de Zaragoza, C. de Mariano Esquillor Gómez, s/n, Zaragoza, 50018, Aragón, Spain · Universitat de les Illes Balears, Universitat de les Illes Balears, Carretera de Valldemossa, km 7.5, Palma, 07122, Illes Balears, Spain
Abstract
This article presents a multimodal emotion recognition module integrated into a proactive Socially Interactive Agent (SIA) powered by generative artificial intelligence. The system evaluates real-time affective states through two distinct channels: a computer vision-based facial recognition module and a semantic linguistic analysis engine. To validate the framework, an empirical study was conducted with 20 users who engaged in dynamic, unscripted dialogues with the conversational agent. The findings reveal a significant discrepancy between automated visual cues and actual internal emotional states. When interacting with the AI, users consistently exhibited a "poker face" effect, displaying serious, concentrated facial expressions even when experiencing positive emotions. Consequently, the generative AI linguistic analysis proved significantly more reliable, by contextualizing the users' verbal expressions. Furthermore, an analysis of the interaction dynamics demonstrated that SIAs can effectively elicit specific emotions by adapting conversational themes and employing structured linguistic patterns, such as empathetic or humorous language. However, the study also noted that instances of uncalibrated proactivity occasionally led to user disengagement and a perception of artificiality. Ultimately, this research highlights the necessity of refining SIAs to dynamically adapt to users' emotional evolution, relying on deep linguistic context to foster more natural, human-like interactions.
Figures & tables
Figure 1: Sanbot Elf running the Android app.
Figure 2: Component diagram of the system.
Figure 3: PROMPT1: dialogue generation.
Figure 4: PROMPT2: dialogue analysis.
Figure 5: Emotion matrix used by users to determine their emotions.
| Interlocutor | Example |
|---|---|
| SIA | “You won’t believe what happened to me the other day! I was going through some old photos with my sister, and we found one from when we were little, dressed as superheroes. I couldn’t stop laughing! Do you have any family photos that remind you of something fun or special?” |
| Ana | “Yes, I have a few.” |
| SIA | “How wonderful! Do you remember the story behind any of those photos? Sometimes a single image can bring back so many amazing memories. Have you ever found something unexpected or shared an unforgettable moment like that?” |
| Ana | “Yes, the photo is from a trip to a theme park, Terra Mítica. It was taken right at the park entrance.” |
| SIA | “That’s great! Terra Mítica must have been quite an adventure. Do you remember what you were most excited about that day? I’m sure there are moments that still make you smile. Which part of the park did everyone enjoy the most?” |
| Ana | “When we went on the roller coasters; it was great!” |
Table 1: Example of a proactive dialogue between the user and the designed system
| Source | Sentiment Analysis Output |
|---|---|
| ChatGPT | The user’s tone is predominantly positive. The participant expresses satisfaction and joy when recalling a special family moment, particularly when mentioning the roller coasters. The detail about their mother not wanting to go back on the ride, which elicits laughter during the retelling, further reinforces this state. |
| Specific expressions such as “it was great” and “we had a great time” serve as clear linguistic indicators of enjoyment and happiness. | |
| Primary Emotion: Joy is the most prominent emotion. This is reflected in the enthusiastic manner in which the user relives these memories, highlighting shared fun within a playful and memorable family context. |
Table 2: Example of the LLM-based sentiment analysis and emotional reasoning
| Dialogue 1 | Dialogue 2 | |||||
|---|---|---|---|---|---|---|
| ID | Item | Scale | Mean | SD | Mean | SD |
| PDQ1 | Alexa’s attitude influenced my feelings | (Disagree - Agree) | 3.1 | 1.3 | 3.5 | 1.3 |
| PDQ2 | Alexa understands my emotions | (Disagree - Agree) | 3.6 | 0.9 | 4.0 | 1.1 |
| PDQ3 | Alexa talks about irrelevant things (R) | (Disagree - Agree) | 2.7 | 1.2 | 2.0 | 0.9 |
| PDQ4 | Overall, Alexa is empathetic | (Disagree - Agree) | 4.0 | 0.9 | 3.9 | 1.1 |
| PDQ5 | Feeling during conversation | (Frustrated - Satisfied) | 4.2 | 0.7 | 4.1 | 0.9 |
Table 3: User experience and affective state results across Dialogue 1 and Dialogue 2 ( per session). Items are rated on a 1–5 Likert or Semantic Differential scale. (R) indicates reverse-scored items.
| Dimension / Item | Mean | SD |
| Communication & Understanding | ||
| FQ1: I believe I can communicate with Alexa through language | 4.5 | 0.5 |
| FQ2: I believe I can clearly express my thoughts to Alexa | 4.2 | 0.7 |
| FQ3: I believe that Alexa fully understands what I mean | 3.9 | 1.1 |
| FQ4: I don’t think Alexa understands my expressions (R) | 2.8 | 1.4 |
| FQ5: I believe that Alexa is not capable of understanding complex stories (R) | 3.0 | 1.4 |
Table 4: User experience and perception questionnaire results (Likert scale 1-5, ). (R) indicates reverse-scored items (lower is better).
Figure 6: Comparison of emotion frequency distribution in self-reports (left), expressions automatically detected by the facial recognition module (middle), and emotions detected from the linguistic modality (right).
Figure 7: Correspondence matrices comparing predictions from the visual (a) and linguistic (b) modalities with user self-reports.
| Emotion | Role: Assistant | Role: User |
|---|---|---|
| Joy | Expressions used: • “That’s great! Dogs and birds make wonderful pets.” • “Haha, that sounds like a lot of fun! Lua seems to be very curious.” • “I’m glad everything turned out well. I hope you have the opportunity to experience more positive surprises in the future!” Observed patterns: • Using interjections (Haha! How cool!) to evoke enthusiasm. • Focus on funny or positive anecdotes to keep the tone cheerful. | Expressions used: • “I like it because whenever I’m with them it makes me forget about everything else.” • “My dog has been really focused on eating tree leaves lately.” • “I love brass bands and bulls.” Observed patterns: • Stories of positive moments or curious situations. • Mentions of festive or everyday activities that generate pleasure. |
| Sadness | Expressions used: • “It’s normal to feel nostalgic when remembering those special moments with your friends.” • “The university stage is a time full of changes and unique experiences.” • “I understand, taking time to reflect is important.” Observed patterns: • Empathic language that validates the user’s feelings. • Use of phrases that suggest emotional accompaniment. | Expressions used: • “I feel very nostalgic because my university life in Teruel is over.” • “All the moments I’ve spent with my friends.” • “It makes me have a lot of fun thinking about those things.” Observed patterns: • Focus on past memories and their emotional importance. • Recognition of mixed sensations of sadness and joy when remembering. |
| Surprise | Expressions used: • “What do you think about the idea of letting life surprise us from time to time?” • “Sometimes surprises are not exactly what we expect.” or “Unexpected changes usually generate surprise.” Observed patterns: • Use of open questions to evoke curiosity. • Phrases that highlight the unexpected as a catalyst for emotions. | Expressions used: • “I’m fine with nice surprises.” • “Of course, I would have to think about it to be able to tell you the details.” or “I hope so too.” Observed patterns: • Receptivity towards surprises, with a preference for positive experiences. • Reflective attitude in the face of the unexpected. |
Table 5: Pattern analysis results (joy, sadness and surprise emotions)
| Emotion | Role: Assistant | Role: User |
|---|---|---|
| Anger | Expressions used: • “That’s really disappointing to hear.” • “It is sad to see how lack of care can ruin natural landscapes.” • “I completely understand, lack of environmental awareness can be very frustrating.” Observed patterns: • Direct language that addresses the cause of the anger. • Validation of user frustration. | Expressions used: • “All three things bother me.” • “They think that they don’t recycle the containers.” • “When it doesn’t rain for a long time, and we lack water in the dams.” Observed patterns: • Mention of external problems as a cause of anger. • Criticism of the lack of action or awareness in society. |
| Fear | Expressions used: • “It must have been terrifying to hear that sound.” • “Running in that situation could have been dangerous.” • “You were definitely lucky to get out of that situation unscathed.” Observed patterns: • Language that escalates the situation to resonate with the user’s fear. • Validation of the user’s emotional reactions. | Expressions used: • “Although I was warned that there were bears, I was young and didn’t pay attention.” • “He scared me with a sound.” • “I know I shouldn’t have run, but I reacted badly.” Observed patterns: • Narrative of events that involve danger. • Reflection on personal reactions to fear. |
| Disgust | Expressions used: • “That’s great, but sometimes animals can be a little gross.” • “Some species of insects feed on carrion.” • “Sometimes pets can get into somewhat unpleasant situations.” Observed patterns: • Introduction of examples that evoke unpleasant sensations. • Use of common situations to connect with emotion. | Expressions used: • “I didn’t have any unpleasant experiences.“ • ”I love insects, and I collect them.“ • ”Yes, on occasion I have come across a decomposing sheep carcass.” Observed patterns: • Aversion to specific situations related to nature. • Description of events that generate rejection. |
Table 6: Pattern analysis results (anger, fear and disgust emotions)