Organizations: Department of Applied Artificial Intelligence, Seoul National University of Science and Technology, Seoul, Republic of Korea · Department of Data Science, Seoul National University of Science and Technology, Seoul, Republic of Korea
In VR interactions with embodied conversational agents, users' emotional intent is often conveyed more by how something is said than by what is said. However, most VR agent pipelines rely on speech-to-text processing, discarding prosodic cues and often producing emotionally incongruent responses despite correct semantics. We propose an emotion-context-aware VR interaction pipeline that treats vocal emotion as explicit dialogue context in an LLM-based conversational agent. A real-time speech emotion recognition model infers users' emotional states from prosody, and the resulting emotion labels are injected into the agent's dialogue context to shape response tone and style. Results from a within-subjects VR study (N=30) show significant improvements in dialogue quality, naturalness, engagement, rapport, and human-likeness, with 93.3% of participants preferring the emotion-aware agent.
Figures & tables
Figure 1. Overview of the emotion-aware interaction pipeline in VR: The Emotion Recognition (ER) condition (left) injects Speech Emotion Recognition (SER)-derived labels into the LLM prompt to ensure affective congruence, whereas the Non-Emotion Recognition (NER) condition (right) relies solely on text. A comparative infographic showing two experimental pipelines. The left side (ER) shows speech input being processed through both STT and HuBERT-based SER to create an emotion-aware prompt for a VR avatar. The right side (NER) shows a standard text-only pipeline bypassing emotion recognition. Outcomes indicate higher rapport and engagement for the ER condition.
Figure 2. Comparison between the Emotion Recognition (ER) and Non-Emotion Recognition (NER) conditions across evaluation metrics. (A) Human–agent interaction quality (HAI: Naturalness, Engagement, Rapport, Human-likeness, Synchrony). (B) System performance and quality (Dialogue Quality, Emotional Responsiveness, Reuse Intention). (C) Emotional response (SAM: Valence, Arousal, Dominance). (D) User experience (UEQ subscales). (E) Intrinsic motivation (IMI: Value/Usefulness, Interest/Enjoyment, Effort/Importance). Boxplots indicate medians, interquartile ranges, and 1.5 × IQR whiskers; circles denote outliers. Significance markers denote paired comparisons: * p<.05 , ** p<.01 , *** p<.001 . A five-panel figure of boxplots comparing two conditions (ER vs NER) on multiple questionnaire-based metrics. Panel A shows higher ER scores on naturalness, engagement, rapport, and human-likeness, with no clear difference on synchrony. Panel B shows higher ER scores on Dialogue quality, emotional responsiveness, and reuse intention. Panel C shows higher ER valence and arousal, with similar dominance. Panel D shows a small advantage for NER on attractiveness with minimal differences on other UEQ subscales. Panel E shows higher ER value/usefulness but higher NER interest/enjoyment, and similar effort/importance.
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 3. Interaction with the ER (Emotion-Recognition) Agent. The agent detects the user’s vocal prosody (Happy, Sad, Angry) and generates an affectively congruent response, recognizing the emotional intent behind the neutral text.
Figure 4. Interaction with the NER (Non-Emotion-Recognition) Agent. The agent relies solely on semantic content, ignoring vocal cues. Consequently, it provides factual or generic responses about "rain" regardless of the user’s emotional tone.
ID
Utterance
Emotionally Neutral Utterances
S1
The professor set the classroom air conditioner to the lowest temperature.
S2
The professor set the classroom online instead of in person.
S3
I am the only one left in the study room.
S5
We were supposed to eat out today, but we ordered chicken instead.
S6
The professor said I have to present first.
Appendix
Table 1. List of Neutral and Biased Utterances used in the experiment.
School of Business, Technical University of Applied Sciences Augsburg, Friedberger Str. 4, 86161 Augsburg, Germany · Data Science und Autonome Systeme Technologietransferzentrum (TTZ), An der Schmiede 19, 86899 Landsberg am Lech, Germany