As embodied agents take on increasingly social and relational roles in VR, visual realism and embodiment alone may be insufficient; users must also perceive these agents as emotionally attuned, supportive, and humanlike. Prior work suggests that verbal attunement and nonverbal mimicry can each improve users' social evaluations of embodied agents. However, behavioral mimicry has largely been studied outside of real-time, conversational AI interactions, leaving limited understanding of how users respond when an agent simultaneously generates contextually responsive dialogue and adapts its nonverbal behavior during an immersive conversation. To address this gap, we developed an embodied AI counselor that combines conversational AI with real-time facial-expression and posture mimicry, while producing either verbally attuned or neutral responses. We evaluated the system in a 2 X 2 within-subjects study with 20 participants, manipulating verbal attunement and behavioral mimicry. Results showed that verbal attunement was the most reliable driver of perceived empathy. Behavioral mimicry showed a marginal relationship with perceived humanness, while greater mimicry exposure showed preliminary, exploratory positive associations with empathy, positivity, and humanness, particularly among female participants. Together, these findings show that multimodal synchrony is not a simple additive strategy for designing empathic conversational agents in VR and underscore the need to consider how verbal and nonverbal behaviors are combined during real-time interaction.
Figures & tables
Figure 1. Representation of a participant speaking to the embodied AI counselor in VR Enjoying the baseball game from the third-base seats. Ichiro Suzuki preparing to bat.
Figure 2. Counselor facial-expression states, shown from left to right: smiling, neutral, and brow lowering.
Figure 3. Counselor behaviors used for mimicry, shown from left to right: default upright posture, slouched posture, and leftward head tilt.
VA
BM
VA × BM
F
p
ηp2
F
p
ηp2
F
p
ηp2
Empathy
7.43
.013
.281
1.28
.273
.063
0.69
.418
.035
General Positivity
3.47
.078
.154
1.34
.261
.066
0.83
.373
.042
Humanness
1.26
.275
.062
4.31
.052
.185
0.03
.861
.002
Eeriness
0.05
.823
.003
0.03
.860
.002
1.07
.314
.053
Verbal Attunement
5.63
.028
.229
2.58
.125
.119
0.33
.571
.017
Table 1. Summary of 2 x 2 repeated-measures ANOVA tests for main and interaction effects of verbal attunement and behavioral mimicry. Bolded values denote statistically significant effects ( p<.05 ) or marginal trends ( p<.10 ).
Measure
NVA+NBM
NVA+BM
VA+NBM
VA+BM
Empathy
42.00 (15.38)
46.75 (13.48)
50.90 (10.57)
51.70 (13.74)
Positivity
34.30 (13.14)
38.10 (12.49)
39.40 (10.08)
40.10 (10.28)
Humanness
6.95 (3.75)
8.40 (3.30)
7.60 (3.40)
8.85 (3.45)
Eeriness
7.55 (3.32)
7.00 (2.70)
6.95 (2.70)
7.20 (2.19)
VA Check
9.40 (4.49)
10.95 (4.38)
11.60 (4.42)
12.40 (4.26)
Table 2. Means and standard deviations for all measurements. Values are M ( SD ).
Figure 4. Mean ratings for significant or marginal main effects. Empathy, general positivity, and verbal attunement are shown by verbal attunement condition, collapsed across behavioral mimicry. Humanness is shown by behavioral mimicry condition, collapsed across verbal attunement.
Nonverbal behavior generation systems for virtual agents often take an utterance as input and generate nonverbal behaviors that emphasize or illustrate the content of the verbal channel. However, human nonverbal behavior is shaped by more than the content of the speech. It is also influenced by speaker roles, interpersonal relationships, social context, and the cognitive and emotional states of the interactants. As a result, the nonverbal channel may reinforce, weaken, qualify, or even contradict the verbal channel. It may also reveal internal states that are hidden or only indirectly implied in speech, including emotional "leakage" that may be incidental to the immediate interaction. Modeling this richer relationship between verbal and nonverbal behavior is important for designing virtual agents that exhibit realistic, human-like behavior. It is especially critical in training contexts that require nuanced social interpretation, such as counseling simulations involving virtual patients. Drawing on Ekman's framework of verbal nonverbal relationships, we propose a taxonomy of categories in which mismatches between verbal and nonverbal behavior can occur. We then examine alternative approaches for realizing these behaviors using large language models, focusing on whether LLMs can select contextually appropriate mismatched verbal and nonverbal behaviors from a given dialogue and social interaction context. Finally, we evaluate the resulting behaviors in a human-subject study, assessing whether context-driven nonverbal behavior, when embodied in a virtual human, produces the intended effects on observers.
Parisa Ghanad Torshizi, Stacy Marsella
Khoury College of Computer Science Northeastern University Boston, USA
AI clones that imitate a specific person typically reproduce what the person says and how they sound, but not how they listen. We investigate whether adding multimodal listening behaviors gives such a clone more presence and authenticity. We integrated verbal backchannels and head nodding, driven by real-time prediction models, into an AI clone equipped with voice cloning and LLM-based responses. In a within-subjects study (N=35), adding these behaviors significantly improved the perceived attentiveness of the avatar, the sense of talking with the real person, and the feeling of co-presence. These results indicate that AI clone fidelity should extend beyond voice and response content to include interactive listening behavior.
Empathetic social robots should respond not only to what users say, but also to how their emotions dynamically evolve during interaction. However, existing empathetic dialogue systems are often text-centered and primarily model empathy as a one-way mapping from the user's emotion to the system response, limiting their ability to capture embodied speaker--listener affective exchange. We present AffectLoop, a multimodal speaker-listener emotion-dynamics-aware spoken dialogue system implemented on the Misty II robot. The system tracks the speaker's verbal and facial affective dynamics, estimates the robot listener's own verbal and behavioral affective state, and conditions LLM-based response generation on both affective streams. The robot then generates a short spoken empathetic response together with emotionally congruent embodied behavior, forming a closed speaker--listener affective loop. We evaluate the system in a pilot within-subject study with five participants, comparing it with an otherwise identical utterance-conditioned baseline that omits the speaker- and listener-affective-state inputs. The proposed system received higher overall impression ratings, especially for empathetic response and user satisfaction. Post-hoc log analysis further showed higher speaker-listener affective alignment and stronger valence-based distress recovery. These preliminary results suggest that explicitly modeling both speaker emotional dynamics and listener affective state can improve embodied empathetic interaction.