LLM-Based Selection of Incongruent Verbal and Nonverbal Behavior for Virtual Humans

Parisa Ghanad Torshizi; Stacy Marsella

cs.AIAug 24, 2026

LLM-Based Selection of Incongruent Verbal and Nonverbal Behavior for Virtual Humans

Authors: Parisa Ghanad Torshizi, Stacy Marsella

Organizations: Khoury College of Computer Science Northeastern University Boston, USA

Abstract

Nonverbal behavior generation systems for virtual agents often take an utterance as input and generate nonverbal behaviors that emphasize or illustrate the content of the verbal channel. However, human nonverbal behavior is shaped by more than the content of the speech. It is also influenced by speaker roles, interpersonal relationships, social context, and the cognitive and emotional states of the interactants. As a result, the nonverbal channel may reinforce, weaken, qualify, or even contradict the verbal channel. It may also reveal internal states that are hidden or only indirectly implied in speech, including emotional "leakage" that may be incidental to the immediate interaction. Modeling this richer relationship between verbal and nonverbal behavior is important for designing virtual agents that exhibit realistic, human-like behavior. It is especially critical in training contexts that require nuanced social interpretation, such as counseling simulations involving virtual patients. Drawing on Ekman's framework of verbal nonverbal relationships, we propose a taxonomy of categories in which mismatches between verbal and nonverbal behavior can occur. We then examine alternative approaches for realizing these behaviors using large language models, focusing on whether LLMs can select contextually appropriate mismatched verbal and nonverbal behaviors from a given dialogue and social interaction context. Finally, we evaluate the resulting behaviors in a human-subject study, assessing whether context-driven nonverbal behavior, when embodied in a virtual human, produces the intended effects on observers.

Figures & tables

Prompt	AUs	Gaze Shift/Avert	Nonverbal Contrast
Utterance	41	4	2/8
Dialogue History	39	7	5/8
Context + History	48	10	7/8

TABLE I : Total Action Units and Gaze Shifts per Prompt Type.

Prompt	POSPOS	NEGNEG	POSNEG	NEGPOS
Utterance	3	3	1	1
Dialogue History	1	2	3	2
Context+History	0	1	4	3

TABLE II : Relation counts per Prompt Type.

Nonverbal Relation	Action Units	Gaze Shifts/Aversions
POSPOS	$5.0$	$0.0$
POSNEG	$5.38$	$1.75$
NEGPOS	$4.33$	$0.33$
NEGNEG	$6.5$	$0.83$

TABLE III : Action Unit and Gaze Shift per Utterance.

Fig. 1 : A selection AU counts per utterance, down selected based on which were notably different between POS_NEG and NEG_POS contrasts

Panel of Figure 5 — Fig. 2 : LLM-selected behavior realized in virtual human framework. The left image shows a frame of the pos-pos condition(positive verbal, positive nonverbal and match between verbal-nonverbal channels)- The right image shows the corresponding frame of the pos-neg condition(positive verbal, negative nonverbal and mismatch between verbal-nonverbal channels)

	Pos Emo	Neg Emo	VB Pos	NVB Pos
pos-pos	2.81 (0.76)	1.58 (0.66)	4.21 (0.52)	4.05 (0.69)
pos-neg	2.40 (0.87)	1.94 (0.84)	3.79 (0.89)	2.97 (1.16)
neg-neg	1.57 (0.73)	2.98 (0.79)	2.36 (1.14)	2.23 (1.09)
neg-pos	1.98 (0.73)	2.44 (0.71)	3.05 (1.10)	3.28 (0.94)

TABLE IV: Descriptive Analysis of Dependent Variables Across Conditions

Fig. 3 : A demonstration of participants’ perception of speaker in different conditions

Fig. 4 : The distribution of speaker’s expressed emotions perceived by participant’s

Dependent Variable	$df$	$F$	$p$	$\eta^{2}_{G}$
Positive Emotion	2.21, 83.82	40.04	$<$ .001	.267
Negative Emotion	2.23, 84.56	37.22	$<$ .001	.336
Verbal Positivity	2.24, 85.15	29.14	$<$ .001	.365
Non-Verbal Positivity	2.66, 101.26	21.92	$<$ .001	.310

TABLE V : One-Way Repeated-Measures ANOVA Results for Each Dependent Variable

Explore similar work

Jun 1, 2026

Unveiling the Limits of Large Language Models in Inferring Pragmatic Meaning from Non-Verbal Responses

Although large language models (LLMs) have shown considerable progress in pragmatic language understanding, prior research has focused mainly on their comprehension of verbal behavior. Nonetheless, non-verbal behavior remains a fundamental component of human communication, especially when deliberately utilized in isolation to convey indirect meanings. In this work, we present the first systematic evaluation of LLMs' ability to infer pragmatic meaning in dialogue consisting solely of non-verbal responses. We explore three research questions: (1) Can LLMs recognize indirect intent conveyed through non-verbal responses? (2) When and how do LLMs fail to capture non-verbal intent? (3) How can we improve LLMs' ability to interpret non-verbal intent?. Through the evaluation, we observe that LLMs struggle to infer underlying meaning from non-verbal responses, with accuracy dropping by up to 60% points compared to verbal ones. Further extensive analysis reveals a behavioral pattern in LLMs' interpretations of non-verbal behavior and demonstrates that in-context learning facilitates pragmatic inference.

Sugyeong Eo, Heuiseok Lim

Department of Software, Yonsei University Mirae Campus, Republic of Korea · Department of Computer Science and Engineering, Korea University, Republic of Korea

LLM Evaluation Pragmatic Reasoning in Language Models

Ask PDF

Sep 23, 2026

Listening and Mirroring: The Effects of Verbal Attunement and Behavioral Mimicry on Social and Empathic Perceptions of Embodied AI Agents in VR

As embodied agents take on increasingly social and relational roles in VR, visual realism and embodiment alone may be insufficient; users must also perceive these agents as emotionally attuned, supportive, and humanlike. Prior work suggests that verbal attunement and nonverbal mimicry can each improve users' social evaluations of embodied agents. However, behavioral mimicry has largely been studied outside of real-time, conversational AI interactions, leaving limited understanding of how users respond when an agent simultaneously generates contextually responsive dialogue and adapts its nonverbal behavior during an immersive conversation. To address this gap, we developed an embodied AI counselor that combines conversational AI with real-time facial-expression and posture mimicry, while producing either verbally attuned or neutral responses. We evaluated the system in a 2 X 2 within-subjects study with 20 participants, manipulating verbal attunement and behavioral mimicry. Results showed that verbal attunement was the most reliable driver of perceived empathy. Behavioral mimicry showed a marginal relationship with perceived humanness, while greater mimicry exposure showed preliminary, exploratory positive associations with empathy, positivity, and humanness, particularly among female participants. Together, these findings show that multimodal synchrony is not a simple additive strategy for designing empathic conversational agents in VR and underscore the need to consider how verbal and nonverbal behaviors are combined during real-time interaction.

Nathalia Gomez, Haig Shamlian, Omar Khan +1

Drexel University Philadelphia, PA, USA

Human-AI Interaction Conversational Agents Embodied AI Human-Robot Interaction Anthropomorphism in AI

Ask PDF

May 19, 2026

Reinforcing Human Behavior Simulation via Verbal Feedback

Humans learn social norms and behaviors from verbal feedback (e.g., a parent saying "that was rude" or a friend explaining "here's why that hurt"). Yet, learning from feedback for LLMs has largely focused on domains like code and math, where RL rewards are directly verifiable and condensed into scalar values. As LLMs are increasingly used to simulate human behavior, e.g., standing in for users, patients, students, and other personas, there is a pressing need to make them more human-like, which requires embracing a fundamentally different kind of signal: feedback that is verbal, subjective, and multi-faceted. We present DITTO, a model trained by treating verbal feedback as a first-class signal in reinforcement learning. After each rollout, DITTO receives verbal feedback and generates a feedback-conditioned improved rollout; both outputs are jointly optimized with GRPO, distilling verbal guidance into the base policy without requiring feedback at test time. We also introduce SOUL (Simulation gym Of hUman-Like behavior), a unified benchmark and training data suite spanning 10 tasks across six categories: Theory of Mind, character role play, social skill, learner simulation, user simulation, and persona simulation. DITTO achieves an average 36% improvement over the base model and exceeds GPT-5.4 on 6 of 10 SOUL benchmarks, demonstrating that RL with verbal feedback is a promising direction for training LLMs to simulate human behavior.

Weiwei Sun, Xuhui Zhou, Jiarui Liu +13

1Carnegie Mellon University · 2Microsoft

RL for Language Models Human Behavior Simulation Large Language Model-Based Social Simulation RL from Human Feedback

Ask PDF