Facial-Expression-Aware Prompting for Empathetic LLM Tutoring
Authors: Shuangquan Feng, Laura Fleig, Ruisen Tu, Philip Chi, Edmund Bu, Melinda Ozel, Junhua Ma, Teng Fei, Virginia R. de Sa
Organizations: Neurosciences Graduate Program, University of California San Diego, La Jolla, USA · Department of Cognitive Science, University of California San Diego, La Jolla, USA · Department of Computer Science and Engineering, University of California San Diego, La Jolla, USA · Department of Electrical and Computer Engineering, University of California San Diego, La Jolla, USA · Department of Mathematics, University of California San Diego, La Jolla, USA · Face the FACS · Halıcıo˘glu Data Science Institute, University of California San Diego, La Jolla, USA
Large language models (LLMs) enable increasingly capable tutoring-style conversational agents, yet effective tutoring requires sensitivity to learners' affective and cognitive states beyond text alone. Facial expressions provide immediate and practical cues of confusion, frustration, or engagement, but remain underexplored in LLM-driven tutoring. We investigate whether facial-expression-aware signals can improve empathetic tutoring responses through prompt-level integration, without end-to-end retraining. We build a scalable simulated tutoring environment where a student agent exhibits diverse facial behaviors from a large unlabeled human facial expression video dataset, and compare four tutor variants: a text-only LLM baseline, a multimodal baseline using a random facial frame, and two Action Unit estimation model (AUM)-based methods that either inject textual AU descriptions or select a peak-expression frame for visual grounding. Across 960 multi-turn conversations spanning three tutor backbones (GPT-5.1, Claude Opus 4.5, and Gemini 2.5 Pro), we evaluate targeted pairwise comparisons with five human raters and an exhaustive AI evaluator. AU-based conditioning consistently improves empathetic responsiveness to facial expressions across all tutor backbones, while AUM-guided peak-frame selection outperforms random-frame visual input. Textual AU abstraction and peak-frame visual injection show model-dependent advantages. Control analyses show that this improvement does not come at the expense of worse pedagogical clarity or responsiveness to textual cues. Overall, our results show that lightweight, structured facial expression representations can meaningfully enhance empathy in LLM-based tutoring systems with minimal overhead. Future work should evaluate these methods using real student facial data collected during authentic learning interactions.
Figures & tables
Figure 1: Facial-expression-aware tutoring workflow and targeted comparisons. Given the same conversation history and a student’s facial expression video, an AU estimation model (AUM) enables two expression-aware integration strategies: (1) LLM+AUM converts estimated AU intensities into a textual description that is prepended to the tutor prompt (AU → Text); (2) MLLM+AUM selects a peak-expression frame as the visual input to a multimodal LLM tutor. Two baselines omit AUM: LLM uses text-only inputs, and MLLM uses a randomly sampled facial frame. Our analysis focuses on three pre-specified pairwise comparisons: (i) LLM+AUM vs. LLM, (ii) MLLM+AUM vs. MLLM, and (iii) LLM+AUM vs. MLLM+AUM.
AU(s)
Base Description
Slightly
Moderately
Strongly
Max(AU1, AU2)
raises eyebrows
1.5 – 2.0
2.0 – 2.8
> 2.8
AU4
knits eyebrows
1.0 – 1.6
1.6 – 2.8
> 2.8
AU5
widens eyes
0.8 – 1.5
1.5 – 2.2
> 2.2
AU9
wrinkles the nose
1.0 – 1.6
1.6 – 2.8
> 2.8
AU12
smiles
1.0 – 1.6
1.6 – 2.8
> 2.8
AU15
downturns the mouth
1.0 – 1.6
1.6 – 2.8
> 2.8
Table 1: Mapping of max-pooled AU activations to natural language descriptions with intensity thresholds
Figure 2: Example of the evaluation interface used by human evaluators. In this example, options A, B, C, and D correspond to responses generated by four AI tutor variants. The variants LLM+AUM , MLLM+AUM , LLM , and MLLM are presented as options D, B, A, and C, respectively, with option ordering randomized to enable blind evaluation.
Figure S1 : A visual reference guide for the AUs used in this study, extracted from [ 43 ] .
AU
1
2
4
5
9
12
15
17
Avg.
ICC(3,1) (higher is better)
CCNN-IT
.18
.15
.61
.07
.55
.82
.44
.37
.40
2DC
.70
.55
.69
.05
.57
.88
.32
.10
.48
SCC-Heatmap
.73
.44
.74
.06
.51
.71
.04
.37
.45
iARL
.13
.36
.68
.22
.36
.86
.52
.37
.44
IR50
.56
.51
.72
.65
.51
.84
.41
.46
.58
Table S1 : Comparison of the IR50 AU estimation model’s performance on the DISFA dataset with other models. Note that only the AUs used in this study are included here.
Empathy is central to human connection, yet people often struggle to express it effectively. In blinded evaluations, large language models (LLMs) generate responses that are often judged more empathic than human-written ones. Yet when a response is attributed to AI, recipients feel less heard than when comparable responses are attributed to a human. We built a conversation platform in which participants are asked to offer empathic support to an LLM expressing realistic troubles and conducted a randomized experiment collecting 33,938 messages spanning 2,904 text-based conversations between 968 participants and their LLM conversational partners. We find participants report feeling empathy but systematically fail to express it, but an LLM coaching intervention offering personalized feedback on effective empathic communication significantly boosts it without homogenizing participants' responses. Moreover, we derive a data-driven taxonomy of idiomatic empathic expressions in naturalistic dialogues across personal and workplace trouble scenarios. These results advance the scientific understanding of how empathy is expressed and demonstrate a scalable, AI-based intervention for scaffolding and cultivating it.
Aakriti Kumar, Nalin Poungpeth, Diyi Yang +2
Kellogg School of Management, Northwestern University. · Northwestern Institute on Complex Systems, Northwestern University. · Ryan Institute on Complexity, Northwestern University. +3
Students increasingly use LLMs as tutors for coursework and problem solving. Little is known about the level of assistance LLMs provide when students use them as tutors in authentic learning interactions. This matters because tutoring responses can differ substantially in how directly they help students complete a task. We operationalize this dimension as scaffolding level and develop a five-level scale, validated against human annotations, that characterizes responses according to the degree of direct assistance they provide. We apply the scale to 14,637 LLM responses from 203 students in a university AI course. Responses are overwhelmingly concentrated at high levels of assistance, with more than 95% classified as either Explaining or Solving. Scaffolding level is systematically associated with students' subsequent conversational behavior, but provides little additional predictive information about performance on three subsequent exams beyond prior achievement and dialogue behavior. These findings provide an empirical baseline for LLM assistance in tutoring interactions and a measurement framework for evaluating how alternative tutoring designs change that assistance.
Activation steering has been used to control traits such as honesty, refusal, and sycophancy, yet supportive empathy is evaluated along multiple dimensions that need not correspond to independently controllable activation directions. Using the EPITOME framework, which decomposes supportive empathy into Emotional Reactions, Interpretations, and Explorations, we study three instruction-tuned LLMs and ask whether candidate directions derived from these labels produce distinguishable intervention effects or instead share structure, and how persona prompts interact with those directions. We find that contrastive activation addition yields a stable middle-layer intervention that consistently shifts the EPITOME proxy scores across models, moving empathy analysis beyond response-level scoring. However, the recovered directions are only partially separable: steering one direction induces off-target shifts, and hand-crafted prompting shifts the empathy profile rather than isolating a single dimension. Persona prompts substantially change EPITOME scores, but a paired activation-shift decomposition shows that the recovered subspace captures only approximately 3 percent of persona-induced squared activation-shift magnitude at layer 15. Under this EPITOME-based definition, expressed empathy is steerable but multi-axial, and controlling persona-conditioned empathy requires targeting structure beyond individual mechanism directions.
JuHeon Ha, Byounghan Lee, Yunseo Choi +1
Department of Artificial Intelligence, Ajou University