EMPA: Evaluating Persona-Aligned Empathy as a Process

Shiya Zhang; Yuhan Zhan; Ruixi Su; Ruihan Sun; Ziyi Song; Zhaohan Chen; Xiaofan Zhang

cs.AIFeb 28, 2026

EMPA: Evaluating Persona-Aligned Empathy as a Process

Authors: Shiya Zhang, Yuhan Zhan, Ruixi Su, Ruihan Sun, Ziyi Song, Zhaohan Chen, Xiaofan Zhang

Organizations: Team Echo, Nature Select · Sun Yat-sen University

Abstract

Evaluating persona-aligned empathy in LLM-based dialogue agents remains challenging. User states are latent, feedback is sparse and difficult to verify in situ, and seemingly supportive turns can still accumulate into trajectories that drift from persona-specific needs. We introduce EMPA, a process-oriented framework that evaluates persona-aligned support as sustained intervention rather than isolated replies. EMPA distills real interactions into controllable, psychologically grounded scenarios, couples them with an open-ended multi-agent sandbox that exposes strategic adaptation and failure modes, and scores trajectories in a latent psychological space by directional alignment, cumulative impact, and stability. The resulting signals and metrics support reproducible comparison and optimization of long-horizon empathic behavior, and they extend to other agent settings shaped by latent dynamics and weak, hard-to-verify feedback.

Figures & tables

Figure 1: Implicit Assumptions in Agent Evaluation and Their Breakdown in Empathy-Oriented Psychological Support

Figure 2: A real EMPA sandbox interaction reveals a failure mode of scalar empathy evaluation: high magnitude scores without directional alignment lead to ineffective support, encouraging verbose but misaligned responses.

Figure 3: Overview of EMPA. Real affective interaction data are distilled into psychologically consistent user profiles and crisis scenarios. The evaluated model then engages in unscripted, multi-turn interaction with user agents endowed with persona and long-term memory. Empathic behavior is finally quantified from the resulting interaction trajectories using EPM.

Figure 4: These dimensions provide an operational basis for analyzing LLM behavior in human–AI interaction and user experience.

Figure 5: Sensitivity and robustness under controlled perturbations

Dataset	$n$	Metric	Mean	Median	$\Pr(d_{i}<0)$	95% CI	$p$
Persona Flip	251	$\Delta E$	$-1.22$	$-0.17$	$72.50\%$	$[-1.47,\,-0.97]$	$<.0001$
Persona Flip	251	$\Delta\cos\theta^{*}$	$-0.38$	$-0.03$	$69.70\%$	$[-0.47,\,-0.29]$	$<.0001$
Sycophancy	98	$\Delta E$	$-4.36$	$-4.68$	$81.60\%$	$[-5.15,\,-3.57]$	$<.0001$
Sycophancy	98	$\Delta\cos\theta^{*}$	$-1.08$	$-1.65$	$84.70\%$	$[-1.27,\,-0.90]$	$<.0001$

Table 5: Perturbation analysis (

\Delta

= perturbed

-

original)

Dataset	$n$	Component	Mean	Median	$\Pr(d_{i}<0)$	Tie rate
Persona Flip	251	EPM (Rubric-Grounded Physics; $\Delta E$ )	$-1.22$	$-0.17$	$72.50\%$	$3.20\%$
Persona Flip	251	Rubric-Only (No Physics)	$-0.09$	$0.00$	$46.20\%$	$44.20\%$
Persona Flip	251	LLM Judge (No-Rubric, Direct)	$-0.25$	$0.00$	$29.10\%$	$52.20\%$
Sycophancy	98	EPM (Rubric-Grounded Physics; $\Delta E$ )	$-4.36$	$-4.68$	$81.60\%$	$3.10\%$
Sycophancy	98	Rubric-Only (No Physics)	$-0.36$	$-0.42$	$85.90\%$	$5.10\%$
Sycophancy	98	LLM Judge (No-Rubric, Direct)	$1.65$	$1.33$	$25.50\%$	$4.10\%$

Table 6: Ablation results (

\Delta

= perturbed

-

original)

Rank	Model	Outcome Quality			Process Efficiency			Strategic Stability			EPM-Q
Rank	Model	RDI	$E_{\mathrm{tot}}$	$S_{\mathrm{net}}$	$\rho$	$S_{\mathrm{proj}}$	$\tau$	$R_{\mathrm{pos}}$	Align	Pen	EPM-Q
1	Claude 4.6 Opus	99.5	117.6	122.0	139.0	128.4	96.9	91.5	92.4	98.9	107.2
2	Gemini 3 Pro Preview	98.1	100.3	113.9	111.4	103.4	95.8	90.7	92.2	97.8	99.8
3	GPT-5.2 Pro	97.7	95.5	113.4	102.5	95.3	94.2	89.9	91.7	97.9	97.6
4	Gemini 2.5 Pro	92.6	90.1	124.2	65.5	62.2	81.4	88.2	87.1	93.8	90.7
5	Qwen 3 235B	92.0	91.7	121.3	88.3	82.6	74.0	80.2	82.0	82.3	89.6

Table 7: EPM Benchmark Leaderboard

Figure 6: Structural distribution of the EPM Benchmark Dataset. (a) Perfect orthogonality across Cognitive (C), Affective (A), and Proactive (P) dimensions. (b) Right-skewed difficulty distribution, with 86.7% of cases classified as Medium difficulty or above. (c) Coverage of six distinct life domains. (d-e) Demanding persona profile where 50% of users possess a "High" empathy threshold.

Figure 7: Stacked bar chart of Success (blue), Failure (pink), and Timeout (not shown) rates across models.

Figure 8: Statistical summary of core EPM-Q metrics (Mean $\pm$ Std). Panels display Outcome Quality (RDI, Total Effective Energy, Total MDEP Score), Process Efficiency (Empathy Density $\rho$ , Effective Projection $S_{proj}$ , Path Tortuosity $\tau$ ), and Process Stability (Positive Energy Ratio $R_{pos}$ , Avg Alignment $\cos\theta$ , Penalty Rate $R_{pen}$ ).

Figure 9: EPM-Q adaptability analysis. Comparison of model performance on Routine (light) vs. Challenging (dark) scenarios across Affective (A), Cognitive (C), and Proactive (P) axes.

Figure 10: EPM-Q performance across six scenario domains.

Figure 11: Performance breakdown by User Need Type (A/C/P) and Empathy Threshold (Receptive vs. Defensive).

Figure 12: Radar chart grid of scenario category performance.

Figure 13: Radar chart grid of mechanism stress test profiles.

Figure 14: Radar chart grid of persona resilience profiles.

Figure 15: Comparative analysis of Cognitive-Affective-Proactive trajectories across model tiers.

Figure 16: Heatmap of EPM-Q scores across all 30 test cases for all 14 models.

Metric Name	Symbol	Core Meaning and Evaluation Value
Task Completion Status	$\mathrm{Status}$	Success/failure is defined by the Trinity Victory Condition—meeting the geometric/positional goal and accumulating sufficient energy.
Relative Distance Improvement	$\mathrm{RDI}$	Measures the thoroughness of healing. Calculates the percentage improvement of the user’s final psychological state relative to the initial deficit.
Cumulative Effective Energy	$E_{\text{total}}$	Measures cumulative effective intervention along the ideal healing direction across the dialogue, serving as a proxy for substantive effort.
Energy Surplus	$E_{\text{surplus}}$	Measures empathy abundance. Calculates the additional energy support provided beyond the basic requirement.
Total MDEP Net Score	$S_{\text{net}}$	Measures total empathy quality. The sum of cumulative net scores obtained in the three dimensions of C/A/P.

Table 8: EPM Outcome Metrics – Measuring Final Efficacy and Total Workload

Table 9: EPM Process Metrics

Metric Name	Symbol	Raw Range	Boundary $x_{0}$ (Score 0)	Boundary $x_{100}$ (Score 100)
Relative Dist. Improvement	$Idx_{\{RDI\}}$	$[-1,1]$	$-1.0$ (Deterioration)	$1.0$ (Full Resolution)
Alignment Index	$Idx_{\{Align\}}$	$[-1,1]$	$-1.0$ (Opposite)	$1.0$ (Perfect Alignment)
Path Tortuosity	$Idx_{\{\tau\}}$	$[1,3]$	$3.0$ (Inefficient)	$1.0$ (Optimal)
Penalty Rate	$Idx_{\{Pen\}}$	$[0,3]$	$3.0$ (High Toxicity)	$0.0$ (Zero Penalty)

Table 10: Metric Specifications for Unified Linear Mapping

Table 11: Initial Empathy Deficit Rating (IEDR)

Weight Class	Multiplier	Indicators	[0]	[1]	[2]	[3]
Standard	$\times 1.0$	C.1, C.2, A.1, P.1	0	$-2$	$-4$	$-6$
Priority	$\times 1.5$	C.3, A.3, P.3	0	$-3$	$-6$	$-9$
Core	$\times 2.0$	A.2, P.2	0	$-4$	$-8$	$-12$

Table 12: IEDR Scoring Key

Table 13: Multi-Dimensional Empathy Progress Rating (MDEP-PR)

Axis	Progress			Regression
Axis	[0]	[1]	[2]	[0]	[ $-1$ ]	[ $-2$ ]
C	0	$+1$	$+3$	0	$-2$	$-4$
A	0	$+1$	$+3$	0	$-2$	$-5$
P	0	$+1$	$+3$	0	$-2$	$-5$

Table 14: MDEP-PR Scoring Key

Table 15: Rubric-to-Trajectory Integration

Stage	Rubric	Output	EPM Usage
$T=0$	IEDR (Table 11 )	9 indicator levels $\rightarrow$ deficit scores (Table 12 )	Initial state vector $\mathbf{P}_{0}=(C_{0},A_{0},P_{0})$ ; normalization constant $r_{0}=\\|\mathbf{P}_{0}\\|$ .
Every $K$ turns	MDEP-PR (Table 13 )	6 channel levels $\rightarrow$ increment scores (Table 14 )	Action vector $\mathbf{v}_{t}$ ; effective work $\Delta E_{t}=\\|\mathbf{v}_{t}\\|\cos\theta_{t}$ ; state update $\mathbf{P}_{t}=\mathbf{P}_{t-1}+\mathbf{v}_{t}$ .
End	—	Full trajectory $\{\mathbf{P}_{0},\ldots,\mathbf{P}_{T}\}$	EPM-Q $=0.4\times\text{Outcome}+0.2\times\text{Efficiency}+0.4\times\text{Stability}$ .

Table 16: Rubric-to-Trajectory Integration

Explore similar work

Empathy Is Steerable but Multi-Axial: Mechanism Geometry and Persona Effects in LLMs
Sep 14, 2026JuHeon Ha, Byounghan Lee, Yunseo Choi +1Language Model Steering LLM Interpretability
Department of Artificial Intelligence, Ajou University
Deep Persona: A Psychologically Grounded Architecture and Evaluation Framework for Role-Playing Agents and Simulations
Sep 6, 2026Rotem Dror, Zohar Elyoseph, Yuval Haber +3Large Language Model-Based Role-Play Simulation Personality Modeling in Language Models
Faculty of Computer and Information Science, University of Haifa, Israel · School of Therapy, Counseling and Human Development, University of Haifa, Israel · The PhD Program of Hermeneutics and Culture, Interdisciplinary Studies Unit, Bar-Ilan University, Israel +1
Practicing with Language Models Cultivates Human Empathic Communication
Mar 16, 2026Aakriti Kumar, Nalin Poungpeth, Diyi Yang +2
Kellogg School of Management, Northwestern University. · Northwestern Institute on Complex Systems, Northwestern University. · Ryan Institute on Complexity, Northwestern University. +3

Sep 14, 2026J/K move · Enter open · S save

Empathy Is Steerable but Multi-Axial: Mechanism Geometry and Persona Effects in LLMs

JuHeon Ha, Byounghan Lee, Yunseo Choi, Kyung-Ah Sohn

Department of Artificial Intelligence, Ajou University

Language Model Steering LLM Interpretability Causal Interventions in Language Models Emotion Representation in Language Models

Open paper AskPDF arXiv

Activation steering has been used to control traits such as honesty, refusal, and sycophancy, yet supportive empathy is evaluated along multiple dimensions that need not correspond to independently controllable activation directions. Using the EPITOME framework, which decomposes supportive empathy into Emotional Reactions, Interpretations, and Explorations, we study three instruction-tuned LLMs and ask whether candidate directions derived from these labels produce distinguishable intervention effects or instead share structure, and how persona prompts interact with those directions. We find that contrastive activation addition yields a stable middle-layer intervention that consistently shifts the EPITOME proxy scores across models, moving empathy analysis beyond response-level scoring. However, the recovered directions are only partially separable: steering one direction induces off-target shifts, and hand-crafted prompting shifts the empathy profile rather than isolating a single dimension. Persona prompts substantially change EPITOME scores, but a paired activation-shift decomposition shows that the recovered subspace captures only approximately 3 percent of persona-induced squared activation-shift magnitude at layer 15. Under this EPITOME-based definition, expressed empathy is steerable but multi-axial, and controlling persona-conditioned empathy requires targeting structure beyond individual mechanism directions.

Abstract

Explore similar work

Empathy Is Steerable but Multi-Axial: Mechanism Geometry and Persona Effects in LLMs

Deep Persona: A Psychologically Grounded Architecture and Evaluation Framework for Role-Playing Agents and Simulations

Practicing with Language Models Cultivates Human Empathic Communication