All four leading LLMs talk more than they listen to personality-verified synthetic help-seekers

Pablo A. Fonseca; Raquel Rodríguez-Carvajal; Rafael A. Calvo

cs.HCAug 23, 2026

All four leading LLMs talk more than they listen to personality-verified synthetic help-seekers

Authors: Pablo A. Fonseca, Raquel Rodríguez-Carvajal, Rafael A. Calvo

Organizations: Dyson School of Design Engineering, Imperial College London, London, United Kingdom · Departamento de Psicología Biológica y de la Salud, Universidad Autónoma de Madrid, Madrid, Spain

Abstract

Large language models are increasingly consulted at moments of distress, yet single-turn benchmarks neither test sustained exchanges nor distinguish between users. We built a personality-aware evaluation in which four widely used models advised several synthetic help-seekers, each given a psychometrically specified profile, in an acute crisis: a caregiver learning of a relative's dementia diagnosis. Auditors blind to the profile prompt recovered the specified bands from dialogue alone with high agreement on every instrument (ICC(2,4) = 0.91; 0.79-0.96 by instrument; band-score r = 0.78), as expected for the Big Five but equally for coping style, coping self-efficacy, resilience and reactance, which the lexical approach never covered. Such evaluation therefore reaches beyond the Five Factor Model to motivational, regulatory and self-appraisal dispositions. The four models were not distinguishable on emotion stabilisation and failed alike, sharing three modes: verbosity, a talk-to-listen ratio above one, and problem-solving before the situation had been explored.

Figures & tables

Provider	Condition	N	r	r 95% CI	Spearman	In-band acc.	acc 95% CI	MAE
GPT-4.1	Specified profile	70	0.779	[0.675, 0.851]	0.771	0.629	[0.543, 0.714]	0.551
GPT-4.1	No-profile baseline	70	0.41	[0.247, 0.580]	0.412	0.529	[0.443, 0.614]	0.827
Claude-Sonnet-4.5	Specified profile	70	0.744	[0.640, 0.826]	0.742	0.586	[0.514, 0.657]	0.601
Claude-Sonnet-4.5	No-profile baseline	70	0.374	[0.254, 0.475]	0.343	0.471	[0.386, 0.543]	0.862
Mistral-Medium	Specified profile	70	0.689	[0.544, 0.794]	0.694	0.557	[0.429, 0.657]	0.651
Mistral-Medium	No-profile baseline	70	0.409	[0.280, 0.498]	0.389	0.486	[0.371, 0.600]	0.818

Table 1: Recovery of specified Advisee profiles by the Auditor ensemble, by instantiating model and condition.

N

is the number of construct instances (10 constructs

\times

7 Advisees). Scores are means across the four-model Auditor ensemble. Correlations are between the recovered score and the centre of the specified band. In-band accuracy is the proportion of instances whose recovered score falls within the specified band. Lower MAE indicates closer agreement with the specified band centre. 95% confidence intervals are obtained by a cluster bootstrap (

B=10{,}000

) that resamples Advisees with replacement.

Provider	$\Delta$ r (profile $-$ baseline)	95% CI	CI excludes 0
GPT-4.1	0.369	[0.225, 0.489]	yes
Claude-Sonnet-4.5	0.37	[0.294, 0.431]	yes
Mistral-Medium	0.28	[0.221, 0.345]	yes
DeepSeek-V3.2	0.425	[0.276, 0.570]	yes

Table 2: Persona effect: the difference in band-score correlation between the specified-profile and no-profile baseline conditions. Intervals and

p

values come from the paired cluster bootstrap over Advisees.

Subset	N targets	ICC(2,1) single judge	ICC(2,4) panel mean
All constructs	280	0.706	0.906
NEO-FFI	84	0.871	0.964
CHIP	112	0.545	0.827
ResQ-Care	28	0.626	0.87
CSES-8	28	0.527	0.817
Reactance	28	0.486	0.791

Table 3: Agreement among the four Auditors, as two-way random-effects intraclass correlations with absolute agreement. ICC(2,1) is the reliability of a single judge; ICC(2,4) is the reliability of the ensemble mean, which is what the study reports.

Judge panel	GPT-4.1	Claude-Sonnet-4.5	Mistral-Medium	DeepSeek-V3.2
All four (as reported)	0.779 (#1)	0.744 (#2)	0.689 (#4)	0.715 (#3)
excluding gpt-4.1	0.768 (#1)	0.723 (#2)	0.681 (#3)	0.681 (#4)
excluding claude-sonnet-4-5	0.750 (#1)	0.706 (#2)	0.686 (#3)	0.672 (#4)
excluding mistral-medium-2505	0.782 (#1)	0.753 (#2)	0.677 (#4)	0.730 (#3)
excluding DeepSeek-V3.2	0.794 (#1)	0.762 (#2)	0.700 (#4)	0.729 (#3)

Table 4: Fidelity recomputed with each Auditor removed in turn, testing whether the ranking depends on any one Auditor.

Judge	GPT-4.1	Claude-Sonnet-4.5	Mistral-Medium	DeepSeek-V3.2
GPT-4.1	0.765	0.748	0.68	0.729
Claude-Sonnet-4.5	0.782	0.709	0.655	0.683
Mistral-Medium	0.657	0.544	0.677	0.349
DeepSeek-V3.2	0.699	0.662	0.641	0.623

Table 5: Cross-tabulation of Auditor (rows) against instantiating model (columns), as band-score correlations. Diagonal cells are Auditors scoring their own model.

Comparison	$\Delta$ r	95% CI	p (boot)	CI excludes 0
GPT-4.1 $-$ Claude-Sonnet-4.5	0.035	[0.000, 0.071]	0.047	yes
GPT-4.1 $-$ Mistral-Medium	0.09	[0.041, 0.154]	0.0	yes
GPT-4.1 $-$ DeepSeek-V3.2	0.064	[-0.039, 0.207]	0.309	no
Claude-Sonnet-4.5 $-$ Mistral-Medium	0.055	[0.022, 0.109]	0.0	yes
Claude-Sonnet-4.5 $-$ DeepSeek-V3.2	0.029	[-0.065, 0.148]	0.616	no
Mistral-Medium $-$ DeepSeek-V3.2	-0.026	[-0.149, 0.103]	0.656	no

Table 6: Paired provider differences in the specified-profile condition.

\Delta r

is the difference in band-score correlation between providers, with 95% intervals from the paired cluster bootstrap using the same resampled Advisee draws for both providers;

p

is the two-sided bootstrap proportion.

Instrument	GPT-4.1	Claude-Sonnet-4.5	Mistral-Medium	DeepSeek-V3.2
NEO-FFI	0.822	0.84	0.802	0.686
CHIP	0.657	0.533	0.539	0.687
ResQ-Care	0.591	0.642	0.733	0.662
CSES-8	0.935	0.859	0.966	0.759
Reactance	0.784	0.862	0.386	0.687

Table 7: Band-score correlation by instrument and instantiating model.

Figure 1: Emotional change by Advisor. Each point is one of the seven Advisees; horizontal bars are the mean and the vertical bars are 95% confidence intervals. For the PSS and PANAS negative, a negative change indicates improvement; for PANAS positive, a positive change indicates an increase.

Advisor	Measure	Start	End	Delta	t(6)	dz	Wilcoxon p	Holm p	Survives Holm
GPT-4.1	PSS (stress)	2.29	1.97	-0.31	-3.93	-1.49	0.0156	0.0308	yes
GPT-4.1	PANAS positive	1.27	1.56	0.29	2.46	0.93	0.0625	0.0988	no
GPT-4.1	PANAS negative	3.2	2.19	-1.01	-3.17	-1.2	0.0469	0.0776	no
Claude-Sonnet-4.5	PSS (stress)	2.23	2.16	-0.07	-0.85	-0.32	0.2969	0.4262	no
Claude-Sonnet-4.5	PANAS positive	1.21	1.57	0.36	4.75	1.8	0.0312	0.0124	yes
Claude-Sonnet-4.5	PANAS negative	3.03	2.8	-0.23	-0.97	-0.37	0.4688	0.3697	no

Table 8: Emotional stabilisation: start and end values, change, paired

t

statistics and corrected

p

values for the twelve pre–post contrasts. All tests are paired and two-tailed,

t(6)

. Shapiro–Wilk did not reject normality for any contrast (all

p\geq.10

); the Wilcoxon signed-rank

p

is reported alongside because that test has low power at

n=7

. Holm correction is applied within instrument, four Advisors per family.

Instrument · Advisor	dz	95% CI of dz	p	CI excludes 0
PSS · GPT-4.1	-1.49	[-2.57, -0.36]	0.0077	yes
PSS · Claude-Sonnet-4.5	-0.32	[-1.07, 0.45]	0.4262	no
PSS · Mistral-Medium	-0.64	[-1.45, 0.20]	0.1392	no
PSS · DeepSeek-V3.2	-0.71	[-1.52, 0.15]	0.1106	no
PANAS+ · GPT-4.1	0.93	[0.00, 1.80]	0.0494	yes
PANAS+ · Claude-Sonnet-4.5	1.8	[0.54, 3.01]	0.0031	yes

Table 9: Effect sizes with confidence intervals for the twelve pre–post contrasts.

d_{z}

is the paired standardised mean difference; intervals are exact, obtained by inverting the noncentral

t

distribution (

n=7

Instrument	Advisor	n	Shapiro-Wilk W	Shapiro p	Normality rejected	paired t p	Wilcoxon p	Holm p (within instrument)	Survives Holm
PSS	GPT-4.1	7	0.84	0.1	no	0.0077	0.0156	0.0308	yes
PSS	Claude-Sonnet-4.5	7	0.944	0.675	no	0.4262	0.2969	0.4262	no
PSS	Mistral-Medium	7	0.902	0.346	no	0.1392	0.25	0.3318	no
PSS	DeepSeek-V3.2	7	0.963	0.845	no	0.1106	0.1562	0.3318	no
PANAS+	GPT-4.1	7	0.969	0.891	no	0.0494	0.0625	0.0988	no
PANAS+	Claude-Sonnet-4.5	7	0.903	0.348	no	0.0031	0.0312	0.0124	yes

Table 10: Normality and multiplicity. Shapiro–Wilk was applied to the seven paired differences behind every contrast; normality was not rejected in any of the twelve (all

p\geq.10

), though the test has low power at

n=7

, so each contrast is also reported with the distribution-free Wilcoxon signed-rank test. Familywise correction uses Holm within each instrument (four Advisors per family).

Outcome	n Advisees	F	df	p	partial eta ˆ 2	Friedman chi2	Friedman p	Advisee share of variance	Advisors differ
PSS	7	2.36	3,18	0.1059	0.282	6.85	0.0768	53%	no
PANAS+	7	0.95	3,18	0.4382	0.136	2.43	0.4887	56%	no
PANAS $-$	7	2.11	3,18	0.1351	0.26	5.23	0.1558	45%	no

Table 11: Omnibus comparison of the four Advisors on the three emotion outcomes, blocking on Advisee.

Figure 2: TENS-Life scores (1–7) by Advisor. Each point is one of the seven Advisees. TENS separates the support and frustration aspects of the effect on basic psychological needs; larger support and smaller frustration values indicate better Advisor performance. Error bars are 95% percentile-bootstrap confidence intervals.

Advisor	Support	Support 95% CI	Frustration	Frustration 95% CI
GPT-4.1	4.9	[4.38, 5.48]	1.1	[1.00, 1.29]
Claude-Sonnet-4.5	4.9	[4.67, 5.14]	1.29	[1.10, 1.57]
Mistral-Medium	3.76	[3.00, 4.57]	2.1	[1.76, 2.43]
DeepSeek-V3.2	5.29	[4.86, 5.71]	1.05	[1.00, 1.14]

Table 12: Basic psychological needs: autonomy. Values are on a 1–7 scale. Where all seven Advisees scored the scale floor, the bootstrap interval is degenerate ([1.00, 1.00]) and reflects a floor effect rather than high precision.

Advisor	Support	Support 95% CI	Frustration	Frustration 95% CI
GPT-4.1	4.38	[3.62, 5.14]	1.14	[1.00, 1.43]
Claude-Sonnet-4.5	5.86	[5.67, 6.00]	1.1	[1.00, 1.29]
Mistral-Medium	3.67	[2.76, 4.67]	2.86	[2.00, 3.76]
DeepSeek-V3.2	5.14	[4.48, 5.76]	1.0	[1.00, 1.00]

Table 13: Basic psychological needs: competence. Values are on a 1–7 scale. Where all seven Advisees scored the scale floor, the bootstrap interval is degenerate ([1.00, 1.00]) and reflects a floor effect rather than high precision.

Advisor	Support	Support 95% CI	Frustration	Frustration 95% CI
GPT-4.1	4.52	[4.00, 5.00]	1.0	[1.00, 1.00]
Claude-Sonnet-4.5	5.05	[4.67, 5.38]	1.0	[1.00, 1.00]
Mistral-Medium	3.86	[3.38, 4.43]	1.14	[1.00, 1.33]
DeepSeek-V3.2	5.1	[4.76, 5.43]	1.0	[1.00, 1.00]

Table 14: Basic psychological needs: relatedness. Values are on a 1–7 scale. Where all seven Advisees scored the scale floor, the bootstrap interval is degenerate ([1.00, 1.00]) and reflects a floor effect rather than high precision.

Advisor	Overall net support
GPT-4.1	3.52
Claude-Sonnet-4.5	4.14
Mistral-Medium	1.73
DeepSeek-V3.2	4.16

Table 15: Basic psychological needs: overall net support, computed as support minus frustration averaged across the three dimensions. Larger values indicate better support of basic psychological needs.

Figure 3: Advisor performance on CEISS total (0–36) and the MITI global scores. Each point is one of the seven Advisees; error bars are 95% percentile-bootstrap confidence intervals.

Advisor	C1	C2	C3	C4	C5	C6	C7	C8	C9	CEISS_total	CEISS_total 95% CI
GPT-4.1	3.29	3.29	3.43	1.14	2.43	1.0	3.0	3.0	2.29	22.86	[20.86, 24.71]
Claude-Sonnet-4.5	3.29	3.86	3.57	3.14	3.43	2.71	3.43	4.0	3.71	31.14	[29.43, 32.71]
Mistral-Medium	1.86	2.57	3.0	0.71	2.0	1.29	2.71	2.86	2.57	19.57	[18.14, 20.71]
DeepSeek-V3.2	3.86	3.29	4.0	1.71	3.43	1.57	3.43	3.86	3.14	28.29	[25.57, 30.71]

Table 16: Support quality: CEISS. Subscale means are per-Advisee averages (items 0–4); the total 95% CI is a percentile-bootstrap interval over the seven Advisees (

B=10{,}000

), used because the scale is bounded and normality is not assumed.

Advisor	Relational global	Technical global	% complex refl.	n def.	Refl.:question	n def.
GPT-4.1	3.64 [3.21, 4.00]	2.64 [2.21, 3.07]	1.0 [1.00, 1.00]	6/7	1.79 [0.70, 2.95]	7/7
Claude-Sonnet-4.5	3.5 [3.21, 3.79]	3.0 [2.71, 3.29]	1.0 [1.00, 1.00]	5/7	0.27 [0.13, 0.40]	7/7
Mistral-Medium	2.14 [1.79, 2.43]	2.57 [2.50, 2.71]	0.93 [0.80, 1.00]	5/7	1.0 [0.00, 2.00]	2/7
DeepSeek-V3.2	3.93 [3.71, 4.14]	3.0 [2.79, 3.21]	1.0 [1.00, 1.00]	6/7	0.85 [0.43, 1.30]	7/7

Table 17: Support quality: MITI. The two ratio indices are reported as descriptive indicators only, with the number of conversations on which each is defined; see Methods for why percent complex reflection degenerates to a presence/absence indicator in this implementation. The relational and technical globals are the interpretable MITI measures here and are the ones entering the composite score.

Figure 4: Problematic Advisor behaviours, counted per conversation. Each point is one of the seven Advisees; error bars are 95% percentile-bootstrap confidence intervals. A larger value means more instances of problematic behaviour.

Advisor	False understanding	Premature trust	False reassurance
GPT-4.1	0.0 [0.00, 0.00]	1.43 [0.57, 2.43]	0.0 [0.00, 0.00]
Claude-Sonnet-4.5	0.57 [0.14, 0.86]	0.14 [0.00, 0.43]	0.29 [0.00, 0.57]
Mistral-Medium	2.71 [2.00, 3.43]	1.14 [0.14, 2.43]	0.14 [0.00, 0.43]
DeepSeek-V3.2	0.0 [0.00, 0.00]	0.14 [0.00, 0.43]	0.0 [0.00, 0.00]

Table 18: Behavioural counter: reactance-producing expressions. Counts are per conversation, averaged across the seven Advisees, with 95% percentile-bootstrap confidence intervals. Behaviours never observed for an Advisor give a degenerate interval at zero, a true absence rather than high precision.

Advisor	Premature problem-solving	Rushing past emotion
GPT-4.1	1.0 [0.43, 1.71]	0.0 [0.00, 0.00]
Claude-Sonnet-4.5	0.71 [0.29, 1.29]	0.0 [0.00, 0.00]
Mistral-Medium	4.29 [3.00, 5.14]	0.0 [0.00, 0.00]
DeepSeek-V3.2	0.71 [0.29, 1.29]	0.0 [0.00, 0.00]

Table 19: Behavioural counter: process failures.

Advisor	Fails to close	Scope overreach
GPT-4.1	0.0 [0.00, 0.00]	0.0 [0.00, 0.00]
Claude-Sonnet-4.5	0.29 [0.00, 0.57]	0.14 [0.00, 0.43]
Mistral-Medium	0.14 [0.00, 0.43]	0.0 [0.00, 0.00]
DeepSeek-V3.2	0.0 [0.00, 0.00]	0.0 [0.00, 0.00]

Table 20: Behavioural counter: boundary and scope.

Advisor	Reactance total	Boundary overstep	Talk-listen ratio	Advisor words (mean)	Long turns
GPT-4.1	1.43 [0.57, 2.43]	0.0 [0.00, 0.00]	1.84 [1.50, 2.16]	312.95 [244.32, 382.48]	4.71 [4.00, 5.43]
Claude-Sonnet-4.5	1.0 [0.57, 1.43]	0.43 [0.14, 0.86]	1.88 [1.45, 2.40]	354.41 [278.87, 444.05]	4.57 [4.14, 5.14]
Mistral-Medium	4.0 [2.57, 5.57]	0.14 [0.00, 0.43]	3.26 [2.69, 3.84]	586.63 [495.68, 687.66]	4.71 [4.29, 5.29]
DeepSeek-V3.2	0.14 [0.00, 0.43]	0.0 [0.00, 0.00]	1.79 [1.24, 2.41]	276.27 [198.35, 368.55]	3.29 [2.57, 4.00]

Table 21: Behavioural counter: overall indices.

Family	Measure	F	df	p	partial eta ˆ 2	Friedman p	Advisee variance	Holm p (within family)	Advisors differ
TENS subscales	autonomy_support	5.66	3,18	0.0065	0.485	0.0184	23%	0.013	yes
TENS subscales	autonomy_frustration	16.67	3,18	0.0	0.735	0.0004	12%	0.0001	yes
TENS subscales	competence_support	7.88	3,18	0.0015	0.568	0.0079	23%	0.0044	yes
TENS subscales	competence_frustration	12.5	3,18	0.0001	0.676	0.0013	13%	0.0006	yes
TENS subscales	relatedness_support	10.13	3,18	0.0004	0.628	0.0054	35%	0.0016	yes
TENS subscales	relatedness_frustration	2.08	3,18	0.1391	0.257	0.1116	20%	0.1391	no

Table 22: Between-Advisor comparisons, blocking on Advisee. One-way repeated-measures analyses with Advisee as the random factor, since the same seven Advisees appear in all four arms. Friedman’s rank test accompanies each contrast as a distribution-free check. Holm correction is applied within three declared families: TENS subscales, support-quality metrics and behaviour counts. Advisee variance is the proportion of total variance attributable to the Advisee rather than the Advisor. Rushing past emotion was constant at zero across the corpus and is not testable.

Figure 5: Overall results from Studies 2, 3 and 4, and the composite score. Confidence intervals for the composite are given in Table 23 .

Rank	Advisor	Composite	95% CI
1	DeepSeek-V3.2	65.0	[62.2, 68.2]
2	GPT-4.1	59.9	[52.2, 64.8]
3	Claude-Sonnet-4.5	59.6	[55.9, 63.6]
4	Mistral-Medium	34.8	[28.2, 41.6]

Table 23: Overall Advisor ranking with confidence intervals. Intervals are percentile bootstrap over the seven Advisees (

B=10{,}000

Comparison	Difference	95% CI	p (boot)	CI excludes 0
GPT-4.1 - Claude-Sonnet-4.5	0.3	[-7.2, 6.5]	0.869	no
GPT-4.1 - Mistral-Medium	25.1	[18.9, 31.5]	0.0	yes
GPT-4.1 - DeepSeek-V3.2	-5.2	[-11.7, -0.3]	0.031	yes
Claude-Sonnet-4.5 - Mistral-Medium	24.7	[16.2, 32.2]	0.0	yes
Claude-Sonnet-4.5 - DeepSeek-V3.2	-5.5	[-10.9, -0.5]	0.034	yes
Mistral-Medium - DeepSeek-V3.2	-30.2	[-34.2, -26.2]	0.0	yes

Table 24: Pairwise differences in composite score, using the same resampled Advisee draws for both members of each pair.

Figure 6: Recovered score against specified level, by instantiating model. Points are mean recovered scores across the seven Advisees and ten constructs for each specified band, with 95% confidence intervals. The dotted line is the target band centre. Recovery is compressed toward the scale midpoint at both extremes, which accounts for the gap between the band-score correlations (

r

= 0.69–0.78) and in-band accuracy (50–63%).

Emotional Support Conversation

Emotional support conversation studies how people and AI systems provide empathetic, safe help through dialogue, including counseling, peer support, and companionship.

See all 60 papers

Same Voice, Different Lab: On the Homogenization of Frontier LLM Personalities
Mar 20, 2026Avinash Krishna, Kalyana Chadalavada, Unso Eun Seo JoPersonality Modeling in Language Models Behavioral Consistency of LLM Agents
Independent · Anthropic · Cornell University
An LLM-Native Psychometric Instrument Reveals a Self-Report--Behavior Gap Across 25 Models
Apr 24, 2026Juan Manuel ContrerasLLM Evaluation Language Model Self-Assessment
Independent Researcher
Evaluation Drift in LLM Personality Induction: Are We Moving the Goalpost?
May 16, 2026Prateek Rajput, Yewei Song, Iyiola E. Olatunji +2Personality Modeling in Language Models LLM Post-Training
University of Luxembourg, Esch-sur-Alzette, Luxembourg

Mar 20, 2026J/K move · Enter open · S save

Same Voice, Different Lab: On the Homogenization of Frontier LLM Personalities

Avinash Krishna, Kalyana Chadalavada, Unso Eun Seo Jo

Independent · Anthropic · Cornell University

Personality Modeling in Language Models Behavioral Consistency of LLM Agents Normative Conformity in Language Models Language Model Post-Training

Open paper AskPDF arXiv

LLM assistant personalities play a critical role in user experience and perceived response quality. We present a large-scale experiment of frontier LLM personalities using external ELO-based traits scoring across 144 traits. We find that all models tested converge on a form of trait expression that is systematic, methodical, and analytical and suppress traits such as remorseful and sycophantic. Moreover, models tend to diverge more in their expression of middle-of-distribution traits such as poetic or playful, but even these so-called creative models tend to have more neutral identities. These similarities suggest an implicit emergence of a standard of optimal assistant behavior. In a landscape of varied training methods, character training, therefore, stands out for its uniformity, offering insight into a tacit consensus between model developers.

Abstract

Emotional Support Conversation

Explore similar work

Same Voice, Different Lab: On the Homogenization of Frontier LLM Personalities

An LLM-Native Psychometric Instrument Reveals a Self-Report--Behavior Gap Across 25 Models

Evaluation Drift in LLM Personality Induction: Are We Moving the Goalpost?