Does Reasoning Improve Psychological Depth in Large Language Models? It Depends on Who's Judging

Ruichen Zheng; Yihe Wang; Fabrice Y Harel-Canada; Sara Khosravi; Zeynep Senahan Yildiz; Amit Sahai; Nanyun Peng

cs.LGSep 12, 2026

Does Reasoning Improve Psychological Depth in Large Language Models? It Depends on Who's Judging

Authors: Ruichen Zheng, Yihe Wang, Fabrice Y Harel-Canada, Sara Khosravi, Zeynep Senahan Yildiz, Amit Sahai, Nanyun Peng

Organizations: University of California, Los Angeles · Tsinghua University · LA General Hospital · Google

Abstract

LLM-as-a-Judge evaluators are increasingly used to score open-ended generation, yet a judge's correlation with human ratings on its development set may not guarantee valid measurement when outputs are closely matched and human preferences are subjective. We study this failure mode through psychological depth in short stories. Seven human readers and an LLM-judge ensemble selected on the original scalar Psychological Depth Scale dataset ( $ρ= 0.646$ ) evaluated 60 blinded, prompt-matched story pairs from GPT-5 vs.\ GPT-4o and DeepSeek-R1 vs.\ DeepSeek-V3. Human preferences showed no universal reasoning advantage: GPT-5 was modestly preferred over GPT-4o (60.0--62.9%), whereas DeepSeek-R1 trailed V3 (42.9%), and inter-reader agreement was near chance (Krippendorff's $α= 0.070$ ), with within-reader consistency and recurring weighting patterns suggesting structured heterogeneity rather than random responding. The judge, by contrast, favored reasoning outputs in 89.0% of dimension-level comparisons and 59 of 60 pairs on aggregate PDS, uniformly across all five evaluator configurations, and its scores were associated with surface features such as sentence length and lexical diversity. These results suggest that development-set performance is insufficient evidence for deployment validity on a shifted distribution, and that point-estimate judges can obscure the heterogeneity in subjective human evaluation.

Figures & tables

Panel of Figure 1 — Figure 1: Human preferences vary by model pairing and dimension; the LLM judge does not. (a) Human preference for the reasoning-model story with pointwise 95% GLMM intervals; the dashed line marks chance. (b) LLM-judge preference for the reasoning-model story pooled over all 60 pairs. Solid bars are reasoning wins; hatched segments credit each exact score tie 0.5 to the reasoning story; labels give the resulting even-split rate, with raw tie counts in parentheses. Aggregate PDS averages the five dimension scores and is a distinct target from the human overall-preference question.

Appendix figures & tables25 assets

Supplementary material from the paper’s appendix.

Appendix

	DeepSeek-R1	GPT-5 auto	GPT-5 high
Dimension	vs. V3	vs. 4o	vs. 4o
Authenticity	47.9% [39.0, 56.4]	55.7% [47.8, 65.6]	52.1% [43.0, 61.6]
Empathy	52.9% [44.4, 62.2]	68.6% [62.7, 78.3]	69.3% [64.4, 80.1]
Engagement	39.3% [29.6, 46.6]	58.6% [50.7, 67.3]	65.0% [58.1, 74.4]
Emotion prov.	61.4% [53.6, 69.9]	70.7% [64.2, 79.3]	62.9% [55.5, 72.3]
Narr. complexity	58.6% [50.9, 67.7]	55.0% [47.0, 64.3]	52.9% [44.5, 62.5]

Appendix

Table 1: Human preference for the reasoning-model output, with pointwise 95% GLMM intervals. Rates above 50% favor the reasoning model. The 18 intervals are unadjusted for multiplicity and interpreted descriptively.

Dimension	Fleiss’ $\kappa$	Kripp. $\alpha$	Raw
Authenticity	0.035	0.037	52.5%
Emotion prov.	0.080	0.082	54.1%
Empathy	0.182	0.185	59.1%
Engagement	0.059	0.061	53.3%
Narr. complexity	0.016	0.018	52.1%
Overall preference	0.035	0.038	51.9%

Appendix

Table 2: Inter-reader agreement across 60 pairs and seven readers. Raw is mean pairwise agreement.

Dimension	All	R1 vs. V3	GPT-5 auto vs. 4o	GPT-5 high vs. 4o
Authenticity	0.037	0.073	0.053	$-0.015$
Emotion prov.	0.082	0.035	0.147	0.072
Empathy	0.185	0.047	0.176	0.290
Engagement	0.061	0.101	0.010	0.080
Narr. complexity	0.018	0.049	$-0.013$	0.005
Overall preference	0.038	0.074	$-0.003$	0.050

Appendix

Table 3: Krippendorff’s

\alpha

by model pairing.

Figure 2: Exploratory reader-level structure. Left: coefficients from separate logistic regressions relating PDS dimensions to each reader’s overall preference. Right: within-reader Cohen’s $\kappa$ between each dimension and overall preference. Similar clusters across analyses are consistent with recurring dimension-weighting patterns, but the seven-reader sample precludes population-level archetype claims.

Pairing	Human overall votes	Judge agg. PDS	$\Delta$	$d_{z}$
GPT-5 auto vs. 4o	84/140 (60.0%)	20/20	$+0.518$	1.14
GPT-5 high vs. 4o	88/140 (62.9%)	20/20	$+0.500$	1.12
DeepSeek-R1 vs. V3	60/140 (42.9%)	19/20	$+0.214$	0.68
Overall	232/420 (55.2%)	59/60	$+0.411$	0.95

Appendix

Table 4: Human overall-preference votes and judge aggregate-PDS pair decisions favoring reasoning-model outputs. These columns measure different targets and are not direct agreement statistics.

\Delta

is the descriptive mean paired judge-score difference (reasoning minus comparison) pooled across the five routed dimensions, and

d_{z}

is the corresponding standardized mean difference. Because dimensions from the same pair are dependent, we do not report naive paired-test inference.

Evaluator	AUT	PROV	EMP	ENG	NCOM	Mean/routed $\rho$
Llama 3.1 70B, 7 demo	0.576	0.674	0.637	0.507	0.637	0.606
Llama 3.1 70B, 5 demo	0.588	0.655	0.596	0.450	0.575	0.573
Llama 3.1 70B, 10 demo	0.561	0.702	0.671	0.387	0.568	0.578
Llama 3.3 70B, 10 demo	0.580	0.676	0.702	0.407	0.550	0.583
Qwen 3.5 397B, 7 demo	0.511	0.663	0.602	0.564	0.496	0.567
Llama 3.1 70B CoT, 10 demo	0.499	0.627	0.658	0.356	0.664	0.561

Appendix

Table 5: Spearman correlation with human PDS annotations on the 80% development/selection split. Bold marks the configuration selected for each dimension; these are not untouched-test estimates. For individual configurations, the final column is the mean of the five displayed correlations; for the routed ensemble, it is computed from routed predictions.

Configuration	Auth.	Emo.	Emp.	Eng.	Narr.	Ties	Decisive	Even-split
Llama 3.1 70B (5d)	43/44 ∗	54/57	49/54	47/51	53/55	39	94.3%	88.5%
Llama 3.1 70B (10d)	43/43	53/53 ∗	48/51	46/46	54/54	53	98.8%	90.2%
Llama 3.3 70B (10d)	32/33	53/55	51/57 ∗	47/58	47/56	41	88.8%	83.5%
Qwen 3.5 397B (7d)	57/57	58/58	55/58	57/59 ∗	57/58	10	97.9%	96.3%
Llama 3.1 70B CoT	7/8	49/50	47/53	43/55	49/59 ∗	75	86.7%	77.5%
Routed ensemble	43/44	53/53	51/57	57/59	49/59	28	93.0%	89.0%

Appendix

Table 6: Reasoning-output preference for five evaluator configurations and the routed ensemble. Dimension cells report reasoning wins / non-tied pairs; ∗ marks the routed dimension. Decisive pools wins over non-tied comparisons; Even-split credits ties 0.5 to each story over all 300 comparisons.

Dimension	Ties	Reasoning wins	Even-split rate
Authenticity	16	43	85.0%
Emotion prov.	7	53	94.2%
Empathy	3	51	87.5%
Engagement	1	57	95.8%
Narr. complexity	1	49	82.5%
Five dimensions pooled	28	253	89.0%

Appendix

Table 7: Tie frequency and reasoning-output preference for the routed ensemble. Even-split rate

=(\text{wins}+0.5\times\text{ties})/60

per dimension, or

/300

for the pooled row. Aggregate PDS is a separate evaluator target, not human overall preference.

Feature	Auth.	Emo.	Emp.	Eng.	Narr.
Word count	$-.28$	$-.19$	$-.12$	$-.18$	$+.09$
Mean sentence length	$-.32$	$-.43$	$-.17$	$-.70$	$-.09$
MATTR	$-.37$	$-.33$	$-.35$	$-.29$	$-.14$
MTLD	$-.24$	$-.19$	$-.25$	$-.09$	$+.03$

Appendix

Table 8: Story-level Pearson correlation between surface features and ensemble judge scores (

n=120

Feature	Auth.	Emo.	Emp.	Eng.	Narr.
Word count	$-.64$	$-.40$	$-.18$	$-.54$	$+.11$
Mean sentence length	$-.56$	$-.47$	$-.27$	$-.59$	$+.03$
MATTR	$-.31$	$-.17$	$-.24$	$-.28$	$+.13$
MTLD	$-.39$	$-.13$	$-.15$	$-.36$	$+.31$

Appendix

Table 9: Within-pair standardized regression coefficients (

\beta

) relating surface features to ensemble judge scores (

n=60

pairs).

Model	Temperature	Other
GPT-4o	1.0 (default)	—
GPT-5 (auto)	1.0 (default)	—
GPT-5 (high)	1.0 (default)	reasoning_effort=high
DeepSeek-V3	1.0	—
DeepSeek-R1	1.0	thinking tags stripped

Appendix

Table 10: Generation parameters. All models used temperature

=1.0

and default settings for top-

p

, frequency penalty, etc. GPT-5 (high) used OpenAI’s reasoning_effort parameter. No max-token limit was set.

Sourced from r/WritingPrompts
ID	Premise
1	While other god’s shrines are magnificent, yours is a bit too humbling. And yet a little girl visits you every year after stumbling upon it, never missing a year even as she grows old. Deeply moved, you decide to give her a parting gift greater than what any other God would dare to give.
2	Elves realized ages ago that, due to the lifespan differences between them, humans will eventually forget the reasoning for various treaties and break them. So it’s understandably shocking to the elves when they discover some long-forgotten treaties in their own records that humans never broke.
3	A princess who is going to be in an arranged marriage runs away. She cuts her hair and pretends to be a man. However, she runs into the prince who was going to get married to her. He also ran away, and he is pretending to be a woman. They instantly recognize each other.
4	"You know… Most vampires I hunted tend to see humans as just cattle." "Oh, don’t misunderstand me. This is just how I see you too." "Really? Then why don’t you hurt people in your town?" "Well, you don’t slaughter dairy cows for their meat, do you?"
5	You’re a hero with a weird name. “Anything for $20”. You gain the ability to do anything, as long as you’re offered$ 20. Everyone takes it as a joke, until one day there’s a cataclysm, and someone offers you $20 to end it.

Appendix

Table 11: All 20 premises used to prompt the models. The selection was intended to include speculative and grounded settings; 15 of 20 premises came from r/WritingPrompts. Premises from r/WritingPrompts are reproduced verbatim, including original spelling and grammatical errors.

Figure 3: Annotation interface: instructions and PDS metric definitions shown to annotators.

Figure 4: Annotation interface: side-by-side story presentation with premise shown above.

Figure 5: Annotation interface: evaluation questions and optional comment field.

Figure 6: Distribution of annotation time per pair for each of the 7 human annotators. Times exceeding 600 s are capped for display; the number of capped outliers is shown in each subplot title. Red dashed lines mark the median; navy dotted lines mark the mean of capped times. The rightmost panel compares median and mean times across annotators. Overall median annotation time is 5.2 minutes (n = 420).

Figure 7: Annotation time per pair plotted chronologically, with linear trends per annotator. Red points denote times capped at 600 s. Dashed black lines show linear trends; colored dotted lines show 5-pair rolling averages. The rightmost panel overlays all trend lines.

Figure 8: Leave-One-Annotator-Out (LOAO) stability analysis: shift in the aggregate reasoning-model win rate when each annotator is removed.

Pair	Ann	Pref.	Comment
Pair 0 (4–3 for A)	1	A	“A: greater focus on the abstract. B: prose heavy, but greater focus on concrete; convoluted”
	5	A	“Story A was much more engaging because it was straightforward in depicting the stages of life…”
	7	B	“Story B delves deeper into the characterization of the humble god…Death of worshiper is more profound”
Pair 24 (5–2 for A)	1	A	“A: good imagery, dialogue. B: interesting parallels, odd tone”
Pair 24 (5–2 for A)	7	B	“A is a step by step of the prompt given. B builds up characters, establishes risk, and is authentic.”
Pair 27 (4–3 for B)	1	A	“A: like a prologue. B: prose, very nonsensical”

Appendix

Table 12: Selected annotator comments across five story pairs. Only annotators who provided free-text justifications are shown.

Figure 9: Evaluating psychological depth on a representative story pair (Pair 24, GPT-5 vs. GPT-4o). Left: human annotators displayed diverse preferences across PDS dimensions and overall preference. Right: all five evaluator configurations favored GPT-5 on every dimension. Blue denotes preference for GPT-5 and red preference for GPT-4o.

Figure 10: Preference rates for reasoning models across all 60 pairs for each of the 7 human annotators, the human average, and the ensemble LLM judge. Red denotes stronger reasoning-model preference, blue stronger base-model preference, and white 50%. Judge percentages credit each exact score tie 0.5 to the reasoning model (even-split rate; Table 7 gives tie counts). The final row compares human overall preference with the judge’s aggregate PDS, which are different targets.

Figure 11: Preference rates for DeepSeek-R1 vs. DeepSeek-V3 (20 pairs) across 7 human annotators and the GLMM group estimate.

Figure 12: Preference rates for GPT-5 (auto) vs. GPT-4o (20 pairs) across 7 human annotators and the GLMM group estimate.

Figure 13: Preference rates for GPT-5 (high) vs. GPT-4o (20 pairs) across 7 human annotators and the GLMM group estimate.

	Auth.	Emo.	Emp.	Eng.	Narr.	Pref.
Reader 1	B	B	B	B	R	B
Reader 2	R	B	B	R	R	R
Reader 3	B	B	B	B	R	B
Reader 4	B	B	B	R	B	B
Reader 5	B	B	B	R	B	B
Reader 6	B	B	B	R	R	B

Appendix

Table 13: Per-reader pairwise preferences for Pair 56 (GPT-5 high vs. GPT-4o). R = prefers reasoning model, B = prefers comparison model.

LLM-as-a-Judge

Large language model (LLM)-as-a-judge research uses language models to score, compare, and verify AI outputs as scalable alternatives to human evaluation.

See all 559 papers

Level-k Distinguishable Mechanisms for Evaluating Bounded Rationality in LLMs
Aug 21, 2026Binchi Zhang, Atrisha SarkarLLM Evaluation Strategic Reasoning Risks in Language Models
University of Western Ontario
How Much Do LLM-as-a-Judge Design Choices Matter? A Systematic Comparison of Prompt Designs, Rating Scales, and Models
Oct 4, 2026Laurène Vaugrante, Thilo HagendorffLLM Evaluation LLM-as-a-Judge
AI Safety Research Group, University of Stuttgart
Think Deep, Not Just Long: Measuring LLM Reasoning Effort via Deep-Thinking Tokens
Feb 13, 2026Wei-Lin Chen, Liqian Peng, Tian Tan +5LLM Inference Efficiency Inference-Time Scaling
University of Virginia · Google

Aug 21, 2026J/K move · Enter open · S save

Level-k Distinguishable Mechanisms for Evaluating Bounded Rationality in LLMs

Binchi Zhang, Atrisha Sarkar

University of Western Ontario

LLM Evaluation Strategic Reasoning Risks in Language Models Reasoning Consistency in Language Models

Open paper AskPDF arXiv

Strategic depth of reasoning is essential for human interaction of Large Language Models (LLMs) operating in boundedly rational environments. However, existing evaluations are primarily based on canonical games prevalent in pretraining corpora, making it difficult to disentangle true strategic reasoning from memorisation. To address this, we formalise a necessary level-K distinguishability condition for strategic depth inference and construct a suite of novel game structures that meet this standard. Using these games, we evaluate strategic depth in LLMs from both the Chain-of-Thought tokens and actual actions under recursive reasoning and an inductive trace of opponent game-play data. Across experimental trials spanning four LLMs, four game structures, and ten levels of iterated reasoning, we find that model models maintain accurate strategic depth under recursive reasoning, with strong internal consistency between stated reasoning and actions at every level. Errors arise from using the wrong number of iterated depth of reasoning steps, not from computing best responses incorrectly. However, inductive inference from opponent play degrades accuracy sharply and unevenly across games, and explicit strategic mentalizing in the chain of thought substantially improves overall performance.

Abstract

LLM-as-a-Judge

Explore similar work

Level-k Distinguishable Mechanisms for Evaluating Bounded Rationality in LLMs

How Much Do LLM-as-a-Judge Design Choices Matter? A Systematic Comparison of Prompt Designs, Rating Scales, and Models

Think Deep, Not Just Long: Measuring LLM Reasoning Effort via Deep-Thinking Tokens