PersEvalBench: Can LLM Evaluators Judge Contextually Appropriate Use of Persona Information?

Xinyi Liu; Hooshang Nayyeri; Emine Yilmaz; Dilek Hakkani-Tur; Charith Peris; Hari Thadakamalla

cs.CLOct 13, 2026

PersEvalBench: Can LLM Evaluators Judge Contextually Appropriate Use of Persona Information?

Authors: Xinyi Liu, Hooshang Nayyeri, Emine Yilmaz, Dilek Hakkani-Tur, Charith Peris, Hari Thadakamalla

Organizations: Amazon · University of Illinois Urbana-Champaign · University College London

Abstract

Large language model (LLM) evaluators are increasingly used to assess personalized assistants, yet whether they can reliably judge personalization has rarely been tested. We introduce PersEvalBench, a benchmark testing whether an evaluator can recognize appropriate and inappropriate uses of user history. It contains 403 multi-episode scenarios where user information should persist, transfer, or cease to apply. The benchmark covers nine controlled error subtypes across three families: under-personalization, over-personalization, and misapplied-state errors. Human audits confirm that generated errors match their intended types and that the prescribed uses of user information are reasonable. Across five LLM evaluators, recognition of controlled personalization errors remains limited even with the full dialogue history. On the human-evaluated subset, under-personalization shows the largest gap: humans recognize 94.4% of these errors, compared with 63.9% for the strongest reported evaluator. Evaluators often recognize that a response is problematic but reverse the error direction, mistaking failure to follow an applicable preference (UNDER) for use of an inapplicable preference (OVER). Easier evidence access offers little help: highlighting the relevant history turns raises error recognition only slightly, suggesting that the remaining difficulty lies in judging what role that information should play in the response.

Figures & tables

Figure 1: Running example. An online-only requirement is superseded by a hybrid preference. Responses may use the update correctly, omit it, import an out-of-scope constraint, or continue using the stale state.

Figure 2: Overview of PersEvalBench . Multi-episode histories determine whether prior user information should persist, transfer, or cease to apply. Each scenario pairs a Right response with one controlled error response, which the evaluator judges from the dialogue history.

Benchmark family	Judge	Multi episode	Contextual applic.	Controlled errors
Memory and preference following	–	✓	$\triangle$	–
Selective personalization	–	$\triangle$	✓	–
Personalized Judge	✓	–	–	–
PersEvalBench	✓	✓	✓	U/O/M 9 subtypes

Table 1: Task-level comparison. Contextual applicability asks whether prior information should affect the target response and with what influence.

\triangle

denotes partial or mixed coverage.

Scale
Scenarios	403
Candidate responses	806
Avg. episodes / turns per dialogue	2.4 / 26.8
Scenario relation
Re-entry	174 (43.2%)
Cross-domain transfer	166 (41.2%)

Table 2: Scale and composition of PersEvalBench .

Model	Under-personalization				Over-personalization				Misapplied-state
	CO	PO	EV	Und.	IU	SL	UI	Ov.	SS	OD	UW	Mis.	All	FAR $\downarrow$
Kimi K2.5	63.6	48.7	44.7	52.3	78.1	70.5	90.7	79.5	61.3	42.6	93.2	65.6	66.5	24.6
GLM-5	50.0	61.5	53.2	54.6	76.6	63.6	86.0	75.5	74.2	27.7	86.4	60.7	64.3	12.2
DeepSeek-V3.2	65.9	51.3	57.4	58.5	62.5	50.0	86.0	65.6	77.4	25.5	93.2	63.1	62.5	10.9
Qwen3-32B	15.9	17.9	23.4	19.2	35.9	29.5	62.8	41.7	25.8	10.6	63.6	33.6	32.0	2.5
Constant- Over	0.0	0.0	0.0	0.0	100.0	100.0	100.0	100.0	100.0	100.0	100.0	100.0	67.7	100.0

Table 3: Error Recognition Rate (%) by error subtype in the main setting, with FAR and the constant- Over baseline. Subtype abbreviations follow Section 3.2 . Und., Ov., and Mis. pool all error responses within each family, and All pools all error responses. Bold marks the best reported evaluator per column.

Judge	Und. $n{=}36$	Ov. $n{=}40$	Mis. $n{=}24$	All	FAR $\downarrow$
Kimi K2.5	63.9	70.0	70.8	68.0	26.0
GLM-5	58.3	60.0	54.2	58.0	9.0
DeepSeek-V3.2	58.3	52.5	54.2	55.0	11.0
Qwen3-32B	25.0	32.5	29.2	29.0	1.0
Annotator A	94.4	62.5	83.3	79.0	4.0
Annotator B	94.4	50.0	79.2	73.0	7.0

Table 4: Human and model performance on the same 100 scenarios under Full context. Each response is evaluated individually.

Figure 3: Verdict behavior in the main setting. (a) Wrong-direction verdicts, pooled over evaluators: the share of under-personalization errors judged Over and of over-personalization errors judged Under . (b) Verdict split by reported evaluator for underweighting errors.

	Context availability					Scenario relation			Applicability type
Model	Target Only	Recent	Full	Full + Highl.	Gold Ev. Only	Re-entry $n{=}174$	Cross-domain $n{=}166$	Out-of-scope $n{=}63$	Categorical boundary	Graded decision
Kimi K2.5	63.5	70.7	66.5	70.5	69.0	62.6	64.5	82.5	72.9	58.8
GLM-5	46.7	66.3	64.3	65.8	62.8	62.6	61.4	76.2	69.6	58.7
DeepSeek-V3.2	50.4	58.3	62.5	65.0	63.0	60.3	64.5	63.5	67.6	56.9
Qwen3-32B	28.0	34.0	32.0	35.2	33.0	31.6	31.9	33.3	33.3	31.2
Mean FAR	22.1	30.6	–	12.5	24.0	–	–	–	–	–

Table 5: ERR (%) by context availability, scenario relation, and applicability type. Categorical (holder, scope, temporal) and Graded (strength, transferability) are unweighted family means. The final row reports mean FAR over all five evaluators for the reduced and augmented context conditions; Full-context FAR for the reported evaluators appears in Table 3 . Bold marks the best reported evaluator per column.

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Benchmark	Evaluates	Longitudinal history	Applicability boundaries	Persona-use error coverage			Matched responses
				Under	Over	Misapplied state
Assistant-side benchmarks
LoCoMo ( Maharana et al., 2024 )	Assistant / memory	✓	–	–	–	–	–
PrefEval ( Zhao et al., 2025a )	Assistant	✓	–	✓	–	–	–
PersonaLens ( Zhao et al., 2025b )	Assistant	✓	$\triangle$	$\triangle$	–	–	–
PersonaMem ( Jiang et al., 2025 )	Assistant	✓	✓ (time)	$\triangle$	–	$\triangle$ (stale)	–

Appendix

Table 6: Comparison with the closest personalization, memory, and personalized-evaluation benchmarks.

		Legal error subtypes
Relation	Central judgment	Under-personalization	Over-personalization	Misapplied-state
Re-entry	Which states remain current?	CO, PO, EV	IU, SL	SS, OD, UW
Cross-domain	What transfers, and with what weight?	CO, PO, EV	IU, UI	SS, OD, UW
Out-of-scope	Which history should be ignored?	–	IU, SL, UI	–

Appendix

Table 7: Scenario grammar: each history–target relation, its central judgment, and the error subtypes that arise naturally within it (19 of 27 relation–subtype combinations are legal).

Family	Relation	Gloss
Categorical boundaries
Holder	same attr., diff. object	held for a different person or object
Scope	same attr., diff. scope	applies in a different task or occasion
Temporal	same attr., diff. time	held at a different time
	old vs. new value	earlier value later updated
Graded decisions

Appendix

Table 8: The eight state relations and their five applicability families, grouped by the categorical vs. graded distinction of Section 5.2 .

Panel A: Scale
Scenario items	403
Candidate responses	806
Task types / task seeds	48 / 480
Avg. episodes / turns per dialogue	2.4 / 26.8
Panel B: Composition
Scenario relation

Appendix

Table 9: Benchmark statistics: scale (Panel A), composition (Panel B), and evidence realization (Panel C; distance bins of Section 3.2 ).

	Evaluation formulation (ERR / FAR)
Model	E2E	EvF	ST
Kimi K2.5	66.5 / 24.6	71.0 / 29.0	60.3 / 16.1
GLM-5	64.3 / 12.2	65.8 / 16.1	62.5 / 10.9
DeepSeek-V3.2	62.5 / 10.9	65.0 / 16.4	54.8 / 11.4
Qwen3-32B	32.0 / 2.5	42.4 / 6.0	18.1 / 1.5

Appendix

Table 10: ERR / FAR (%) by evaluation formulation under Full context (E2E/EvF/ST = End-to-end/Evidence-first/State-table; bold = highest ERR and lowest FAR among reported evaluators per column).

Condition	ERR	FAR $\downarrow$	Pair acc.
Target Only	46.0	20.8	38.8
Full $-$ relevant history	60.8	67.6	19.6
Full $-$ unrelated context	62.8	14.4	60.8
Full	64.4	12.4	64.0

Appendix

Table 11: Relevant-history ablation: five-evaluator mean ERR, FAR, and pair accuracy (%) on 50 scenarios.

Subtype	$n$	Confirmed (%)	Revised
Complete omission	12	83	2
Partial omission	12	100	0
Explicit violation	12	67	4
Irrelevant use	18	89	2
Scope leakage	10	70	3
Unsupported inference	12	100	0

Appendix

Table 12: Intervention fidelity (Study 1): per-subtype consensus confirmation rate (%) on raw pipeline generations and the number of scenarios revised in the final benchmark.

LLM Evaluation

Large language model evaluation measures what models can do, how reliably they do it, and whether benchmark scores predict real-world performance.

See all 2,408 papers

Re-Centering Humans in LLM Personalization
Jun 4, 2026Lechen Zhang, Jiarui Liu, Tal AugustHuman Preference Evaluation LLM Evaluation
†University of Illinois Urbana-Champaign · ♯Carnegie Mellon University
Neither Over- nor Under-Constrained: Evaluating LLM Personalization under a Constraint Qualification Audit
Oct 19, 2026Xuran Li, Guanqin Zhang, Imran Razzak +4LLM Evaluation LLM Personalization
University of New South Wales · Mohamed bin Zayed University of Artificial Intelligence · The Technology Innovation Institute +1
LUNAR: Benchmarking Personalized Large Language Models on UNiversal User BehAvioR Logs
Aug 5, 2026Jiahao Zhang, Yongzhi Tong, Zelin Fu +4LLM Evaluation LLM Personalization
1Shenzhen University of Advanced Technology · ♣Work done at Ant Group · 2Ant Group +1

Jun 4, 2026J/K move · Enter open · S save

Re-Centering Humans in LLM Personalization

Lechen Zhang, Jiarui Liu, Tal August

†University of Illinois Urbana-Champaign · ♯Carnegie Mellon University

Human Preference Evaluation LLM Evaluation LLM Personalization Personalized Language Models User Modeling Personalized Text Generation+1

Open paper AskPDF arXiv

Despite growing interest, most evaluations of large language models' (LLMs') personalization abilities have relied on synthetic data. It remains unclear how well current personalization systems work for real users. In this paper, we study the gap in LLM personalization performance when using synthetic versus human data. We collect human conversations (550 conversations) and judgments across three stages of personalization: extracting user attributes from conversations (5,949 judgments), pairing relevant attributes with new prompts (11,919), and incorporating relevant attributes into a personalized response (1,101). Incorporating human data reveals system limitations at each stage. Models struggle to extract attributes from human conversations, disagree with human judgments on relevant attributes, and generate personalized responses that humans judge no better than generic responses (though that LLM judges widely rate as better). We introduce two lightweight training-based interventions that shift automated personalization evaluation closer to human data in our first two stages. However, in our third stage we find that learned reward models achieve only modest correlation with human ratings, suggesting that human-aligned personalization quality judgments are difficult to model directly. Our collected data provides a foundation for studying how models should extract, select, and incorporate user information in ways that humans find useful.

Abstract

LLM Evaluation

Explore similar work

Re-Centering Humans in LLM Personalization

Neither Over- nor Under-Constrained: Evaluating LLM Personalization under a Constraint Qualification Audit

LUNAR: Benchmarking Personalized Large Language Models on UNiversal User BehAvioR Logs