BenchPreS: A Benchmark for Context-Aware Personalized Preference Selectivity of Persistent-Memory LLMs

Sangyeon Yoon; Hyesoo Hong; Wonje Jeung; Yongil Kim; Wooseok Seo; Heuiyeen Yeen; Albert No; Sunkyoung Kim

cs.AIMar 17, 2026

BenchPreS: A Benchmark for Context-Aware Personalized Preference Selectivity of Persistent-Memory LLMs

Authors: Sangyeon Yoon, Hyesoo Hong, Wonje Jeung, Yongil Kim, Wooseok Seo, Heuiyeen Yeen, Albert No, Sunkyoung Kim

Organizations: Yonsei University · LG AI Research

Abstract

Large language models (LLMs) increasingly store user preferences in persistent memory to support personalization across interactions. However, in third-party communication settings governed by social and institutional norms, some user preferences may be inappropriate to apply. We introduce BenchPreS, which evaluates whether memory-based user preferences are appropriately applied or suppressed across communication contexts. Using two complementary metrics, Misapplication Rate (MR) and Appropriate Application Rate (AAR), we find even frontier LLMs struggle to apply preferences in a context-sensitive manner. Models with stronger preference adherence exhibit higher rates of over-application, and neither reasoning capability nor prompt-based defenses fully resolve this issue. These results suggest current LLMs treat personalized preferences as globally enforceable rules rather than as context-dependent normative signals.

Figures & tables

Figure 1: Preference selectivity across models. Lower Misapplication Rate (MR) and higher Appropriate Application Rate (AAR) indicate stronger selectivity, with the ideal point at (0, 100). Many models lie near the dashed line (y = x), indicating limited selectivity.

Figure 2: BenchPreS setup overview. Given a task prompt and persistent memory containing user preferences, the model must generate responses that apply contextually appropriate preferences while suppressing inappropriate ones. The top example succeeds, whereas the bottom example fails.

Model	MR $\downarrow$	AAR $\uparrow$	AAR - MR $\uparrow$
Gemini 3 Pro	86.48% $\pm$ 0.72	88.69% $\pm$ 0.58	2.21
DeepSeek V3.2	61.15% $\pm$ 1.14	87.63% $\pm$ 1.26	26.48
Claude-4.5 Sonnet	52.99% $\pm$ 1.37	87.93% $\pm$ 0.81	34.94
GPT-5.2	40.95% $\pm$ 2.54	87.33% $\pm$ 0.34	46.38
Qwen3 235B A22B Thinking 2507	84.62% $\pm$ 1.06	82.85% $\pm$ 1.44	-1.77
gpt-oss-120b	74.28% $\pm$ 0.63	87.40% $\pm$ 0.89	13.12

Table 1: Quantitative Results across 10 widely used LLMs. Misapplication Rate (MR), Appropriate Application Rate (AAR), and their difference (AAR - MR). Asterisk (*) indicates non-reasoning models. Models are separated by size using 500B parameters as the cutoff. Bold indicates best-performing model for each metric.

Figure 3: Qualitative Failure Cases. Examples where models apply user preferences that should be suppressed. Segments highlighted in red denote preference reflections that are normatively inappropriate for the given context.

Panel of Figure 4 — Figure 4: Performance comparison of non-reasoning and reasoning-enabled model variants in terms of Misapplication Rate (MR), Appropriate Application Rate (AAR), and IFBench score.

Model	Prompt Strategy	MR $\downarrow$	AAR $\uparrow$
Gemini 3 Pro	Default	86.48%	88.69%
Gemini 3 Pro	Mitigation	12.80% (-73.68 pp)	84.87% (-3.82 pp)
DeepSeek V3.2	Default	61.15%	87.63%
DeepSeek V3.2	Mitigation	40.68% (-20.47 pp)	84.84% (-2.79 pp)
Claude-4.5 Sonnet	Default	52.99%	87.93%
Claude-4.5 Sonnet	Mitigation	14.04% (-38.95 pp)	84.75% (-3.18 pp)

Table 2: Effect of selectivity-oriented prompting on Misapplication Rate (MR) and Appropriate Application Rate (AAR). Values in parentheses denote percentage-point (pp) changes relative to the default setting.

Figure 5: Effect of selectivity-oriented prompting on MR and AAR for Qwen3 235B A22B 2507. Hatched regions indicate changes from the default setting.

Panel of Figure 6 — Figure 6: Performance comparison across five communication domains. Results are reported for Gemini 3 Pro, GPT-5.2, Claude-4.5 Sonnet, K-EXAONE-236B-A23B, and Llama-3.3 70B Instruct.

	Task Completeness (1–5)		Preference Selectivity
Model	Without Preferences $\uparrow$	With Preferences $\uparrow$	AAR $-$ MR $\uparrow$
Gemini 3 Pro	4.855	3.746 (-1.109)	2.21
DeepSeek V3.2	4.734	4.734 (0.000)	26.48
Claude-4.5 Sonnet	4.637	4.183 (-0.454)	34.94
GPT-5.2	4.925	4.957 (+0.032)	46.38

Table 3: Task completeness with and without preferences stored in persistent memory. Scores are measured on a 1–5 scale. Parentheses in the With Preferences column denote the difference relative to Without Preferences .

Figure 7: Misapplication Rate (MR) across preference categories. Results are reported for GPT-5.2, Claude-4.5 Sonnet, DeepSeek V3.2, Qwen3 235B A22B Thinking 2507, and Gemini 3 Pro.

Figure 8: Example of reasoning for selective preference regulation. The reasoning trace shows how the model evaluates preferences under the given communication context and suppresses those that conflict with the task. Segments highlighted in red indicate the model’s justification for excluding inappropriate preferences.

Benchmark	Focus	Persistent-memory	Application	Suppression
LaMP ( Salemi et al., 2024 )	Profile-based personalization	$\times$	$\checkmark$	$\times$
PrefEval ( Zhao et al., 2025 )	Multi-session preference application	$\times$	$\checkmark$	$\times$
RoleLLM ( Wang et al., 2024 )	Persona consistency	$\times$	$\checkmark$	$\times$
FollowBench ( Jiang et al., 2024 )	Instruction constraint compliance	$\times$	$\checkmark$	$\times$
BenchPreS	Preference selectivity	$\checkmark$	$\checkmark$	$\checkmark$

Table 4: BenchPreS differs from prior benchmarks by evaluating whether persistent-memory LLMs can both apply appropriate preferences and suppress inappropriate ones under the current communication context.

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

	Value
Profiles	10
Attributes/Profile	152
Contexts/Profile	39
Preferences ( $g{=}1$ )/Context	1.7
Preferences ( $g{=}0$ )/Context	3.3

Appendix

Table 5: Summary statistics of BenchPreS.

Evaluator Pair	Agreement	Cohen’s $\kappa$
DeepSeek-R1 vs GPT-5-mini	92.67%	0.853
DeepSeek-R1 vs Human	88.67%	0.773
GPT-5-mini vs Human	84.00%	0.680

Appendix

Table 6: Judge validation on 300 sampled instances.

Figure 9: Example of reasoning for a failure case where inappropriate preferences are applied. Bold text highlights instances where the model treats inappropriate preferences as instructions to be followed.

Recipient	Task	Public Domain
Bank Loan Officer	Apply for personal loan	finance
Credit Union Representative	Consolidate debt
Financial Advisor	Plan investment portfolio
Mortgage Broker	Refinance existing mortgage
IRS Agent	Resolve tax discrepancy
Defense Attorney	Prepare case strategy

Appendix

Table 7: List of recipient–task pairs grouped by formal communication domain.

Panel of Figure 10 — Figure 10: Prompt Template for Task Solving.

Panel of Figure 11 — Figure 11: Prompt-Based Defense Template for Task Solving.

Panel of Figure 12 — Figure 12: Prompt Template for LLM-as-Judge Evaluation.

Panel of Figure 13 — Figure 13: Prompt Template for Evaluating Task Completeness.

Personalized Language Models

Personalized language models adapt responses to individual users’ preferences, histories, writing styles, and changing needs.

See all 89 papers

PairPref: When Should Memory Guide the Answer? A Benchmark for Contextual Preference Use
Sep 28, 2026Mingfei Lu, Mengjia Wu, Yi ZhangUser Preference Modeling Memory-Augmented Language Model Agents
AAII, University of Technology Sydney
Towards Natural Personalization: Evaluating Long-Horizon Preference Following in Personalized User-LLM Interactions
Mar 4, 2026Qianyun Guo, Yibo Li, Yue Liu +1LLM Evaluation LLM Personalization
National University of Singapore, Singapore.
AlpsBench: An LLM Personalization Benchmark for Real-Dialogue Memorization and Preference Alignment
Mar 9, 2026Jianfei Xiao, Xiang Yu, Chengbing Wang +8LLM Personalization Personalized Language Model Alignment
University of Science and Technology of China · National University of Singapore

Sep 28, 2026J/K move · Enter open · S save

PairPref: When Should Memory Guide the Answer? A Benchmark for Contextual Preference Use

Mingfei Lu, Mengjia Wu, Yi Zhang

AAII, University of Technology Sydney

User Preference Modeling Memory-Augmented Language Model Agents

Open paper AskPDF arXiv

Memory-augmented assistants use retrieved preferences to guide their responses. A small change in the situation can change whether a preference is appropriate while barely affecting its retrieval similarity. Memory benchmarks typically test whether systems store and retrieve preferences, with less attention to when those preferences should apply. We introduce PairPref, a benchmark of contextual preference use. Each pair changes only the situation, keeping the preference, request, and four candidate replies fixed. The preference remains valid in both situations. In the selection track, models must choose the reply that applies the preference only where appropriate. In the free-generation track, they must decide when to apply it without seeing candidate replies. Both tracks use the same 1,227 pairs across 45 preferences and eight situation categories. We evaluate eight models, most of which achieve selection scores (

Δ

) of 51 to 65 points. In free generation, however, both responses are appropriate for their respective situations in only 3.6% to 18.3% of pairs. Models continue to apply the preference in both situations even with fewer retrieved memories, alternative presentation formats, and a stricter prompt. These results show that models still struggle to judge when user preferences apply and respond accordingly.

Abstract

Personalized Language Models

Explore similar work

PairPref: When Should Memory Guide the Answer? A Benchmark for Contextual Preference Use

Towards Natural Personalization: Evaluating Long-Horizon Preference Following in Personalized User-LLM Interactions

AlpsBench: An LLM Personalization Benchmark for Real-Dialogue Memorization and Preference Alignment