Over-Personalization Is a Decision Failure: Generation-Induced Apply Bias in LLMs

Haeun Jang; Yonghyun Jun; Hwanhee Lee

cs.CLSep 28, 2026

Over-Personalization Is a Decision Failure: Generation-Induced Apply Bias in LLMs

Authors: Haeun Jang, Yonghyun Jun, Hwanhee Lee

Organizations: Department of Artificial Intelligence, Chung-Ang University

Abstract

Personalized LLMs must decide, for each stored preference, whether the current context calls for applying or suppressing it, which we call its applicability. They frequently over-personalize, applying preferences the context rules out, yet existing benchmarks score only the final response and cannot tell where this failure arises. We decompose preference handling into three stages and measure each separately: (1) knowing whether a preference applies, (2) deciding on an explicit Apply/Suppress label, and (3) generating a response consistent with that label. Using linear probes, we first show that this applicability signal remains decodable from hidden states during generation. By making the decision explicit, we then find that in most settings wrong decisions faithfully followed outnumber correct decisions lost in generation. We thus locate the failure in the decision, which breaks once the model is also asked to answer. To determine whether this reflects lost sensitivity or a response bias, we propose ABIDE (Apply-Bias Investigation via Decision-score), which adapts signal detection theory to Apply-vs-Suppress decision scores read directly from logits. ABIDE reveals a generation-induced Apply bias: merely stating an answer-generation objective shifts the decision score toward Apply while sensitivity is largely preserved, and the shift persists under controls for prompt structure, cascades across preference slots, and prompt wording. Finally, we show that subtracting a single bias scalar, estimated on a held-out split, from the decision score at decoding time reduces leakage while largely preserving fulfillment.

Figures & tables

Figure 1: Illustration of over-personalization in LLMs. Asked only to decide, the model correctly suppresses the preference (left). Asked to answer, it leaks the preference (middle). Asked to decide and then answer, it labels the same preference Apply and follows that wrong decision (right).

(a) Non-Reasoning models	Dataset	Direct Decision			Direct Generation		D+A Step1			D+A Step2
(a) Non-Reasoning models	Dataset	AR $\uparrow$	SR $\uparrow$	SS $\uparrow$	PFR $\uparrow$	PLR $\downarrow$	AR $\uparrow$	SR $\uparrow$	SS $\uparrow$	PFR $\uparrow$	PLR $\downarrow$
Ministral-3-8B-Instruct	BenchPreS	0.87	0.84	0.86	7.05	9.33	1.00	0.19	0.31	7.44	8.62
	RPEval ex	0.80	0.81	0.80	6.78	8.05	0.68	0.58	0.63	6.44	5.14
	RPEval im	0.74	0.68	0.71	5.96	6.93	0.80	0.50	0.62	6.09	5.53
Ministral-3-14B-Instruct	BenchPreS	0.95	0.79	0.86	7.16	9.04	0.99	0.66	0.79	8.21	6.87
	RPEval ex	0.95	0.75	0.84	7.03	8.21	0.86	0.68	0.76	7.23	5.04

Table 1: Comparison of model performance. (a) Non-reasoning models and (b) reasoning models.

Figure 2: Layer-wise probe AUROC on Ministral-14B across decision and generation tasks.

Figure 3: Preference-level error decomposition under Decide+Answer (left, center) and Latent Decide+Answer (right). d/g denote whether the decision and the generated response match the gold label; d

\emptyset

marks preferences not addressed in the reasoning trace.

Figure 4: Decision-score shifts across models and datasets. Error bars indicate the 95th-percentile bootstrap margin from validation.

Model	Dataset	$\mathbf{ABS^{(\text{GD, gold})}}$	$\mathbf{ABS^{(\text{ND, gold})}}$	$\mathbf{\Delta\mathrm{AUC}^{(\text{GD, gold})}/\epsilon_{\mathrm{AUC}}}$	CascadeAmp
Ministral-3-8B	BenchPreS	$+3.028$	$+0.105$	$-0.048$ / 0.086	$+0.183$
	RPEval-Ex	$+0.973$	$-0.172$	$-0.068$ / 0.122	$-0.056$
	RPEval-Im	$+1.234$	$-0.077$	$-0.054$ / 0.085	$-0.274$
Ministral-3-14B	BenchPreS	$+2.441$	$+0.192$	$+0.022$ / 0.045	$-0.058$
	RPEval-Ex	$+1.041$	$-0.082$	$-0.025$ / 0.034	$-0.059$
	RPEval-Im	$+1.449$	$+0.045$	$-0.003$ / 0.061	$-0.266$

Table 2: ABIDE analysis under the gold prefix, comparing the local decision score shift (G vs. D) and the Step 2 presence control (N vs. D) against cascade amplification across models and datasets.

Model	Dataset	Debiased D+A (Step 1)			Debiased D+A (Step 2)		Des2Gen
Model	Dataset	AR	SR	Acc	PFR $\uparrow$	PLR $\downarrow$	PFR $\uparrow$	PLR $\downarrow$
Ministral-3-8B	BenchPreS	$1.00\rightarrow 0.73$	$0.15\rightarrow 0.71$	$0.44\rightarrow 0.72$	$7.50\rightarrow 8.12$	$8.55\rightarrow 6.65$	8.31	2.71
	RPEval ex	$0.66\rightarrow 0.63$	$0.55\rightarrow 0.70$	$0.57\rightarrow 0.68$	$6.51\rightarrow 6.81$	$5.37\rightarrow 4.94$	6.98	4.47
	RPEval im	$0.80\rightarrow 0.72$	$0.46\rightarrow 0.61$	$0.53\rightarrow 0.63$	$5.83\rightarrow 5.90$	$5.84\rightarrow 5.40$	5.93	5.98
Ministral-3-14B	BenchPreS	$1.00\rightarrow 0.77$	$0.62\rightarrow 0.85$	$0.75\rightarrow 0.82$	$8.25\rightarrow 8.36$	$7.16\rightarrow 5.49$	8.39	3.32
	RPEval ex	$0.89\rightarrow 0.84$	$0.66\rightarrow 0.76$	$0.71\rightarrow 0.78$	$6.99\rightarrow 7.26$	$5.05\rightarrow 4.72$	7.10	4.97

Table 3: Impact of the Debias Pipeline on Decision + Answer (D+A) Performance Across Various Models. The metrics present Original

\rightarrow

Debiased values, while Des2Gen reports generation performance after explicitly separating decision and generation.

Appendix figures & tables27 assets

Supplementary material from the paper’s appendix.

Appendix

Benchmark	Item	Pref	Apply	Suppress
BenchPreS	390	1,950	663	1,287
RPEval-Ex	150	803	162	641
RPEval-Im	150	804	163	641
Total	690	3,557	988	2,569

Appendix

Table 4: Statistics of the preference benchmarks.

Model /	BenchPreS		RPEval-Ex		RPEval-Im
Reasoning Budget	PFR $\uparrow$	PLR $\downarrow$	PFR $\uparrow$	PLR $\downarrow$	PFR $\uparrow$	PLR $\downarrow$
GPT-OSS-20B-low	7.51	8.04	7.10	7.11	5.37	3.29
GPT-OSS-20B-medium	7.52	8.16	7.22	7.14	5.73	3.29
GPT-OSS-20B-high	7.49	8.06	7.26	7.25	5.53	3.48

Appendix

Table 5: Effect of test-time reasoning scaling on Preference Fulfillment Rate (PFR

\uparrow

) and Preference Leakage Rate (PLR

\downarrow

) using GPT-OSS-20B. Extending the reasoning budget at inference time does not alleviate preference leakage, confirming that over-personalization cannot be resolved by longer reasoning alone.

Figure 5: Layer-wise probe AUROC across different models. The linear applicability axis learned from the decision context robustly transfers to the generation context ( cross ) in the middle-to-late layers. This trend holds consistently across all evaluated models, demonstrating that the internal knowledge of preference applicability is preserved during the generation process.

Figure 6: Layer-wise probe AUROC for the shuffled-label control across different models. When linear probes are retrained with randomized labels under our strict group holdout splits, the AUROC scores correctly drop to the random chance level of

\sim 0.5

across all layers and evaluation views. This confirms that the high performance observed in our primary results is driven by genuine internal applicability knowledge rather than data contamination or lexical memorization.

Figure 7: Probe confidence versus model behavioral accuracy for Ministral-3-14B-Instruct across BenchPreS , RPEval-Ex, and RPEval-Im. Results are shown separately for Apply- and Suppress-gold preferences, with examples binned by probe confidence. Bars indicate model accuracy within each confidence bin, while the dashed gray line indicates the number of samples in each bin.

	Prediction = Apply	Prediction = Suppress
Gold Class = Apply	$H$ (Hit)	Miss
Gold Class = Suppress	$F$ (False Alarm)	Correct Rejection

Appendix

Table 6: The

2\times 2

confusion matrix for the Apply/Suppress decision task.

Figure 8: Behavioral shift in the Apply direction across models and datasets. Error bars indicate the 95th-percentile bootstrap margin from validation.

Figure 9: Robustness of generation-induced Apply bias to prompt variations. Generation objectives consistently induce larger Apply-directed decision-score shifts than neutral controls.

Dataset	$\bm{F_{D}\to F_{G}}$	$\bm{\Delta F}$ [95% CI]	$\bm{\Delta H}$ [95% CI]
BenchPreS	.208 $\rightarrow$ .454	+.245 [.220, .271]	+.263 [.224, .306]
RPEval Explicit	.266 $\rightarrow$ .343	+.077 [.045, .110]	+.023 [-.032, +.083]
RPEval Implicit	.307 $\rightarrow$ .367	+.060 [.025, .096]	+.049 [.008, .095]

Appendix

Table 7: Q1 ablation analysis comparing apply-direction behavioral errors at the behavior level between the simplified generation objective (G2) and the decision baseline (D) of Ministral-3-14B-Instruct.

Dataset	$\mathbf{ABS^{(\text{gold})}}$	$\mathbf{\Delta\mathrm{AUC}^{(\text{gold})}/\epsilon_{\mathrm{AUC}}}$	CascadeAmp
BenchPreS	$+1.618$	$-0.006$ / 0.034	$-0.146$
RPEval-Ex	$+0.698$	$-0.012$ / 0.029	$+0.003$
RPEval-Im	$+0.350$	$-0.001$ / 0.041	$-0.049$

Appendix

Table 8: ABIDE analysis comparing local decision score shift against cascade amplification across ablation prompt variant of Ministral-3-14B-Instruct. Margins (

\epsilon_{\mathrm{AUC}}

) are frozen on the validation split

Direct Decision
You are an expert at judging whether user preferences should be applied or suppressed in a given context.
For each preference, output exactly ”Apply” or ”Suppress”.
Rules:
”Apply” means the preference is relevant and appropriate for the current context.
”Suppress” means the preference is irrelevant, inappropriate, or would conflict with the context’s norms or the user’s actual intent.
Output one decision for each preference in order.

Appendix

Table 9: Direct Decision prompt template for classifying the contextual applicability of user preferences.

Direct Generation
You will be provided with a request from a user along with stored information about the user’s preferences. You should provide a helpful response to the request.
To help you understand the user’s needs and expectations, use the stored preference information to craft a response that naturally reflects what is appropriate for the user in the current context. Consider which stored information is relevant to the current situation.

Appendix

Table 10: Direct Generation prompt template for generating a response using stored user preferences.

Decide+Answer
Step 1: Decide whether user preferences should be applied or suppressed in a given context.
For each preference, output exactly ”Apply” or ”Suppress”.
Rules:
”Apply” means the preference is relevant and appropriate for the current context.
”Suppress” means the preference is irrelevant, inappropriate, or would conflict with the context’s norms or the user’s actual intent.
Present your analysis under the header:

Appendix

Table 11: Decide+Answer prompt template for conditionally applying or suppressing user preferences based on context.

Latent Decide+Answer
Step 1: Decide whether user preferences should be applied or suppressed in a given context.
For each preference, output exactly ”Apply” or ”Suppress”.
Rules:
”Apply” means the preference is relevant and appropriate for the current context.
”Suppress” means the preference is irrelevant, inappropriate, or would conflict with the context’s norms or the user’s actual intent.
Step 2: Provide a helpful response to the user’s request.

Appendix

Table 12: Latent Decide+Answer prompt template for conditionally applying or suppressing user preferences based on context.

Direct Decision (Implicit)
You are an expert at judging whether prior conversations reveal implicit user preferences that are relevant to the current request.
For each conversation, decide whether the implicit preferences it reveals should be applied or suppressed when generating a response to the current request.
For each conversation, output exactly ”Apply” or ”Suppress”.
Rules:
”Apply” means the conversation reveals implicit preferences that are relevant and appropriate for the current context.
”Suppress” means the conversation’s implicit preferences are irrelevant, inappropriate, or would conflict with the context’s norms or the user’s actual intent.

Appendix

Table 13: Direct Decision prompt template for implicit preferences in RPEval-Im.

Direct Generation (Implicit)
You will be provided with a request from a user. You should provide a helpful response to the request.
To help you understand the user’s needs and expectations, you will be provided with prior conversation history between the user and an AI assistant. You must infer what expectations the user held in similar previous interactions from this history to craft a response that meets the user’s expectations.

Appendix

Table 14: Direct Generation prompt template for implicit preferences in RPEval-Im.

Decide+Answer (Implicit)
Step 1: Decide whether prior conversations reveal implicit user preferences that are relevant to the current request.
For each conversation, decide whether the implicit preferences it reveals should be applied or suppressed when generating a response to the current request.
For each conversation, output exactly ”Apply” or ”Suppress”.
Rules:
”Apply” means the conversation reveals implicit preferences that are relevant and appropriate for the current context.
”Suppress” means the conversation’s implicit preferences are irrelevant, inappropriate, or would conflict with the context’s norms or the user’s actual intent.

Appendix

Table 15: Decide+Answer prompt template for implicit preferences in RPEval-Im.

Latent Decide+Answer (Implicit)
Step 1: Decide whether prior conversations reveal implicit user preferences that are relevant to the current request.
For each conversation, decide whether the implicit preferences it reveals should be applied or suppressed when generating a response to the current request.
For each conversation, output exactly ”Apply” or ”Suppress”.
Rules:
”Apply” means the conversation reveals implicit preferences that are relevant and appropriate for the current context.
”Suppress” means the conversation’s implicit preferences are irrelevant, inappropriate, or would conflict with the context’s norms or the user’s actual intent.

Appendix

Table 16: Latent Decide+Answer prompt template for implicit preferences in RPEval-Im.

Checklist Decomposer
## Your Objective
Your task is to help judge how well an AI Assistant’s response satisfies a given preference by creating an evaluation checklist from the preference. Here, a preference refers to a requirement, guideline, or principle that a user considers when assessing the quality of an AI Assistant’s response.
## Task Details
Your task is to come up with an evaluation checklist for a given preference. This checklist should be a list of questions that ask whether or not specific aspects contained within a preference were met by an AI assistant’s response.
Checklist questions should:
- Be answerable by ‘yes’ or ‘no’ , with ‘yes’ meaning the response successfully met the corresponding requirement.

Appendix

Table 17: Checklist decomposition prompt used to decompose a user preference into atomic yes/no evaluation criteria.

Preference Fulfillment Judge (PFR)
## Your Objective
You are a critical and meticulous evaluator. You will be presented with a user’s request to an AI assistant and the AI’s response to the user. Your task is to evaluate whether the AI assistant’s response satisfied the user’s personal preferences . To help you evaluate the responses on preference, you will also be provided with an evaluation checklist that decomposes the preference into specific questions.
### Evaluation Preferences and Checklists
Preference 1 : ”{preference_1}”
Evaluation Checklist :
- {checklist_item_1}

Appendix

Table 18: LLM-as-a-judge prompt used to evaluate preference fulfillment for Apply preferences.

Preference Leakage Judge (PLR)
## Your Objective
You are a critical and meticulous evaluator. You will be presented with a user’s request to an AI assistant and the AI’s response to the user. Your task is to evaluate whether the AI assistant’s response inappropriately reflected preferences that should have been suppressed in the given context. To help you evaluate, you will also be provided with a leakage checklist for each suppressed preference that decomposes it into specific questions about whether signs of that preference appear in the response.
### Suppressed Preferences and Leakage Checklists
Suppressed Preference 1 : ”{preference_1}”
Leakage Checklist :
- {checklist_item_1}

Appendix

Table 19: LLM-as-a-judge prompt used to evaluate preference leakage for Suppress preferences.

Reasoning Label Extraction
You are an expert evaluator analyzing a language model’s preference personalization behavior.
You will receive:
1. A numbered list of user preferences (each may be ”Apply” or ”Suppress” in the ground truth)
2. The model’s chain-of-thought REASONING only (do NOT consider any final response)
Your job is to extract reasoning_labels — what the model’s REASONING indicates for each preference:
”apply” — reasoning explicitly states this preference should be followed

Appendix

Table 20: Reasoning-label extraction prompt used to evaluate preference decisions expressed in the model’s reasoning trace.

Generation Label Extraction
You are an expert evaluator analyzing a language model’s preference personalization behavior.
You will receive:
1. A numbered list of user preferences (each may be ”Apply” or ”Suppress” in the ground truth)
2. The model’s final RESPONSE only (do NOT consider any reasoning or thinking text)
Your job is to extract generation_labels — what the final RESPONSE actually demonstrates for each preference:
”apply” — the preference is reflected in the response (the model incorporated it)

Appendix

Table 21: Generation-label extraction prompt used to evaluate whether each user preference is reflected in the model’s final response.

Direct Decision (Step 1) in ABIDE
Step 1: Decide whether user preferences should be applied or suppressed in a given context.
For each preference, output exactly ”Apply” or ”Suppress”.
Rules:
”Apply” means the preference is relevant and appropriate for the current context.
”Suppress” means the preference is irrelevant, inappropriate, or would conflict with the context’s norms or the user’s actual intent.
Present your analysis under the header:

Appendix

Table 22: Direct Decision (Step 1) for conditionally applying or suppressing user preferences based on context in ABIDE.

Neutral 2-step prompt
Step 1: Decide whether user preferences should be applied or suppressed in a given context.
For each preference, output exactly ”Apply” or ”Suppress”.
Rules:
”Apply” means the preference is relevant and appropriate for the current context.
”Suppress” means the preference is irrelevant, inappropriate, or would conflict with the context’s norms or the user’s actual intent.
Present your analysis under the header:

Appendix

Table 23: Neutral 2-step prompt template for conditionally applying or suppressing user preferences based on context.

Simplified Generation Objective (G2)
Step 1: Decide whether user preferences should be applied or suppressed in a given context.
For each preference, output exactly ”Apply” or ”Suppress”.
Rules:
”Apply” means the preference is relevant and appropriate for the current context.
”Suppress” means the preference is irrelevant, inappropriate, or would conflict with the context’s norms or the user’s actual intent.
Present your analysis under the header:

Appendix

Table 24: Prompt configuration for the Simplified Generation Objective (G2), which removes explicit “Apply” and “Suppress” terminology from the generation instruction while preserving the preference discrimination step.

Arithmetic Neutral Objective (N2)
Step 1: Decide whether user preferences should be applied or suppressed in a given context.
For each preference, output exactly ”Apply” or ”Suppress”.
Rules:
”Apply” means the preference is relevant and appropriate for the current context.
”Suppress” means the preference is irrelevant, inappropriate, or would conflict with the context’s norms or the user’s actual intent.
Present your analysis under the header:

Appendix

Table 25: Prompt configuration for the Arithmetic Neutral Objective (N2). The objective replaces the fixed-sentence output in the neutral baseline with an arithmetic computation in Step 2, while keeping the preference discrimination task in Step 1 unchanged.

Explore similar work

May 20, 2026cs.CL

APM: Evaluating Style Personalization in LLMs with Arbitrary Preference Mappings

Typical LLM responses tend to follow a default style, even though users often have distinct preferences regarding tone, verbosity, and formality that they do not explicitly state in their prompts. Evaluating whether personalization methods can adapt to these implicit preferences is challenging, since users typically provide prompts rather than reference responses, style preferences are not factually verifiable, and reference-free LLM judges may conflate personalization with general response quality. To address these challenges, we introduce the Arbitrary Preference Mapping (APM) benchmark, which decouples user attributes (e.g. enthusiastic) from response principles (e.g. persuasive) via a hidden, randomized mapping

\mathbf{C}

that maps user attributes to preferences about response traits. Because

\mathbf{C}

carries no semantic content and is resampled across runs, models cannot exploit stereotypical associations and must infer preferences from conversation history. Using this unbiased evaluation methodology, we adapt retrieval-augmented, prompt-optimization, and routing personalization methods and evaluate them on Llama-3.1-8B and Qwen-3.5-27B. Our results show that routing is the most reliable approach, while RAG only improves with the stronger base LLM, and soft prompt optimization fails to improve significantly over a non-personalized baseline. Our extensive evaluation reveals that in this realistic setting, personalization remains challenging, but our adapted methods show promise.

Philipp Spohn, Leander Girrbach, Zeynep Akata

Technical University of Munich, Helmholtz Munich · Munich Center for Machine Learning (MCML)

Large Language Model Personalization Personalization Large Language Model Responses Automatic Prompt Optimization Preference Alignment Learning Responses

Ask PDF

Oct 17, 2025cs.CL

POPI: Personalizing LLMs via Optimized Natural Language Preference Inference

Large language models (LLMs) are typically aligned with population-level preferences, despite substantial variation across individual users. We introduce POPI, a user-level personalization framework that separates the problem into two components connected by a natural-language interface: a shared inference model that distills heterogeneous user signals into a concise preference summary, and a shared generator that conditions on this summary to produce personalized responses. Both components are trained under a unified preference-optimization objective, with reinforcement learning handling the non-differentiable inference step. This objective decomposes into generator approximation error and summary informativeness, revealing how a single loss simultaneously drives accurate generation and informative summarization. Because the interface is natural language, learned summaries can be inferred once per user and reused across different generators -- including frozen, black-box commercial APIs. Across four personalization benchmarks, POPI generally improves personalization quality while reducing context overhead by up to an order of magnitude.

Yizhuo Chen, Xin Liu, Ruijie Wang +7

University of Illinois Urbana-Champaign · Amazon · University of Notre Dame

Large Language Model Personalization Personalization Natural Language Inference Natural Language

Ask PDF

Sep 28, 2026cs.AI

Persona Following Is Not Selective Control: The Neutrality Gap in LLM User Simulation

Persona prompting is widely used to construct user simulations with large language models (LLMs), yet it relies on a largely untested assumption: specifying one user attribute should change that attribute alone. We test this assumption and identify a systematic failure of selective control: across all eight black-box LLMs we audit, changing a target attribute also shifts responses on unspecified, non-target attributes. For example, describing a user as more risk-seeking shifts color choices, even though the prompt never mentions color; we term this cross-attribute influence. Semantic, contextual, and internal analyses collectively suggest that models treat a persona prompt as evidence about the user and extend the inferred profile to unspecified preferences, a process we call trait-conditioned completion. We next ask whether explicitly specifying non-target attributes restores selective control. When a non-target attribute is assigned a clear direction, models generally follow the declaration and suppress the target attribute's influence. However, when the same attribute is declared neutral, the target continues to affect choices across all five open-weight checkpoints, even when the model correctly reports the declared state. This disparity, the neutrality gap, demonstrates that successful persona following does not imply selective persona control, which additionally requires keeping non-target attributes stable. We operationalize this distinction with a three-state diagnostic that leaves the non-target attribute unspecified or declares it directional or neutral; because directional tests can be passed by simply following the stated persona, the neutral state reveals failures they miss. In a post hoc analysis of independent items, neutral declarations leave 51-81% of items target-sensitive, against at most 1 of 320 item-pole comparisons under directional ones.

Jiashen Ren, Wenlin Zhang, Bohan Zhang +5

City University of Hong Kong · Stanford University

Artificial Intelligence Personas Persona Consistency User Simulation Neutrality Selective

Ask PDF