Response-Aware User Memory Selection for LLM Personalization

Jillian Fisher; Jennifer Neville; Chan Young Park

cs.AIApr 15, 2026

Response-Aware User Memory Selection for LLM Personalization

Authors: Jillian Fisher, Jennifer Neville, Chan Young Park

Organizations: Department of Computer Science and Engineering, University of Washington, Seattle, WA, United States of America · Microsoft Research, Redmond, WA, United States of America

Abstract

A common approach to personalization in large language models (LLMs) is to incorporate a subset of the user memory into the prompt at inference time to guide the model's generation. Existing methods select these subsets primarily using similarity between user memory items and input queries, ignoring how features actually affect the model's response distribution. We propose Response-Utility optimization for Memory Selection (RUMS), a novel method that selects user memory items by measuring the mutual information between a subset of memory and the model's outputs, identifying items that reduce response uncertainty and sharpen predictions beyond semantic similarity. We demonstrate that this information-theoretic foundation enables more principled user memory selection that aligns more closely with human selection compared to state-of-the-art methods, and models $400\times$ larger. Additionally, we show that memory items selected using RUMS result in better response quality compared to existing approaches, while having up to $95\%$ reduction in computational cost.

Figures & tables

Figure 1: Overview of RUMS. RUMS-Utility quantifies how much memory subsets reduce response uncertainty (training phase), while RUMS-Models efficiently selects informative items at inference.

Panel of Figure 2 — Figure 2: Results for H1 analysis.

Dataset	Metric	RUMS-Subset	RUMS-Binary	RUMS-Multi	Semantic Sim.	GPT4-Zero	GPT4-Few
	Precision	0.61	0.60	0.23	0.09	0.26	0.53
Synthetic	Recall	0.70	0.89	0.39	0.23	0.42	0.65
	F1	0.70	0.78	0.31	0.14	0.35	0.64
	Precision	0.48	0.32	0.30	0.17	0.39	0.42
Real World	Recall	0.36	0.60	0.38	0.27	0.52	0.39
	F1	0.53	0.45	0.35	0.20	0.43	0.39

Table 1: Precision, recall, and F1 scores of memory items selected compared to human gold label annotation across datasets and methods.

Synthetic LLaMA 70B GPT-4 Utility Binary Multi Utility Binary Multi None 0.11 0.25 -0.13 0.20 0.26 0.36 All 0.18 0.16 -0.19 0.20 -0.16 0.02 Random 0.44 0.34 0.13 0.10 0.06 0.19 Semantic Sim. 0.43 0.27 0.09 -0.12 -0.03 0.05 BM25 0.03 0.19 -0.13 0.18 0.20 0.32 ReContriever 0.31 0.27 -0.03 -0.05 -0.04 0.03 GPT4-Few 0.07 0.18 -0.16 -0.07 -0.09 -0.01

Real World LLaMA 70B GPT-4 Utility Binary Multi Utility Binary Multi -0.05 0.05 0.03 0.10 0.20 0.17 0.03 -0.04 -0.10 -0.14 -0.14 -0.06 0.25 0.11 0.13 0.05 0.10 0.04 0.16 -0.03 0.00 -0.06 -0.02 0.00 0.06 0.01 0.05 0.10 0.18 0.13 0.22 0.06 0.01 0.07 -0.05 0.13 0.01 0.00 -0.08 -0.08 0.04 0.11

Table 2: Average win rate of our method against baselines on Synthetic (left) and Real World (right) datasets. Cells in green mark cases where our method outperforms the baseline, and cells in red mark cases where the baseline does better. Values with magnitude above 0.1 are bolded to highlight strong differences. Results are reported for two response models (LLaMA 70B Instruct and GPT-4) and two RUMS variants: RUMS-Binary (fewer items) and RUMS-Multi (more items).

Panel of Figure 3 — Figure 3: Cost Analysis

Appendix figures & tables84 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 4 — Figure 4: Distribution of maximum utility scores for a personalized (blue) and non-personalized (orange) dataset differing by type of model used for entropy calculation.

Panel of Figure 5 — Figure 5: Distribution of maximum utility scores for a personalized (blue) and non-personalized (orange) dataset differing by size of model used for entropy calculation.

Panel of Figure 6 — Figure 6: Distribution of maximum utility scores for a personalized (blue) and non-personalized (orange) dataset calculated using models of different families.

Panel of Figure 7 — Figure 7: Distribution of maximum utility scores for a personalized (blue) and non-personalized (orange) dataset differing by decoding type.

Figure 8: Average entropy by the index of the token being generated.

Panel of Figure 9 — Figure 9: Distribution of maximum utility scores for a personalized (blue) and non-personalized (orange) dataset using different number of tokens.

Panel of Figure 10 — Figure 10: Distribution of maximum utility scores for a personalized (blue) and non-personalized (orange) dataset differing number of Monte Carlo samples average over.

Panel of Figure 11 — Figure 11: Distribution of maximum utility scores for a personalized (blue) and non-personalized (orange) dataset differing by number of users averaged over.

	LLaMA 70B	GPT-4
Method	Human Win Rate	Human Win Rate
None	0.31	0.16
All	0.28	-0.26
Random	0.42	0.06
Semantic Similarity	0.38	0.01
BM25	0.18	0.26

Appendix

Table 3: Win rate of responses preferred using human selected memory subsets compared to other methods. A number above 0 is the percentage of responses generated by Humans chosen subsets that were preferred.

Dataset	Metric	RUMS-Binary	RUMS-Multi
	Precision	0.73	0.33
Synthetic	Recall	0.90	0.44
	F1	0.78	0.37
	Precision	0.26	0.35
Real World	Recall	0.42	0.35
	F1	0.32	0.34

Appendix

Table 4: Precision, recall, and F1 scores of memory items selected compared to human gold label annotation across datasets and methods.

Dataset	Subset Gap	Sequential Gap
Synthetic	0.23	0.11
Real World	1.01	0.09

Appendix

Table 5: Results from the analysis of the combinatorial approximation gap between RUMS-Utility and RUMS-Binary. We show the average normalized subset interaction gap and the average sequential conditional interaction gap for personalized datasets.

Figure 24: Optimal threshold calculated for each

n=10

users based on utility scores. It shows the optimal thresholds for the 10 users (solid blue line), as well as the mean threshold (dotted red line) and

\pm 1

standard deviation (gray shaded area)

Threshold	PersonaFeedback	FreebaseQA	WildChat-Personal	WildChat-NonPersonal
-100	2.76	1.83	2.77	2.70
0.00	2.76	1.56	2.73	2.57
0.29	2.43	0.21	1.31	1.38
0.50	1.11	0.00	0.29	0.27
0.80	0.14	0.00	0.00	0..00

Appendix

Table 6: Average number of memory items selected at different thresholds used in RUMS-Utility.

User Input	Humans*	RUMS-Binary	GPT4-ZeroShot	GPT4-FewShot	Semantic Similarity
[PERSONALIZED] I became a millionaire recently and want to give money to some local businesses	Location Hobbies and Interests Volunteer Activities	Location Hobbies and interests Volunteer activities Personal values Industry of employment	Location Occupation Industry of employment Current projects Long-term aspirations	Location Recent life events	Location Volunteer activities Financial situation Annual Income Pet ownership
[NON-PERSONALIZED] Who was the Prime Minister of Gt. Britain at the outbreak of The First World War?	No features selected	No features selected	Location Occupation Name Political affiliation	Location Occupation	Travel history Occupation Name Political affiliation Current projects

Appendix

Table 7: Examples of memory items selected by each method compared to human gold labels. Overlapping memory with items human annotation is bolded .

Table 26

Figure 25: Estimated monthly cost of additional prompt tokens (1M queries per day, 30 days, GPT-4 input pricing at $0.01 per 1K tokens).

Method	Average Time (s)
All	0.0
None	0.0
Random	0.0
Semantic Similarity	0.21
BM25	0.01
ReContriever	0.08

Appendix

Table 10: Average time for inference of one sample, calculated over

n=3

trials. All values are in units of seconds.

Dataset	RUMS-Utility Winrate
Synthetic	0.33
Real World	-0.03

Appendix

Table 11: Winrate of responses generated using memory subset of RUMS-Utility vs. GPT4-Few, determined by human evaluation.

Dataset	Model	Recall	Specificity
Synthetic	Llama-7B	0.94	0.94
Real World	Llama-7B	0.92	0.96
Synthetic	Mistral-7B	0.83	0.88
Real World	Mistral-7B	0.92	0.77

Appendix

Table 12: H1 Results

Dataset	Model	Precision	Recall	F1
Synthetic	Llama-7B	0.61	0.70	0.70
Real World	Llama-7B	0.48	0.36	0.53
Synthetic	Mistral-7B	0.61	0.62	0.69
Real World	Mistral-7B	0.61	0.30	0.69

Appendix

Table 14: H1 Results

Dataset	Model	Precision	Recall	F1
Synthetic Personalized	GPT-4	0.24	0.77	0.49
	Llama-70B	0.23	0.74	0.47
	Qwen-72B	0.21	0.74	0.45
	Gemma-27B	0.22	0.66	0.45
Real World Personalized	GPT-4	0.46	0.70	0.59
	Llama-70B	0.41	0.69	0.53

Appendix

Table 16: Model Performance on Personalized Datasets

	Humans	GPT-4	Llama-70B	Qwen-72B	Gemma-27B
Humans	—	—	—	—	—
GPT-4	0.4	—	—	—	—
Llama-70B	0.5	0.7	—	—	—
Qwen-72B	0.5	0.8	0.6	—	—
Gemma-27B	0.5	0.6	0.7	0.7	—

Appendix

Table 18: Pairwise Agreement Between Models and Humans

Table 20: Average winrate comparison on Synthetic data under Not Personal (left) and Personal (right) conditions. Positive values (green) indicate better performance, negative values (red) indicate worse performance. Values greater than 0.1 in magnitude are bolded.

Table 21: Average winrate comparison on Real World data under Not Personal (left) and Personal (right) conditions. Positive values (green) indicate better performance, negative values (red) indicate worse performance. Values greater than 0.1 in magnitude are bolded.

Category	%	Example		Chosen Items
		User Input	Humans (All)	Humans ( $>1$ )	RUMS-Utility
Correct	38	“In May 1994 the Channel Tunnel was opened by Queen Elizabeth II and which French President?”	[]	[]	[]
Underselect	15	“How to use a vpn while vacationing in china”	[Tech savvy]	[Tech savvy, Feedback style, Comm mode, Tech use]	[none]
All	13	“Choose between new zara sneaker or used new balance”	[]	[Hobbies, Income, Aspirations, Finances, Exercise, Personality, Job, Industry, Pastimes, Events, Gender, Location]	[Hobbies, Income]
Some	22	“Buying a new notebook to write a script”	[]	[Location, Income, Tech savvy, Finances, Personality]	[Hobbies, Tech savvy, Fav books]
None	5	“My PC is r6 3600 and 1660s need new GPU”	[Projects, Finances]	[Income, Job, Hobbies, Industry, Projects, Aspirations, Finances, Pastimes]	[Tech savvy]

Appendix

Table 22: Breakdown of selection behavior categories with representative user inputs and selected memory items by humans (all agree), humans (at least one agree), and RUMS Utility.

Dataset	Threshold
All	-1.0
Synthetic	-1.0
Real World	0.5

Appendix

Table 23: Threshold for H1 analysis of semantic similarity baseline. If at least one memory item had a semantic similarity higher than this threshold, then the query was labeled as needing personalization.

Method	Model	# Parameters
RUMS	microsoft/deberta-v3-large	418M
Semantic Similarity	all-MiniLM-L6-v2	$\sim$ 22.7M
GPT4o	GPT4o	Not disclosed (est. $>$ 200B)
ReContriever	Yibin-Lei/ReContriever	$\sim$ 110M

Appendix

Table 24: Baseline methods and their corresponding model sizes.

	Personalized	Not-Personalized	Difference
Human Annotation	6.45	1.43	5.02
RUMS-Utility	1.87	0.80	1.07
RUMS-Binary	4.87	1.56	3.31
RUMS-Multi	3.27	1.67	1.60
Semantic Similarity	5.00	5.00	0.00
GPT4-ZeroShot	3.30	2.61	0.69

Appendix

Table 25: Average number of features selected for personalized vs. not-personalized inputs. Methods closes to human performance are bolded .

Figure 26: Distribution of Name

Figure 27: Distribution of Gender

Figure 28: Distribution of Ethnicity

Figure 29: Distribution of Race

Figure 30: Distribution of Annual Household Income

Figure 31: Distribution of Language spoken at home

Figure 32: Distribution of Language preference

Figure 33: Distribution of Sexual Preference

Figure 34: Distribution of Disabilities

Figure 35: Distribution of Occupation

Figure 36: Distribution of Political affiliation

Figure 37: Distribution of Education level

Figure 38: Distribution of Hobbies and interests

Figure 39: Distribution of Favorite sports

Figure 40: Distribution of Preferred music genre

Figure 41: Distribution of Favorite books

Figure 42: Distribution of Preferred movie genre

Figure 43: Distribution of Favorite TV shows

Figure 44: Distribution of Favorite foods

Figure 45: Distribution of Dietary restrictions (e.g., vegetarian, vegan)

Figure 46: Distribution of Relationship status

Figure 47: Distribution of Number of children

Figure 48: Distribution of Pet ownership (types of pets)

Figure 49: Distribution of Travel history (countries visited)

Figure 50: Distribution of Travel preferences (adventurous, relaxing)

Figure 51: Distribution of Social media platforms used

Figure 52: Distribution of Level of tech-savviness

Figure 53: Distribution of Preferred mode of communication (text, voice, video)

Figure 54: Distribution of Work schedule (full-time, part-time)

Figure 55: Distribution of Industry of employment

Figure 56: Distribution of Current projects or goals

Figure 57: Distribution of Long-term aspirations

Figure 58: Distribution of Financial situation (broad categories)

Figure 59: Distribution of Health and fitness goals

Figure 60: Distribution of Preferred exercise activities

Figure 61: Distribution of Level of environmental consciousness

Figure 62: Distribution of Volunteer activities or interests

Figure 63: Distribution of Current challenges or pain points

Figure 64: Distribution of Personality traits (e.g. introverted, extroverted)

Figure 65: Distribution of Favorite pastimes

Figure 66: Distribution of Personal values or beliefs

Figure 67: Distribution of Cultural background

Figure 68: Distribution of Religious beliefs

Figure 69: Distribution of Recent life events (e.g. change in job, moved)

Figure 70: Distribution of Technology usage (devices, apps)

Figure 71: Distribution of Frequency of using the service (daily, weekly)

Figure 72: Distribution of Reasons for using the service

Figure 73: Distribution of Feedback preferences (detailed, brief)

Figure 74: Distribution of Preferred tone of communication (formal, casual)

Figure 75: Distribution of Country

LLM Personalization

LLM personalization adapts language models to users’ preferences, histories, expertise, and changing goals while balancing usefulness, privacy, and control.

See all 130 papers

POPI: Personalizing LLMs via Optimized Natural Language Preference Inference
Oct 17, 2025Yizhuo Chen, Xin Liu, Ruijie Wang +7LLM Personalization Personalized Language Models
University of Illinois Urbana-Champaign · Amazon · University of Notre Dame
Latent Personal Memory: Represent personal memory as dynamic soft prompts
Jun 18, 2026Debrup Das, Avinash Amballa, Yashas Malur Saidutta +3LLM Inference Efficiency Memory-Efficient Fine-Tuning
University of Massachusetts Amherst · Samsung Research America
Memory-Aware Reward Modeling: Learning Human Preferences for LLM Memory Use
Oct 22, 2026Yijiang River Dong, Zexue He, Cheng Yu +9Memory-Augmented Language Models Reward Modeling
University of Cambridge · Stanford University · Technical University of Munich +2

Oct 17, 2025J/K move · Enter open · S save

POPI: Personalizing LLMs via Optimized Natural Language Preference Inference

Yizhuo Chen, Xin Liu, Ruijie Wang, Zheng Li+6

University of Illinois Urbana-Champaign · Amazon · University of Notre Dame

LLM Personalization Personalized Language Models Personalized Language Model Alignment Language Model Post-Training User Preference Modeling

Open paper AskPDF arXiv

Large language models (LLMs) are typically aligned with population-level preferences, despite substantial variation across individual users. We introduce POPI, a user-level personalization framework that separates the problem into two components connected by a natural-language interface: a shared inference model that distills heterogeneous user signals into a concise preference summary, and a shared generator that conditions on this summary to produce personalized responses. Both components are trained under a unified preference-optimization objective, with reinforcement learning handling the non-differentiable inference step. This objective decomposes into generator approximation error and summary informativeness, revealing how a single loss simultaneously drives accurate generation and informative summarization. Because the interface is natural language, learned summaries can be inferred once per user and reused across different generators -- including frozen, black-box commercial APIs. Across four personalization benchmarks, POPI generally improves personalization quality while reducing context overhead by up to an order of magnitude.

Abstract

LLM Personalization

Explore similar work

POPI: Personalizing LLMs via Optimized Natural Language Preference Inference

Latent Personal Memory: Represent personal memory as dynamic soft prompts

Memory-Aware Reward Modeling: Learning Human Preferences for LLM Memory Use