Memory-Aware Reward Modeling: Learning Human Preferences for LLM Memory Use

Yijiang River Dong; Zexue He; Cheng Yu; Tiancheng Hu; Michael J Ryan; Jan Batzner; Zheng Hui; Shaolong Wu

cs.CLOct 22, 2026

Memory-Aware Reward Modeling: Learning Human Preferences for LLM Memory Use

Authors: Yijiang River Dong, Zexue He, Cheng Yu, Tiancheng Hu, Michael J Ryan, Jan Batzner, Zheng Hui, Shaolong Wu,

Organizations: University of Cambridge · Stanford University · Technical University of Munich · Harvard University ·

Abstract

Memory enables language model assistants to maintain continuity across sessions and generate personalized responses, but knowing more about a user does not necessarily produce responses they prefer. Existing evaluations assess memory capabilities without directly establishing whether memory use benefits the user. To study this gap, we collect approximately 1,000 pairwise judgments from 87 participants using their own ChatGPT queries and personal memories. We find that memory-conditioned responses are not consistently preferred over responses generated without memory and in some cases, users strongly prefer the latter. Automatic evaluators also show limited agreement with users and favor memory-conditioned responses substantially more often than users do. Motivated by these gaps, we introduce a memory-aware reward model that learns from user judgments, achieving 72.8% preference prediction accuracy and outperforming the existing baselines. Our findings motivate evaluation and optimization objectives that align memory use with the preferences of the people whose information is being used.

Figures & tables

Figure 1: Overview of memory-aware preference learning. Given a query and retrieved user memories, the same generator produces paired responses with (

y^{m}

) and without (

y^{0}

) memory conditioning. A rubric evaluator assesses memory-related and general response characteristics, yielding pairwise features

x

. Feature weights are learned from annotators’ pairwise preferences, allowing the model to predict which response is preferred. Importantly, annotators evaluate responses generated using their own memories. The example and feature contributions are illustrative; Section 4 describes the full model.

Original strength	Same side on repeat	Rate
Slightly	17/29	58.6%
Much	34/46	73.9%
Overall	51/75	68.0%

Table 1: Test–retest agreement among 10 participants after three days. “Same side” counts comparisons in which the same response is preferred in both sessions; groups are defined by the original preference strength.

Prompting regime
	Preference (%)
Condition	Memory	No memory	Tie
Conservative	43.3	40.6	16.1
Ordinary	38.7	41.9	19.4
Aggressive	34.6	49.3	16.2
Response generator (ordinary prompting)

Table 2: Annotator preferences by prompting regime and response generator. Percentages include ties. Generator comparisons are restricted to ordinary prompting to hold the personalization instructions constant.

	Accuracy (%)
Predictor	Avg.	GPT	Gemini	Claude
Majority	55.0	51.8	70.6	75.6
Skywork (with memory)	52.9	63.6	37.5	73.3
Skywork (no memory)	58.8	55.5	66.9	42.2
Direct judge	58.1	53.6	58.8	66.7
Rubric judge	59.6	52.3	68.8	50.0

Table 3: Preference prediction accuracy. Majority is calculated by always choosing without memory. Skywork is the Reward-V2 Llama-3.1-8B checkpoint, scoring each response independently and then compare the score. We include two variants of Skywork: query only (no memory) and query with the retrieved memories. For both LLM Judge and Skywork, we varied several different prompts/formats and the results listed is the best performance.

Feature	$\beta$	Mem.
Heading difference. Headings in the memory-conditioned response minus those in the other response.	$+0.646$	No
Relative length. Log length ratio of the memory-conditioned to the no-memory response.	$+0.571$	No
Memory relevance. Relevance of the retrieved memories to the query, scored without either response.	$+0.386$	Yes
Assumption proportionality. Whether personal assumptions are justified by the memory evidence.	$+0.323$	Yes
Unwarranted personal assumptions. Degree to which a response assumes more about the user than the query establishes.	$-0.201$	Yes

Table 4: The five features with the largest standardized coefficients in the descriptive fit of our model to the 291 clear-preference pairs. Positive coefficients favor the memory-conditioned response; Mem. marks memory-related features.

	Coverage
Predictor	100%	75%	50%	25%
Ours (Response only)	67.0	75.5	82.2	83.1
Ours (Memory only)	62.5	66.1	68.8	64.2
Ours	72.8	76.2	83.7	88.8

Table 5: Accuracy (%) after retaining the most confident comparisons within each pipeline.

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 2: Persona collection interface (Step 1), showing the prompt-copy control, an example structured profile, and the transition to event collection. The displayed profile is illustrative.

Figure 3: Recent-event collection interface (Step 2), with task instructions, examples of sufficiently specific events, collection progress, and text/voice entry controls.

Feature	Definition	$\beta$	Mem.
Head	Heading difference. Number of headings in the memory-conditioned response minus that in the response without memory.	$+0.646$	No
Len	Relative length. Log ratio of response lengths, oriented from the memory-conditioned to the no-memory response.	$+0.571$	No
Rel	Memory relevance. Relevance of the retrieved memories to the current query, scored without viewing either response.	$+0.386$	Yes
Prop	Assumption proportionality. Whether personal assumptions are justified by the available memory evidence, compared between responses.	$+0.323$	Yes
Assump	Unwarranted personal assumptions. Degree to which a response assumes more about the user than the query establishes, compared between responses.	$-0.201$	Yes
Closure	Actionable closure. Whether a response supplies a clear next action rather than only considerations to weigh, compared between responses.	$+0.191$	No

Appendix

Table 6: The twelve features of our model, ordered by absolute standardized L2 coefficient in the descriptive fit to 291 clear-preference pairs (

C=0.1

). Mem. identifies the memory-related feature block. Positive coefficients favor the memory-conditioned response. These coefficients describe the full-data fit, rather than an average of the nested-validation fits.

Response generator	Judge variant	$n$	Majority (%)	Accuracy (%)
Pooled	Voting	291	55.0	58.1
Pooled	Both orders agree	226	51.8	60.2
GPT-5.2	Voting	110	51.8	55.5
GPT-5.2	Both orders agree	80	57.5	57.5
Gemini-3.1-Pro	Voting	136	70.6	58.8
Gemini-3.1-Pro	Both orders agree	110	69.1	59.1

Appendix

Table 7: Zero-shot direct GPT-5.2 judging on the clear-preference evaluation stratum. Majority rates are computed separately for each evaluation subset.

Memory-Augmented Language Models

Memory-augmented language models combine persistent or temporary memory with neural computation to retain, retrieve, and use information beyond the active context.

See all 291 papers

Know It, Act on It: Investigating Memory Utilization in LLM Personalization
Jul 31, 2026Zhaoxin Feng, Jianfei Ma, Emmanuele ChersoniLLM Personalization LLM Agent Memory
The Hong Kong Polytechnic University
Response-Aware User Memory Selection for LLM Personalization
Apr 15, 2026Jillian Fisher, Jennifer Neville, Chan Young ParkLLM Personalization Memory-Augmented Language Models
Department of Computer Science and Engineering, University of Washington, Seattle, WA, United States of America · Microsoft Research, Redmond, WA, United States of America
PairPref: When Should Memory Guide the Answer? A Benchmark for Contextual Preference Use
Sep 28, 2026Mingfei Lu, Mengjia Wu, Yi ZhangUser Preference Modeling Memory-Augmented Language Model Agents
AAII, University of Technology Sydney

Jul 31, 2026J/K move · Enter open · S save

Know It, Act on It: Investigating Memory Utilization in LLM Personalization

Zhaoxin Feng, Jianfei Ma, Emmanuele Chersoni

The Hong Kong Polytechnic University

LLM Personalization LLM Agent Memory LLM Agent Evaluation User Preference Modeling

Open paper AskPDF arXiv

As large language model (LLM) agents evolve into personalized companions, memory has emerged as a core capability. However, LLMs face a knowledge utilization problem: they may fail to act on relevant user preferences even when they are fully present in context. When an agent fails to tailor its response in a context where previously shared user preferences should matter, it is unclear whether the model failed to remember that information or remembered it but failed to use it. To isolate this breakdown, we introduce a decoupled evaluation paradigm that administers paired Know and Act tests to the same user preference. We conduct large-scale experiments across 16 systems and five memory architectures, evaluating 1,000 preferences embedded at three levels of expression strength. Our results show a large gap between Know and Act outcomes: agents often pass the recall test for a user preference but fail to reflect that same preference in the paired behavioral scenario. While memory architectures reduce this gap, utilization remains especially weak for health and therapy-related preferences, where failures to act carry the greatest real-world stakes.

Abstract

Memory-Augmented Language Models

Explore similar work

Know It, Act on It: Investigating Memory Utilization in LLM Personalization

Response-Aware User Memory Selection for LLM Personalization

PairPref: When Should Memory Guide the Answer? A Benchmark for Contextual Preference Use