A Controlled Audit of Personal AI Memory for Rating Prediction

Shivam Gupta

cs.LGOct 2, 2026

A Controlled Audit of Personal AI Memory for Rating Prediction

Authors: Shivam Gupta

Organizations: Independent research.

Abstract

In structured rating prediction, does a personal AI use historical item-rating associations, or mainly the user's rating tendencies? We audit this distinction by permuting historical ratings within each user while preserving the exact rating distribution, item support, and metadata. We combine this control with full history, native memory extraction, and matched numerical readers in a publicly frozen evaluation of 400 held-out user profiles and 6,160 target ratings across Coat and MovieLens. On Coat, the tested Qwen-written Mem0 pipeline increases user-macro mean absolute error relative to full history by 0.084 for Qwen and 0.149 for Phi; both family-adjusted bootstrap intervals exclude zero. Correct historical assignments help both readers on Coat, but the corresponding MovieLens effects are smaller and inconclusive after adjustment. A history-only ridge reader outperforms Qwen in both domains and Phi on MovieLens, while the Coat Phi comparison is unresolved. All 2,800 reader calls, including 150 invalid outputs, are retained under a fixed fallback rule. A separate implementation verifies inputs, metrics, and all ten primary contrasts. The contribution is a reproducible diagnostic study showing why extraction, association use, output reliability, and reader choice require separate evaluation.

Figures & tables

Figure 1: The same source history supports three evidence controls and, on Coat, native memory extraction. Target outcomes never enter construction or prediction. Each reader receives the same new target list in every condition. A separate ridge reader uses the same observed history and metadata.

	Coat	MovieLens
History ratings / user	24	24
New target ratings	2,960	3,200
Numerical dimensions	32	20
Reader conditions	4	3
Reader models	2	2
Native writer calls	200	0

Table 1: Evaluation design. Each domain has a separate 30-user development set and 200-user evaluation set. Calls exclude fitting-user preflights.

System / evidence	Coat MAE	Valid	MovieLens MAE	Valid
Scale midpoint	1.236	200/200	0.984	200/200
Ridge / permuted	1.129	200/200	0.865	200/200
History mean	0.991	200/200	0.839	200/200
History median	0.923	200/200	0.817	200/200
History-only ridge	0.914	200/200	0.845	200/200
Qwen / no history	1.694	199/200	1.029	197/200

Table 2: Operational user-macro MAE and valid-output coverage. Invalid reader arrays use the predeclared constant-3 fallback. Numerical methods always produce valid predictions. Native memory is evaluated on Coat only. Lower MAE is better. The Coat midpoint row is an auxiliary descriptive reference; ridge permutation is a frozen secondary control.

Figure 2: All ten predeclared contrasts in user-macro MAE. Points show paired means; thick bars are ordinary 95% percentile intervals and thin bars are family-adjusted 99.5% intervals. Positive values mean extraction increases error, correct assignments help, or ridge has lower error, respectively. Each all-user comparison contains 200 users.

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 3: Descriptive MSE decomposition into squared user-mean error and centered residual error. Q and P denote Qwen and Phi; none, full, perm., and memory denote the four evidence conditions. Both components use target outcomes only during evaluation. Lower mean error can coexist with worse centered residual error; neither component identifies uniquely personal preferences.

System	MAE	RMSE	Signed error	Pairwise	Valid-only MAE	Valid
History mean	0.991	1.166	0.406	0.500	0.991	200/200
History median	0.923	1.220	0.251	0.500	0.923	200/200
History-only ridge	0.914	1.136	0.330	0.637	0.914	200/200
Phi / full	0.910	1.249	0.214	0.599	0.888	185/200
Phi / memory	1.059	1.347	0.452	0.562	1.035	184/200
Phi / none	1.236	1.463	0.764	0.500	1.259	177/200

Appendix

Table 3: Coat complete metric summary. Pairwise concordance averages eligible users; ties receive half credit. All primary operational summaries include every user. Valid-only subsets can differ. Population and Pop. rows use additional fitting-user data; their equations are in Appendix D .

Contrast	All users	Adjusted interval	Common valid	95% interval	Users
Qwen / extraction	0.084	[0.012, 0.160]	0.084	[0.032, 0.138]	198
Qwen / association	0.144	[0.072, 0.215]	0.144	[0.094, 0.195]	200
Qwen / numerical reader	0.160	[0.095, 0.225]	0.160	[0.116, 0.206]	200
Phi / extraction	0.149	[0.072, 0.229]	0.147	[0.096, 0.200]	171
Phi / association	0.205	[0.119, 0.295]	0.220	[0.153, 0.287]	168
Phi / numerical reader	-0.004	[-0.067, 0.059]	-0.025	[-0.067, 0.016]	185

Appendix

Table 4: Coat predeclared primary contrasts and paired common-valid sensitivity. The all-user intervals use alpha 0.005 for the family of ten comparisons; the descriptive common-valid intervals shown here are ordinary 95% intervals. Positive effects follow the definitions in the main text.

System	MAE	RMSE	Signed error	Pairwise	Valid-only MAE	Valid
Scale midpoint	0.984	1.234	-0.541	0.500	0.984	200/200
History mean	0.839	1.054	0.030	0.500	0.839	200/200
History median	0.817	1.127	0.121	0.500	0.817	200/200
History-only ridge	0.845	1.042	-0.109	0.558	0.845	200/200
Phi / full	0.921	1.213	-0.035	0.551	0.921	199/200
Phi / none	0.986	1.234	-0.538	0.499	0.985	141/200

Appendix

Table 5: MovieLens complete metric summary. Pairwise concordance averages eligible users; ties receive half credit. All primary operational summaries include every user. Valid-only subsets can differ.

Contrast	All users	Adjusted interval	Common valid	95% interval	Users
Qwen / association	0.020	[-0.022, 0.062]	0.025	[-0.006, 0.056]	193
Qwen / numerical reader	0.091	[0.049, 0.135]	0.087	[0.056, 0.116]	195
Phi / association	0.029	[-0.011, 0.069]	0.027	[-0.001, 0.055]	198
Phi / numerical reader	0.076	[0.038, 0.117]	0.075	[0.049, 0.103]	199

Appendix

Table 6: MovieLens predeclared primary contrasts and paired common-valid sensitivity. The all-user intervals use alpha 0.005 for the family of ten comparisons; the descriptive common-valid intervals shown here are ordinary 95% intervals. Positive effects follow the definitions in the main text.

Domain	Model	Stage / evidence	Calls	Prompt tokens	Output tokens
Coat	Qwen	Native writing	200	1,782,117	191,987
Coat	Qwen	none	200	118,318	9,109
Coat	Qwen	full	200	299,835	9,080
Coat	Qwen	permuted	200	299,835	9,080
Coat	Qwen	memory	200	237,828	9,074
Coat	Phi	none	200	111,798	16,400

Appendix

Table 7: Recorded model usage, excluding separately reported preflights and development. Native writing counts responses recorded by the native provider callback; reader calls count attempted condition calls. Token totals do not measure full storage or commercial cost.

Domain	Reader	Evidence	Invalid	Non-stop	Contract
Coat	Qwen	full	0	0	0
Coat	Qwen	memory	2	0	2
Coat	Qwen	none	1	0	1
Coat	Qwen	permuted	0	0	0
Coat	Phi	full	15	10	5
Coat	Phi	memory	16	9	7

Appendix

Table 8: Output failures out of 200 attempts per row. Non-stop counts responses whose generation did not end normally, including token-limit termination. Transport counts attempts without a model response. Contract counts normally stopped responses that fail the exact-array parser. Categories are disjoint and sum to Invalid. All are retained through the constant-3 fallback.

Domain	System	Users	Normal	Augmented	Near-tie	Disagreements
Coat	History-only ridge	199	0.637462	0.637287	0.637462	17
Coat	Ridge / permuted	199	0.481885	0.481725	0.481859	24
MovieLens	History-only ridge	200	0.557539	0.557547	0.557539	2
MovieLens	Ridge / permuted	200	0.501437	0.501458	0.501437	3

Appendix

Table 9: Post-freeze numerical precision sensitivity, specified before accuracy inspection. Columns show mean pairwise concordance from the original normal-equation solve, augmented least squares, and a common tie tolerance of 1e-10. Disagreements count users whose strict concordance differs between solvers. Primary MAE comparisons are unchanged.

Memory-Augmented Language Models

Memory-augmented language models combine persistent or temporary memory with neural computation to retain, retrieve, and use information beyond the active context.

See all 292 papers

Memory-Aware Reward Modeling: Learning Human Preferences for LLM Memory Use
Oct 22, 2026Yijiang River Dong, Zexue He, Cheng Yu +9Memory-Augmented Language Models Reward Modeling
University of Cambridge · Stanford University · Technical University of Munich +2
To Memories and Beyond: From Remembering to Knowing You across Long-Term Multimodal Personal Archives
Sep 11, 2026Wenqi Zhou, Zhuorui Yu, Kaiao Wen +6Multimodal Memory User Preference Modeling
University of Bristol · Memories.ai Research
PersEvalBench: Can LLM Evaluators Judge Contextually Appropriate Use of Persona Information?
Oct 13, 2026Xinyi Liu, Hooshang Nayyeri, Emine Yilmaz +3LLM Evaluation LLM-as-a-Judge
Amazon · University of Illinois Urbana-Champaign · University College London

Oct 22, 2026J/K move · Enter open · S save

Memory-Aware Reward Modeling: Learning Human Preferences for LLM Memory Use

Yijiang River Dong, Zexue He, Cheng Yu, Tiancheng Hu+8

University of Cambridge · Stanford University · Technical University of Munich+2

Memory-Augmented Language Models Reward Modeling User Preference Modeling

Open paper AskPDF arXiv

Memory enables language model assistants to maintain continuity across sessions and generate personalized responses, but knowing more about a user does not necessarily produce responses they prefer. Existing evaluations assess memory capabilities without directly establishing whether memory use benefits the user. To study this gap, we collect approximately 1,000 pairwise judgments from 87 participants using their own ChatGPT queries and personal memories. We find that memory-conditioned responses are not consistently preferred over responses generated without memory and in some cases, users strongly prefer the latter. Automatic evaluators also show limited agreement with users and favor memory-conditioned responses substantially more often than users do. Motivated by these gaps, we introduce a memory-aware reward model that learns from user judgments, achieving 72.8% preference prediction accuracy and outperforming the existing baselines. Our findings motivate evaluation and optimization objectives that align memory use with the preferences of the people whose information is being used.

Abstract

Memory-Augmented Language Models

Explore similar work

Memory-Aware Reward Modeling: Learning Human Preferences for LLM Memory Use

To Memories and Beyond: From Remembering to Knowing You across Long-Term Multimodal Personal Archives

PersEvalBench: Can LLM Evaluators Judge Contextually Appropriate Use of Persona Information?