A common test of personalization replaces a user's history with someone else's and measures the drop in accuracy. But this swap also changes the topics in the history, so the drop may reflect topic mismatch. We examine this problem with a frozen language model on LaMP-2 news categorization, using histories from similar users and synthetic histories assembled from other users' articles. The user's own history improves accuracy by 2.3 percentage points over a model trained without history. Removing history only at evaluation makes this gain appear larger, at 4.8 points. Category labels alone reproduce 86% of the accuracy gap between the user's own history and a random user's history. With synthetic histories matched on labels and text similarity, the gap falls to 0.9 points. Across two user splits and three training seeds, the advantage over a similar user's history is 3.2 points for users seen during training and 0.65 points for unseen users, where the evidence for an advantage remains inconclusive. A small movie-tagging experiment with one training seed also finds no advantage over other users' histories. These findings show why personalization should be evaluated on unseen users, with histories of similar content and a model trained without history as comparisons. We also find that adding gated cross-attention improves on a prefix-only model under our training setup; whether this holds more broadly remains open.
Figures & tables
Figure 1: LaMP-2 news controls. (a) Illustrative histories show why a random-user swap mixes a change of owner with a change of topic. A similar-user history reduces content mismatch; the query and trained checkpoint stay fixed. (b) Three-seed mean accuracy relative to a separately trained no-history model. (c) Matched-minus-swapped accuracy against profile cosine, separately for random and similar users; bands resample users, retaining their repeated measurements. The dashed line is the 2.3-point gain over the trained comparator. The panels do not decompose a causal identity effect.
Condition
Seed 7
Seed 42
Seed 123
Mean
Valid
Matched
0.7975
0.7882
0.7950
0.7936
0.997
Composite, label-matched
0.7844
0.7844
0.7844
0.7844
0.997
Composite, cos=0.95
0.7857
0.7801
0.7801
0.7819
0.998
Composite, cos=0.90
0.7595
0.7595
0.7558
0.7583
0.997
Similar-user
0.7551
0.7645
0.7539
0.7578
0.996
Labels only, own
0.7682
0.7688
0.7626
0.7666
0.997
Table 1: LaMP-2 news accuracy. All history interventions use the same three matched-trained checkpoints; the last row trains and evaluates with history masked. Labels-only inputs contain the six categories without article text. Valid is the mean exact-label output rate. Intervals for paired contrasts are in Appendix L .
Control
Seen users
Unseen users
History masked
3.7[1.7,5.6]
0.8[−0.7,2.3]
Random-user
11.0[8.0,14.1]
7.9[5.4,10.4]
Similar-user
3.2[0.9,5.7]
0.65[−1.1,2.2]
Composite
0.6[−0.6,2.0]
0.4[−0.6,1.5]
Composite, label-matched
−0.1[−1.2,1.0]
−0.4[−1.5,0.5]
Table 2: Matched minus each control, accuracy points (95% bootstrap intervals), three seeds and two user folds. The masked condition uses the same checkpoint; no separate no-history model was trained within these folds.
Decoder
Statistic
Observed
Homogeneous-effect null
Zero-effect null
0.5B
Gini (∣Δu∣)
0.727±0.016
0.720
0.760
0.5B
SD (Δu)
0.352
0.318
–
3B
Gini (∣Δu∣)
0.761±0.017
0.758
0.791
3B
SD (Δu)
0.323
0.294
–
Table 3: Per-user decomposition against permutation nulls at matched item counts: Gini of the per-user effect magnitude and the standard deviation of the per-user effect across users (not reported under the zero-effect null). The observed Gini sits next to the homogeneous-effect null and below the zero-effect null; only dispersion exceeds its null.
Task
Metric
Matched
History masked
Random-user
Matched − Random-user
LaMP-1, citation choice
Accuracy
0.6707
0.6757
0.6700
+0.0007
LaMP-3, product rating
MAE ↓
0.436
0.447
0.446
−0.010
LaMP-4, news headline
ROUGE-1
0.1923
0.1886
0.1859
+0.0064
LaMP-5, scholarly title
ROUGE-1
0.5008
0.4966
0.5006
+0.0002
Table 4: The four other LaMP tasks under the three conditions. Three seeds for LaMP-3/5 and two for LaMP-1/4; history controls are counterfactual evaluations of each matched-trained checkpoint. Lower is better for mean absolute error (MAE); higher is better otherwise. One metric per task is reported; the benchmark also defines RMSE for LaMP-3 and ROUGE-L for LaMP-4 and LaMP-5. Differences between conditions are of the same size as the seed spread.
Figure 2: Accuracy against non-shared trainable parameters, excluding the shared 109M-parameter input encoder. Error bars are standard deviations over three seeds. Cross-attention models retain the prefix; these are not exactly budget-matched route comparisons. Full metrics appear in Table 5 .
Appendix figures & tables10 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 3: Left: within-user replacement of k out of six items, three-seed means relative to the trained no-history model; the dotted line is masking on the matched-trained model. Lines connect measured doses and carry no uncertainty band. Right: Matched-minus-control contrasts for seen and unseen users, with seed–fold/user bootstrap intervals (Table 2 ).
Model
Trainable
× prefix
Accuracy
Macro-F1
Valid
Train loss
Prefix (2-layer Q-Former)
129,970,944
1.00
0.7389±0.0059
0.4896±0.0044
0.984
0.0041
Prefix, 4-layer
144,146,688
1.11
0.7385±0.0019
0.484
0.981
0.0039
Prefix, 8-layer
172,498,176
1.33
0.7180±0.0013
0.468
0.982
0.0126
Prefix, 16-layer
229,201,152
1.76
0.6685±0.0174
0.375
0.973
0.0283
Gated prefix, 2-layer
134,165,249
1.03
0.7275±0.0221
0.478
0.979
0.0087
XAttn-1
148,339,457
1.14
0.7782±0.0097
0.5614±0.0114
0.995
0.0011
Appendix
Table 5: Injection route against trainable parameters on LaMP-2, frozen Qwen2.5-3B, learning rate 3×10−5 , three seeds, sample standard deviations (for macro-F1 where recorded). Upper block: the prefix’s Q-Former depth scaled at fixed conditioning length, and the prefix with a gated zero-initialized branch (§ C.2 ). Lower block: XAttn- k with injected layers range(18, 36, s) of the 36-layer decoder; strides s=18,8,4,2 give k=1,3,5,9 . Each injected layer adds 16,789,505 trainable parameters; the first also adds a 1,579,008-parameter projection of Z . Train loss: median per-step loss over the final third of training (Appendix I ), not reported for XAttn-3 and XAttn-5.
Setting
Value
History encoder
BAAI/bge-base-en-v1.5 , frozen, max 512 tokens per item
Control encoder
BAAI/bge-base-en-v1.5 ; robustness checks repeat the similarity assignment and the composite with intfloat/e5-base-v2
Input encoder
separate BGE encoder; all parameters trainable
Q-Former
8 queries, 2 layers, width 768 unless varied in the sweep
Projection width
1024 hidden units
Optimizer
AdamW, weight decay 10−3 , gradient clipping at 1.0
Appendix
Table 6: Complete hyperparameter listing.
Wout=0
Wout random
Accuracy, α0=0
0.480±0.178
0.677±0.033
Accuracy, α0=1.5
0.726±0.012
0.711±0.031
Adapter gradient norm, α0=0 / 1.5
0.0 / 5.7
2.1 / 7.4
Gate ∣tanhα∣ , α0=0 / 1.5
0.000000 / 0.904092
0.006787 / 0.903341
Appendix
Table 7: Initialization ablation on LaMP-2 with Qwen2.5-1.5B, 600 steps, 300 evaluation examples, three seeds per cell, sample standard deviations. The doubly-zero branch has exactly zero gradient; the short-run accuracy differences are not a stable effect-size estimate. Gradient norm and gate magnitude are measured at step 590 of the seed-42 run.
Figure 4: Accuracy before and after auditing the parser, identical subset and seed: prefix 0.7300 to 0.7750, cross-attention 0.8375 in both cases. This diagnostic subset is separate from the full-validation architecture comparison.
Figure 5: Memory injection routes; all three share the encoders and the Q-Former that produce Z∈R8×768 from the history items and the current query. (a) Prefix: Z is projected and prepended to the input embeddings. (b) XAttn- k : the prefix is retained and k decoder layers receive the gated update of Eq. ( 1 ); this hybrid is the cross-attention model throughout. (c) Deep key-value prefix, an appendix experiment. Decoder and history-item encoder are frozen; input encoder, Q-Former and injection modules train.
Model
Q-Former params
Injection params
Q-Former
Injection
Prefix, 2-layer
18,909,696
1,579,008
1.82×10−3
9.28×10−3
Prefix, 16-layer
118,139,904
1,579,008
5.36×10−4
1.35×10−2
XAttn-1
18,909,696
19,947,521
1.03×10−3
1.75×10−3
Appendix
Table 8: Root-mean-square gradient per parameter reaching the Q-Former and the injection module, averaged over the final quarter of training, seed 42.
Work
Setting
Replacement
Graded by similarity
Fixed checkpoint
Contrast vs. no or masked history
Negative rebuilt
LaMP [ Salemi et al., 2024 ]
retrieval of text items
random items from the user’s own profile, or from the whole corpus
two levels
zero-shot and fine-tuned
yes
–
Wu et al. [2024]
retrieval of text items (Flan-T5 fine-tuned; frozen Llama-2)
query-relevant items from a pool of other users’ items
by query relevance, not user similarity
trained per augmentation
yes
–
PRIME [ Zhang et al., 2025 ]
textual profile plus per-user LoRA
whole profile and LoRA of another user
four levels (most similar to dissimilar)
no: LoRA swapped
Non-P
–
ENOUGH [ Liu et al., 2026a ]
retrieval of text records, frozen generator
one record replaced by a content-matched other-user record; also random user and whole-profile swap
matched on context, relevance, length, time
yes
yes
matching audit
Zheng et al. [2026]
multimodal recommenders
shuffle (permutation) of per-user weights, plus a population-weight contrast and a signal-implant dose–response
dose–response
yes
–
–
This paper
eight-vector compressed memory around a frozen decoder, and raw text
whole profile: random user, similarity-matched user, composite of other users’ items
similarity and dose curves; category matching
yes
yes, masked and trained
two encoders, three greedy builds
Appendix
Table 9: Replacement controls in prior and concurrent work and in this paper.
Contrast
Seed 7
Seed 42
Seed 123
Mean
Matched − History masked (gain over masked history)
5.5
4.4
4.5
4.8
History masked − Random-user (harm)
5.2
4.4
6.2
5.3
Matched − Random-user
10.7
8.8
10.8
10.1
Matched − Similar-user
4.2
2.4
4.1
3.6
Matched − Composite ( cos=0.90 )
3.8
2.9
3.9
3.5
Matched − Composite ( cos=0.95 )
1.2
0.8
1.5
1.2
Appendix
Table 10: Contrasts by seed (points) on the 3B XAttn-9 checkpoints, and descriptive ratios relative to masking (not the trained no-history comparator). The last block gives the alternative constructions of the negative: two greedy rebuilds of the composite at seed 42, and the Similar-user and composite negatives rebuilt under the E5 encoder (three-seed means).
Control
Seed 7
Seed 42
Seed 123
t , 2 df
Bootstrap
Random-user
10.72
8.85
10.78
[7.39,12.84]
[8.01,12.30]
Similar-user
4.24
2.37
4.11
[0.98,6.17]
[2.09,5.23]
Composite, 0.95
1.18
0.81
1.50
[0.31,2.02]
[0.40,1.89]
Composite, labels
1.31
0.37
1.06
[−0.29,2.12]
[0.09,1.79]
History masked
5.48
4.42
4.55
[3.38,6.26]
[3.41,6.34]
Own labels only
2.93
1.93
3.24
[1.00,4.40]
[1.25,4.21]
Appendix
Table 11: Matched minus each control, accuracy points. The t -interval uses three seed-level contrasts; the bootstrap resamples seeds and users (10,000 replicates).
Large language model (LLM) evaluators are increasingly used to assess personalized assistants, yet whether they can reliably judge personalization has rarely been tested. We introduce PersEvalBench, a benchmark testing whether an evaluator can recognize appropriate and inappropriate uses of user history. It contains 403 multi-episode scenarios where user information should persist, transfer, or cease to apply. The benchmark covers nine controlled error subtypes across three families: under-personalization, over-personalization, and misapplied-state errors. Human audits confirm that generated errors match their intended types and that the prescribed uses of user information are reasonable. Across five LLM evaluators, recognition of controlled personalization errors remains limited even with the full dialogue history. On the human-evaluated subset, under-personalization shows the largest gap: humans recognize 94.4% of these errors, compared with 63.9% for the strongest reported evaluator. Evaluators often recognize that a response is problematic but reverse the error direction, mistaking failure to follow an applicable preference (UNDER) for use of an inapplicable preference (OVER). Easier evidence access offers little help: highlighting the relevant history turns raises error recognition only slightly, suggesting that the remaining difficulty lies in judging what role that information should play in the response.
Xinyi Liu, Hooshang Nayyeri, Emine Yilmaz +3
Amazon · University of Illinois Urbana-Champaign · University College London
Despite growing interest, most evaluations of large language models' (LLMs') personalization abilities have relied on synthetic data. It remains unclear how well current personalization systems work for real users. In this paper, we study the gap in LLM personalization performance when using synthetic versus human data. We collect human conversations (550 conversations) and judgments across three stages of personalization: extracting user attributes from conversations (5,949 judgments), pairing relevant attributes with new prompts (11,919), and incorporating relevant attributes into a personalized response (1,101). Incorporating human data reveals system limitations at each stage. Models struggle to extract attributes from human conversations, disagree with human judgments on relevant attributes, and generate personalized responses that humans judge no better than generic responses (though that LLM judges widely rate as better). We introduce two lightweight training-based interventions that shift automated personalization evaluation closer to human data in our first two stages. However, in our third stage we find that learned reward models achieve only modest correlation with human ratings, suggesting that human-aligned personalization quality judgments are difficult to model directly. Our collected data provides a foundation for studying how models should extract, select, and incorporate user information in ways that humans find useful.
Lechen Zhang, Jiarui Liu, Tal August
†University of Illinois Urbana-Champaign · ♯Carnegie Mellon University
In structured rating prediction, does a personal AI use historical item-rating associations, or mainly the user's rating tendencies? We audit this distinction by permuting historical ratings within each user while preserving the exact rating distribution, item support, and metadata. We combine this control with full history, native memory extraction, and matched numerical readers in a publicly frozen evaluation of 400 held-out user profiles and 6,160 target ratings across Coat and MovieLens. On Coat, the tested Qwen-written Mem0 pipeline increases user-macro mean absolute error relative to full history by 0.084 for Qwen and 0.149 for Phi; both family-adjusted bootstrap intervals exclude zero. Correct historical assignments help both readers on Coat, but the corresponding MovieLens effects are smaller and inconclusive after adjustment. A history-only ridge reader outperforms Qwen in both domains and Phi on MovieLens, while the Coat Phi comparison is unresolved. All 2,800 reader calls, including 150 invalid outputs, are retained under a fixed fallback rule. A separate implementation verifies inputs, metrics, and all ten primary contrasts. The contribution is a reproducible diagnostic study showing why extraction, association use, output reliability, and reader choice require separate evaluation.