cs.AINov 6, 2026
SaveNot Your Diary, Just Your Topic: Grading the Wrong-User Control
Organizations: Yandex · Accenture · RawModels · Independent Researcher ·
Abstract
A common test of personalization replaces a user's history with someone else's and measures the drop in accuracy. But this swap also changes the topics in the history, so the drop may reflect topic mismatch. We examine this problem with a frozen language model on LaMP-2 news categorization, using histories from similar users and synthetic histories assembled from other users' articles. The user's own history improves accuracy by 2.3 percentage points over a model trained without history. Removing history only at evaluation makes this gain appear larger, at 4.8 points. Category labels alone reproduce 86% of the accuracy gap between the user's own history and a random user's history. With synthetic histories matched on labels and text similarity, the gap falls to 0.9 points. Across two user splits and three training seeds, the advantage over a similar user's history is 3.2 points for users seen during training and 0.65 points for unseen users, where the evidence for an advantage remains inconclusive. A small movie-tagging experiment with one training seed also finds no advantage over other users' histories. These findings show why personalization should be evaluated on unseen users, with histories of similar content and a model trained without history as comparisons. We also find that adding gated cross-attention improves on a prefix-only model under our training setup; whether this holds more broadly remains open.
Figures & tables
Figure 1: LaMP-2 news controls. (a) Illustrative histories show why a random-user swap mixes a change of owner with a change of topic. A similar-user history reduces content mismatch; the query and trained checkpoint stay fixed. (b) Three-seed mean accuracy relative to a separately trained no-history model. (c) Matched-minus-swapped accuracy against profile cosine, separately for random and similar users; bands resample users, retaining their repeated measurements. The dashed line is the 2.3-point gain over the trained comparator. The panels do not decompose a causal identity effect.
| Condition | Seed 7 | Seed 42 | Seed 123 | Mean | Valid |
|---|---|---|---|---|---|
| Matched | 0.7975 | 0.7882 | 0.7950 | 0.7936 | 0.997 |
| Composite, label-matched | 0.7844 | 0.7844 | 0.7844 | 0.7844 | 0.997 |
| Composite, | 0.7857 | 0.7801 | 0.7801 | 0.7819 | 0.998 |
| Composite, | 0.7595 | 0.7595 | 0.7558 | 0.7583 | 0.997 |
| Similar-user | 0.7551 | 0.7645 | 0.7539 | 0.7578 | 0.996 |
| Labels only, own | 0.7682 | 0.7688 | 0.7626 | 0.7666 | 0.997 |
Table 1: LaMP-2 news accuracy. All history interventions use the same three matched-trained checkpoints; the last row trains and evaluates with history masked. Labels-only inputs contain the six categories without article text. Valid is the mean exact-label output rate. Intervals for paired contrasts are in Appendix L .
| Control | Seen users | Unseen users |
|---|---|---|
| History masked | ||
| Random-user | ||
| Similar-user | ||
| Composite | ||
| Composite, label-matched |
Table 2: Matched minus each control, accuracy points (95% bootstrap intervals), three seeds and two user folds. The masked condition uses the same checkpoint; no separate no-history model was trained within these folds.
| Decoder | Statistic | Observed | Homogeneous-effect null | Zero-effect null |
|---|---|---|---|---|
| 0.5B | Gini | 0.720 | 0.760 | |
| 0.5B | SD | 0.352 | 0.318 | – |
| 3B | Gini | 0.758 | 0.791 | |
| 3B | SD | 0.323 | 0.294 | – |
Table 3: Per-user decomposition against permutation nulls at matched item counts: Gini of the per-user effect magnitude and the standard deviation of the per-user effect across users (not reported under the zero-effect null). The observed Gini sits next to the homogeneous-effect null and below the zero-effect null; only dispersion exceeds its null.
| Task | Metric | Matched | History masked | Random-user | Matched Random-user |
|---|---|---|---|---|---|
| LaMP-1, citation choice | Accuracy | 0.6707 | 0.6757 | 0.6700 | |
| LaMP-3, product rating | MAE | 0.436 | 0.447 | 0.446 | |
| LaMP-4, news headline | ROUGE-1 | 0.1923 | 0.1886 | 0.1859 | |
| LaMP-5, scholarly title | ROUGE-1 | 0.5008 | 0.4966 | 0.5006 |
Table 4: The four other LaMP tasks under the three conditions. Three seeds for LaMP-3/5 and two for LaMP-1/4; history controls are counterfactual evaluations of each matched-trained checkpoint. Lower is better for mean absolute error (MAE); higher is better otherwise. One metric per task is reported; the benchmark also defines RMSE for LaMP-3 and ROUGE-L for LaMP-4 and LaMP-5. Differences between conditions are of the same size as the seed spread.
Figure 2: Accuracy against non-shared trainable parameters, excluding the shared 109M-parameter input encoder. Error bars are standard deviations over three seeds. Cross-attention models retain the prefix; these are not exactly budget-matched route comparisons. Full metrics appear in Table 5 .
Appendix figures & tables10 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 3: Left: within-user replacement of out of six items, three-seed means relative to the trained no-history model; the dotted line is masking on the matched-trained model. Lines connect measured doses and carry no uncertainty band. Right: Matched-minus-control contrasts for seen and unseen users, with seed–fold/user bootstrap intervals (Table 2 ).
| Model | Trainable | prefix | Accuracy | Macro-F1 | Valid | Train loss |
| Prefix (2-layer Q-Former) | 129,970,944 | 1.00 | 0.984 | 0.0041 | ||
| Prefix, 4-layer | 144,146,688 | 1.11 | 0.484 | 0.981 | 0.0039 | |
| Prefix, 8-layer | 172,498,176 | 1.33 | 0.468 | 0.982 | 0.0126 | |
| Prefix, 16-layer | 229,201,152 | 1.76 | 0.375 | 0.973 | 0.0283 | |
| Gated prefix, 2-layer | 134,165,249 | 1.03 | 0.478 | 0.979 | 0.0087 | |
| XAttn-1 | 148,339,457 | 1.14 | 0.995 | 0.0011 |
Table 5: Injection route against trainable parameters on LaMP-2, frozen Qwen2.5-3B, learning rate , three seeds, sample standard deviations (for macro-F1 where recorded). Upper block: the prefix’s Q-Former depth scaled at fixed conditioning length, and the prefix with a gated zero-initialized branch (§ C.2 ). Lower block: XAttn- with injected layers range(18, 36, s) of the 36-layer decoder; strides give . Each injected layer adds 16,789,505 trainable parameters; the first also adds a 1,579,008-parameter projection of . Train loss: median per-step loss over the final third of training (Appendix I ), not reported for XAttn-3 and XAttn-5.
| Setting | Value |
|---|---|
| History encoder | BAAI/bge-base-en-v1.5 , frozen, max 512 tokens per item |
| Control encoder | BAAI/bge-base-en-v1.5 ; robustness checks repeat the similarity assignment and the composite with intfloat/e5-base-v2 |
| Input encoder | separate BGE encoder; all parameters trainable |
| Q-Former | 8 queries, 2 layers, width 768 unless varied in the sweep |
| Projection width | 1024 hidden units |
| Optimizer | AdamW, weight decay , gradient clipping at 1.0 |
Table 6: Complete hyperparameter listing.
| random | ||
| Accuracy, | ||
| Accuracy, | ||
| Adapter gradient norm, / | 0.0 / 5.7 | 2.1 / 7.4 |
| Gate , / | 0.000000 / 0.904092 | 0.006787 / 0.903341 |
Table 7: Initialization ablation on LaMP-2 with Qwen2.5-1.5B, 600 steps, 300 evaluation examples, three seeds per cell, sample standard deviations. The doubly-zero branch has exactly zero gradient; the short-run accuracy differences are not a stable effect-size estimate. Gradient norm and gate magnitude are measured at step 590 of the seed-42 run.
Figure 4: Accuracy before and after auditing the parser, identical subset and seed: prefix 0.7300 to 0.7750, cross-attention 0.8375 in both cases. This diagnostic subset is separate from the full-validation architecture comparison.
Figure 5: Memory injection routes; all three share the encoders and the Q-Former that produce from the history items and the current query. (a) Prefix: is projected and prepended to the input embeddings. (b) XAttn- : the prefix is retained and decoder layers receive the gated update of Eq. ( 1 ); this hybrid is the cross-attention model throughout. (c) Deep key-value prefix, an appendix experiment. Decoder and history-item encoder are frozen; input encoder, Q-Former and injection modules train.
| Model | Q-Former params | Injection params | Q-Former | Injection |
|---|---|---|---|---|
| Prefix, 2-layer | 18,909,696 | 1,579,008 | ||
| Prefix, 16-layer | 118,139,904 | 1,579,008 | ||
| XAttn-1 | 18,909,696 | 19,947,521 |
Table 8: Root-mean-square gradient per parameter reaching the Q-Former and the injection module, averaged over the final quarter of training, seed 42.
| Work | Setting | Replacement | Graded by similarity | Fixed checkpoint | Contrast vs. no or masked history | Negative rebuilt |
|---|---|---|---|---|---|---|
| LaMP [ Salemi et al., 2024 ] | retrieval of text items | random items from the user’s own profile, or from the whole corpus | two levels | zero-shot and fine-tuned | yes | – |
| Wu et al. [2024] | retrieval of text items (Flan-T5 fine-tuned; frozen Llama-2) | query-relevant items from a pool of other users’ items | by query relevance, not user similarity | trained per augmentation | yes | – |
| PRIME [ Zhang et al., 2025 ] | textual profile plus per-user LoRA | whole profile and LoRA of another user | four levels (most similar to dissimilar) | no: LoRA swapped | Non-P | – |
| ENOUGH [ Liu et al., 2026a ] | retrieval of text records, frozen generator | one record replaced by a content-matched other-user record; also random user and whole-profile swap | matched on context, relevance, length, time | yes | yes | matching audit |
| Zheng et al. [2026] | multimodal recommenders | shuffle (permutation) of per-user weights, plus a population-weight contrast and a signal-implant dose–response | dose–response | yes | – | – |
| This paper | eight-vector compressed memory around a frozen decoder, and raw text | whole profile: random user, similarity-matched user, composite of other users’ items | similarity and dose curves; category matching | yes | yes, masked and trained | two encoders, three greedy builds |
Table 9: Replacement controls in prior and concurrent work and in this paper.
| Contrast | Seed 7 | Seed 42 | Seed 123 | Mean |
|---|---|---|---|---|
| Matched History masked (gain over masked history) | 5.5 | 4.4 | 4.5 | 4.8 |
| History masked Random-user (harm) | 5.2 | 4.4 | 6.2 | 5.3 |
| Matched Random-user | 10.7 | 8.8 | 10.8 | 10.1 |
| Matched Similar-user | 4.2 | 2.4 | 4.1 | 3.6 |
| Matched Composite ( ) | 3.8 | 2.9 | 3.9 | 3.5 |
| Matched Composite ( ) | 1.2 | 0.8 | 1.5 | 1.2 |
Table 10: Contrasts by seed (points) on the 3B XAttn-9 checkpoints, and descriptive ratios relative to masking (not the trained no-history comparator). The last block gives the alternative constructions of the negative: two greedy rebuilds of the composite at seed 42, and the Similar-user and composite negatives rebuilt under the E5 encoder (three-seed means).
| Control | Seed 7 | Seed 42 | Seed 123 | , 2 df | Bootstrap |
|---|---|---|---|---|---|
| Random-user | 10.72 | 8.85 | 10.78 | ||
| Similar-user | 4.24 | 2.37 | 4.11 | ||
| Composite, 0.95 | 1.18 | 0.81 | 1.50 | ||
| Composite, labels | 1.31 | 0.37 | 1.06 | ||
| History masked | 5.48 | 4.42 | 4.55 | ||
| Own labels only | 2.93 | 1.93 | 3.24 |
Table 11: Matched minus each control, accuracy points. The -interval uses three seed-level contrasts; the bootstrap resamples seeds and users (10,000 replicates).