With ever increasing personalized large language model (LLM) services, the next question to answer is how to improve the quality of services/users' experiences, which is indicated by personalized LLM generation accuracy/quality and inference latency. Although state-of-the-art designs, e.g., parameter-efficient fine-tuning (PEFT), retrieval-augmented generation (RAG) or Full PEFT+RAG, can help, they largely ignore the heterogeneous semantic concentrations of users' historical data and cannot fully unleash the potential of personal data for personalized LLM performance enhancement. In this paper, we propose PEFT Union RAG (PUR), a novel fast and accurate'' approach that exploits the semantic concentration of users' personal historical data to split workload between PEFT and RAG, aiming to improve personalized LLM performance in terms of better generation accuracy/quality and lower inference latency. Briefly, PUR partitions user's personal historical data and assigns semantic concentrated core'' information to PEFT for offline parametric fine-tuning, while retaining dispersed information for RAG. During the online LLM inference, PUR directly generates responses for core'' queries covered by the PEFT region and invokes RAG only for sporadic peripheral'' queries. Experimental results show that, by appropriately splitting workload between PEFT and RAG, PUR consistently achieves the best generation accuracy/quality for various users across multiple tasks, while reducing inference latency by 14.4%--46.6% compared with the best performed method.
Figures & tables
Figure 1: Preliminary results. (a) The generation accuracy/quality of PEFT, RAG, and Full PEFT+RAG across users with different semantic concentration scores. (b) Average inference latency under different methods, where k denotes the number of retrieved texts used at inference.
Figure 2: An overview of our proposed PEFT Union RAG (PUR).
Sconc Range
Compared Methods
PUR (PEFT Allocated Ratio)
PEFT
RAG
Full PEFT+RAG
10%
20%
30%
40%
50%
60%
70%
80%
90%
Personalized Movie Tagging (Acc. ↑ )
0.66–0.68
52.7
59.4
57.8
59.4
59.7
60.9
59.8
58.1
56.4
56.0
54.9
53.3
0.68–0.70
53.6
61.9
59.4
62.0
63.9
64.6
62.5
60.7
57.9
57.1
55.2
53.1
0.70–0.72
54.9
61.0
63.5
60.8
62.5
63.0
64.4
63.7
61.8
59.6
57.1
55.8
0.72–0.74
57.0
60.8
63.4
61.4
61.9
63.3
64.1
65.4
62.2
60.0
58.9
57.0
Table 1: Performance under different semantic concentration range and PEFT allocation ratios. For each user group, we compare other methods with PUR instantiated using PEFT allocation ratios from 10% to 90%, to examine how the optimal allocation varies with user semantic concentration. The best score is in bold and the second best is underlined .
Sconc Range
PEFT Ratio
0.66–0.70
30%
0.70–0.72
40%
0.72–0.74
50%
0.74–0.76
60%
0.76–0.80
70%
0.80–0.84
80%
Table 2: Data allocation rule for PUR.
Metrics
PRQ Perf.
RRQ Perf.
Overall Perf.
Avg. Infer. Time (s) ↓
PRQ Perf.
RRQ Perf.
Overall Perf.
Avg. Infer. Time (s) ↓
PRQ Perf.
RRQ Perf.
Overall Perf.
Avg. Infer. Time (s) ↓
Personalized Movie Tagging (Acc. ↑ )
Method
User1 ( Sconc : 0.689 / PRQ Ratio: 34.5%)
User2 ( Sconc : 0.752 / PRQ Ratio: 56.5%)
User3 ( Sconc : 0.811 / PRQ Ratio: 83.6%)
PEFT
56.7
52.6
54.0
0.85
57.7
55.0
56.5
0.90
68.3
56.3
66.3
0.79
RAG
63.3
61.4
62.1
1.75
61.5
62.5
62.0
1.83
67.1
56.3
66.3
1.78
Full PEFT+RAG
63.3
57.9
59.8
1.77
63.5
65.0
64.1
1.81
68.3
68.8
68.4
1.74
PUR (Ours)
70.0
61.4
64.4
1.41
67.3
65.0
66.3
1.29
70.7
68.8
70.4
0.93
Table 3: Detailed LLM inference performance analysis on representative users with different semantic concentration levels. Each user’s new queries are partitioned into PEFT-Region Queries (PRQ) and RAG-Region Queries (RRQ) according to PUR’s inference path selection.
Figure 3: Personalized generation accuracy/quality on the four remaining evaluated LaMP tasks not used in the probing study. PUR applies the allocation ratios obtained from the probing study to evaluate cross-task generalization.
Figure 4: Impact of historical data size on the relative performance of the methods. PUR consistently achieves the best performance across different user historical data sizes, while the relative ranking of the other methods remains largely unchanged within the same semantic concentration interval.
Figure 5: Effect of the long-tail compensation term on the relationship between Sconc and the optimal PEFT allocation ratio. Each point represents a sampled user. Adding the compensation term ( λ=1 ) yields better alignment in the score space than using average similarity alone ( λ=0 ).
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
Team
( C1Sconc : 0.745, C2Sconc : 0.752)
( C1Sconc : 0.811, C2Sconc : 0.838)
( C1Sconc : 0.769, C2Sconc : 0.823)
Method
User1 Perf.
User2 Perf.
Overall Perf.
User1 Perf.
User2 Perf.
Overall Perf.
User1 Perf.
User2 Perf.
Overall Perf.
PEFT
47.5
48.3
47.9
49.1
49.9
49.5
48.0
49.5
48.9
RAG
49.3
49.6
49.5
49.4
49.2
49.3
47.7
48.8
48.3
Full PEFT+RAG
50.1
50.7
50.4
50.8
50.4
50.6
48.6
50.2
49.5
PUR (Ours)
51.5
52.3
51.9
51.6
52.0
51.8
50.1
52.9
51.7
Appendix
Table 4: Performance comparison under multi-cluster historical data. Each team consists of two users whose merged historical data is clearly detected by HDBSCAN as forming two clusters. We report each user’s individual evaluation performance and the average performance over all queries from both users (Task: Personalized Scholarly Title Generation).
User Group 1
User Group 2
User Group 3
Embedding model
Method
Sconc Range
(R-1 ↑ )
PEFT Subset Overlap
Sconc Range
(R-1 ↑ )
PEFT Subset Overlap
Sconc Range
(R-1 ↑ )
PEFT Subset Overlap
BGE-Base-en-v1.5
PEFT
–
46.4
–
–
48.3
–
–
49.6
–
RAG
–
48.5
–
–
49.0
–
–
49.3
–
Full PEFT+RAG
–
48.3
–
–
50.1
–
–
50.3
–
PUR (Ours)
0.68-0.70
49.4
100%
0.74-0.76
51.5
100%
0.80-0.82
51.7
100%
E5-Base-v2
PEFT
–
46.4
–
–
48.3
–
–
49.6
–
Appendix
Table 5: Performance comparison under different embedding models. For the same user groups, we report the resulting Sconc range, generation quality, and PUR’s PEFT subset overlap under different embedding models (Task: Personalized Scholarly Title Generation).
User 1 (Low Sconc )
User 2 (Medium Sconc )
User 3 (High Sconc )
Method
Offline Comp. (PFLOPs) ↓
Inference Comp. (TFLOPs) ↓
Offline Comp. (PFLOPs) ↓
Inference Comp. (TFLOPs) ↓
Offline Comp. (PFLOPs) ↓
Inference Comp. (TFLOPs) ↓
PEFT
1.096
30.7
1.107
34.7
1.209
36.1
RAG
0.002
170.9
0.002
208.8
0.002
265.4
Full PEFT+RAG
1.098
170.9
1.109
208.8
1.211
265.4
PUR (Ours)
0.329
122.5
0.665
110.4
0.967
73.7
Appendix
Table 6: Computational cost comparison of different methods in the offline and online inference stages. We report the computational cost for the same three users analyzed in Table 3 on the Personalized Movie Tagging task.
Figure 6: Distribution of Savg and the long-tail compensation term πtail,ihtail,i for the users involved in Fig. 5 . Most users have relatively small compensation values, while a smaller subset exhibits substantially larger values and corresponds closely to the points that deviate from the overall trend when λ=0 .
Average historical data size of sampled users within each Sconc interval
Sconc Range
LaMP-1
LaMP-2
LaMP-3
LaMP-4
LaMP-5
LaMP-7
0.66–0.68
238
181
884
93
341
36
0.68–0.70
335
126
1006
82
450
48
0.70–0.72
403
124
946
119
580
59
0.72–0.74
432
113
923
87
551
85
0.74–0.76
388
137
891
127
528
77
Appendix
Table 7: Average historical data size of the sampled evaluation users across different Sconc intervals and LaMP tasks.
Large language models (LLMs) are typically aligned with population-level preferences, despite substantial variation across individual users. We introduce POPI, a user-level personalization framework that separates the problem into two components connected by a natural-language interface: a shared inference model that distills heterogeneous user signals into a concise preference summary, and a shared generator that conditions on this summary to produce personalized responses. Both components are trained under a unified preference-optimization objective, with reinforcement learning handling the non-differentiable inference step. This objective decomposes into generator approximation error and summary informativeness, revealing how a single loss simultaneously drives accurate generation and informative summarization. Because the interface is natural language, learned summaries can be inferred once per user and reused across different generators -- including frozen, black-box commercial APIs. Across four personalization benchmarks, POPI generally improves personalization quality while reducing context overhead by up to an order of magnitude.
Yizhuo Chen, Xin Liu, Ruijie Wang +7
University of Illinois Urbana-Champaign · Amazon · University of Notre Dame
Personalizing large language models requires adapting model behavior to individual users while preserving robustness and deployment-scale efficiency. Existing approaches typically personalize LLMs either at the input level, by retrieving user histories or constructing profile prompts, or at the parameter level, by maintaining user-specific parameter-efficient modules. The former makes personalization sensitive to retrieval quality and prompt design, whereas the latter incurs storage and maintenance costs that grow with the user population. To address these limitations, we propose TAP-PER (Temporal Attentive Prefix for PERsonalization), a prefix-based framework that encodes user preferences as learnable representations, avoiding the serialization of user histories into prompts and replacing heavy per-user adapters with lightweight user-state prefix embeddings. Inspired by personalized recommendation systems, TAP-PER decomposes user modeling into user-state and query-conditioned components, and incorporates temporal signals to capture the evolving nature of user interests. Experiments on six LaMP tasks show that TAP-PER consistently outperforms prompt-based and model-based baselines across classification, rating, and generation settings. Moreover, TAP-PER uses 130x fewer per-user parameters than OPPU and roughly half the total parameter footprint of PER-PCS at the 1,000-user scale, demonstrating scalable personalization without prompt-serialized histories or heavy per-user adapters.
Heng Cao, Fan Zhang, Jian Yao +8
*Microsoft · †The Hong Kong Polytechnic University · ⋄Shanghai International Studies University +1
A common approach to personalization in large language models (LLMs) is to incorporate a subset of the user memory into the prompt at inference time to guide the model's generation. Existing methods select these subsets primarily using similarity between user memory items and input queries, ignoring how features actually affect the model's response distribution. We propose Response-Utility optimization for Memory Selection (RUMS), a novel method that selects user memory items by measuring the mutual information between a subset of memory and the model's outputs, identifying items that reduce response uncertainty and sharpen predictions beyond semantic similarity. We demonstrate that this information-theoretic foundation enables more principled user memory selection that aligns more closely with human selection compared to state-of-the-art methods, and models 400× larger. Additionally, we show that memory items selected using RUMS result in better response quality compared to existing approaches, while having up to 95% reduction in computational cost.
Jillian Fisher, Jennifer Neville, Chan Young Park
Department of Computer Science and Engineering, University of Washington, Seattle, WA, United States of America · Microsoft Research, Redmond, WA, United States of America