Fast and Accurate: Improving Personalized LLM Performance by Semantic Concentration Inspired Workload Split between RAG and Fine-Tuning

Xingke Yang; Sicong Li; Lingfeng Yao; Rui Chen; Changqing Luo; Xin Fu; Miao Pan

cs.LGOct 11, 2026

Fast and Accurate: Improving Personalized LLM Performance by Semantic Concentration Inspired Workload Split between RAG and Fine-Tuning

Authors: Xingke Yang, Sicong Li, Lingfeng Yao, Rui Chen, Changqing Luo, Xin Fu, Miao Pan

Organizations: University of Houston, USA · Independent Researcher, USA

Abstract

With ever increasing personalized large language model (LLM) services, the next question to answer is how to improve the quality of services/users' experiences, which is indicated by personalized LLM generation accuracy/quality and inference latency. Although state-of-the-art designs, e.g., parameter-efficient fine-tuning (PEFT), retrieval-augmented generation (RAG) or Full PEFT+RAG, can help, they largely ignore the heterogeneous semantic concentrations of users' historical data and cannot fully unleash the potential of personal data for personalized LLM performance enhancement. In this paper, we propose PEFT Union RAG (PUR), a novel fast and accurate'' approach that exploits the semantic concentration of users' personal historical data to split workload between PEFT and RAG, aiming to improve personalized LLM performance in terms of better generation accuracy/quality and lower inference latency. Briefly, PUR partitions user's personal historical data and assigns semantic concentrated core'' information to PEFT for offline parametric fine-tuning, while retaining dispersed information for RAG. During the online LLM inference, PUR directly generates responses for core'' queries covered by the PEFT region and invokes RAG only for sporadic peripheral'' queries. Experimental results show that, by appropriately splitting workload between PEFT and RAG, PUR consistently achieves the best generation accuracy/quality for various users across multiple tasks, while reducing inference latency by 14.4%--46.6% compared with the best performed method.

Figures & tables

Panel of Figure 1 — Figure 1: Preliminary results. (a) The generation accuracy/quality of PEFT, RAG, and Full PEFT+RAG across users with different semantic concentration scores. (b) Average inference latency under different methods, where k denotes the number of retrieved texts used at inference.

Figure 2: An overview of our proposed PEFT Union RAG (PUR).

$S_{\mathrm{conc}}$ Range	Compared Methods			PUR (PEFT Allocated Ratio)
$S_{\mathrm{conc}}$ Range	PEFT	RAG	Full PEFT+RAG	10%	20%	30%	40%	50%	60%	70%	80%	90%
Personalized Movie Tagging (Acc. $\uparrow$ )
0.66–0.68	52.7	59.4	57.8	59.4	59.7	60.9	59.8	58.1	56.4	56.0	54.9	53.3
0.68–0.70	53.6	61.9	59.4	62.0	63.9	64.6	62.5	60.7	57.9	57.1	55.2	53.1
0.70–0.72	54.9	61.0	63.5	60.8	62.5	63.0	64.4	63.7	61.8	59.6	57.1	55.8
0.72–0.74	57.0	60.8	63.4	61.4	61.9	63.3	64.1	65.4	62.2	60.0	58.9	57.0

Table 1: Performance under different semantic concentration range and PEFT allocation ratios. For each user group, we compare other methods with PUR instantiated using PEFT allocation ratios from 10% to 90%, to examine how the optimal allocation varies with user semantic concentration. The best score is in bold and the second best is underlined .

$S_{\mathrm{conc}}$ Range	PEFT Ratio
0.66–0.70	30%
0.70–0.72	40%
0.72–0.74	50%
0.74–0.76	60%
0.76–0.80	70%
0.80–0.84	80%

Table 2: Data allocation rule for PUR.

Metrics	PRQ Perf.	RRQ Perf.	Overall Perf.	Avg. Infer. Time (s) $\downarrow$	PRQ Perf.	RRQ Perf.	Overall Perf.	Avg. Infer. Time (s) $\downarrow$	PRQ Perf.	RRQ Perf.	Overall Perf.	Avg. Infer. Time (s) $\downarrow$
Personalized Movie Tagging (Acc. $\uparrow$ )
Method	User1 ( $S_{\mathrm{conc}}$ : 0.689 / PRQ Ratio: 34.5%)				User2 ( $S_{\mathrm{conc}}$ : 0.752 / PRQ Ratio: 56.5%)				User3 ( $S_{\mathrm{conc}}$ : 0.811 / PRQ Ratio: 83.6%)
PEFT	56.7	52.6	54.0	0.85	57.7	55.0	56.5	0.90	68.3	56.3	66.3	0.79
RAG	63.3	61.4	62.1	1.75	61.5	62.5	62.0	1.83	67.1	56.3	66.3	1.78
Full PEFT+RAG	63.3	57.9	59.8	1.77	63.5	65.0	64.1	1.81	68.3	68.8	68.4	1.74
PUR (Ours)	70.0	61.4	64.4	1.41	67.3	65.0	66.3	1.29	70.7	68.8	70.4	0.93

Table 3: Detailed LLM inference performance analysis on representative users with different semantic concentration levels. Each user’s new queries are partitioned into PEFT-Region Queries (PRQ) and RAG-Region Queries (RRQ) according to PUR’s inference path selection.

Panel of Figure 3 — Figure 3: Personalized generation accuracy/quality on the four remaining evaluated LaMP tasks not used in the probing study. PUR applies the allocation ratios obtained from the probing study to evaluate cross-task generalization.

Panel of Figure 4 — Figure 4: Impact of historical data size on the relative performance of the methods. PUR consistently achieves the best performance across different user historical data sizes, while the relative ranking of the other methods remains largely unchanged within the same semantic concentration interval.

Panel of Figure 5 — Figure 5: Effect of the long-tail compensation term on the relationship between $S_{\mathrm{conc}}$ and the optimal PEFT allocation ratio. Each point represents a sampled user. Adding the compensation term ( $\lambda=1$ ) yields better alignment in the score space than using average similarity alone ( $\lambda=0$ ).

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Team	( $C_{1}$ $S_{\mathrm{conc}}$ : 0.745, $C_{2}$ $S_{\mathrm{conc}}$ : 0.752)			( $C_{1}$ $S_{\mathrm{conc}}$ : 0.811, $C_{2}$ $S_{\mathrm{conc}}$ : 0.838)			( $C_{1}$ $S_{\mathrm{conc}}$ : 0.769, $C_{2}$ $S_{\mathrm{conc}}$ : 0.823)
Method	User1 Perf.	User2 Perf.	Overall Perf.	User1 Perf.	User2 Perf.	Overall Perf.	User1 Perf.	User2 Perf.	Overall Perf.
PEFT	47.5	48.3	47.9	49.1	49.9	49.5	48.0	49.5	48.9
RAG	49.3	49.6	49.5	49.4	49.2	49.3	47.7	48.8	48.3
Full PEFT+RAG	50.1	50.7	50.4	50.8	50.4	50.6	48.6	50.2	49.5
PUR (Ours)	51.5	52.3	51.9	51.6	52.0	51.8	50.1	52.9	51.7

Appendix

Table 4: Performance comparison under multi-cluster historical data. Each team consists of two users whose merged historical data is clearly detected by HDBSCAN as forming two clusters. We report each user’s individual evaluation performance and the average performance over all queries from both users (Task: Personalized Scholarly Title Generation).

		User Group 1			User Group 2			User Group 3
Embedding model	Method	$S_{\mathrm{conc}}$ Range	(R-1 $\uparrow$ )	PEFT Subset Overlap	$S_{\mathrm{conc}}$ Range	(R-1 $\uparrow$ )	PEFT Subset Overlap	$S_{\mathrm{conc}}$ Range	(R-1 $\uparrow$ )	PEFT Subset Overlap
BGE-Base-en-v1.5	PEFT	–	46.4	–	–	48.3	–	–	49.6	–
	RAG	–	48.5	–	–	49.0	–	–	49.3	–
	Full PEFT+RAG	–	48.3	–	–	50.1	–	–	50.3	–
	PUR (Ours)	0.68-0.70	49.4	100%	0.74-0.76	51.5	100%	0.80-0.82	51.7	100%
E5-Base-v2	PEFT	–	46.4	–	–	48.3	–	–	49.6	–

Appendix

Table 5: Performance comparison under different embedding models. For the same user groups, we report the resulting

S_{\mathrm{conc}}

range, generation quality, and PUR’s PEFT subset overlap under different embedding models (Task: Personalized Scholarly Title Generation).

	User 1 (Low $S_{\mathrm{conc}}$ )		User 2 (Medium $S_{\mathrm{conc}}$ )		User 3 (High $S_{\mathrm{conc}}$ )
Method	Offline Comp. (PFLOPs) $\downarrow$	Inference Comp. (TFLOPs) $\downarrow$	Offline Comp. (PFLOPs) $\downarrow$	Inference Comp. (TFLOPs) $\downarrow$	Offline Comp. (PFLOPs) $\downarrow$	Inference Comp. (TFLOPs) $\downarrow$
PEFT	1.096	30.7	1.107	34.7	1.209	36.1
RAG	0.002	170.9	0.002	208.8	0.002	265.4
Full PEFT+RAG	1.098	170.9	1.109	208.8	1.211	265.4
PUR (Ours)	0.329	122.5	0.665	110.4	0.967	73.7

Appendix

Table 6: Computational cost comparison of different methods in the offline and online inference stages. We report the computational cost for the same three users analyzed in Table 3 on the Personalized Movie Tagging task.

Panel of Figure 6 — Figure 6: Distribution of $S_{\mathrm{avg}}$ and the long-tail compensation term $\pi_{\mathrm{tail},i}h_{\mathrm{tail},i}$ for the users involved in Fig. 5 . Most users have relatively small compensation values, while a smaller subset exhibits substantially larger values and corresponds closely to the points that deviate from the overall trend when $\lambda=0$ .

Average historical data size of sampled users within each $S_{\mathrm{conc}}$ interval
$S_{\mathrm{conc}}$ Range	LaMP-1	LaMP-2	LaMP-3	LaMP-4	LaMP-5	LaMP-7
0.66–0.68	238	181	884	93	341	36
0.68–0.70	335	126	1006	82	450	48
0.70–0.72	403	124	946	119	580	59
0.72–0.74	432	113	923	87	551	85
0.74–0.76	388	137	891	127	528	77

Appendix

Table 7: Average historical data size of the sampled evaluation users across different

S_{\mathrm{conc}}

intervals and LaMP tasks.

Personalized Language Models

Personalized language models adapt responses to individual users’ preferences, histories, writing styles, and changing needs.

See all 89 papers

POPI: Personalizing LLMs via Optimized Natural Language Preference Inference
Oct 17, 2025Yizhuo Chen, Xin Liu, Ruijie Wang +7LLM Personalization Personalized Language Models
University of Illinois Urbana-Champaign · Amazon · University of Notre Dame
Beyond Retrieval: Learning Compact User Representations for Scalable LLM Personalization
Jun 3, 2026Heng Cao, Fan Zhang, Jian Yao +8LLM Personalization Prompt Tuning
*Microsoft · †The Hong Kong Polytechnic University · ⋄Shanghai International Studies University +1
Response-Aware User Memory Selection for LLM Personalization
Apr 15, 2026Jillian Fisher, Jennifer Neville, Chan Young ParkLLM Personalization Memory-Augmented Language Models
Department of Computer Science and Engineering, University of Washington, Seattle, WA, United States of America · Microsoft Research, Redmond, WA, United States of America

Oct 17, 2025J/K move · Enter open · S save

POPI: Personalizing LLMs via Optimized Natural Language Preference Inference

Yizhuo Chen, Xin Liu, Ruijie Wang, Zheng Li+6

University of Illinois Urbana-Champaign · Amazon · University of Notre Dame

LLM Personalization Personalized Language Models Personalized Language Model Alignment Language Model Post-Training User Preference Modeling

Open paper AskPDF arXiv

Large language models (LLMs) are typically aligned with population-level preferences, despite substantial variation across individual users. We introduce POPI, a user-level personalization framework that separates the problem into two components connected by a natural-language interface: a shared inference model that distills heterogeneous user signals into a concise preference summary, and a shared generator that conditions on this summary to produce personalized responses. Both components are trained under a unified preference-optimization objective, with reinforcement learning handling the non-differentiable inference step. This objective decomposes into generator approximation error and summary informativeness, revealing how a single loss simultaneously drives accurate generation and informative summarization. Because the interface is natural language, learned summaries can be inferred once per user and reused across different generators -- including frozen, black-box commercial APIs. Across four personalization benchmarks, POPI generally improves personalization quality while reducing context overhead by up to an order of magnitude.

Abstract

Personalized Language Models

Explore similar work

POPI: Personalizing LLMs via Optimized Natural Language Preference Inference

Beyond Retrieval: Learning Compact User Representations for Scalable LLM Personalization

Response-Aware User Memory Selection for LLM Personalization