Iterative Critique-Refine Framework for Enhancing LLM Personalization

Durga Prasad Maram; Dhruvin Gandhi; Zonghai Yao; Gayathri Akkinapalli; Franck Dernoncourt; Yu Wang; Ryan A. Rossi; Nesreen K. Ahmed

cs.CLOct 28, 2025

Iterative Critique-Refine Framework for Enhancing LLM Personalization

Authors: Durga Prasad Maram, Dhruvin Gandhi, Zonghai Yao, Gayathri Akkinapalli, Franck Dernoncourt, Yu Wang, Ryan A. Rossi, Nesreen K. Ahmed

Organizations: University of Massachusetts Amherst · Adobe Research · University of Oregon · Cisco AI Research

Abstract

Personalized text generation requires models not only to produce coherent text but also to align with a target user's style, tone, and topical focus. Existing retrieval-augmented approaches such as LaMP and PGraphRAG enrich profiles with user and neighbor histories, but they stop at generation and often yield outputs that drift in tone, topic, or style. We present PerFine, a unified, training-free critique-refine framework that enhances personalization through iterative, profile-grounded feedback. In each iteration, an LLM generator produces a draft conditioned on the retrieved profile, and a critic LLM - also conditioned on the same profile - provides structured feedback on tone, vocabulary, sentence structure, and topicality. The generator then revises, while a novel knockout strategy retains the stronger draft across iterations. We further study additional inference-time strategies such as Best-of-N and Topic Extraction to balance quality and efficiency. Across Yelp, Goodreads, and Amazon datasets, PerFine consistently improves personalization over PGraphRAG, with G-Eval gains of +8.5-12%, steady improvements over 3-5 refinement iterations, and scalability with increasing critic size. These results highlight that post-hoc, profile-aware feedback offers a powerful paradigm for personalized LLM generation that is both training-free and model-agnostic.

Figures & tables

Figure 1: Overview of our framework for personalized text generation. User profile information is retrieved to guide the generator, whose outputs are iteratively critiqued and refined by PerFine, enabling multi-round personalization.

Dataset	Method	METEOR	G-Eval
Yelp	LaMP	0.156	0.435
	PGraphRAG	0.180	0.496
	PerFine + Knockout	0.194 ±0.0003	0.543 ±0.0008
Goodreads	LaMP	0.193	0.528
	PGraphRAG	0.206	0.534
	PerFine + Knockout	0.214 ±0.0015	0.581 ±0.0031

Table 1: Baseline Comparison . The table compares the performance of PerFine+Knockout (mean

{}_{\pm\text{std}}

) against two baselines: LaMP , where the user profile contains samples only from the user’s own history, and PGraphRAG , which also includes samples from the profile histories of neighboring users. The output of PGraphRAG is taken as the initial generation

y_{0}

, over which PerFine+Knockout performs the feedback refinement process. Qwen-2.5-14B-Instruct is used as the critic LLM, while Llama-3.1-8B-Instruct serves as the generator LLM for PerFine and the baseline methods.

Dataset	Critic LLM	METEOR	G-Eval
Yelp	Llama-3.1-8B	0.200	0.521
	Qwen-2.5-7B	0.191	0.531
	Qwen-2.5-14B	0.195	0.544
	Qwen-2.5-32B	0.208	0.577
	gpt-5-mini	0.211	0.590
Goodreads	Llama-3.1-8B	0.211	0.558

Table 2: Evaluation of different critic models in the PerFine+Knockout setup (all are instruct models). Llama-3.1-8B-Instruct is the generator.

Figure 2: Performance across iterations on Yelp, Goodreads, and Amazon. PerFine+Knockout starts from the PGraphRAG baseline and exhibits steady improvements, with gains plateauing after a few iterations.

Figure 3: Visualization of the critic’s token usage (prompt + completion) vs normalized G-Eval performance. Notably, PerFine+Knockout improves performance, while PerFine+Knockout+Best-of-N achieves the highest scores, with increased token cost. Considering both efficiency and effectiveness, we ultimately select PerFine+Knockout.

Figure 4: Figure showing generator’s token usage (prompt + completion) vs normalized G-Eval performance. While the token usage for PerFine and PerFine+Knockout is similar, the token footprint increases for PerFine+Knockout+Best-of-N due to the sampling of multiple revisions, while yielding only a marginal improvement in performance.

Dataset	Method	METEOR	G-Eval	# token	# token
				(critic)	(gen)
Yelp	PerFine	0.185	0.505	10.77	9.54
	+Knockout	0.195	0.544	19.23	9.49
	+ Best-of-N	0.197	0.549	28.36	10.63
Goodreads	PerFine	0.209	0.565	11.65	10.57
	+Knockout	0.216	0.585	22	10.53

Table 3: Comparison of performance and token usage (prompt+completion) across various inference strategies. We observe that the scores improve as we scale the inference time token usage. However, the additional gains from Best-of-N sampling are marginal, while the associated token usage increases significantly. Token count is per query for 5 iterations in K. Llama-3.1-8B is the generator, and Qwen-2.5-14B is the critic.

Dataset	G-Eval-Style		G-Eval-Content
	PGraphRAG	PerFine	PGraphRAG	PerFine
	PGraphRAG	+ Knockout		+ Knockout
Yelp	0.405	0.473	0.427	0.502
Goodreads	0.444	0.498	0.461	0.528
Amazon	0.425	0.491	0.448	0.527

Table 4: Performance of PerFine+Knockout across style and content dimensions. Llama-3.1-8B-Instruct is the generator and gpt-5-mini is the critic.

Method	METEOR	G-Eval
PGraphRAG	0.184	0.524
PerFine+Knockout
Original prompts	0.200	0.589
Critic-Variant-1	0.201	0.583
Critic-Variant-2	0.197	0.584
Knockout-Variant-1	0.201	0.592

Table 5: Prompt sensitivity study of PerFine+Knockout on a 500-sample subset of the Amazon dataset. Each variant perturbs a single prompt while keeping all other prompts unchanged.

Dataset	Generator LLM	METEOR		G-Eval
Dataset	Generator LLM	PGraphRAG	PerFine+Knockout	PGraphRAG	PerFine+Knockout
Yelp	Llama-3.1-8B-Instruct	0.180	0.195	0.496	0.544
	DeepSeek-R1-Distill-Qwen-7B	0.167	0.174	0.435	0.461
	Ministral-3-3B-Instruct	0.191	0.194	0.550	0.567
	Ministral-3-8B-Instruct	0.193	0.203	0.566	0.580
	Ministral-3-14B-Instruct	0.200	0.204	0.581	0.600

Table 6: Comparison of different generator LLMs in the PerFine+Knockout setup. Qwen-2.5-14B-Instruct is the critic. PerFine+Knockout consistently improves performance across all generator LLMs.

Dataset	PGraphRAG	PerFine+Knockout
Yelp	0.59	0.64
Goodreads	0.67	0.69
Amazon	0.63	0.66

Table 7: Performance of PerFine+Knockout in the user profile-based evaluation setting, where top-

k

samples (

k

=4) from the author’s writing are used as the reference text. Llama-3.1-8B-Instruct is used as the generator and gpt-5-mini as the critic.

LLM-Judge	Amazon	Yelp	Goodreads
Gemma-3-27B-It	4.76%	8.47%	2.99%
Qwen3-32B	6.45%	10.00%	4.54%
Llama-3.3-70B-It	8.19%	14.00%	7.69%

Table 8: G-Eval percentage gains of PerFine+Knockout over PGraphRAG baseline under different LLM judges in the profile-referenced evaluation setting.

G-Eval (Ground-truth reference)
	LaMP	PerFine+Knockout
METEOR	0.257	0.265
Gemma-3-27B-It	0.450	0.470
Qwen3-32B	0.333	0.350
Llama-3.3-70B-It	0.362	0.420
G-Eval (Profile reference)

Table 9: Performance of PerFine+Knockout and the LaMP baseline on the CMCC email-writing dataset. G-Eval scores are reported under different LLM judges, using either the ground-truth email or the author’s past emails as the reference.

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Dataset	Method	METEOR	G-Eval	# token	# token	# token
				(critic)	(gen)	(topics)
Yelp	PerFine	0.185	0.505	10.77	9.54	-
Yelp	PerFine + Topic_Extraction	0.188	0.513	7.73	9.48	2.60
Goodreads	PerFine	0.209	0.565	11.65	10.57	-
Goodreads	PerFine + Topic_Extraction	0.209	0.557	7.73	10.48	2.81
Amazon	PerFine	0.197	0.558	9.58	8.27	-

Appendix

Table 10: Performance and Token usage for Topic Extraction method. The topic extraction method outperforms the baselines and achieves performance comparable to the vanilla setting, while requiring fewer critic tokens during feedback rounds. However, it introduces an initial overhead for extracting topics, which increases the overall token consumption. Qwen-2.5-32B-Instruct is used for extracting the topics.

Dataset	PerFine	PerFine+Knockout	PerFine+Knockout+Best-of-N
Yelp	2.29	3.69	5.68
Goodreads	2.62	4.17	6.65
Amazon	3.48	4.73	6.53

Appendix

Table 11: Per-query average wall-clock time (in seconds) for different refinement strategies.

LLM Personalization

LLM personalization adapts language models to users’ preferences, histories, expertise, and changing goals while balancing usefulness, privacy, and control.

See all 130 papers

MIRROR: From Imitation to Internalization in LLM Personalization
Oct 7, 2026Huayi Lai, Jicheng Yang, Min Yi +1Personalized Language Models Personalized Language Model Alignment
Baidu Inc.
PrefReward: Learning User Preference Matrix for Personalized Text Generation
Jul 23, 2026Yue Wu, Chengbing Wang, Yimeng Bai +3Personalized Language Models Personalized Text Generation
University of Science and Technology of China · The Chinese University of Hong Kong · National University of Singapore
POPI: Personalizing LLMs via Optimized Natural Language Preference Inference
Oct 17, 2025Yizhuo Chen, Xin Liu, Ruijie Wang +7LLM Personalization Personalized Language Models
University of Illinois Urbana-Champaign · Amazon · University of Notre Dame

Oct 7, 2026J/K move · Enter open · S save

MIRROR: From Imitation to Internalization in LLM Personalization

Huayi Lai, Jicheng Yang, Min Yi, Chong Meng

Baidu Inc.

Personalized Language Models Personalized Language Model Alignment Language Model Distillation On-Policy Self-Distillation

Open paper AskPDF arXiv

The demand for personalized LLMs is shifting from style imitation toward content quality. We investigate whether self-distillation can bridge this gap in existing fine-tuning paradigm. To address this limitation, we introduce MIRROR(Meta- personalization by Internalizing Reference-Revealed On-policy Reflections), a novel self-distillation framework that shifts LLM personalization from imitation toward preference internalization. First, we replace reference-token imitation with reference-revealed on-policy self-distillation, aligning the model's next-token distributions along its own generation trajectories with those of its reference-conditioned self, thereby internalizing user preferences rather than reproducing reference wording.Second, we introduce MIRROR-F, a focal plug-in that augments on-policy distributional alignment with selective supervision over informative reference tokens, thereby strengthening content generation while preserving user-specific expression. Across three personalized generation benchmarks, two model scales, and complementary reference-based and LLM-based evaluations, MIRROR and MIRROR-F achieve leading overall personalization performance and superior text quality, while exhibiting less catastrophic forgetting than SFT-based baselines on three unseen personalized generation tasks. The gains are consistent across model scales and application scenarios, translating to improved performance in LLM personalization tasks.

Abstract

LLM Personalization

Explore similar work

MIRROR: From Imitation to Internalization in LLM Personalization

PrefReward: Learning User Preference Matrix for Personalized Text Generation

POPI: Personalizing LLMs via Optimized Natural Language Preference Inference