PeReGrINE: Evaluating Personalized Review Fidelity with User Item Graph Context

Steven Au; Baihan Lin

cs.IRApr 9, 2026

PeReGrINE: Evaluating Personalized Review Fidelity with User Item Graph Context

Authors: Steven Au, Baihan Lin

Organizations: Icahn School of Medicine at Mount Sinai

Abstract

We introduce PeReGrINE, a benchmark and evaluation framework for personalized review generation grounded in graph-structured user--item evidence. PeReGrINE restructures Amazon Reviews 2023 into a temporally consistent bipartite graph, where each target review is conditioned on bounded evidence from user history, item context, and neighborhood interactions under explicit temporal cutoffs. To represent persistent user preferences without conditioning directly on sparse raw histories, we compute a User Style Parameter that summarizes each user's linguistic and affective tendencies over prior reviews. This setup supports controlled comparison of four graph-derived retrieval settings: product-only, user-only, neighbor-only, and combined evidence. Beyond standard generation metrics, we introduce Dissonance Analysis, a macro-level evaluation framework that measures deviation from expected user style and product-level consensus. We also study visual evidence as an auxiliary context source and find that it can improve textual quality in some settings, while graph-derived evidence remains the main driver of personalization and consistency. Across product categories, PeReGrINE offers a reproducible way to study how evidence composition affects review fidelity, personalization, and grounding in retrieval-conditioned language models.

Figures & tables

Figure 1: Overview of PeReGrINE. The system computes a user style summary, retrieves item-side and user-side evidence from the graph, and then conditions a language model on the resulting context.

Method	Text R-L	Text B-F1	Text M	Title R-L	Title B-F1	Title M	MAE	RMSE
LaMP	0.1452	0.7573	0.1180	0.1593	0.7477	0.1468	0.30	0.6164
PGraphRAG	0.1475	0.7551	0.1259	0.1459	0.7365	0.1303	0.29	0.5916
PeReGrINE	0.1595	0.7636	0.1443	0.1474	0.7400	0.1343	0.33	0.6708
PeReGrINE-joint	0.1277	0.7543	0.1665	0.0465	0.7092	0.0304	0.94	1.5427

Table 1: Matched baseline comparison on the metadata-required 100-example All Beauty development subset. Higher is better for ROUGE-L, BERTScore-F1, and METEOR. Lower is better for MAE and RMSE.

Task	Metric	Product	User	Neighbor	Both
Text	ROUGE-L ( $\uparrow$ )	0.110	0.114	0.115	0.123
	BLEU ( $\uparrow$ )	0.008	0.011	0.005	0.016
	METEOR ( $\uparrow$ )	0.181	0.160	0.163	0.173
	BERT-F1 ( $\uparrow$ )	0.742	0.743	0.746	0.752
Title	ROUGE-L ( $\uparrow$ )	0.058	0.032	0.044	0.060
Title	BERT-F1 ( $\uparrow$ )	0.707	0.699	0.697	0.707

Table 2: Micro metrics for the text-only evidence ablation on All Beauty using Qwen2.5-3B-Instruct. Best values are in bold.

Metric $\downarrow$	Product	User	Neighbor	Both
User Dissonance	0.274	0.199	0.257	0.251
Product Dissonance	0.358	0.503	0.400	0.399
Sentiment Dissonance	0.164	0.164	0.169	0.162

Table 3: Dissonance metrics for the text-only evidence ablation on All Beauty using Qwen2.5-3B-Instruct. Lower is better.

Task	Metric	Product	User	Neighbor	Both
Text	ROUGE-L ( $\uparrow$ )	0.127	0.120	0.124	0.131
	BLEU ( $\uparrow$ )	0.012	0.009	0.008	0.014
	METEOR ( $\uparrow$ )	0.183	0.135	0.164	0.192
	BERT-F1 ( $\uparrow$ )	0.752	0.744	0.753	0.759
Title	ROUGE-L ( $\uparrow$ )	0.072	0.049	0.040	0.059
Title	BERT-F1 ( $\uparrow$ )	0.717	0.701	0.712	0.719

Table 4: Micro metrics for the multimodal evidence ablation on All Beauty using Qwen3-VL-8B-Instruct.

Metric $\downarrow$	Product	User	Neighbor	Both
User Dissonance	0.253	0.197	0.253	0.252
Product Dissonance	0.374	0.501	0.379	0.374
Sentiment Dissonance	0.150	0.158	0.161	0.159

Table 5: Dissonance metrics for the multimodal evidence ablation on All Beauty. Lower is better.

Appendix figures & tables13 assets

Supplementary material from the paper’s appendix.

Appendix

Category	Split	# Inst.	Mean $\|H_{u}\|$	Mean $\|H_{i}\|$	Img. %
All Beauty	Dev	218	6.07	6.46	17.0%
All Beauty	Test	399	9.72	5.81	21.6%
Baby	Dev	500	5.53	15.53	0.6%
Baby	Test	500	6.53	185.92	13.0%
Arts & Crafts	Dev	500	5.53	12.20	0.8%
Arts & Crafts	Test	500	7.39	69.13	12.6%

Appendix

Table 6: Dataset statistics. All instances satisfy

|H_{u}|\geq 4

and

|H_{i}|\geq 3

	Raw Distribution		PeReGrINE Test Set
Category	Avg. Degree	Sparsity	Avg. Degree	Density Shift
All Beauty	6.23	High	5.81	-6.7%
Baby Products	27.56	Med	185.92	+574%
Arts, Crafts & Sewing	11.23	Med	69.13	+515%
Pet Supplies	34.10	Low	343.47	+907%
Toys & Games	18.30	Med	97.92	+435%

Appendix

Table 7: Comparison of the raw data distribution and the denser PeReGrINE benchmark subset. The benchmark intentionally shifts toward higher-degree items so that graph-conditioned generation is well defined.

Panel of Figure 2 — Figure 2: Prompt template for the combined-evidence setting. Other retrieval settings use the same template with the relevant evidence blocks removed.

Figure 5: Retrieval paradigms supported by the bipartite formulation. User-history retrieval corresponds to the red edges, item-neighbor retrieval corresponds to the green edges, and the combined setting uses both.

Category	ROUGE-L	BLEU	METEOR	BERTScore-F1
Baby Products	0.1131	0.0066	0.1863	0.7485
Beauty & Personal Care	0.1086	0.0087	0.1736	0.7414
Pet Supplies	0.1088	0.0045	0.1764	0.7430
Sports & Outdoors	0.1043	0.0072	0.1725	0.7426
Toys & Games	0.1009	0.0076	0.1810	0.7401
All Beauty	0.1434	0.0136	0.2119	0.7689

Appendix

Table 8: Category-wise review-text generation results. Higher values indicate stronger alignment with reference reviews.

Category	Title ROUGE-L	Title BERTScore-F1	Title-Text Consistency
Baby Products	0.0626	0.7099	0.4286
Beauty & Personal Care	0.0600	0.7066	0.4217
Pet Supplies	0.0783	0.7146	0.4810
Sports & Outdoors	0.0688	0.7175	0.4413
Toys & Games	0.0657	0.7140	0.4601
All Beauty	0.0640	0.7199	0.4892

Appendix

Table 9: Category-wise title generation results.

Category	Accuracy	MAE	RMSE
Baby Products	0.6160	0.8370	1.5190
Beauty & Personal Care	0.6100	0.9660	1.7181
Pet Supplies	0.5740	1.0360	1.7672
Sports & Outdoors	0.6460	0.6990	1.3544
Toys & Games	0.6520	0.8050	1.5305
All Beauty	0.4937	0.7657	1.2163

Appendix

Table 10: Category-wise rating prediction results. Lower is better for MAE and RMSE.

Category	User Dissonance	Product Dissonance	Sentiment Dissonance
Baby Products	0.3054	0.4733	0.1964
Beauty & Personal Care	0.3265	0.4866	0.2162
Pet Supplies	0.3114	0.4946	0.2271
Sports & Outdoors	0.3147	0.4949	0.1868
Toys & Games	0.3358	0.4896	0.1898
All Beauty	0.2385	0.2972	0.1534

Appendix

Table 11: Category-wise Dissonance metrics. Lower values indicate better alignment.

Task	Metric	LLaMA-3.1	GPT-5	Qwen3-VL	Claude-4.5
Text	ROUGE-L ( $\uparrow$ )	0.1284	0.1226	0.1258	0.1245
	BLEU ( $\uparrow$ )	0.0197	0.0113	0.0160	0.0150
	METEOR ( $\uparrow$ )	0.2072	0.1841	0.1867	0.1972
	BERT-F1 ( $\uparrow$ )	0.7494	0.7474	0.7549	0.7569
Title	ROUGE-L ( $\uparrow$ )	0.0467	0.0515	0.0591	0.0649
	BERT-F1 ( $\uparrow$ )	0.7044	0.7093	0.7148	0.7186

Appendix

Table 12: Micro metrics for the text-only model comparison on the combined All Beauty setting.

Metric ( $\downarrow$ )	LLaMA-3.1	GPT-5	Qwen3-VL	Claude-4.5
User Dissonance	0.2651	0.2446	0.2452	0.2843
Product Dissonance	0.3790	0.3843	0.3819	0.3808
Sentiment Dissonance	0.1458	0.1518	0.1663	0.1585

Appendix

Table 13: Dissonance metrics for the text-only model comparison. Lower is better.

Figure 6: Illustration of the input structure. The user profile provides stylistic evidence, while product information provides grounding context.

Model	Rating	Generated review and short analysis
Example A: correcting sentiment drift
Gold	4.0	Title: Rich shades of red. The gold review is positive overall, with mild complaints about wear and lip dryness.
Text-only	2.0	The text-only system hallucinates a packaging or shipping failure and drifts negative. This appears to come from over-reliance on user-history complaints.
Multimodal	4.0	The multimodal output aligns with the product context and keeps the review near the correct positive rating.
Example B: detecting negative quality cues
Gold	2.0	Title: I loved the idea of this organic nail polish remover. The gold review is clearly negative despite an initially positive setup.

Appendix

Table 14: Short qualitative examples comparing text-only and multimodal behavior.

Personalized Text Generation

Personalized text generation adapts language-model outputs to a user's writing style, preferences, history, and context.

See all 22 papers

Iterative Critique-Refine Framework for Enhancing LLM Personalization
Oct 28, 2025Durga Prasad Maram, Dhruvin Gandhi, Zonghai Yao +5LLM Personalization Personalized Language Models
University of Massachusetts Amherst · Adobe Research · University of Oregon +1
Personalization Matters: Long-Horizon Conversation Agent with User-Centric Information in Online Shopping Interactions
Oct 8, 2026Rena Gao, Yue Dai, Hao Guan +4Agentic Commerce Recommender Systems
ByteDance · The University of Melbourne · University of Technology Sydney +1
Preference-Aware Rubric Learning for Personalized Evaluation
May 29, 2026Yilun Qiu, Xiaoyan Zhao, Yang Zhang +7LLM Evaluation LLM Alignment
National University of Singapore · Xiaohongshu Inc. · The University of Tokyo

Oct 28, 2025J/K move · Enter open · S save

Iterative Critique-Refine Framework for Enhancing LLM Personalization

Durga Prasad Maram, Dhruvin Gandhi, Zonghai Yao, Gayathri Akkinapalli+4

University of Massachusetts Amherst · Adobe Research · University of Oregon+1

LLM Personalization Personalized Language Models Persona-Conditioned Generation LLM Self-Refinement

Open paper AskPDF arXiv

Personalized text generation requires models not only to produce coherent text but also to align with a target user's style, tone, and topical focus. Existing retrieval-augmented approaches such as LaMP and PGraphRAG enrich profiles with user and neighbor histories, but they stop at generation and often yield outputs that drift in tone, topic, or style. We present PerFine, a unified, training-free critique-refine framework that enhances personalization through iterative, profile-grounded feedback. In each iteration, an LLM generator produces a draft conditioned on the retrieved profile, and a critic LLM - also conditioned on the same profile - provides structured feedback on tone, vocabulary, sentence structure, and topicality. The generator then revises, while a novel knockout strategy retains the stronger draft across iterations. We further study additional inference-time strategies such as Best-of-N and Topic Extraction to balance quality and efficiency. Across Yelp, Goodreads, and Amazon datasets, PerFine consistently improves personalization over PGraphRAG, with G-Eval gains of +8.5-12%, steady improvements over 3-5 refinement iterations, and scalability with increasing critic size. These results highlight that post-hoc, profile-aware feedback offers a powerful paradigm for personalized LLM generation that is both training-free and model-agnostic.

Abstract

Personalized Text Generation

Explore similar work

Iterative Critique-Refine Framework for Enhancing LLM Personalization

Personalization Matters: Long-Horizon Conversation Agent with User-Centric Information in Online Shopping Interactions

Preference-Aware Rubric Learning for Personalized Evaluation