Personalized Language Model Evaluation

Follow this topic Ask about this topic

Latest papers

8 papers1 paper in the last four weeks, with none the four weeks before. 0.0% of all new papers.

Not Your Diary, Just Your Topic: Grading the Wrong-User Control
Nov 6, 2026Karina Romanova, Igor Kiselev, Vladimir Platonov +5Personalized Language Model Evaluation Personalized Language Models
Yandex · Accenture · RawModels +4
PersEvalBench: Can LLM Evaluators Judge Contextually Appropriate Use of Persona Information?
Oct 13, 2026Xinyi Liu, Hooshang Nayyeri, Emine Yilmaz +3Personalized Language Model Evaluation LLM Evaluation
Amazon · University of Illinois Urbana-Champaign · University College London
VIBE-Bench: Evaluating Personalized Large Language Models When Profiles Don't Mean Preferences
Sep 1, 2026Yiwen Jiang, Yang Deng, Stephanie Fong +9LLM Evaluation LLM Personalization
1Monash University · 2Singapore Management University · University of Liverpool +1
LUNAR: Benchmarking Personalized Large Language Models on UNiversal User BehAvioR Logs
Aug 5, 2026Jiahao Zhang, Yongzhi Tong, Zelin Fu +4LLM Evaluation LLM Personalization
1Shenzhen University of Advanced Technology · ♣Work done at Ant Group · 2Ant Group +1
Beyond Borrowed Histories: Person-Aligned User Simulation for Interactive Role-Playing Evaluation
Jul 30, 2026Yuhang Zhu, Mingxuan Du, Benfeng Xu +3LLM Evaluation Large Language Model-Based Role-Play Simulation
University of Science and Technology of China · MetaStone · MetaStone Technology, Beijing, China
Preference-Aware Rubric Learning for Personalized Evaluation
May 29, 2026Yilun Qiu, Xiaoyan Zhao, Yang Zhang +7LLM Evaluation LLM Alignment
National University of Singapore · Xiaohongshu Inc. · The University of Tokyo
Personalized Benchmarking: Evaluating LLMs by Individual Preferences
Apr 21, 2026Cristina Garbacea, Heran Wang, Chenhao TanLLM Evaluation User Preference Modeling
University of Chicago
From Feelings to Metrics: Understanding and Formalizing How Users Vibe-Test LLMs
Apr 15, 2026Itay Itzhak, Eliya Habba, Gabriel Stanovsky +1Personalized Language Model Evaluation LLM Evaluation
Technion – Israel Institute of Technology · The Hebrew University of Jerusalem

Nov 6, 2026J/K move · Enter open · S save

Not Your Diary, Just Your Topic: Grading the Wrong-User Control

Karina Romanova, Igor Kiselev, Vladimir Platonov, Viliana Devbunova+4

Yandex · Accenture · RawModels+4

Personalized Language Model Evaluation Personalized Language Models User Preference Modeling

Open paper AskPDF arXiv

A common test of personalization replaces a user's history with someone else's and measures the drop in accuracy. But this swap also changes the topics in the history, so the drop may reflect topic mismatch. We examine this problem with a frozen language model on LaMP-2 news categorization, using histories from similar users and synthetic histories assembled from other users' articles. The user's own history improves accuracy by 2.3 percentage points over a model trained without history. Removing history only at evaluation makes this gain appear larger, at 4.8 points. Category labels alone reproduce 86% of the accuracy gap between the user's own history and a random user's history. With synthetic histories matched on labels and text similarity, the gap falls to 0.9 points. Across two user splits and three training seeds, the advantage over a similar user's history is 3.2 points for users seen during training and 0.65 points for unseen users, where the evidence for an advantage remains inconclusive. A small movie-tagging experiment with one training seed also finds no advantage over other users' histories. These findings show why personalization should be evaluated on unseen users, with histories of similar content and a model trained without history as comparisons. We also find that adding gated cross-attention improves on a prefix-only model under our training setup; whether this holds more broadly remains open.