cs.CLOct 30, 2026

PersonaChatBench: Evaluating Chat Assistants with Diverse Simulated Users

Authors: Zheyuan Deng, Taoran Li, Jingwen Zhang, Yuante Li, Dachuan Zhao, Fulei Zhang, Yunkai Zhang, Zhikun Xu, 

Organizations: Brown University · Texas A&M University · Harvard University · Carnegie Mellon University · 

Abstract

Conversational assistants are expected to help people with diverse backgrounds, preferences, and communication styles. Yet evaluations centered on a single cooperative user provide limited insight into how reliably an assistant serves users with different personas. We introduce PersonaChatBench, a framework for evaluating task competence and responsiveness to diverse user needs. PersonaChatBench pairs the same underlying task with multiple simulated users, each guided by a private persona card that specifies distinct needs, preferences, and interaction behaviors. By holding the task fixed while varying the user, the framework examines how assistant performance changes across user conditions. We evaluate 20 language models across six domains, separately assessing native task outcomes, fulfillment of expressed persona needs, and simulator fidelity. Our results show that task performance and need fulfillment vary across personas and domains. Strong task performance does not necessarily translate into satisfying individual needs, and models' relative strengths in meeting those needs are not consistent across domains. PersonaChatBench makes these differences measurable, revealing gaps obscured by aggregate scores and helping identify which users an assistant serves well and where adaptation is needed. It provides a basis for evaluating consistency across users and guiding the development of assistants that better accommodate diverse needs in everyday applications.

Figures & tables

Appendix figures & tables34 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. PersonaCUABench: Benchmarking Computer-Use Agents Serving Diverse Personas

    Oct 16, 2026Haoming Li, Defu Cao, Xu Hu +23Computer-Use Agent EvaluationComputer-Use Agent Benchmarks

  2. Eval4Sim: An Evaluation Framework for Persona Simulation

    Mar 3, 2026Eliseo Bao, Anxo Perez, Javier Parapar +1Large Language Model-Based Role-Play SimulationLarge Language Model-Based Social Simulation