PersonaChatBench: Evaluating Chat Assistants with Diverse Simulated Users
Organizations: Brown University · Texas A&M University · Harvard University · Carnegie Mellon University ·
Abstract
Conversational assistants are expected to help people with diverse backgrounds, preferences, and communication styles. Yet evaluations centered on a single cooperative user provide limited insight into how reliably an assistant serves users with different personas. We introduce PersonaChatBench, a framework for evaluating task competence and responsiveness to diverse user needs. PersonaChatBench pairs the same underlying task with multiple simulated users, each guided by a private persona card that specifies distinct needs, preferences, and interaction behaviors. By holding the task fixed while varying the user, the framework examines how assistant performance changes across user conditions. We evaluate 20 language models across six domains, separately assessing native task outcomes, fulfillment of expressed persona needs, and simulator fidelity. Our results show that task performance and need fulfillment vary across personas and domains. Strong task performance does not necessarily translate into satisfying individual needs, and models' relative strengths in meeting those needs are not consistent across domains. PersonaChatBench makes these differences measurable, revealing gaps obscured by aggregate scores and helping identify which users an assistant serves well and where adaptation is needed. It provides a basis for evaluating consistency across users and guiding the development of assistants that better accommodate diverse needs in everyday applications.
Figures & tables
Appendix figures & tables34 assets
Supplementary material from the paper’s appendix.
Appendix
| Role | Information used | Output |
|---|---|---|
| Tested assistant | Public task context, dialogue, permitted tools and policies. | Replies, tool calls, or a final task artifact. |
| User simulator | Public dialogue, authorized user facts, and assigned persona content. | A user message or a decision to stop. |
| Native task evaluator | The artifact or conversation required by the source rubric, together with its tests or references. | Native task result. |
| Persona-needs judge | Public dialogue and pre-recorded needs with their criteria. | Per-need judgments and supporting evidence. |
| Simulator auditor | Dialogue, authorized task facts, and assigned persona. | Findings about simulator fidelity. |
| Status | Credit | Interpretation |
|---|---|---|
| Scored: an expressed need with a response opportunity | ||
| satisfied | 1 | The evidence supports full satisfaction of the relevant criteria. |
| partially_satisfied | 0.5 | The relevant criteria are only partly met. |
| not_satisfied | 0 | The relevant criteria are not met. |
| Unscored: applicability or evidence does not support a numeric judgment | ||
| inapplicable | – | Declared inapplicable under the domain’s pre-answer rules. |
| Task outcomes | Expressed needs | Attribute evidence | |||||
|---|---|---|---|---|---|---|---|
| Assistant | Task | Facts | Needs | Demo. | |||
| GPT-6 Astra | 1,011 | 59.00 | 64.37 | 712 | 99.73 | 828 | 71.13 |
| GPT-5.6 Sol | 1,198 | 56.69 | 61.73 | 834 | 98.90 | 1,000 | 67.46 |
| GPT-6 Sol | 1,195 | 41.00 | 36.99 | 291 | 98.54 | 1,000 | 44.33 |
| Grok 4.6 | 1,199 | 59.02 | 63.74 | 899 | 97.04 | 1,000 | 72.23 |
| GPT-6 Luna | 1,198 | 39.69 | 45.46 | 353 | 96.53 | 1,000 | 44.01 |
| User condition | Task | Facts | Needs | Demo. | Turns |
|---|---|---|---|---|---|
| Neutral | 55.85 | 63.67 | – | – | 4.14 |
| P1: Receptive novice | 51.87 | 52.81 | 94.14 | 53.43 | 4.53 |
| P2: Informed questioner | 57.89 | 67.86 | 95.18 | 71.52 | 4.81 |
| P3: Private, time-conscious novice | 51.03 | 51.66 | 81.13 | 34.06 | 4.12 |
| P4: Guarded, time-conscious novice | 51.24 | 52.29 | 80.51 | 17.94 | 4.49 |
| P5: Receptive, informed contributor | 58.05 | 67.55 | 94.97 | 81.06 | 4.56 |
| Model | Passes/scored | Pass rate | Needs | Needs |
|---|---|---|---|---|
| GPT-6 Astra | 681/700 | 97.29 | 96.49 | 24 |
| Fable 5.1 | 662/693 | 95.53 | 90.86 | 592 |
| Claude Opus 5.5 | 663/700 | 94.71 | 95.42 | 596 |
| Claude Sonnet 5.5 | 661/699 | 94.56 | 88.22 | 591 |
| GPT-6 Sol | 653/700 | 93.29 | 94.32 | 129 |
| GPT-5.6 Sol | 646/700 | 92.29 | 95.03 | 143 |
| User condition | Example preferences | Task | Needs |
|---|---|---|---|
| Detailed typed professional | Detailed reasoning, docstrings, modular code, library tools, explicit types | 81.39 | 96.49 |
| Performance-focused student | Custom code, performance comparisons, detailed explanations, comments, types | 85.87 | 93.50 |
| Deadline-driven technologist | Usable delivery, visible implementation steps, readable code, plain language | 88.43 | 92.73 |
| Examples-first minimalist | Brief explanations and examples, simple structure, no code comments | 89.53 | 92.24 |
| Plain-language technologist | Accessible trade-offs, meaningful comments, low complexity, supportive wording | 88.35 | 87.92 |
| Examples-first learner | Concrete walkthroughs, accessible explanations, compact names, supportive wording | 89.46 | 84.08 |
| Requested support | Not fully met (%) | Unmet/scored |
|---|---|---|
| P5: Plain-language technologist | ||
| Acknowledge the user’s difficulty | 30.6 | 182/594 |
| Explain trade-offs | 30.2 | 190/630 |
| Keep test explanations brief | 27.3 | 35/128 |
| Use plain language | 26.4 | 162/614 |
| Include meaningful code comments | 17.5 | 110/629 |
| Attribute | P1 | P2 | P3 | P4 | P5 | P6 |
|---|---|---|---|---|---|---|
| Explanation detail | Compact | Detailed | — | — | Compact | Detailed |
| Presentation | Principles first | Principles first | Examples first | Examples first | — | — |
| Tone | Direct | — | Supportive | Direct | Supportive | Direct |
| Terminology | — | Explain terms | Explain terms | — | Explain terms | — |
| Reasoning style | Logical | — | — | Intuitive | — | Intuitive |
| Help preference | Hint first | Demonstrate first | Hint first | Demonstrate first | Demonstrate first | Hint first |
| Source | Tasks | Task performance | Needs satisfaction | Tutor replies |
|---|---|---|---|---|
| SimulatorArena–MATH | 37 | 99.15 (5,168/5,180) | 89.57 (4,430/4,440) | 2.66 |
| HARP | 139 | 93.69 (19,005/19,460) | 82.94 (16,276/16,680) | 2.97 |
| IMO-ProofBench | 29 | 75.40 (3,542/4,060) | 73.70 (3,039/3,480) | 5.38 |
| Instructional need | Scored (%) | Satisfaction | Scored conversations |
|---|---|---|---|
| Explanation detail | 70.1 | 82.6 | 11,108 |
| Presentation | 93.1 | 83.6 | 14,731 |
| Tone | 23.6 | 77.3 | 4,662 |
| Terminology | 37.7 | 83.3 | 4,459 |
| Reasoning style | 67.8 | 84.9 | 8,073 |
| Help preference | 94.5 | 80.2 | 22,443 |
| D1: Cautious owner on a tight schedule | D3: Methodical bookkeeper who verifies claims | |
|---|---|---|
| Background | Runs a small energy business alone and has a short window before field work. | Keeps the books of an energy firm and can spend as long as needed. |
| Domain familiarity | Low: unfamiliar with most financial terms. | High: reads financial statements comfortably. |
| Initial trust | Low: wants reasons before accepting an answer. | High: gives weight to a well-presented answer. |
| Uncertainty tolerance | Low: wants a definite takeaway. | High: works with uncertainty if next steps are clear. |
| Risk sensitivity | Low: focuses on the immediate loss. | High: attends to what could go wrong. |
| Belief revision | Updates after a clear explanation. | Needs checkable evidence to drop the “guarantee” view. |
| Condition | Critical needs | Expr. | Needs | Turns | Rubric | Below | Above | |
|---|---|---|---|---|---|---|---|---|
| Neutral | none | – | – | 1.81 | 43.6 | – | – | – |
| Communication presets | ||||||||
| Careful learner | plain language, keeps the substance | 92.5 | 74.0 | 3.06 | 43.9 | 0.4 | 0 | 3 |
| Skeptical questioner | grounded reasoning, calibrated uncertainty | 96.1 | 81.5 | 3.96 | 44.8 | 1.6 | 0 | 4 |
| Concise pragmatist | focused response, essential caveats | 72.0 | 67.0 | 2.55 | 43.2 | 0.3 | 1 | 0 |
| Pressure presets | ||||||||
| Rubric percent | Pass rates | Needs | |||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Model | Gen. | All | Persona | Neutral | Advice | Persona | Joint | Value | Sim. valid | Cap | Turns | ||
| Fable 5.1 | 1,164 | 57.1 | 1,164 | 57.2 | 56.0 | 67.9 | 25.2 | 16.8 | 70.1 | 1,005 | 94.2 | 58 | 3.1 |
| Claude Sonnet 5.5 | 1,164 | 52.3 | 1,164 | 52.3 | 52.2 | 55.3 | 26.9 | 16.2 | 72.7 | 990 | 95.3 | 23 | 2.8 |
| GPT-5.6 Sol | 1,164 | 50.6 | 1,146 | 50.8 | 48.7 | 53.5 | 46.7 | 23.6 | 85.3 | 909 | 95.6 | 2 | 2.1 |
| GPT-6 Astra | 1,154 | 50.6 | 1,108 | 50.8 | 48.1 | 53.9 | 62.9 | 32.9 | 90.0 | 846 | 96.4 | 2 | 2.0 |
| GPT-5.5 | 1,164 | 48.8 | 1,122 | 49.2 | 44.8 | 53.1 | 33.9 | 17.2 | 78.7 | 982 | 93.9 | 29 | 2.6 |
| Condition | Critical need | Expr. | Needs | Compl. | Significant drops | ||
|---|---|---|---|---|---|---|---|
| Neutral | none | – | – | – | 72.5 | – | – |
| Frustrated on refusal | reason and recourse after a refusal | 1,219 | 75.9 | 87.9 | 72.1 | 0.5 | 1 |
| Self-doubt | facts checked against the record | 935 | 55.8 | 37.1 | 73.2 | 0.7 | 0 |
| Gives up early | none (behavior only) | – | – | – | 65.3 | 7.2 | 7 |
| Money matters | exact amount and timing | 1,004 | 61.7 | 62.0 | 71.8 | 0.7 | 2 |
| Wants the reason | rule behind the outcome | 1,719 | 99.8 | 75.3 | 65.1 | 7.5 | 7 |
| Task | Needs | Failed tasks | |||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|
| Model | Gen. | Complete | Decision | Value | Pass | Sim. valid | Decision | Process | Ending | Cap | |
| Qwen 3.8 27B | 1,452 | 71.3 | 95.8 | 74.0 | 706 | 57.1 | 94.5 | 61 | 294 | 62 | 0 |
| Muse Glimmer 30B | 1,452 | 35.5 | 52.9 | 47.5 | 683 | 32.8 | 92.6 | 684 | 220 | 32 | 23 |
| GPT-6 Luna | 1,452 | 75.2 | 95.2 | 69.2 | 665 | 52.3 | 94.8 | 70 | 222 | 68 | 0 |
| Gemini 3.7 Flash | 1,452 | 92.6 | 96.2 | 75.9 | 677 | 60.8 | 94.1 | 55 | 29 | 23 | 0 |
| Gemini 3.8 Flash | 1,452 | 94.1 | 97.3 | 78.4 | 695 | 66.3 | 94.7 | 39 | 20 | 26 | 0 |
| Model | By rule, route (68) | Escalation req. (34) | Misremembered (12) | Identity (10) | Conditional (5) | By rule, no route (3) |
|---|---|---|---|---|---|---|
| Qwen 3.8 27B | 70.5 | 63.4 | 75.0 | 87.3 | 85.5 | 87.9 |
| Muse Glimmer 30B | 36.4 | 31.3 | 20.5 | 57.3 | 50.9 | 27.3 |
| GPT-6 Luna | 73.0 | 71.1 | 77.3 | 94.5 | 76.4 | 97.0 |
| Gemini 3.7 Flash | 89.7 | 93.6 | 97.0 | 100.0 | 100.0 | 93.9 |
| Gemini 3.8 Flash | 91.6 | 96.3 | 97.0 | 100.0 | 96.4 | 93.9 |
| Kimi K3 | 72.7 | 74.1 | 67.4 | 88.2 | 83.6 | 93.9 |
| Model | Neutral | Frustrated | Self-doubt | Gives up | Money | Reason | Bundles | Risk | Options | Significant drops |
|---|---|---|---|---|---|---|---|---|---|---|
| Qwen 3.8 27B | 72.7 | 1 | ||||||||
| Muse Glimmer 30B | 35.1 | 0 | ||||||||
| GPT-6 Luna | 78.3 | 2 | ||||||||
| Gemini 3.7 Flash | 93.9 | 1 | ||||||||
| Gemini 3.8 Flash | 96.5 | 2 | ||||||||
| Kimi K3 | 74.2 | 0 |
| Card | Description shown to the simulated traveler on official tasks (verbatim) |
|---|---|
| Careful planner | You prefer to work through a complicated trip one aspect at a time. You need accessible explanations and a short recap to feel confident. Ask naturally when overwhelmed or unsure; you can stop once helped. |
| Skeptical comparer | You like comparing plausible alternatives and understanding the basis for a recommendation, including what a displayed price covers. Ask for evidence when an explanation seems too confident. Do not invent extra travel constraints. |
| Concise decider | You want efficient, direct exchanges and compact reasons rather than repeated long lists. You still want control before a final selection. Say what would make the answer easier to use; do not force every need into every turn. |
| Efficient comparer | You prefer an efficient exchange that leads with a recommendation, followed by a focused comparison with a plausible alternative. Ask which details are catalog facts and which remain uncertain when that matters. Do not invent extra travel constraints or force every preference into every turn. |
| Concise planner | You prefer compact, accessible replies and a short recap of your expressed requirements to feel ready for the next decision. Ask naturally for clarification when needed; you can stop once helped. Do not invent extra travel constraints or require repeated reassurance. |
| Condition | Critical needs (third need) | Expr. | Needs | Share | Turns | Reward | Below | Above | |
|---|---|---|---|---|---|---|---|---|---|
| Neutral | none | – | – | 78.8 | 3.53 | 73.4 | – | – | – |
| Careful planner stepwise, plain, support | paced questions, plain terms (recap before committing) | 54.2 | 94.5 | 67.3 | 4.72 | 62.8 | 10.4 | 13 | 0 |
| Skeptical comparer stepwise, comparative, evidence | comparison, price breakdown (facts vs. assumptions) | 98.7 | 97.9 | 53.2 | 4.14 | 49.5 | 24.1 | 19 | 0 |
| Concise decider efficient, plain, evidence | concise replies, answer first (reason before an irrevocable choice) | 35.0 | 64.8 | 60.7 | 4.23 | 56.8 | 16.7 | 16 | 0 |
| Efficient comparer efficient, comparative, evidence | comparison, answer first (facts vs. assumptions) | 66.5 | 96.9 | 65.0 | 3.79 | 60.1 | 13.3 | 13 | 0 |
| Concise planner efficient, plain, support | concise replies, plain terms (recap before committing) | 8.6 | 88.6 † | 77.3 | 3.83 | 72.0 | 1.4 | 3 | 2 |
| Choice reward | Recorded | Needs | |||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Model | Gen. | All | Persona | Neutral | Valid | Share | Credit | Rev. | Value | Pass | Sim. valid | Cap | Turns | ||
| Gemini 3.7 Flash | 432 | 86.6 | 402 | 85.6 | 91.8 | 87.0 | 92.5 | 93.6 | 30 | 85.2 | 251 | 71.4 | 96.3 | 0 | 4.4 |
| Fable 5.1 | 432 | 79.7 | 402 | 77.9 | 88.8 | 80.2 | 83.7 | 95.2 | 30 | 87.8 | 288 | 80.9 | 94.2 | 0 | 3.8 |
| Claude Sonnet 5.5 | 432 | 79.3 | 402 | 77.5 | 87.9 | 79.6 | 84.0 | 94.4 | 30 | 84.2 | 294 | 86.2 | 94.9 | 0 | 4.1 |
| Gemini 3.8 Flash | 432 | 76.3 | 402 | 73.9 | 88.4 | 76.8 | 79.9 | 95.5 | 30 | 88.1 | 254 | 82.6 | 94.9 | 0 | 4.2 |
| Grok 4.6 | 429 | 75.6 | 399 | 75.2 | 77.3 | 77.9 | 81.1 | 93.2 | 30 | 85.2 | 275 | 80.6 | 89.8 | 1 | 3.9 |
AI Agent Benchmarks
AI agent benchmarks measure how reliably autonomous systems plan, use tools, interact, and complete tasks across controlled and real-world settings.
See all 1,571 papers