Organizations: Brown University · Texas A&M University · Harvard University · Carnegie Mellon University · Columbia University · University of California, Berkeley · Arizona State University · Northeastern University · University of Southern California · University of Chicago · The University of Texas at Dallas · University of California, Los Angeles · California Institute of Technology · University of Illinois Chicago · University of Michigan · University of Illinois at Urbana-Champaign · Massachusetts Institute of Technology · Amherst College · Purdue University · The Ohio State University
Conversational assistants are expected to help people with diverse backgrounds, preferences, and communication styles. Yet evaluations centered on a single cooperative user provide limited insight into how reliably an assistant serves users with different personas. We introduce PersonaChatBench, a framework for evaluating task competence and responsiveness to diverse user needs. PersonaChatBench pairs the same underlying task with multiple simulated users, each guided by a private persona card that specifies distinct needs, preferences, and interaction behaviors. By holding the task fixed while varying the user, the framework examines how assistant performance changes across user conditions. We evaluate 20 language models across six domains, separately assessing native task outcomes, fulfillment of expressed persona needs, and simulator fidelity. Our results show that task performance and need fulfillment vary across personas and domains. Strong task performance does not necessarily translate into satisfying individual needs, and models' relative strengths in meeting those needs are not consistent across domains. PersonaChatBench makes these differences measurable, revealing gaps obscured by aggregate scores and helping identify which users an assistant serves well and where adaptation is needed. It provides a basis for evaluating consistency across users and guiding the development of assistants that better accommodate diverse needs in everyday applications.
Figures & tables
Figure 1: Overview of PersonaChatBench. The same task is evaluated with different simulated users through multi-turn conversations, with task performance, persona-needs satisfaction, and simulator fidelity assessed separately.
Figure 2: Task and persona composition across six domains. Inner and outer rings show domains and task sources, with sector sizes proportional to task counts. Cards summarize user conditions and planned conversations per assistant.
Table 1: Task performance across five domains. Avg. is the equally weighted domain mean. The highest score in each domain is bold and the second-highest is underlined.
Table 2: Persona-needs satisfaction across six domains. Avg. is the equally weighted domain mean. The highest score in each domain is bold and the second-highest is underlined.
Figure 3: Task performance across persona conditions in five domains. Branches group personas by domain; leaf values are model-equal mean task scores. Gray leaves mark the neutral condition. Repeated trials are averaged within task.
Figure 4: Selected within-model persona contrasts on matched tasks across five domains. Open and filled markers denote lower and higher task means; gaps are score-point differences.
Figure 5: Persona-needs satisfaction by domain and persona. Equal-angle sectors show model-equal mean scores on a 0–100 scale. Green and red denote scores above and below the domain’s persona mean, shown at the center.
Appendix figures & tables34 assets
Supplementary material from the paper’s appendix.
Appendix
Role
Information used
Output
Tested assistant
Public task context, dialogue, permitted tools and policies.
Replies, tool calls, or a final task artifact.
User simulator
Public dialogue, authorized user facts, and assigned persona content.
A user message or a decision to stop.
Native task evaluator
The artifact or conversation required by the source rubric, together with its tests or references.
Native task result.
Persona-needs judge
Public dialogue and pre-recorded needs with their criteria.
Per-need judgments and supporting evidence.
Simulator auditor
Dialogue, authorized task facts, and assigned persona.
Findings about simulator fidelity.
Appendix
Table 4: Information available to each role. Private persona cards and evaluation references are not shown to the tested assistant.
Status
Credit
Interpretation
Scored: an expressed need with a response opportunity
satisfied
1
The evidence supports full satisfaction of the relevant criteria.
partially_satisfied
0.5
The relevant criteria are only partly met.
not_satisfied
0
The relevant criteria are not met.
Unscored: applicability or evidence does not support a numeric judgment
inapplicable
–
Declared inapplicable under the domain’s pre-answer rules.
Appendix
Table 5: Need-level judgments. Only numeric credits enter the satisfaction score; unscored statuses retain their distinct meanings.
Figure 6: Five Health personas defined by attributes relevant to a consultation. H and L denote high and low levels. P2/P5 and P3/P4 differ only in initial trust at the attribute level. The neutral condition is an ordinary cooperative user with no assigned attributes or private needs.
Task outcomes
Expressed needs
Attribute evidence
Assistant
N
Task
Facts
N
Needs
N
Demo.
GPT-6 Astra
1,011
59.00
64.37
712
99.73
828
71.13
GPT-5.6 Sol
1,198
56.69
61.73
834
98.90
1,000
67.46
GPT-6 Sol
1,195
41.00
36.99
291
98.54
1,000
44.33
Grok 4.6
1,199
59.02
63.74
899
97.04
1,000
72.23
GPT-6 Luna
1,198
39.69
45.46
353
96.53
1,000
44.01
Appendix
Table 6: Health outcomes for 20 assistants, ordered by needs satisfaction. Task and Facts cover the six user conditions; Needs and attribute demonstration (Demo.) cover the five personas. Each N counts conversations with that measure. Scores are on a 0–100 scale; bold denotes the largest value in each score column. Demo. measures observable attribute evidence, not a simulator pass rate.
Figure 7: Health task coverage and expressed-needs satisfaction for each assistant and user condition. N denotes neutral; P1–P5 are defined in Figure 6 . Models are ordered by their overall Health needs satisfaction. Each cell averages the available scored conversations for that model and condition. Both panels use a fixed 0–100 scale.
User condition
Task
Facts
Needs
Demo.
Turns
Neutral
55.85
63.67
–
–
4.14
P1: Receptive novice
51.87
52.81
94.14
53.43
4.53
P2: Informed questioner
57.89
67.86
95.18
71.52
4.81
P3: Private, time-conscious novice
51.03
51.66
81.13
34.06
4.12
P4: Guarded, time-conscious novice
51.24
52.29
80.51
17.94
4.49
P5: Receptive, informed contributor
58.05
67.55
94.97
81.06
4.56
Appendix
Table 7: Health outcomes by user condition, giving each of the 20 models equal weight. Turns counts doctor replies, including the shared first reply. Neutral has no assigned needs or attributes. P2 and P5, shaded, are the medically familiar, proactive profiles.
Figure 8: Paired Health task-score difference between P5 and P3, with models ordered by needs satisfaction. Points show each model’s mean over shared cases; bars are pointwise 95% percentile bootstrap intervals. The diamond and shaded band show the equally weighted cross-model mean and its task-cluster interval. Pair counts are shown at right. Intervals use 5,000 resamples and are not adjusted for multiple comparisons. ∗ The Opus contrast also varies in supporting-model configuration.
Model
Passes/scored
Pass rate
Needs
Needs n
GPT-6 Astra
681/700
97.29
96.49
24
Fable 5.1
662/693
95.53
90.86
592
Claude Opus 5.5
663/700
94.71
95.42
596
Claude Sonnet 5.5
661/699
94.56
88.22
591
GPT-6 Sol
653/700
93.29
94.32
129
GPT-5.6 Sol
646/700
92.29
95.03
143
Appendix
Table 8: Coding scores and coverage across 20 assistants. Pass rate and needs satisfaction are on a 0–100 scale. Each model has 700 planned task episodes, including neutral, and 600 planned persona episodes; needs n counts episodes with a numeric needs score.
Figure 9: Coding performance by assistant and user condition. Cells show the mean available score (0–100) and scored-conversation count n . Color limits are 55–100 for native pass rate and 60–100 for needs satisfaction. N denotes neutral; P1–P6 follow Table 9 . Models are ordered by overall needs satisfaction. Cells marked n=0 have no needs score. Native outcomes retain archived judgments, and scored populations differ across cells.
Table 9: Coding personas and condition-level scores. Preferences are illustrative rather than mandatory requests. Both metrics use a 0–100 scale and equal model weights: 20 models for task performance and a fixed 19-model subset for needs satisfaction.
Figure 10: Unmet persona needs despite passing code. Each bar gives the percentage of persona conversations with an archived native pass and a numeric needs score in which at least one scored persona-specific need receives partial or zero credit, (a) for each assistant and (b) for each persona. Labels on the right give unmet/scored conversations; the pooled rate is 43.8% (2,014/4,594). Scored counts differ widely across assistants because of judgment coverage and acceptance policies. Missing scores are excluded; rates describe the selected task panel.
Requested support
Not fully met (%)
Unmet/scored
P5: Plain-language technologist
Acknowledge the user’s difficulty
30.6
182/594
Explain trade-offs
30.2
190/630
Keep test explanations brief
27.3
35/128
Use plain language
26.4
162/614
Include meaningful code comments
17.5
110/629
Appendix
Table 10: Unmet persona-specific needs for P5 and P6. Each row covers an expressed, scored need in conversations with passing code and a numeric needs score. Partial and zero credit both count as not fully met. Counts give conversations with that need unmet out of those with the need scored. Rows may overlap and their percentages are not additive. All persona-specific need types for P5 and P6 are shown.
Table 12: Math results by source. Task performance includes all seven conditions; needs satisfaction includes the six personas. Parentheses give scored/planned conversations pooled across assistants. Tutor replies are averaged over completed dialogues with recorded lengths, including confirmation and closure. Missing grades are not assigned zero.
Instructional need
Scored (%)
Satisfaction
Scored conversations
Explanation detail
70.1
82.6
11,108
Presentation
93.1
83.6
14,731
Tone
23.6
77.3
4,662
Terminology
37.7
83.3
4,459
Reasoning style
67.8
84.9
8,073
Help preference
94.5
80.2
22,443
Appendix
Table 13: Fulfillment of assigned instructional needs. Scored (%) is the fraction receiving numeric credit among conversations assigned that need with an interpretable needs judgment. Satisfaction averages numeric credits. Both give assistants equal weight; counts pool scored conversations. The overall needs score averages credits within conversations before aggregation.
Figure 11: Help-preference satisfaction across 20 tutor models. Each row connects a tutor’s mean scores for hint-first and demonstration-first profiles on tasks with numeric help grades for all six personas. The comparison includes 2,676 assistant–task sets, spanning all 205 tasks and 60–167 tasks per assistant. The bottom row gives equal-assistant means: 75.3 for hints and 86.6 for demonstrations. The mean gap is 11.3 points (95% source-stratified task-bootstrap interval [10.3, 12.4]); it remains 11.3 across the 2,139 sets with correct mathematics in all six conversations.
D1: Cautious owner on a tight schedule
D3: Methodical bookkeeper who verifies claims
Background
Runs a small energy business alone and has a short window before field work.
Keeps the books of an energy firm and can spend as long as needed.
Domain familiarity
Low: unfamiliar with most financial terms.
High: reads financial statements comfortably.
Initial trust
Low: wants reasons before accepting an answer.
High: gives weight to a well-presented answer.
Uncertainty tolerance
Low: wants a definite takeaway.
High: works with uncertainty if next steps are clear.
Risk sensitivity
Low: focuses on the immediate loss.
High: attends to what could go wrong.
Belief revision
Updates after a clear explanation.
Needs checkable evidence to drop the “guarantee” view.
Appendix
Table 14: Two task-specific personas for the same question. The question asks which derivative a bank would offer an oil producer worried about falling prices. Rows show a subset of the 20 attributes. Needs marked with ∗ are critical.
Condition
Critical needs
Expr.
Needs
Turns
Rubric
Δ
Below
Above
Neutral
none
–
–
1.81
43.6
–
–
–
Communication presets
Careful learner
plain language, keeps the substance
92.5
74.0
3.06
43.9
+ 0.4
0
3
Skeptical questioner
grounded reasoning, calibrated uncertainty
96.1
81.5
3.96
44.8
+ 1.6
0
4
Concise pragmatist
focused response, essential caveats
72.0
67.0
2.55
43.2
− 0.3
1
0
Pressure presets
Appendix
Table 15: Finance conditions in the main cohort. Critical needs lists the two critical needs of each preset. The task-specific personas D1–D5 are written separately for each question, so their needs vary. Numeric columns other than Below and Above are unweighted means of per-model values over the 18 main-cohort models. Expr. is the share of pre-recorded needs that the client expressed, in percent, among completed conversations whose simulator audit was not invalid. Needs is the persona-needs score in percent, Turns the mean number of assistant turns per generated conversation, and Rubric the rubric percent. Δ is the paired persona-minus-neutral change in rubric percent, in percentage points, over 65 to 97 questions per model. Below and Above count the models whose 95% task-cluster bootstrap interval lies entirely below or above zero. Pooled rows combine each model’s persona conditions in a group into one paired contrast.
Rubric percent
Pass rates
Needs
Model
Gen.
All
n
Persona
Neutral
Advice
Persona
Joint
Value
n
Sim. valid
Cap
Turns
Fable 5.1
1,164
57.1
1,164
57.2
56.0
67.9
25.2
16.8
70.1
1,005
94.2
58
3.1
Claude Sonnet 5.5
1,164
52.3
1,164
52.3
52.2
55.3
26.9
16.2
72.7
990
95.3
23
2.8
GPT-5.6 Sol
1,164
50.6
1,146
50.8
48.7
53.5
46.7
23.6
85.3
909
95.6
2
2.1
GPT-6 Astra
1,154
50.6
1,108
50.8
48.1
53.9
62.9
32.9
90.0
846
96.4
2
2.0
GPT-5.5
1,164
48.8
1,122
49.2
44.8
53.1
33.9
17.2
78.7
982
93.9
29
2.6
Appendix
Table 16: Finance: all models and secondary metrics. Values are percentages unless stated; n counts episodes. Gen. : generated conversations of 1,164 planned. Rubric percent : headline PRBench expert-rubric percent over the whole conversation for all 12 conditions, its n , and separately for the 11 persona conditions and the neutral user. Pass rates (persona conditions only): advice pass (rubric percent ≥50 ), persona pass (every critical need met, over conversations with a decided pass, Appendix efapp:finance-grading) and both together. Needs : score over expressed and answered needs (strict needs mode) and n scoreable persona episodes. Sim. valid : valid share of audited episodes. Cap : conversations stopped at the 8-turn cap; Turns : mean assistant turns per generated conversation. Within the main cohort, rows are sorted by rubric percent, and bold and underline mark the best and second-best values in the rubric, pass-rate and needs-value columns. Gray values with markers are outside the main cohort: † simulator and judges not all GPT-6 Luna at medium effort (GPT-5.6 Sol at high effort for some or all roles); ‡ score-only package without full dialogue evidence (registry status unverified; in Math these packages also lack an exact panel record and use GPT-6 Luna at high or unrecorded effort); § run listed as partial in the registry. ∘ fewer than 80% of planned episodes generated. n.r. not recorded. Gemini 3.7 / 3.8 Flash follow the original validated judge outputs, as all models do; under the raw-judge score update that their packages also publish they would read Gemini 3.7: task 44.1 ( n = 1,164), needs 61.7 ( n = 1,036); Gemini 3.8: task 45.5 ( n = 1,164), needs 63.5 ( n = 1,016).
Figure 12: Length of the shared first reply against the two grades, 18 main-cohort models. Each point is one model, colored by provider. The horizontal axis is the median word count of the first reply over the 97 questions. It excludes punctuation-only tokens such as Markdown symbols, so it is slightly lower than the whitespace counts quoted in the text. Left: rubric percent over all conditions. Right: persona pass rate. Dashed lines are least-squares fits over the 18 models. Longer first replies earn more rubric credit and tend to pass fewer persona needs.
Figure 13: Persona failures hidden by the advice grade. For each main-cohort model, the bar gives the share of persona conversations that pass the advice threshold (rubric percent of at least 50) but fail the persona pass. For most models, most conversations that an advice-only evaluation would count as successes leave a critical client need unmet.
Figure 14: Rubric credit lost after the shared first reply, main cohort. Each bar is the share of rubric items earned by the neutral conversation, which ended after the shared reply, but not earned in a continued persona conversation on the same question and model. Orange: items judged on the position the assistant holds at the end, with the two presets that question the answer in dark orange. Gray: items credited if any reply covers them, whose loss can only come from judge inconsistency, and the shaded band spans their range. Only the skeptical questioner and the confident contrarian lose final-position credit well above the gray level.
Condition
Critical need
n
Expr.
Needs
Compl.
Δ
Significant drops
Neutral
none
–
–
–
72.5
–
–
Frustrated on refusal
reason and recourse after a refusal
1,219
75.9
87.9
72.1
− 0.5
1
Self-doubt
facts checked against the record
935
55.8
37.1
73.2
+ 0.7
0
Gives up early
none (behavior only)
–
–
–
65.3
− 7.2
7
Money matters
exact amount and timing
1,004
61.7
62.0
71.8
− 0.7
2
Wants the reason
rule behind the outcome
1,719
99.8
75.3
65.1
− 7.5
7
Appendix
Table 17: Customer Service persona conditions in the main cohort. Numeric columns other than n and Significant drops are unweighted means of per-model values over the 14 main-cohort models, in percent except Δ . n : persona conversations with a numeric needs score, pooled over the 14 models. Expr. : share of simulator-valid generated conversations in which the customer actually voiced the need. Needs : persona-needs score. Compl. : task completion over generated conversations; the neutral row covers all three neutral trials. Δ : paired change in completion against the item’s neutral trials, in percentage points, averaged over the 14 models (per-model values in Table 20 ). Significant drops : number of the 14 models for which the 95% bootstrap interval of Δ over items lies below zero (no correction for multiple comparisons). Only one interval lies entirely above zero: GLM 5.3 Flash with the money-conscious customer. Persona conditions are not pooled.
Task
Needs
Failed tasks
Model
Gen.
Complete
Decision
Value
n
Pass
Sim. valid
Decision
Process
Ending
Cap
Qwen 3.8 27B
1,452
71.3
95.8
74.0
706
57.1
94.5
61
294
62
0
Muse Glimmer 30B
1,452
35.5
52.9
47.5
683
32.8
92.6
684
220
32
23
GPT-6 Luna
1,452
75.2
95.2
69.2
665
52.3
94.8
70
222
68
0
Gemini 3.7 Flash
1,452
92.6
96.2
75.9
677
60.8
94.1
55
29
23
0
Gemini 3.8 Flash
1,452
94.1
97.3
78.4
695
66.3
94.7
39
20
26
0
Appendix
Table 18: Customer Service: all models and secondary metrics. Values are percentages unless stated; n counts episodes. Gen. : generated conversations of 1,452 planned. Task : headline rule-based completion over all 1,452 planned episodes (policy-correct decision, required look-ups, matching execution and ending; incomplete episodes count 0) and the policy decision alone. Needs : score over simulator-valid completed persona episodes, its n , and persona pass (every critical need met). Sim. valid : valid share of audited completed episodes. Failed tasks : graded episodes that were not completed, by first failure: wrong decision, process gap (right decision without the required look-ups or execution), or wrong ending. Cap : conversations stopped at the 24-message cap. Bold marks the best main-cohort headline. Gray values with markers are outside the main cohort: † simulator and judges not all GPT-6 Luna at medium effort (GPT-5.6 Sol at high effort for some or all roles); ‡ score-only package without full dialogue evidence (registry status unverified); § run listed as partial in the registry. ∘ fewer than 80% of planned episodes generated. n.r. not recorded. For Gemini 3.7 / 3.8 Flash the raw-judge score update published by their packages gives the same headline values.
Model
By rule, route (68)
Escalation req. (34)
Misremembered (12)
Identity (10)
Conditional (5)
By rule, no route (3)
Qwen 3.8 27B
70.5
63.4
75.0
87.3
85.5
87.9
Muse Glimmer 30B
36.4
31.3
20.5
57.3
50.9
27.3
GPT-6 Luna
73.0
71.1
77.3
94.5
76.4
97.0
Gemini 3.7 Flash
89.7
93.6
97.0
100.0
100.0
93.9
Gemini 3.8 Flash
91.6
96.3
97.0
100.0
96.4
93.9
Kimi K3
72.7
74.1
67.4
88.2
83.6
93.9
Appendix
Table 19: Customer Service: task completion by policy class, per model. Percent of planned conversations, all eleven conversations per item pooled within the model; item counts in parentheses. By rule : approve or refuse by rule, with or without an escalation route after a refusal; Escalation req. : escalation required; Misremembered : the customer misremembers a fact that the record contradicts; Identity : identity check before acting; Conditional : the correct action depends on whether the customer insists. Gray rows are outside the main cohort.
Model
Neutral
Frustrated
Self-doubt
Gives up
Money
Reason
Bundles
Risk
Options
Significant drops
Qwen 3.8 27B
72.7
+5.3
+2.3
−5.3
+1.5
−10.6
−1.5
−4.5
−3.0
1
Muse Glimmer 30B
35.1
−0.3
+0.5
−0.3
+6.6
−4.8
+2.8
−2.5
+2.8
0
GPT-6 Luna
78.3
−2.5
−1.0
−3.3
−3.3
−9.3
−4.8
−7.8
−1.8
2
Gemini 3.7 Flash
93.9
+1.5
+1.5
−8.3
−1.5
−4.5
−4.5
+0.8
+0.8
1
Gemini 3.8 Flash
96.5
−1.0
−0.3
−9.3
−1.8
−1.0
−7.1
−2.5
−2.5
2
Kimi K3
74.2
+6.8
+1.5
−3.8
+4.5
+1.5
−3.8
−3.0
+0.8
0
Appendix
Table 20: Customer Service: paired change in task completion under each persona, per model. Neutral : completion over the three neutral trials (percent of planned conversations). Each persona column is the mean over the 132 items of the persona conversation’s completion (0 or 1) minus the item’s mean over its three neutral trials, in percentage points; the population is all planned conversations, so the means differ by a few tenths from Table 17 . Bold marks a 95% item-bootstrap interval that excludes zero (no correction for multiple comparisons); Significant drops counts the personas whose interval lies below zero. Gray rows are outside the main cohort (Appendix B.3 ). Personas are never pooled.
Figure 15: Customer Service: where the persona effects come from (main cohort, 14 models). (a) Share of planned conversations in each failure type by condition, model-equal means. (b) Wrong endings on the 34 items that require an escalation after a refusal. (c) Approved cases handed to a manager, on the 41 approval items with an escalation route. Panels (b) and (c) pool conversations over the 14 models.
Card
Description shown to the simulated traveler on official tasks (verbatim)
Careful planner
You prefer to work through a complicated trip one aspect at a time. You need accessible explanations and a short recap to feel confident. Ask naturally when overwhelmed or unsure; you can stop once helped.
Skeptical comparer
You like comparing plausible alternatives and understanding the basis for a recommendation, including what a displayed price covers. Ask for evidence when an explanation seems too confident. Do not invent extra travel constraints.
Concise decider
You want efficient, direct exchanges and compact reasons rather than repeated long lists. You still want control before a final selection. Say what would make the answer easier to use; do not force every need into every turn.
Efficient comparer
You prefer an efficient exchange that leads with a recommendation, followed by a focused comparison with a plausible alternative. Ask which details are catalog facts and which remain uncertain when that matters. Do not invent extra travel constraints or force every preference into every turn.
Concise planner
You prefer compact, accessible replies and a short recap of your expressed requirements to feel ready for the next decision. Ask naturally for clarification when needed; you can stop once helped. Do not invent extra travel constraints or require repeated reassurance.
Appendix
Table 21: Travel persona cards. Each card’s description as shown to the simulated traveler on the official tasks. The card also lists its three needs with their satisfaction behaviors, critical flags and applicability (Table 22 ). On the authored revision tasks, the concise decider’s description instead asks for a leading recommendation with a brief explanation and states that recommendations are revisable.
Condition
Critical needs (third need)
Expr.
Needs
Share
Turns
Reward
Δ
Below
Above
Neutral
none
–
–
78.8
3.53
73.4
–
–
–
Careful planner stepwise, plain, support
paced questions, plain terms (recap before committing)
concise replies, plain terms (recap before committing)
8.6
88.6 †
77.3
3.83
72.0
− 1.4
3
2
Appendix
Table 22: Travel conditions in the main cohort. Critical needs lists the two critical needs of each card and, in parentheses, its third, non-critical need; the line under each card gives its decision pace, explanation depth and trust style. Numeric columns other than Below and Above are unweighted means of per-model values over the 19 main-cohort models. Expr. is the share of applicable needs that the traveler expressed, in percent, among completed conversations whose simulator audit was not invalid. Needs is the persona-needs score in percent over persona conversations on the 67 official and five authored tasks; † under the GPT-6 Luna simulator the concise-planner traveler rarely voiced its needs, so this value rests on only 189 scored conversations, about ten per model. Share is the share of requested aspects with a recorded choice, Turns the mean number of assistant replies, and Reward the choice reward on the 0–100 scale, all over official-task conversations with a score. Δ is the paired persona-minus-neutral change in reward, in points, over 47 to 67 official tasks per model. Below and Above count the models whose 95% task-cluster bootstrap interval lies entirely below or above zero. The pooled row combines each model’s five persona conditions into one paired contrast; its Needs value is the mean headline needs score of Table 23 .
Choice reward
Recorded
Needs
Model
Gen.
All
n
Persona
Neutral
Valid
Share
Credit
Rev.
Value
n
Pass
Sim. valid
Cap
Turns
Gemini 3.7 Flash
432
86.6
402
85.6
91.8
87.0
92.5
93.6
30
85.2
251
71.4
96.3
0
4.4
Fable 5.1
432
79.7
402
77.9
88.8
80.2
83.7
95.2
30
87.8
288
80.9
94.2
0
3.8
Claude Sonnet 5.5
432
79.3
402
77.5
87.9
79.6
84.0
94.4
30
84.2
294
86.2
94.9
0
4.1
Gemini 3.8 Flash
432
76.3
402
73.9
88.4
76.8
79.9
95.5
30
88.1
254
82.6
94.9
0
4.2
Grok 4.6
429
75.6
399
75.2
77.3
77.9
81.1
93.2
30
85.2
275
80.6
89.8
1
3.9
Appendix
Table 23: Travel: all models and secondary metrics. Values are percentages unless stated; n counts conversations. Gen. : conversations completed of 432 planned; unlike in Table 16 , conversations stopped at a cap or by an error are not counted. Choice reward : headline UserBench reward (1 / 0.8 / 0 per requested aspect, averaged over the two aspects; an aspect without a recorded choice earns 0) over all official-task conversations with a score from the six conditions, its n , the same separately for the five persona conditions and the neutral traveler, and over simulator-valid completed conversations only ( Valid ; 306 to 386 per model). Recorded : share of requested aspects with a recorded choice ( Share ) and mean credit of the recorded choices ( Credit ), over the same conversations as All . Rev. : successes on the 30 authored revision conversations (final-state success), never pooled with the reward. Needs : persona-needs score over expressed and answered needs in persona conversations on the 67 official and five authored tasks, its n of scoreable persona conversations, and the persona pass rate among persona conversations with a decided pass (68 to 133 per main-cohort model, together 29.6% of main-cohort persona conversations, 58% of them under the skeptical comparer). Sim. valid : valid share of audited conversations. Cap : conversations stopped at the 10-reply cap (for Muse Glimmer 30B, 3 of its 11 at the 20-step tool cap). Turns : mean assistant replies per scored official-task conversation. Within the main cohort, rows are sorted by the headline reward, and bold and underline mark the best and second-best values in the reward, needs-value and pass columns. Gray values are outside the main cohort; † simulator and judges GPT-5.6 Sol at high effort instead of GPT-6 Luna at medium effort. Gemini 3.7 / 3.8 Flash follow the original validated judge outputs, as all models do; under the raw-judge score update that their packages also publish, their task values are unchanged and their needs values would read 85.7 ( n = 289) and 88.6 ( n = 287). The main-text Table 2 computes Travel needs over the official tasks only and with this raw-judge update, so its Travel values differ from the Needs column by up to 1.4 points and rank Muse Glimmer 30B first and GPT-6 Astra second.
Figure 16: Persona effects on the reward come from fewer recorded choices, main cohort. Over a set of conversations, the reward equals the share of requested aspects with a recorded choice times the mean credit of the recorded choices. Left: each point is one model and persona condition (95 cells), paired with the neutral condition over the same official tasks; the dashed line is the change expected if the credit of recorded choices stayed at its pooled 93%. Colors encode persona conditions, not providers. Right: the mean paired reward change split into the part due to fewer recorded choices and the part due to the credit of the recorded choices, which together add up to the change.
Figure 17: Persona cards whose labels differ in one attribute, main cohort. Each row compares two cards as written on the same official tasks within each model (first card minus second card); the cards also differ in their descriptions and needs (Table 21 ). Gray dots are models, and the orange diamond is the mean over the 19 models with a 95% bootstrap interval over models. Left: change in choice reward; the note gives the number of models whose task-bootstrap interval lies entirely below or above zero. Right: change in the share of requested aspects with a recorded choice.
Figure 18: The task-needs association rests largely on the request for a reason before an irrevocable choice, main cohort. Left: Spearman correlation across the 19 models between the choice reward and the persona-needs score recomputed without one need, with 95% model-bootstrap intervals; the dashed line uses all needs. Right: concise-decider conversations on official tasks, grouped by the judge’s verdict on this need, by the number of requested aspects with a recorded choice, with the mean reward of each group and of its conversations that recorded both aspects. A not-satisfied verdict requires a recorded choice, and satisfying the need often means holding off on the choice that the reward requires.
Figure 19: Selected excerpts from paired conversations on the same task and opening. The Health and Coding pairs are discussed in Appendices I.1.2 and I.2.1 , respectively. Different follow-ups accompany changes in disclosed information, assistant responses, and task outcomes. Health scores measure inquiry coverage; Coding outcomes come from saved native tests.
Computer-use agents (CUAs) are commonly evaluated against fixed task specifications, but users may omit requirements, add constraints, or require approval before consequential actions. Completing the benchmark instruction does not guarantee serving the person who asked. We introduce PersonaCUABench, which holds 74 tasks across 11 domains fixed and varies only who asks. A persona agent, playing one of 44 personas in four groups, rewrites each instruction in that user's own words and answers the CUA's questions, while the CUA sees only the dialogue, never the profile or the original instruction. Across nine backbones (3,330 runs), mean task completion falls by 36% relative to the original requests. The losses have two sources: in 59.9% of cells that lose points, the agent engages with what the user asked for and still gets it wrong, and 11.2% are stops the user requested that the checker scores as failures. Even when a persona request is fully completed, the user gives the top satisfaction rating in only 62.7% of cells, against 84.1% for the original requests. CUAs should therefore be evaluated not only on task completion but also on whether they meet the requirements and constraints users express. Code: https://github.com/MatrAIx-ai/PersonaCUABench. Project page: https://matraix.ai/research/persona-cua.html
Haoming Li, Defu Cao, Xu Hu +23
The University of Texas at Dallas · University of Southern California · Carnegie Mellon University +14
Large Language Model personas, explicit profiles specifying a user's attributes, preferences, and behavioural tendencies, are increasingly used to simulate human conversations for user modelling, social reasoning, and behavioural analysis. Evaluating whether such simulations faithfully reflect human conversational behaviour is critical, yet current practice often relies on LLM-as-a-judge approaches that provide limited grounding in observable behaviour and produce opaque scalar scores. We present Eval4Sim, an evaluation framework that measures alignment between simulated and human conversations across three dimensions: adherence, whether persona traits are recoverable from dialogue via dense retrieval; consistency, whether a persona maintains a distinguishable stylistic identity via authorship verification; and naturalness, whether conversations exhibit human-like turn-to-turn flow via dialogue NLI. Unlike optimization-oriented metrics, each dimension takes a human corpus as a reference baseline and penalizes deviations in both directions, distinguishing insufficient persona encoding from over-optimized, unnatural behaviour. The framework is corpus-agnostic: any persona-annotated conversational dataset can serve as the reference. Evaluated over ten simulation corpora, Eval4Sim surfaces systematic trade-offs invisible to single-score methods.
Eliseo Bao, Anxo Perez, Javier Parapar +1
IRLab, CITIC, Universidade da Coruña A Coruña, Spain · University of Sheffield Sheffield, United Kingdom
We introduce AgentPersonaBench (APB), a benchmark evaluating whether persona conditioning faithfully steers downstream agent behavior. While language models are increasingly deployed for persona-driven user simulation, existing benchmarks primarily evaluate conversational styling or self-reports rather than authentic behavioral fidelity. APB evaluates latent persona adherence one trait at a time, embedding each target trait within a complete synthetic profile without explicitly naming the trait or disclosing the test. Ground-truth adherence is verified strictly from observable actions across four interaction surfaces of increasing realism: survey, chat, web (interactive web environments), and app (desktop software environments). APB comprises 2,460 tasks spanning 867 traits, verified through automated audits and expert review. Our evaluation of 20 frontier model arms demonstrates that high-fidelity user simulation is already attainable: leading models achieve up to 84.7% full-pass adherence under unprompted conditions. At the same time, APB identifies clear behavioral boundaries: adherence drops across interaction modalities (only 37.9-64.3% pass all four surfaces), multi-attribute demands degrade retention, and competing model families exhibit pronounced behavioral divergence.