Organizations: University of California, Los Angeles · University of Southern California · University of Illinois at Urbana-Champaign · University of Illinois Chicago · Massachusetts Institute of Technology · University of Chicago · Texas A&M University · Carnegie Mellon University · University of the Arts London · Amherst College · Northwestern University · Harvard University · The University of Texas at Dallas · Northeastern University · Purdue University · The Ohio State University
Coding agents must deliver correct software while respecting users' expertise, preferences, and working conditions. Existing benchmarks increasingly incorporate interaction, yet personalized requirement fulfillment remains underexplored. We introduce PersonaCodeBench, which turns user characteristics into verifiable obligations in persona-driven multi-turn coding sessions. Our persona pool spans roles, technical abilities, interaction preferences, and practical constraints, yielding task-grounded evolved requirements for implementation, permissions, testing, explanation, and delivery. At runtime, a controller governs user simulator responses and requirement disclosure, using online requirement evaluation to provide feedback and advance the interaction. Requirements emerge progressively, with their content, disclosure conditions, and acceptance criteria fixed before evaluation. A completion verifier and a personalization verifier separately assess native correctness and persona fulfillment. Joint pass requires both adherence to the declared budget and compliance with validity conditions. We evaluate all major frontier agents, establish a comprehensive leaderboard, and conduct detailed experimental analyses to validate effectiveness. PersonaCodeBench supports reproducible evaluation of coding agents' ability to meet explicit, personalized user expectations alongside coding objectives.
Figures & tables
Figure 1: PersonaCodeBench results overview. (a) Agent cost versus Joint pass rate, with reasoning effort and time-budget multipliers annotated. (b) Joint pass rates at the best-performing effort shown for each model.
Figure 2: Overview of the persona-driven multi-turn coding sessions. The controller governs the disclosure of evolved requirements and uses the online requirement evaluator to validate responses. Satisfied requirements permit advancement and Unmet requirements return feedback to the coding agent for revision.
Figure 3: Metadata of PersonaCodeBench. (A) language coverage. (B) evolved requirement counts. (C) persona role families.
Figure 4: Schematic of the benchmark construction process.
Source benchmark
Full
Lite
Terminal-bench 3
52
23
DeepSWE
64
28
SWE-bench-Live
45
19
SWE-Together
9
4
SWE-bench Pro
61
26
Total pairs
231
100
Table 1: Composition of the development collection and public Lite release.
Table 2: The pass rate pooled over all 100 tasks in PersonaCodeBench. † only counts the textual part.
Persona
Terminal- bench 3
SWE- bench Pro
SWE- bench- Live
SWE- Together
DeepSWE
Student Requiring explanations
0.0% 0/4
20.0% 1/5
0.0% 0/5
0.0% 0/1
20.0% 1/5
First PR contributor
0.0% 0/1
33.3% 3/9
–
0.0% 0/1
0.0% 0/8
Outcome-oriented manager
0.0% 0/3
–
50.0% 1/2
–
7.7% 1/13
Test-focused lead
0.0% 0/2
66.7% 2/3
0.0% 0/1
–
–
Senior reviewer
0.0% 0/2
–
50.0% 1/2
100.0% 2/2
–
Analyst Without index permissions
–
25.0% 1/4
100.0% 2/2
–
–
Table 3: Joint pass outcomes by persona and source subset for GPT-6 Luna.
Ablation
Joint pass
completion pass
personalization pass
PersonaCodeBench
19.0%
24.0%
50.0%
PersonaCodeBench w. Opening
20.0%
26.0%
51.0%
PersonaCodeBench w/o Persona
45.0%
45.0%
–
Table 4: Ablation Study of key interactive design on GPT-6 Luna.
Appendix figures & tables11 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 5: Detailed evaluation flow across the six execution profiles. The online requirement evaluator assesses agent responses and the controller governs transitions and qingwen simulator responses. Final verifiers produce completion and personalization outcomes used to determine joint pass.
Snapshot → completion verifier → personalization verifier (deterministic handoff checker on the last eligible reply).
Appendix
Table 5: Principal finalization order in the reviewed runtime. Arrows show the main path; conditional hooks and recovery branches remain profile-specific.
Figure 6: Persona attribute composition in Full and Lite. Panels show interaction, expertise/background, trust/skepticism, explanation, and verification. Each assignment contributes one category per panel; annotations give rounded Full → Lite percentages.
Figure 7: Evolved-requirement category coverage in Lite. Each cell is the percentage of tasks in a source subset with at least one added requirement in that category; column labels give denominators. Categories overlap, and displayed percentages are rounded.
Profile
Protection of the delivered contract
Terminal-bench 3
Host assembly appends frozen native facts and the due persona disclosure; completion is gated on the required disclosure and progress state.
SWE-bench Pro
Native wording inventory and disclosure ledger; delivery audit checks recorded agent inputs or mini-agent trajectory receipts, including missing or late disclosures.
SWE-bench-Live
Frozen opening followed by the complete original issue on user turn 2; the persona handoff contract is fixed separately.
SWE-Together
Exact inclusion of the due persona statement; invalid proposals are retried, and acceptance requires all persona requests to be disclosed and assessed as Satisfied. Native follow-ups remain context-dependent.
DeepSWE
Frozen staged native units and persona requests; validated document selection and host expansion preserve required public blocks, with simulator response validation of surrounding user prose.
Appendix
Table 6: Requirement-delivery safeguards in the reviewed source profiles. These are implementation mechanisms, not a measured benchmark-wide semantic fidelity rate.
Figure 8: Checker effectiveness and judge consistency. (A) The pinned SWE-bench-Live handoff checker matches all 133 specified control outcomes; missing evidence correctly yields Unresolved . (B) Among 1,962 complete run–task records, 179 of 874 native-test passes fail persona assessment. (C) Final three-vote DeepSWE assessments disagree in 9/36 and 6/73 decisions for two coding-agent trajectory sets, with the same grading model. (D) A dateutil trajectory receives three online Satisfied judgments at turn 8 but three final Unmet judgments. The online requirement evaluator and personalization verifier observe different evidence. Each panel has its own denominator and scope, detailed below.
Mechanism
Tasks
Recorded construction
Concise summary
75
Terminal-bench 3: 52; SWE-bench Pro: 23. The opening summarizes the native goal and defers details.
Complete issue deferral
45
SWE-bench-Live. A generic repository-help request precedes the complete original issue on user turn 2.
Staged native disclosure
64
DeepSWE. S1 native units appear initially; later stages disclose the remaining native units.
Native description upfront
47
SWE-bench Pro: 38; SWE-Together: 9. Later interaction can still introduce evolved requirements or native follow-ups.
Appendix
Table 7: Opening mechanisms in the 231-task Full collection. Counts refer to task–persona pairs, with one selected persona per original task.
Figure 9: Task-level initial-intent blurring, shown over two pages for readability. Part 1 contains Terminal-bench 3, SWE-bench-Live, and SWE-Together. Bars show native-description shortening bi ; colors identify the opening mechanism. Right-hand labels give examples of delayed native details. The measure concerns text length, not semantic information loss.
Figure 10: Task-level initial-intent blurring (continued): DeepSWE and SWE-bench Pro. All 231 Full tasks are shown across the two pages. Zero shortening is scoped to the native text definition and does not imply the absence of subsequent evolved requirements.
Figure 11: One continuous line per model connects all available settings, including max at 1× , 2× , and 3× . Every setting has one point labeled with its pass rate (%). Rates at 1× come from the full effort ladder, including its max- 1× value; max- 2× and max- 3× rates come from the time-budget ladder. Lines are visual guides: the task set can change between 1× and 2 – 3× , so a connected series is not necessarily a matched-task comparison throughout. Claude Opus 5.5 uses the same seven tasks across all six settings. Med. denotes medium effort.
Baseline
Pivot
Task disclosure
Full at start
Progressive; stops at S2
Agent calls / tools / replies
18 / 17 / 1
35 / 24 / 11
User turns
1
12
Hidden fail-to-pass tests
19/20
7/20
Hidden pass-to-pass tests
3/3
3/3
Native full acceptance
False
False
Appendix
Table 8: Recorded outcomes for the selected pair. Public validation and hidden native acceptance have different scopes.