Organizations: University of California, Los Angeles · University of Southern California · University of Illinois at Urbana-Champaign · University of Illinois Chicago · Massachusetts Institute of Technology · University of Chicago · Texas A&M University · Carnegie Mellon University · University of the Arts London · Amherst College · Northwestern University · Harvard University · The University of Texas at Dallas · Northeastern University · Purdue University · The Ohio State University
Coding agents must deliver correct software while respecting users' expertise, preferences, and working conditions. Existing benchmarks increasingly incorporate interaction, yet personalized requirement fulfillment remains underexplored. We introduce PersonaCodeBench, which turns user characteristics into verifiable obligations in persona-driven multi-turn coding sessions. Our persona pool spans roles, technical abilities, interaction preferences, and practical constraints, yielding task-grounded evolved requirements for implementation, permissions, testing, explanation, and delivery. At runtime, a controller governs user simulator responses and requirement disclosure, using online requirement evaluation to provide feedback and advance the interaction. Requirements emerge progressively, with their content, disclosure conditions, and acceptance criteria fixed before evaluation. A completion verifier and a personalization verifier separately assess native correctness and persona fulfillment. Joint pass requires both adherence to the declared budget and compliance with validity conditions. We evaluate all major frontier agents, establish a comprehensive leaderboard, and conduct detailed experimental analyses to validate effectiveness. PersonaCodeBench supports reproducible evaluation of coding agents' ability to meet explicit, personalized user expectations alongside coding objectives.
Figures & tables
Figure 1: PersonaCodeBench results overview. (a) Agent cost versus Joint pass rate, with reasoning effort and time-budget multipliers annotated. (b) Joint pass rates at the best-performing effort shown for each model.
Figure 2: Overview of the persona-driven multi-turn coding sessions. The controller governs the disclosure of evolved requirements and uses the online requirement evaluator to validate responses. Satisfied requirements permit advancement and Unmet requirements return feedback to the coding agent for revision.
Figure 3: Metadata of PersonaCodeBench. (A) language coverage. (B) evolved requirement counts. (C) persona role families.
Figure 4: Schematic of the benchmark construction process.
Source benchmark
Full
Lite
Terminal-bench 3
52
23
DeepSWE
64
28
SWE-bench-Live
45
19
SWE-Together
9
4
SWE-bench Pro
61
26
Total pairs
231
100
Table 1: Composition of the development collection and public Lite release.
Table 2: The pass rate pooled over all 100 tasks in PersonaCodeBench. † only counts the textual part.
Persona
Terminal- bench 3
SWE- bench Pro
SWE- bench- Live
SWE- Together
DeepSWE
Student Requiring explanations
0.0% 0/4
20.0% 1/5
0.0% 0/5
0.0% 0/1
20.0% 1/5
First PR contributor
0.0% 0/1
33.3% 3/9
–
0.0% 0/1
0.0% 0/8
Outcome-oriented manager
0.0% 0/3
–
50.0% 1/2
–
7.7% 1/13
Test-focused lead
0.0% 0/2
66.7% 2/3
0.0% 0/1
–
–
Senior reviewer
0.0% 0/2
–
50.0% 1/2
100.0% 2/2
–
Analyst Without index permissions
–
25.0% 1/4
100.0% 2/2
–
–
Table 3: Joint pass outcomes by persona and source subset for GPT-6 Luna.
Ablation
Joint pass
completion pass
personalization pass
PersonaCodeBench
19.0%
24.0%
50.0%
PersonaCodeBench w. Opening
20.0%
26.0%
51.0%
PersonaCodeBench w/o Persona
45.0%
45.0%
–
Table 4: Ablation Study of key interactive design on GPT-6 Luna.
Appendix figures & tables11 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 5: Detailed evaluation flow across the six execution profiles. The online requirement evaluator assesses agent responses and the controller governs transitions and qingwen simulator responses. Final verifiers produce completion and personalization outcomes used to determine joint pass.
Snapshot → completion verifier → personalization verifier (deterministic handoff checker on the last eligible reply).
Appendix
Table 5: Principal finalization order in the reviewed runtime. Arrows show the main path; conditional hooks and recovery branches remain profile-specific.
Figure 6: Persona attribute composition in Full and Lite. Panels show interaction, expertise/background, trust/skepticism, explanation, and verification. Each assignment contributes one category per panel; annotations give rounded Full → Lite percentages.
Figure 7: Evolved-requirement category coverage in Lite. Each cell is the percentage of tasks in a source subset with at least one added requirement in that category; column labels give denominators. Categories overlap, and displayed percentages are rounded.
Profile
Protection of the delivered contract
Terminal-bench 3
Host assembly appends frozen native facts and the due persona disclosure; completion is gated on the required disclosure and progress state.
SWE-bench Pro
Native wording inventory and disclosure ledger; delivery audit checks recorded agent inputs or mini-agent trajectory receipts, including missing or late disclosures.
SWE-bench-Live
Frozen opening followed by the complete original issue on user turn 2; the persona handoff contract is fixed separately.
SWE-Together
Exact inclusion of the due persona statement; invalid proposals are retried, and acceptance requires all persona requests to be disclosed and assessed as Satisfied. Native follow-ups remain context-dependent.
DeepSWE
Frozen staged native units and persona requests; validated document selection and host expansion preserve required public blocks, with simulator response validation of surrounding user prose.
Appendix
Table 6: Requirement-delivery safeguards in the reviewed source profiles. These are implementation mechanisms, not a measured benchmark-wide semantic fidelity rate.
Figure 8: Checker effectiveness and judge consistency. (A) The pinned SWE-bench-Live handoff checker matches all 133 specified control outcomes; missing evidence correctly yields Unresolved . (B) Among 1,962 complete run–task records, 179 of 874 native-test passes fail persona assessment. (C) Final three-vote DeepSWE assessments disagree in 9/36 and 6/73 decisions for two coding-agent trajectory sets, with the same grading model. (D) A dateutil trajectory receives three online Satisfied judgments at turn 8 but three final Unmet judgments. The online requirement evaluator and personalization verifier observe different evidence. Each panel has its own denominator and scope, detailed below.
Mechanism
Tasks
Recorded construction
Concise summary
75
Terminal-bench 3: 52; SWE-bench Pro: 23. The opening summarizes the native goal and defers details.
Complete issue deferral
45
SWE-bench-Live. A generic repository-help request precedes the complete original issue on user turn 2.
Staged native disclosure
64
DeepSWE. S1 native units appear initially; later stages disclose the remaining native units.
Native description upfront
47
SWE-bench Pro: 38; SWE-Together: 9. Later interaction can still introduce evolved requirements or native follow-ups.
Appendix
Table 7: Opening mechanisms in the 231-task Full collection. Counts refer to task–persona pairs, with one selected persona per original task.
Figure 9: Task-level initial-intent blurring, shown over two pages for readability. Part 1 contains Terminal-bench 3, SWE-bench-Live, and SWE-Together. Bars show native-description shortening bi ; colors identify the opening mechanism. Right-hand labels give examples of delayed native details. The measure concerns text length, not semantic information loss.
Figure 10: Task-level initial-intent blurring (continued): DeepSWE and SWE-bench Pro. All 231 Full tasks are shown across the two pages. Zero shortening is scoped to the native text definition and does not imply the absence of subsequent evolved requirements.
Figure 11: One continuous line per model connects all available settings, including max at 1× , 2× , and 3× . Every setting has one point labeled with its pass rate (%). Rates at 1× come from the full effort ladder, including its max- 1× value; max- 2× and max- 3× rates come from the time-budget ladder. Lines are visual guides: the task set can change between 1× and 2 – 3× , so a connected series is not necessarily a matched-task comparison throughout. Claude Opus 5.5 uses the same seven tasks across all six settings. Med. denotes medium effort.
Baseline
Pivot
Task disclosure
Full at start
Progressive; stops at S2
Agent calls / tools / replies
18 / 17 / 1
35 / 24 / 11
User turns
1
12
Hidden fail-to-pass tests
19/20
7/20
Hidden pass-to-pass tests
3/3
3/3
Native full acceptance
False
False
Appendix
Table 8: Recorded outcomes for the selected pair. Public validation and hidden native acceptance have different scopes.
Computer-use agents (CUAs) are commonly evaluated against fixed task specifications, but users may omit requirements, add constraints, or require approval before consequential actions. Completing the benchmark instruction does not guarantee serving the person who asked. We introduce PersonaCUABench, which holds 74 tasks across 11 domains fixed and varies only who asks. A persona agent, playing one of 44 personas in four groups, rewrites each instruction in that user's own words and answers the CUA's questions, while the CUA sees only the dialogue, never the profile or the original instruction. Across nine backbones (3,330 runs), mean task completion falls by 36% relative to the original requests. The losses have two sources: in 59.9% of cells that lose points, the agent engages with what the user asked for and still gets it wrong, and 11.2% are stops the user requested that the checker scores as failures. Even when a persona request is fully completed, the user gives the top satisfaction rating in only 62.7% of cells, against 84.1% for the original requests. CUAs should therefore be evaluated not only on task completion but also on whether they meet the requirements and constraints users express. Code: https://github.com/MatrAIx-ai/PersonaCUABench. Project page: https://matraix.ai/research/persona-cua.html
Haoming Li, Defu Cao, Xu Hu +23
The University of Texas at Dallas · University of Southern California · Carnegie Mellon University +14
The recent emergence of vibe-coding workflows is changing what coding agents are expected to do. Instead of merely completing code under fully specified instructions, agents are increasingly expected to transform incomplete product intent into working software by combining various abilities including planning, requirement clarification, tool use, debugging, and repository-level construction. Yet existing benchmarks have not fully caught up with this shift, evaluating agents on static, fully specified tasks. In this paper, we introduce ICAE-Bench, a benchmark for evaluating coding agents under interactive project-building settings. The basic idea is to start from a fuzzy product requirement, simulating the dynamic paradigm with an automated User Agent. To make this setting both realistic and evaluable, ICAE-Bench introduces three key designs. First, to avoid the ambiguity of unconstrained fuzzy requirements, each task derives ambiguity from a precise real open-source repository with executable behavior. Second, to ensure high-quality and reproducible user simulation, ICAE-Bench grounds interaction through User Agent Data, allowing the User Agent to reveal hidden constraints without inventing new requirements or leaking implementation artifacts. Third, to evaluate open-ended repositories fairly, ICAE-Bench uses standardized black-box tests together with multi-dimensional diagnostics, including functional correctness, semantic and API similarity, structural fidelity, design quality, and interaction quality.
Zhongyuan Peng, Dan Huang, Chuyu Zhang +8
Fudan University · Meituan Group · Singapore Management University +1
Most coding-agent benchmarks are static: an agent receives a complete task description up front and is judged only by its final code. Real coding assistance is interactive, with users clarifying goals, adding constraints, and correcting mistakes over multiple turns. We introduce SWE-Together, a multi-turn benchmark reconstructed from real user-agent coding sessions. To make real interactions verifiable, we curate 109 repository-level tasks from 11,260 recorded sessions, selecting sessions with recoverable repository states, clear user goals, and observable outcomes. To replay these interactions across agents, we build a reactive LLM-based user simulator that preserves the original users' intents and provides feedback when the coding agent's progress requires it. To evaluate agents as collaborators, we measure both final repository correctness and the number of corrective feedback turns required during the interaction. Experiments with frontier coding agents show that stronger agents generally achieve higher final success rates while requiring fewer interventions, suggesting an improved user experience.