SWE-WhatIf: Simulating Real Users of Coding Agents from Recorded Session
Organizations: KAIST · SkillBench · Carnegie Mellon University
Abstract
Recorded sessions between users and coding agents offer a way to evaluate how agents complete real-world tasks and interact with users. However, these sessions cannot simply be replayed with new agents, because recorded user replies may no longer apply when an agent takes a different trajectory. We introduce SWE-WhatIf, which replays recorded sessions with new agents by modeling the user's intents, the criteria for satisfying them, and the conditions under which they arise. Using this model, a user simulator generates replies that adapt to unseen agent turns while remaining grounded in the recording. Across 89 real-world sessions, SWE-WhatIf generated the replies most similar to those of real users and matched the strongest baseline when sessions were rewritten. To test adaptation, we replay sessions with agents instructed to avoid or repeat actions that the recorded user pushed back on. SWE-WhatIf produced the highest win rate for avoiding these actions (66.3%), significantly above chance. We further demonstrate its utility by ranking five LLMs as coding agents, where the highest-ranked model depended on how users specified and developed their intents, and a study with 16 users, where replaying their own sessions changed 14 participants' rankings and informed their future agent use.
Figures & tables
| Replication | Robustness | Adaptation | |||||
| User Simulator | Single | Multi | Avg. | Single | Multi | Avg. | Win Rate |
| Goal-Prompted | 26.3 | 24.8 | 25.6 | 25.9 | 24.4 | 25.2 | 55.1 ( =.306) |
| SWE-Interact | 35.0 | 31.4 | 33.2 | 35.8 | 30.3 | 33.1 | 50.6 ( =1.00) |
| SWE-Together | 51.3 | 40.4 | 45.9 | 48.1 | 41.3 | 44.7 | 55.6 ( =.314) |
| SWE-WhatIf | 53.2 | 44.9 | 49.1 | 49.5 | 40.7 | 45.1 | 66.3 ( =.001) |
| Style profile | 51.8 | 42.0 | 46.9 | 47.3 | 40.0 | 43.7 | 62.9 ( =.012) |
| User Simulator | Premature | Extra | Passive | Omitted |
|---|---|---|---|---|
| Goal-Prompted | 77.5% | 64.0% | 1.69% | 18.5% |
| SWE-Interact | 53.4% | 70.2% | 3.37% | 23.6% |
| SWE-Together | 12.9% | 44.9% | 96.1% | 35.4% |
| SWE-WhatIf | 33.1% | 52.2% | 5.62% | 21.3% |
| Style profile | 31.5% | 47.2% | 4.49% | 25.3% |
Appendix figures & tables36 assets
Supplementary material from the paper’s appendix.
Appendix
| Median | Min | Max | |
| User messages | 6 | 2 | 27 |
| Characters per message | 82 | 21 | 227 |
| Intents | 5 | 1 | 16 |
| Criteria | 10 | 2 | 41 |
| Conditions | 2 | 0 | 9 |
| Intent | Criteria | Raised when | |
|---|---|---|---|
| I1 | entire explain prints its help text when given an argument without a qualifier flag. | (a) Print the help text for an argument without any of the qualifier flags ( --session , --commit , --checkpoint ). (b) Command with no argument, or with a qualifier flag, should behave as it did before the patch. | Opening request |
| I2 | The patch is on a new branch. | (a) A new branch is created and holds the changes of the patch. (b) None of the changes is committed outside the new branch. | No condition |
| I3 | The code changes are reviewed with a verdict. | (a) The agent reviews the changes of the patch and should decide to approve or request changes. | I1 satisfied |
| I4 | Add a short hint that explains why the help text is shown, if the change is small. | (a) Print a short hint that explains why the help text before it. (b) Adding the hint is a small change that does not affect the rest of the command. | C1 met |
| C1 : A review of the patch suggests printing a message that explains why the help text is shown. | |||
| SWE-Together (65 tasks, by task_id ) | ||
| agent-swarm-task-4a881b | cli-task-ea3f8f | pi-mono-auto-6f8288bb |
| agent-swarm-task-ea4bd8 | comfyui-gemma3-sliding-window | pi-mono-auto-90748a5d |
| amytis-task-e3714e | comfyui-jina-clip-v2-full | pi-mono-auto-93c17d3b |
| arr-monitor-add-processes-flag | comfyui-lumina-axes-lens | pi-mono-auto-95de1efe |
| cli-task-0ec2e9 | comfyui-lumina2-lora-prefix | pi-mono-auto-a4fca584 |
| cli-task-2a55af | comfyui-newbie-lumina-refactor | pi-mono-auto-cad40af4 |
| Replication | Robustness | |||
| User simulator | Single | Multi | Single | Multi |
| Goal-Prompted | 36.6 / 16.1 | 42.2 / 7.4 | 36.3 / 15.6 | 42.3 / 6.5 |
| SWE-Interact | 36.3 / 33.6 | 40.0 / 22.8 | 36.7 / 34.9 | 38.3 / 22.3 |
| SWE-Together | 49.0 / 53.7 | 39.1 / 41.7 | 47.2 / 49.0 | 40.6 / 41.9 |
| SWE-WhatIf | 49.5 / 57.0 | 42.7 / 47.2 | 47.2 / 51.8 | 41.2 / 40.2 |
| Style profile | 49.5 / 54.2 | 41.8 / 42.3 | 46.4 / 48.2 | 41.7 / 38.4 |
| User messages | Up to 5 | 6 to 8 | 9 or more |
|---|---|---|---|
| Single-turn (39, 24, and 26 tasks) | |||
| Goal-Prompted | 32.9 | 26.4 | 24.0 |
| SWE-Interact | 37.1 | 34.4 | 36.0 |
| SWE-Together | 51.1 | 49.1 | 49.3 |
| SWE-WhatIf | 55.4 | 50.9 | 50.2 |
| Style profile | 52.5 | 49.6 | 49.3 |
| Error | What the simulated user does |
|---|---|
| Premature | Asks for something before the point where the real user asked for it, or bundles requests that the real user made one at a time. |
| Extra | Asks for work that the real user never asked for (e.g., diffs, tests, commits, or approvals). |
| Passive | Lets the agent keep working where the real user stepped in with feedback or a correction, or keeps the session going after the real user stopped. |
| Omitted | Leaves out a request or detail of the real user that still applies to the agent’s work. A correction for a mistake that the agent did not make is not counted. |
| User Simulator | Turn-Level | Session-Level |
|---|---|---|
| Goal-Prompted | 42.2 | 44.4 |
| SWE-Interact | 26.0 | 28.7 |
| SWE-Together | 26.3 | 47.8 |
| SWE-WhatIf | 19.7 | 24.7 |
| Style profile | 19.3 | 24.7 |
| SWE-WhatIf | ||||||
|---|---|---|---|---|---|---|
| Model | Succ. | Msgs. | Score | DeepSWE | SWE-T. | Arena |
| Opus 5.5 | 65.3 | 6.4 | 81.6 | 68.4 | 61.5 | 17.0 |
| Sonnet 5.5 | 65.1 | 6.7 | 78.0 | 72.0 | 51.4 | 12.2 |
| GPT-6 Sol | 61.9 | 6.3 | 77.2 | 69.0 | 50.5 | 10.7 |
| GPT-6 Luna | 57.9 | 6.2 | 71.7 | 63.7 | 47.7 | 2.8 |
| Kimi K3 | 58.7 | 7.1 | 66.2 | 68.4 | 53.2 | 3.9 |
| Without Reactive Intents ( ) | With Reactive Intents ( ) | |||||
|---|---|---|---|---|---|---|
| Model | Succ. | Msgs. | Score | Succ. | Msgs. | Score |
| Opus 5.5 | 71.9 | 5.6 | 79.4 | 55.4 | 7.5 | 85.0 |
| Sonnet 5.5 | 71.6 | 6.7 | 70.6 | 55.1 | 6.8 | 89.2 |
| GPT-6 Sol | 71.9 | 5.5 | 86.3 | 46.6 | 7.5 | 63.2 |
| GPT-6 Luna | 67.5 | 5.7 | 76.9 | 43.4 | 6.9 | 63.7 |
| Kimi K3 | 63.6 | 6.0 | 68.2 | 51.1 | 8.7 | 63.0 |
| Agent | Predicted | Final |
|---|---|---|
| Claude Code sessions (7 participants) | ||
| Claude Opus 5.5 | 6 | 3 |
| Claude Haiku 4.5 | 0 | 2 |
| GPT-6 Sol (Codex) | 1 | 2 |
| Codex sessions (9 participants) | ||
| GPT-6 Sol | 4 | 4 |
| Theme | Description | Example | |
| What participants learned | |||
| Smaller models were enough | A smaller and cheaper model met the same goals as the larger one, or did better. | 12 | “can still do decent investigative work” (P11, on Claude Haiku 4.5) |
| Agents worked differently with the user | Agents differed in how they asked, listened, and explained, and this affected which one the participant preferred. | 8 | “felt the most thorough, the most like a team member with competence” (P13) |
| Beliefs about harnesses were tested | The replays confirmed or revised which harness the participant expected to be better. | 4 | “Claude Code still outdoes ChatGPT.” (P12) |
| How agents are steered matters | What the agent is told, and how early, shapes the result. | 2 | “All three needed more context for a reliable calculation.” (P05) |
| What participants will do differently | |||
User Simulation
User simulation models human behavior and interaction to test, train, and evaluate AI systems with synthetic users.
See all 142 papers