RehearsalBench: Can AI Predict What Happens When You Change a Company?
Organizations: Afterflow Inc. · Independent contributor
Abstract
Organizations increasingly ask AI systems what will happen if they change how they operate: which work to automate, which team to resize, what a voice agent will do to waiting times. We introduce RehearsalBench, a benchmark that measures whether AI models can diagnose an operation, choose an intervention, and forecast its consequences. Each model is placed in a simulated bank contact center fitted to real call-level records, sees only what a manager would see, and submits a plan and forecasts that are then played out across 200 seeded futures and three plausible versions of the organization. Decisions are scored in dollars at the stakeholders' stated valuations; forecasts are scored on the effect of each change relative to doing nothing. Ten models made 280 attempts across 14 organizations. The best model, GPT-6 Astra, improved on doing nothing by 6.8% of the no-change cost and made 77% less forecast error than a forecaster who always predicts no effect. On average, every model forecast the direction of every specified effect correctly, but models erred on the size of the same effects in the same direction, overstating a voice agent's effect on repeat contacts about five-fold. No model ever recommended holding off when doing nothing was the right choice. To test external validity, twelve models forecast 33 real organizational changes with known outcomes. Under verdict rules fixed in advance, the benchmark's forecast ranking agreed with the real-world ranking (Spearman , 95% lower bound 0.50). We report the result alongside an inconclusive first test, the analysis choices made after outcomes were seen, and evidence that part of the agreement reflects how well models size their uncertainty.
Figures & tables
| Data | Role | Used in |
|---|---|---|
| Bank A, Jan–Jun 1999 | Estimate call statistics (calibration) | Section 3.1 |
| Bank A, Jul–Dec 1999 | Held-out validation: same bank, later period | Section 3.3 |
| Bank B, consumer loans, 2001–2003 | External validation: different bank | Section 3.3 |
| 14 fictional banks | Benchmark test set, generated by the validated simulator | Sections 6 – 7 |
| 33 real organizational changes | Retrospective test of the forecast ranking | Section 8 |
| Intervention | What it does | Changes |
|---|---|---|
| Staffing | Changes how many staff are on duty in each interval | |
| Opening hours | Extends or shortens the hours in which calls are answered | |
| Routing | Changes how waiting calls are assigned to staff | |
| Callbacks | Offers waiting callers a call back instead of holding | , |
| AI voice agent | Resolves some calls without staff | , , |
| AI assistance for staff | Helps staff handle calls faster |
| Model | Short name | Developer | Access route |
|---|---|---|---|
| GPT-6 Astra | Astra | OpenAI | Developer API |
| GPT-6.1 Sol | Sol | OpenAI | Developer API |
| GPT-6 Luna | Luna | OpenAI | OpenRouter |
| Claude Sonnet 5.5 | Sonnet | Anthropic | OpenRouter |
| Claude Fable 5.1 | Fable | Anthropic | Both |
| Claude Haiku 5.5 | Haiku | Anthropic | Developer API |
| Outcome | Forecast | Simulated outcome |
|---|---|---|
| Abandonment rate (%) | 11.0 (9.8–12.4) | 10.6 |
| Mean wait (seconds) | 40 (33–48) | 38 |
| Staff cost (USD) | 306,185 | 305,970 |
| Criterion | Threshold | Result |
|---|---|---|
| Rank correlation ( ) | 0.92 | |
| Lower bound of 95% bootstrap interval | 0.50 | |
| Lowest after omitting one source | 0.73 (US bank omitted) | |
| Lowest after omitting one model | 0.90 | |
| Models differ on real changes (permutation test) | ||
| excluding recognized changes † | 0.83 |
AI-Assisted Decision Making
AI-assisted decision making studies how computational systems can inform human choices while accounting for context, uncertainty, and consequences.
See all 157 papers