cs.AINov 17, 2026

RehearsalBench: Can AI Predict What Happens When You Change a Company?

Authors: Mika Siddiqui, Tharaka Perera

Organizations: Afterflow Inc. · Independent contributor

Abstract

Organizations increasingly ask AI systems what will happen if they change how they operate: which work to automate, which team to resize, what a voice agent will do to waiting times. We introduce RehearsalBench, a benchmark that measures whether AI models can diagnose an operation, choose an intervention, and forecast its consequences. Each model is placed in a simulated bank contact center fitted to real call-level records, sees only what a manager would see, and submits a plan and forecasts that are then played out across 200 seeded futures and three plausible versions of the organization. Decisions are scored in dollars at the stakeholders' stated valuations; forecasts are scored on the effect of each change relative to doing nothing. Ten models made 280 attempts across 14 organizations. The best model, GPT-6 Astra, improved on doing nothing by 6.8% of the no-change cost and made 77% less forecast error than a forecaster who always predicts no effect. On average, every model forecast the direction of every specified effect correctly, but models erred on the size of the same effects in the same direction, overstating a voice agent's effect on repeat contacts about five-fold. No model ever recommended holding off when doing nothing was the right choice. To test external validity, twelve models forecast 33 real organizational changes with known outcomes. Under verdict rules fixed in advance, the benchmark's forecast ranking agreed with the real-world ranking (Spearman ρ=0.92ρ= 0.92, 95% lower bound 0.50). We report the result alongside an inconclusive first test, the analysis choices made after outcomes were seen, and evidence that part of the agreement reflects how well models size their uncertainty.

Figures & tables

Explore similar work

CardsList
  1. ForecastBench-Sim: A Simulated-World Forecasting Benchmark

    Jun 17, 2026Jaeho Lee, Nick Merrill, Ezra KargerForecasting BenchmarksProbabilistic Forecasting

  2. Generative AI, Managerial Expectations, and Economic Activity

    Oct 4, 2024Manish Jha, Jialin Qian, Michael Weber +1Time Series Forecasting