As LLM agents tackle increasingly complex tasks, a critical question is whether they can maintain strategic coherence over long horizons: planning under uncertainty, learning from delayed feedback, and adapting when early mistakes compound. We introduce YC-Bench, a benchmark that evaluates these capabilities by tasking an agent with running a simulated startup over a one-year horizon spanning hundreds of turns. The agent must manage employees, select task contracts, and maintain profitability in a partially observable environment where adversarial clients and growing payroll create compounding consequences for poor decisions. We evaluate 12 models, both proprietary and open source, across 3 seeds each. Only three models consistently surpass the starting capital of $200K, with Claude Opus 4.6 achieving the highest average final funds at $1.27 M, followed by GLM-5 at $1.21 M at 11× lower inference cost. Scratchpad usage, the sole mechanism for persisting information across context truncation, is the strongest predictor of success, and adversarial client detection is the primary failure mode, accounting for 47% of bankruptcies. Our analysis reveals that frontier models still fail through distinct failure modes such as over-parallelization, demonstrating the capability gaps for long-horizon performance. YC-Bench is open-source, reproducible, and configurable.
Figures & tables
Figure 1: Overview of YC-Bench . The agent interacts with the environment through CLI commands ( blue ) and receives structured observations ( green ). The environment tracks observable state (tasks, employees, finance, prestige, client trust) and one hidden element: adversarial clients whose work inflation must be inferred from repeated task failures.
Figure 2: Out of the 12 models that we benchmark on YC-Bench , 5 models are profitable and only 3 turn a substantial profit ( 5× profit). The figure plots the funds across time averaged across three seeds for each model. Comprehensive results can be found in Appendix D .
Figure 3: We observe that better models are able to build client trust over time by strategically selecting clients. What is surprising is smaller distilled models (Sonnet-4.6) do worse than contemporaries (Gemini-3-Flash) unlike VB.
Figure 4: Analyzing how the models deal with adversarial clients, who have appealing rewards when accepting a task but have a lot more work than claimed (scope creep).
Figure 5: Other than identifying adversarial clients there are two other primary task failure modes, especially wrong employee assignment. We observe that Kimi-K.5 is the most cost efficient, whereas the second highest ranking model, GLM-5 is substantially ( 10× ) better than Opus-4.6 on cost efficiency.
Figure 6: Representative failure moments for four models. Each panel shows the scratchpad state, agent action, and outcome. From left to right: Opus violates its own blacklist; Flash blindly accepts adversarial tasks through a rigid loop; Sonnet writes a correct rule and immediately ignores it; Grok diagnoses its bankruptcy but takes no corrective action.
Appendix figures & tables8 assets
Supplementary material from the paper’s appendix.
Appendix
Category
Command
Effect
Observe
company status
Funds, prestige, payroll
Observe
employee list
Names, tiers, salaries, productivity
Observe
market browse
Available tasks with client, reward, domains
Observe
task list
Accepted tasks with status and progress
Observe
task inspect --task-id T
Per-domain progress, deadline, assignments
Observe
client list
Client trust levels and tiers
Appendix
Table 1: Agent action space. All actions are CLI commands with arguments constructed as free-form text. Multiple actions may be issued per turn.
Category
Parameter
Value
Description
Simulation
Horizon
1 yr
Evaluation period
Simulation
Auto-advance
5 turns
Idle turns before forced advance
Simulation
Business hours
9–18
Weekdays only
Workforce
Employees
8
Fixed roster
Workforce
Initial funds
$200K
Starting capital
Workforce
Salary bump
1%/task
All assigned employees
Appendix
Table 2: Default configuration hyperparameters for YC-Bench .
Figure 7: Average final funds across three seeds for all 12 models and the greedy baseline, sorted by performance. Only three models (Claude Opus 4.6, GLM-5, GPT-5.4) consistently surpass the starting capital.
Figure 8: Monthly funds trajectory across three seeds.
Figure 9: Average number of tool calls per run for the top three and bottom three performing models, grouped by command type. Top models execute 3-4x more actions overall, with the largest gaps in task inspection and scratchpad usage.
Model
SP/100T
Inspect/Accept
Avg Conc.
Cmd/Turn
Claude Opus 4.6
5.6
1.10
2.0
1.41
GLM-5
2.7
0.26
1.5
2.16
GPT-5.4
10.6
1.43
2.5
2.63
Kimi-K2.5
0.6
0.00
1.3
1.18
Gemini 3 Flash
0.2
0.11
1.2
0.24
Gemini 3.1 Flash Lite
0.9
0.02
1.2
2.16
Appendix
Table 3: Behavioral statistics per model, averaged across 3 seeds and sorted by average final funds. The top three models use the scratchpad and task inspection significantly more than others. Sonnet’s average concurrency of 7.2 is a clear outlier, consistent with its high understaffing failure rate. GPT-5.4 Mini and Nano execute fewer than 0.1 commands per turn despite running thousands of turns, indicating most turns produce reasoning without action. SP/100T : scratchpad writes per 100 turns. Inspect/Accept : ratio of task inspect to task accept calls. Avg Conc. : mean simultaneously active tasks. Cmd/Turn : commands executed per turn.
Figure 10: Average final funds vs. K (number of retained turns), across three seeds.
Model
Bankrupt
Cost ($)
Tokens
Runtime (min)
Claude Opus 4.6
0/3
86.07 ± 41.17
16.7M
70 ± 36
GLM-5
0/3
7.62 ± 1.06
8.3M
40 ± 7
GPT-5.4
0/3
23.08 ± 7.07
9.5M
19 ± 5
Kimi-K2.5
1/3
1.79 ± 2.74
3.8M
21 ± 30
Gemini 3 Flash
0/3
1.83 ± 0.32
3.8M
6 ± 1
Gemini 3.1 Flash Lite
1/3
0.59 ± 0.22
2.6M
6 ± 2
Appendix
Table 4: Computational cost per run, averaged across three seeds ( μ±σ ). Models are sorted by average final funds.
Language model agents are becoming proficient executors at isolated, short-horizon tasks such as software engineering and customer service. Yet real-world challenges require a combination of sophisticated skills that remain largely untested in agents: (1) navigating long horizons amid uncertainty; (2) acquiring information in noisy environments; (3) adapting to a changing world; (4) orchestrating multiple moving parts toward a coherent goal. We introduce CEO-Bench, which evaluates these capabilities together by simulating a representative real-world task: operating a startup for 500 days. An agent manages pricing, marketing, budgeting, and many other aspects of a fictional company through a programmable Python interface, operating in the same environment and facing the same challenges as a human CEO. Success demands analyzing noisy, interconnected business databases, translating signals into sound strategy, and coordinating many decisions with programming. The strongest agents write sophisticated code that simulates customer cohorts to forecast future cash and mines negotiation history to uncover hidden customer preferences. Even so, most state-of-the-art models struggle in this environment. Only Claude Opus 4.8 and GPT-5.5 finish above the $1M starting balance, and neither consistently turns a profit. CEO-Bench takes a first step toward measuring the intelligence required to drive sustained, adaptive progress over time.
Large language model (LLM) agents are increasingly tested on complex tasks, but their ability to allocate scarce resources over long horizons remains unclear. Unlike reactive tasks with immediate feedback, this setting requires agents to make binding commitments under partial observability, delayed consequences, hard resource budgets, and shifting dynamics. We introduce EnterpriseArena, a 132-month CFO simulator that evaluates long-horizon resource allocation under uncertainty in a FinTech lending firm. Agents must manage liquidity, close books, gather costly signals, and request equity or debt financing across changing macroeconomic regimes. The simulator is built from transformed firm-level financial data, anonymized business documents, decade-scale macroeconomic and industry signals, and expert-validated operating rules. Experiments across 23 LLMs and four agent frameworks show that current agents remain far from robust: only 15.4% of trials survive the full horizon, larger models do not reliably outperform smaller ones, and failures cascade across observation, action timing, and capital sizing. These findings establish long-horizon resource allocation under uncertainty as a distinct capability gap for LLM agents.
Yi Han, Yan Wang, Lingfei Qian +12
Georgia Institute of Technology · The Fin AI · Stevens Institute of Technology +7
As LLM agents become capable of increasingly long-horizon tasks, evaluating their performance in economic systems is becoming increasingly important. Unlike existing benchmarks that primarily evaluate a single agent interacting with a passive environment, economic systems are inherently multi-agent, requiring autonomous agents to communicate, negotiate, and transact while pursuing their own objectives over extended periods. We introduce CoffeeBench, a benchmark for evaluating LLM agents in a long-horizon multi-agent economy composed of heterogeneous firms. In CoffeeBench, two farmers, two roasters, and two retailers autonomously operate their businesses over a 90-day simulation, each seeking to maximize cumulative net income through communication and transactions while managing cash, inventory, and pricing. The evaluated model controls one coffee roaster, while the remaining firms are controlled by fixed reference agents. Across several recent open-weight and proprietary LLMs, all models outperform a passive baseline that takes no actions, with most achieving positive net income. Analysis of agent behavior reveals substantial differences in long-horizon economic interaction: higher-performing models communicate more actively with other firms, whereas ClaudeHaiku4.5 exhibits an idle-drift failure mode, repeatedly choosing inaction despite producing coherent assessments and plans. We release our code and agent trajectories to support future research.