cs.AIAug 19, 2026

FM-Bench: A Benchmark for Long-Horizon Management with Competing Agents

Authors: Tianyou Wang, Chongyang Gao, Kezhen Chen, Dong Chen, Yinghao He, Donghan Li, Wangcheng Xu, Hongjiu Zhang, 

Organizations: AnalogyAI

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Language model agents now execute bounded tasks reliably. Whether they can sustain effective decision-making over long horizons, where actions have cumulative consequences and the environment responds to their choices, remains largely unmeasured. FM-Bench (Football Management Benchmark) measures this. An LLM agent runs a football club for 20 in-game years through 26 tools and roughly 340 to 400 decision stops. It drafts a squad on the same budget as every rival, trades players, negotiates contracts, invests in facilities and youth, sets lineups, and answers to a board that can fire it, while a deterministic engine accumulates every year into one final score with no LLM judge or human rater. The solo track plays each of 15 frontier models against a frozen scripted world, and the Arena places the same models plus a scripted anchor in one shared 20-year world; to our knowledge, the first head-to-head evaluation at this scale. We measure six behavioral capabilities behind the score. Across three seeds, all 15 models complete every horizon while the blind scripted baselines die out in most of theirs, and claude-fable-5 tops the solo board on mean score and the Arena, where the title nonetheless rotates among ten models. Neither scale, price, nor vendor predicts the order; the order settles only late in the horizon, and the best first-play human lands only at the bottom of the model board. What separates the models is managerial behavior rather than computation. Higher-scoring models reduce slow-payoff investment near the end, keep cash invested rather than idle, and open renewals well before the deadline, while token spend predicts nothing. No model learns the market's hidden prices from hundreds of rejected bids, and self-managed memory fails in two opposite modes: an archive that only grows or a plan rewritten every season. Code is available at https://github.com/Analogy-AI/fm-bench.

Figures & tables

Figure 1: How a run works. One run spans 20 in-game years and roughly 340 to 400 decision stops, ending in a single composite score. At each stop, the clock freezes and the agent takes as many tool-call turns as it wants (read →\to act →\to advance ). The agent’s only cross-stop memory is its self-written notebook (§ 3.2 ); the environment side holds the club (net worth, squad with permanently hidden true ability, a board that can fire the manager) and the world (15 rival clubs, a counter-adaptive market, delayed-payoff investments).
demandmechanismtargeted abilitycalibration lever
Hidden informationscout bands with a permanent per-scout bias; hidden player traits (injury proneness, development rate, aging onset); hidden asks in negotiationvaluation calibration under noisebandwidth, bias sigma
Cumulative consequencesUpside: youth development and facility investment pay off over years. Downside: insolvency ends in administration (points penalty, forced sales), confidence collapse ends in firing; both compound yet remain recoverable. Honors accrue into the final score season by seasonlong-range credit assignment; trend recognition, loss-cuttinggrowth and aging curves; spiral thresholds, warning cadence; honors weights
Counter-adaptive marketrejected bids raise the hidden ask; repeat-pair markups (bargains included); anti-inversion counters; negotiation cooldowns; per-seed mispricingstrategy adaptationmarkup and decay rates
Multi-objective pressurethe board judges results and financial discipline jointly; season targets scale with squad strengthmulti-constraint balancingtarget cushion, board patience
Table 1: The four demands: the mechanism that instantiates each, the ability it targets, and the calibration lever that tunes it.
seatSfinalS_{\text{final}}tokens
oracle (privileged)95.54±4.6895.54\pm 4.68—
claude-fable-590.94±5.2090.94\pm 5.2024M
kimi-k2.688.49±0.1588.49\pm 0.1587M
gpt-5.6-terra86.66±1.2086.66\pm 1.2028M
gpt-5.6-sol86.40±2.5386.40\pm 2.5362M
muse-spark-1.183.19±12.0383.19\pm 12.0373M
Table 2: The Solo track results. The four non-LLM seats are reference policies. The oracle reads true hidden state through the same interface (§ 4.1 , Appendix 13 ); heuristic is a disciplined blind script, idle never acts, and random acts arbitrarily (Appendix 10 ).
Figure 2: Per-season solo trajectories for all 15 models in four channels. Lines are the mean over three seeds, shading spans the best and worst seed, and the five models with the highest mean are colored and named in the legend; the grey lines are the remaining ten models.
playerSfinalS_{\text{final}}settledeathsstopstime
H174.64completed039410 h
H259.95completed03984 h
H35.39fired, t=10.8t{=}10.842024 h
H42.47fired, t=10.7t{=}10.741985 h
H50.41fired, t=7.8t{=}7.841651.5 h
H6−3.78-3.78fired, t=12.3t{=}12.342372 h
Table 3: The six first-play human runs (§ 4.2.2 ). Deaths are capped revivals spent before the settling firing; tt is the game-year at which a fired run settled.
Figure 3: Credit assignment. Left: discretionary spend bucketed by payoff horizon, with run totals at right. Right: mean season-end idle-cash ratio vs. final score, rs=−0.50r_{s}=-0.50 , negative on every seed.
Figure 4: Proactive control. Left: distribution of contract months remaining when each renewal negotiation episode opens (rows sorted by final score; shaded band = last-minute, ≤6\leq 6 months). Right: per-model median lead vs. final score, rs=+0.45r_{s}=+0.45 , positive on every seed.
Figure 5: Capability matrix over the 15 solo models, six axes. Each column rank-normalizes one behavioral metric averaged over the three seeds (1 = best of 15; construction in Appendix 15 ), and rows are sorted by mean final score. The winner is a generalist, with no axis below 0.79 and mean 0.94, rather than the leader of any single one; the mid-table pairs a genuine strength with a decisive gap; and the bottom of the board is weak on most axes at once.
#seatSfinalS_{\text{final}}deathssettletokens
1claude-fable-576.260completed∼{\sim} 34M
2muse-spark-1.162.470completed∼{\sim} 80M
3deepseek-v4-pro52.090completed∼{\sim} 111M
4glm-5.251.440completed∼{\sim} 86M
5grok-4.550.890completed∼{\sim} 138M
6gpt-5.6-sol47.940completed∼{\sim} 78M
Table 4: The Arena results.
Figure 6: Arena trajectories, all 16 seats. The six highest-scoring seats are colored and named in the legend; the grey lines are the remaining ten seats. Left: composite SfinalS_{\text{final}} at each season end. Right: league-table position.
#seatproviderrole
1claude-fable-5 ( Anthropic, 2026a )Anthropicnew flagship
2claude-opus-4.8 ( Anthropic, 2026b )Anthropicprevious flagship
3claude-haiku-4.5 ( Anthropic, 2025 )Anthropicsmall tier
4gpt-5.6-sol ( OpenAI, 2026 )OpenAInew flagship
5gpt-5.6-terra ( OpenAI, 2026 )OpenAIbalanced tier
6claude-sonnet-5 ( Anthropic, 2026c )Anthropicworkhorse mid (4-point family curve)
Table 5: The roster: 15 frontier LLM seats plus one scripted anchor, July 2026.
levereasymediumhard
scouting noise, multiplier on the belief sigma1.31.00.35
lineup rotates for fatiguenoyesyes
market days acted onevery 2ndeveryevery
gap required before buying an upgrade1.51.00.25
wage budget, multiplier on the board target0.921.01.10
Table 6: The five competence levers. Medium is the neutral setting.
Figure 7: Final score per model, mean with standard deviation over the three seeds, individual seeds shown as points. Dashed and dotted lines mark the oracle and heuristic means. Stability separates models that a single-seed board cannot tell apart, from kimi-k2.6 at 0.15 to claude-haiku-4.5 at 22.73.
seatscoresdprofile
claude-fable-590.95.2generalist, no axis below 0.79; best bidder in the field at 9 offers per signing and the sharpest endgame reduction
kimi-k2.688.50.1the steadiest model in the campaign, scoring within a third of a point on three different worlds
gpt-5.6-terra86.71.2minimalist: lowest token spend in the field (28M) after the winner and an early endgame reduction, but the shortest renewal leads among the top models
gpt-5.6-sol86.42.5archivist memory (similarity 0.91, append-only) and the weakest bidding of the top group (45 offers per signing)
muse-spark-1.183.212.0strong on two worlds and 20 points weaker on the third; one of two models that raise long-horizon spending at the end
glm-5.283.20.9stable across seeds but loose with cash (135% idle ratio)
Table 7: Per-model profiles over the three seeds. Score is the mean and SD the standard deviation of SfinalS_{\text{final}} ; axis extremes come from the matrix of Figure 5 .
Figure 8: Transfer offers per season, solo track (rows sorted by final score; totals at right). The winner’s 9 offers are all in the first two seasons; the field-wide offer-to-completion conversion is 2–5%.
Figure 9: Transfer offers per season in the Arena (approximate year mapping; daggered models settled early). The winner’s offer count rises from 9 (solo) to 91: the same model under a different, correct judgment of market liquidity.
Figure 10: Season-by-season cumulative API cost vs. composite score, solo track. Steep curves convert dollars into score throughout; flat long curves do not. The full 20-year runs span 18to18 to 191.
Figure 11: The failed metric: share of stops carrying an engine warning, by type (left) and against final score (right), means over the three seeds. The per-seed coefficient runs +0.45+0.45 / +0.07+0.07 / −0.78-0.78 and averages to nothing, so warning exposure is not reported as a capability. Zero warnings conflates genuine anticipation with do-nothing conservatism.
Figure 12: Both panels are means over the three seeds. Left: youth harvest rate (share of promotions later fielded in ≥5\geq 5 lineups); the correlation that looked informative on seed 1 does not survive three ( −0.52-0.52 / −0.18-0.18 / +0.04+0.04 ). Right: endgame shift in long-horizon actions per season, years 2–16 (blue) vs. 17–20 (red); endgame-aware models reduce late investment, one ramps up.
Figure 13: Memory-curation regimes on the solo track. Left: season-to-season TF–IDF cosine similarity of each model’s reconstructed season-end notebook. Right: notebook size over the run. The right panel decodes the left: the same “high consistency” is a 200k-character append-only archive for one model and a 3–6k curated document for the winner.
Figure 14: Compute allocation. Mean tokens per run against mean solo score, both over the three seeds, on a log axis. A null result ( rs=−0.19r_{s}=-0.19 , p=0.50p=0.50 ) across a sevenfold spend range.
Figure 15: The asset channels behind Figure 6 : per-season net worth and squad value for all 16 Arena seats, the six highest-scoring seats colored as in Figure 6 and the grey lines the remaining ten (numeric snapshots in Table 9 ). The winner’s mid-table seasons are asset accumulation (the net-worth curve keeps climbing while the standings do not), and the conservative cash-holders’ net worth grows on cash the composite discounts.
seatY5Y10Y15Y20
claude-fable-54th ⋅\cdot 140H ⋅\cdot 293M ⋅\cdot 130M ⋅\cdot 33.31st ⋅\cdot 480H ⋅\cdot 470M ⋅\cdot 218M ⋅\cdot 50.31st ⋅\cdot 900H ⋅\cdot 1048M ⋅\cdot 443M ⋅\cdot 65.91st ⋅\cdot 1400H ⋅\cdot 1742M ⋅\cdot 634M ⋅\cdot 94.6
muse-spark-1.13rd ⋅\cdot 120H ⋅\cdot 302M ⋅\cdot 158M ⋅\cdot 31.11st ⋅\cdot 460H ⋅\cdot 468M ⋅\cdot 196M ⋅\cdot 48.71st ⋅\cdot 960H ⋅\cdot 1037M ⋅\cdot 457M ⋅\cdot 66.51st ⋅\cdot 1460H ⋅\cdot 1298M ⋅\cdot 315M ⋅\cdot 91.2
grok-4.54th ⋅\cdot 60H ⋅\cdot 289M ⋅\cdot 169M ⋅\cdot 23.91st ⋅\cdot 460H ⋅\cdot 318M ⋅\cdot 107M ⋅\cdot 46.61st ⋅\cdot 880H ⋅\cdot 618M ⋅\cdot 225M ⋅\cdot 58.11st ⋅\cdot 1380H ⋅\cdot 1106M ⋅\cdot 634M ⋅\cdot 89.7
kimi-k2.63rd ⋅\cdot 120H ⋅\cdot 344M ⋅\cdot 116M ⋅\cdot 31.91st ⋅\cdot 440H ⋅\cdot 430M ⋅\cdot 99M ⋅\cdot 46.71st ⋅\cdot 940H ⋅\cdot 668M ⋅\cdot 276M ⋅\cdot 61.41st ⋅\cdot 1440H ⋅\cdot 733M ⋅\cdot 400M ⋅\cdot 88.5
gemini-3-flash3rd ⋅\cdot 160H ⋅\cdot 247M ⋅\cdot 165M ⋅\cdot 32.81st ⋅\cdot 500H ⋅\cdot 339M ⋅\cdot 111M ⋅\cdot 42.11st ⋅\cdot 1000H ⋅\cdot 675M ⋅\cdot 400M ⋅\cdot 61.51st ⋅\cdot 1420H ⋅\cdot 812M ⋅\cdot 618M ⋅\cdot 86.6
qwen3.7-max3rd ⋅\cdot 120H ⋅\cdot 404M ⋅\cdot 127M ⋅\cdot 34.01st ⋅\cdot 360H ⋅\cdot 582M ⋅\cdot 140M ⋅\cdot 48.01st ⋅\cdot 780H ⋅\cdot 917M ⋅\cdot 256M ⋅\cdot 62.31st ⋅\cdot 1200H ⋅\cdot 1305M ⋅\cdot 248M ⋅\cdot 86.1
Table 8: Solo season snapshots (seed 1). Cell format: position ⋅\cdot honors ⋅\cdot net worth ⋅\cdot squad value ⋅\cdot score. Rows sorted by final score.
seatY5Y10Y15Y20
claude-fable-52nd ⋅\cdot 180H ⋅\cdot 326M ⋅\cdot 105M ⋅\cdot 47.45th ⋅\cdot 240H ⋅\cdot 394M ⋅\cdot 196M ⋅\cdot 56.45th ⋅\cdot 440H ⋅\cdot 568M ⋅\cdot 250M ⋅\cdot 71.33rd ⋅\cdot 620H ⋅\cdot 507M ⋅\cdot 298M ⋅\cdot 76.3
muse-spark-1.18th ⋅\cdot 0H ⋅\cdot 260M ⋅\cdot 84M ⋅\cdot 18.57th ⋅\cdot 40H ⋅\cdot 197M ⋅\cdot 117M ⋅\cdot 24.02nd ⋅\cdot 160H ⋅\cdot 272M ⋅\cdot 222M ⋅\cdot 46.21st ⋅\cdot 420H ⋅\cdot 306M ⋅\cdot 246M ⋅\cdot 62.5
deepseek-v4-pro9th ⋅\cdot 0H ⋅\cdot 279M ⋅\cdot 91M ⋅\cdot 20.010th ⋅\cdot 140H ⋅\cdot 342M ⋅\cdot 91M ⋅\cdot 44.54th ⋅\cdot 200H ⋅\cdot 397M ⋅\cdot 122M ⋅\cdot 52.25th ⋅\cdot 200H ⋅\cdot 363M ⋅\cdot 160M ⋅\cdot 52.1
glm-5.24th ⋅\cdot 40H ⋅\cdot 275M ⋅\cdot 107M ⋅\cdot 30.31st ⋅\cdot 160H ⋅\cdot 335M ⋅\cdot 78M ⋅\cdot 46.18th ⋅\cdot 200H ⋅\cdot 346M ⋅\cdot 71M ⋅\cdot 49.36th ⋅\cdot 200H ⋅\cdot 356M ⋅\cdot 125M ⋅\cdot 51.4
grok-4.57th ⋅\cdot 20H ⋅\cdot 197M ⋅\cdot 134M ⋅\cdot 21.16th ⋅\cdot 260H ⋅\cdot 145M ⋅\cdot 145M ⋅\cdot 40.07th ⋅\cdot 320H ⋅\cdot 108M ⋅\cdot 185M ⋅\cdot 36.48th ⋅\cdot 520H ⋅\cdot 139M ⋅\cdot 199M ⋅\cdot 50.9
gpt-5.6-sol6th ⋅\cdot 100H ⋅\cdot 258M ⋅\cdot 115M ⋅\cdot 36.94th ⋅\cdot 120H ⋅\cdot 239M ⋅\cdot 86M ⋅\cdot 36.81st ⋅\cdot 240H ⋅\cdot 281M ⋅\cdot 229M ⋅\cdot 52.32nd ⋅\cdot 280H ⋅\cdot 182M ⋅\cdot 294M ⋅\cdot 47.9
Table 9: Arena season snapshots (seed 7, one shared world); the 15 models, as on the public page (the scripted anchor settled at t≈2.6t\approx 2.6 y). Cell format as in Table 8 . Seats that exhausted the revival cap keep their frozen score afterward, visible as repeated values ( claude-haiku-4.5 from Y15, gemini-3.5-flash from Y10). Rows sorted by final score.
toolwhat it does
Query (12), charged to the 30-query budget:
get_club_overviewthe agent’s club at a glance: division, cash, board, facilities, tactics
get_squadthe senior squad with ability bands, form, contracts, wages
get_playerdetailed view of any player by id
get_league_tableleague standings
get_fixturesthe club’s season fixtures and results
Table 10: The 26-tool interface: 12 query tools, 11 action tools, 2 notebook tools, and 1 control tool. Descriptions follow the tool schemas served to the agent.
stop (day of the 360-day year)what the world wants decided
Scheduled, 13 per season:
Preseason board (2)the board sets the season target; plan the year
Summer window plan (5)the transfer window opens; build the squad
Summer deadline (59)last actions before the window closes
Lineup lock (63)commit lineup and tactics before round 1
Monthly report (90, 120, 150, 240)finances and form check-in
Table 11: The decision-stop calendar: 13 scheduled stops per season on the 360-day year, plus event-driven stops the world raises unsolicited. Several types can fire at one stop.
layerwho writes itcontentsdelivery
Packet auto-contextenginedashboard digest, inbox (all events since last stop), agent’s last 10 engine-mutating actions, pending offerspushed free into every stop packet
Archiveenginehonors / season / transfer history of the whole runget_history query tool, on demand
Notebookthe agentintentions, reasoning, plans (“why I bought him”, “sell Y in winter”)injected into every packet; written via append_note / rewrite_notes
Audit logrunnerevery successful actionnever fed back : replay/anti-cheat/resume only
Table 12: The four memory layers (§ 3.2 ). What happened is recorded automatically; what it means and what to do next must be written by the agent itself.
quantitymedianrange
decision stops374341–390
of which scheduled∼260{\sim}260 (13/season)rest event-triggered (Table 11 )
matches simulated in the world4,800(30 rounds ×\times 8 matches ×\times 20 seasons)
model turns (API calls)∼1,760{\sim}1{,}7601,184–6,969
state-changing actions—849–2,198
tokens (manifest accounting)61M24M–274M
Table 13: One 20-year run, by the numbers: medians and ranges across the 15 solo models.
More in this topic

Long-Horizon Agent Evaluation

Long-Horizon Agent Evaluation measures whether AI agents can plan, act, remember, and stay reliable across extended tasks and changing environments.

See all 357 papers

Explore similar work

CardsList
Sep 28, 2026

FromPitch2Board: Benchmarking LLM Agents in Long-Horizon Football Management

Long-horizon agent benchmarks typically report how far an agent progresses, but do not identify whether its performance comes from the foundation model, scaffold, responsibility scope, match-control granularity, or horizon. We introduce FromPitch2Board, a deterministic football-management benchmark that studies five configurable factors through controlled comparisons on a single simulator, using paired seeds and a frozen calibration. We evaluate four foundation models and four agent scaffolds. In the Model Track, Coach points Z-scores span 0.19, while Manager points Z-scores span 0.68, with GPT-5.6 showing a sharp rise in passivity under responsibility expansion. Its responsibility ladder rises from 46.1 to 58.1 points with recruitment, then falls to 46.8 under full management, localizing the regression to the final responsibility boundary. Across that boundary, its skipped-decision rate rises from 1.1% to 57.9%. Within the Flash-Pro pair crossed across every scaffold, scaffold choice changes Manager points Z-scores by up to 0.48 relative to the fixed stateless scaffold. The 3Y cohort shows a directional reversal in mean ranking between years one and three, while a selected Claude Code+Pro configuration peaks in year three and remains below that peak, showing that responsibility scope and horizon expose behavior changes that a single headline score conceals.

Peiyu Zang

School of Mathematical Sciences, Beijing Normal University · CoRe Lab, Institute for Artificial Intelligence, Peking University

Long-Horizon Agent EvaluationAI Agent BenchmarksLong-Horizon LLM Agents
AskPDF
Apr 30, 2026

KellyBench: A Benchmark for Long-Horizon Sequential Decision Making

Language models are saturating benchmarks for procedural tasks with narrow objectives. But they are increasingly being deployed in long-horizon, non-stationary environments with open-ended goals. In this paper we introduce KellyBench, an environment for evaluating sequential decision-making in sports betting markets. Agents are placed in a sequential simulation of the 2023-24 English Premier League season and tasked with maximising their long-term bankroll growth. They are given detailed historical data, including advanced statistics, lineups, and public odds. To succeed they must build machine learning models, identify edge in public markets, and adapt as the environment changes over time. We find that all frontier models evaluated lose money on average over the course of the season for five seeds. The best performing model achieves an average return of -8%, and many models experiencing ruin across seeds. To judge strategy sophistication, we use a human expert rubric to grade each model and find their approaches to be unsophisticated compared to human baselines; Claude Opus 4.6 achieves a rubric score of 26.5%, which means there is significant room for improvement. KellyBench is available as an open-access API endpoint at https://openreward.ai/GeneralReasoning/KellyBench.

Thomas Grady, Kip Parker, Iliyan Zarov +3

General Reasoning, Inc.

Sequential Decision MakingLong-Horizon Agent EvaluationAI Agent BenchmarksLLM AgentsFinancial Decision-Making
AskPDF
Apr 1, 2026

YC-Bench\texttt{YC-Bench}: Benchmarking AI Agents for Long-Term Planning and Consistent Execution

As LLM agents tackle increasingly complex tasks, a critical question is whether they can maintain strategic coherence over long horizons: planning under uncertainty, learning from delayed feedback, and adapting when early mistakes compound. We introduce YC-Bench\texttt{YC-Bench}, a benchmark that evaluates these capabilities by tasking an agent with running a simulated startup over a one-year horizon spanning hundreds of turns. The agent must manage employees, select task contracts, and maintain profitability in a partially observable environment where adversarial clients and growing payroll create compounding consequences for poor decisions. We evaluate 12 models, both proprietary and open source, across 3 seeds each. Only three models consistently surpass the starting capital of $200K, with Claude Opus 4.6 achieving the highest average final funds at $1.27 M, followed by GLM-5 at $1.21 M at 11×\times lower inference cost. Scratchpad usage, the sole mechanism for persisting information across context truncation, is the strongest predictor of success, and adversarial client detection is the primary failure mode, accounting for 47%47\% of bankruptcies. Our analysis reveals that frontier models still fail through distinct failure modes such as over-parallelization, demonstrating the capability gaps for long-horizon performance. YC-Bench\texttt{YC-Bench} is open-source, reproducible, and configurable.

Muyu He, Adit Jain, Anand Kumar +4

Long-Horizon Agent EvaluationAI Agent BenchmarksLLM Agent PlanningLLM Agent Reliability
AskPDF