Frontier AI Forecasting Has a Measurement Problem: An Audit of Progress Evidence
Authors: Fabricio F Costa
Organizations: AIx4All, LLC, Sunnyvale, CA, USA · HCLTech, Santa Clara, CA, USA · Cancer Biology and Epigenomics Program, Stanley Manne Children’s Research Institute, Ann & Robert H. Lurie Children’s Hospital of Chicago, Northwestern University Feinberg School of Medicine, Chicago, IL, USA
Quantitative forecasts of frontier artificial intelligence often connect dated targets to trends in benchmark scores, training compute, release time, or expert belief. This paper audits whether the public measurement record supports those connections before another trend is fitted. I construct a frozen, event-centric record through 12 August 2026 with 62 selected systems, 12 versioned benchmarks, seven capability or impact criteria, 144 graded events, 27 source records, and 408 typed relations. The record is an audit sample, not a census. Only seven systems jointly observe estimated training compute and a METR 50 percent task horizon. Training compute is absent for 19 of 27 closed systems, including every selected closed release from 2026, while none of the 35 open-weight systems has a METR horizon observation. Benchmark succession creates a second break: a seven-system link from METR Time Horizon 1.0 to 1.1 has a log-scale slope of 1.206 (95 percent CI 1.021 to 1.390), whereas a six-system MMLU to MMLU-Pro comparison appears shift-like under logit and probit links but not under linear or logarithmic links. The observed bridges have about 80 percent power only for slope departures near 25 percent. Provenance is concentrated: 52 of 71 substantive quantitative events, or 73.2 percent, come from one measurement programme, and 76.1 percent are laboratory releases. A review of 56 methodological and empirical sources identifies 16 complementary measurement directions spanning resources, inference budgets, reliability, agentic work, safety, human preference, field outcomes, and forecast backtesting. No direction supplies a replacement scalar. The result is not that frontier AI forecasting is impossible, but that a defensible dated forecast is a claim about a versioned measurement system with explicit joins, protocols, links, and source dependence, not merely a fitted curve or calendar date.
Figures & tables
Figure 1: Method used in the audit. Public evidence is frozen under an explicit cutoff and sampling contract, normalized into versioned entities, represented as events with provenance, tested for missing joins and instrument changes, and then evaluated for forecastability. The lower topology shows the minimum graph structure used to connect models, events, benchmark versions, criteria and sources.
Figure 2: Observability of the selected 62-system record. Panel (a) shows the intended compute–horizon join as four mutually exclusive observation patterns: compute only, horizon only, both and neither. Panel (b) shows that missingness follows the closed/open-weight split and extends to non-METR and peer-reviewed measurements. Panel (c) displays the seven jointly observed systems; the fitted line is descriptive and is not interpreted causally or as a stable scaling law.
Figure 3: Benchmark lifecycle and source dependence. Panel (a) places benchmark versions and recorded methodology, correction, contamination and measurement-ceiling events on a common timeline. Panel (b) combines a Pareto view of the 71 substantive quantitative events, their cumulative source share and their venue-class composition. METR supplies 52 events (73.2%); the source Herfindahl index is 0.542, or 1.85 effective equally represented sources. Repeated rows from one programme are not independent replication.
Figure 4: Linking diagnostics. Panel (a) links METR Time Horizon 1.0 to 1.1 on a logarithmic scale. Panel (b) shows the six common MMLU/MMLU-Pro systems with paired uncertainty intervals in raw percentage space; MMLU-Pro changes content, choice structure and reasoning demands, so this is a cross-benchmark stress test rather than a claim of construct invariance. Panel (c) shows how the fitted slope changes with score representation; the vertical line marks a shift-like slope of one. Panel (d) shows that detectability depends jointly on anchor count and the observed score dispersion.
Bridge and scale
n
95% CI
80% MDE
Power observed
n for 10%
METR TH1.0 → TH1.1, log 2
7
[1.021, 1.390]
0.252
63%
31
MMLU → MMLU-Pro, logit
6
[0.793, 1.159]
0.248
6%
23
Table 1: Power of the observed linking designs. The final column is an illustrative projection assuming the same residual noise and transformed-score dispersion. Deliberately spanning the old score range can materially change the required sample.
Figure 5: Sixteen complementary measurement directions identified in the expanded source audit. Circle size and label give a literature-based design assessment from zero (absent) to three (comparatively mature) for public longitudinal record, model-level joinability, protocol control, external validity and current readiness. These are transparent synthesis scores, not empirical estimates. The right column states the role each direction could play in a forecast.
Figure 6: Synthesis of the paper. A dated frontier-AI forecast links resources, the system as used, measured behavior and deployment outcomes. Missingness, protocol dependence, benchmark drift and source dependence can break that chain. The corresponding remedies are disclosure with uncertainty bounds, response curves at matched budgets, versioned anchors and independent replication with backtesting.
Evaluating the true forecasting ability of AI agents requires environments that are resistant to environments resistant to overfitting, free from centralized trust, and grounded in incentive-compatible scoring. Existing benchmarks either rely on static datasets vulnerable to training-data contamination, or measure trading PnL -- a metric conflating predictive accuracy with timing, sizing, and risk appetite. We introduce Foresight Arena, the first permissionless, on-chain benchmark for evaluating AI forecasting agents on real-world prediction markets. Agents submit probabilistic forecasts on binary Polymarket markets via a commit-reveal protocol enforced by Solidity smart contracts on Polygon PoS; outcomes are resolved trustlessly through the Gnosis Conditional Token Framework. Performance is measured by the Brier Score and a novel Alpha Score -- proper scoring rules that incentivize honest probability reporting and isolate predictive edge over market consensus. We provide a formal analysis: closed-form variance for per-market Alpha, the connection to Murphy's classical Brier decomposition, and a power analysis characterizing the number of rounds required to reliably distinguish agents of different skill levels. We show that detecting a true edge of α∗=0.02 at 80% power requires approximately 350 resolved binary predictions (50 rounds of 7 markets), while α∗=0.01 requires four times more. We complement these analytical results with a deterministic, seed-controlled simulation study calibrated to literature-reported Brier-score ranges, illustrating how Murphy decomposition distinguishes well-calibrated agents from market-tracking agents that fail through reduced resolution. Live results from the deployed benchmark will be reported in a future revision. All smart contracts and evaluation infrastructure are open-source.
Maksym Nechepurenko, Pavel Shuvalov
Director of Research, Devnull, Dubai, UAE · Chief Technology Officer, Devnull, Dubai, UAE
Public AI evaluations are often read as terminal leaderboards, yet the underlying evidence is a selective time series shaped by reporting rules, benchmark revisions, and missingness. Repeated public archives for LiveBench and Open LLM Leaderboard v2 serve as the primary longitudinal record; LMArena provides a preference stress test; and GAIA and tau-bench contribute limited agentic pilots. Together, these archives instantiate a Bayesian inference problem: under a fixed reporting convention, one constructed terminal-only example over 1,000 systems is compatible with two pre-terminal histories, yielding times of 23.03 or 75.13 to reach within 0.05 of the ceiling under the same terminal-tail model. In synthetic posterior comparisons, action-facing diagnostics differ across observation regimes. The candidate selection-aware frontier model fails synthetic recovery, objective-archive prediction, preference transfer, and uncertainty calibration; correspondingly, fixed audit gates reject its stronger claims. An archive-and-adjudication protocol reconstructs public evaluation histories, isolates a verified timing boundary, and falsifies unsupported frontier claims.
We introduce TFRBench, the first benchmark designed to evaluate the reasoning capabilities of forecasting systems. Traditionally, time-series forecasting has been evaluated solely on numerical accuracy, treating foundation models as ``black boxes.'' Unlike existing benchmarks, TFRBench provides a protocol for evaluating the reasoning generated by forecasting systems--specifically their analysis of cross-channel dependencies, trends, and external events. To enable this, we propose a systematic multi-agent framework that utilizes an iterative verification loop to synthesize numerically grounded reasoning traces. Spanning ten datasets across five domains, our evaluation confirms that this reasoning is causally effective; useful for evaluation; and prompting LLMs with our generated traces significantly improves forecasting accuracy compared to direct numerical prediction (e.g., avg. ∼40.2%→56.6%), validating the quality of our reasoning. Conversely, benchmarking experiments reveal that off-the-shelf LLMs consistently struggle with both reasoning (lower LLM-as-a-Judge scores) and numerical forecasting, frequently failing to capture domain-specific dynamics. TFRBench thus establishes a new standard for interpretable, reasoning-based evaluation in time-series forecasting. Our benchmark is available at: https://tfrbench.github.io