From Forecasting Leaderboards to Deployment Decisions: A Fail-Closed Certification Protocol

Estimating how often an ML model will fail at deployment scale is central to pre-deployment safety assessment, but a feasible evaluation set is rarely large enough to observe the failures that matter. Jones et al. (2025) address this by extrapolating from the largest k failure scores in an evaluation set to predict deployment-scale failure rates. We give a finite-k decomposition of this estimator's forecast error and show that it has a built-in bias toward over-prediction in the typical case, which is the safety-favorable direction. This bias is offset when the evaluation set misses a rare high-failure mode that the deployment set contains, leaving the forecast to under-predict at deployment scale. We propose a fine-tuning objective, the forecastability loss, that addresses this failure mode. In two proof-of-concept experiments, a language-model password game and an RL gridworld, fine-tuning substantially reduces held-out forecast error while preserving primary-task capability and achieving safety similar to that of supervised baselines.

Temporal reports are increasingly emitted alongside numerical forecasts and are often interpreted as statements about the computation producing those forecasts. We formalize the resulting certification problem as three distinct stages: \emph{recoverability}, \emph{report correctness}, and \emph{functional use}. For point delays, an exact finite-sample recovery--substitutability identity ties structural discrimination and proxy prediction to the same realized shift geometry while placing them on different scales: structural evidence grows with $nη_n$, whereas the normalized predictive penalty is governed by $η_n$. A delay can therefore be statistically decisive while an alternative lag remains near-oracle. Guided by this regime, we evaluate TCN- and N-HiTS-based systems on the strict intersection of recoverable trajectories, correct reports, and near-oracle one-step predictions. Their dominant forecast dependence remains far from the reported delay under masking, finite perturbations, local Jacobians, and in-distribution conditional replacement, and the separation persists across a 100-fold forecast-loss sweep. A no-bypass factorization then provides an explicit access certificate for the final stage, while architecture-matched multi-seed, post-hoc, and gate-destruction controls identify report-coordinate access as the mechanism governing alignment. The resulting framework separates statistical evidence for \emph{what can be identified} from computational evidence for \emph{what the forecast uses}.

In algorithmic markets, predictive models become part of the data-generating process they aim to forecast. Once their outputs are converted into trades, allocations, execution schedules, or risk controls, they change the future data on which they are evaluated. I introduce algometrics, a framework for time series whose evolution depends on the predictive algorithms forecasting them. The framework distinguishes historical risk, measured under passive forecasting, from deployment risk, measured when forecasts drive actions. I prove three results. First, deployment risk is not identifiable from passive historical data alone: even in a one-step linear feedback model, infinitely many algorithm-mediated environments induce the same historical law while implying different deployment risks for the same forecaster. Second, historical model rankings can invert under crowding, so a predictor with lower passive error can have higher deployment error once similar algorithms are adopted. Third, randomized or instrumented actions identify short-horizon linear feedback, and I derive a finite-sample bound for deployment-risk estimation. These results suggest that time-series benchmarks in algorithmic markets should report feedback sensitivity alongside predictive accuracy.