cs.LGOct 12, 2026
SaveLLM Evaluation is a Forecasting Problem
Organizations: Department of Electrical Engineering, Columbia University. · Department of Computer Science, Columbia University.
Abstract
Standard procedure in machine learning is to randomly split train and test data, emulating IID draws from one underlying data distribution. In practice, models are trained on past data and deployed in the future, which may come from distinct distributions. In this paper, we find that web text is indeed highly non-stationary, as text distributions shift over time. We further observe that model performance changes continuously over time across diverse benchmarks. Consequently, future model performance is better predicted by measurements made on recent evaluation data, and is best predicted by time series forecasters applied to historic performance rather than by historic performance itself. Choosing which model to deploy based on forecasted accuracy enables more reliable selection of performant models, thus improving downstream accuracy in the regime practitioners care about: deployment. Taken together, our experiments suggest that time should be treated as a fundamental variable in LLM evaluation. We release our code and data.
Figures & tables
Figure 1: Recent evaluations and time series forecasting outperform temporally pooled benchmark scores for both future performance prediction and model selection. We evaluate 19 open-source models on 12 temporally labeled datasets, using the historical mean as the pooled baseline. (Left) The most recent year and Chronos-2 yield fewer predictions exceeding each error threshold. (Right) Both improve annual model selection, with larger gains as the candidate set grows.
Figure 2: Surface-level, syntactic, topical, and semantic properties of pretraining corpora systematically shift over time. Across FineWeb-Edu, DCLM-Baseline, and Nemotron-CC v1, document length and tokenizer efficiency increase while byte compressed-size ratio decreases. Part-of-speech and domain distributions, along with annual document-embedding centroids, increasingly diverge from their initial values. Shading shows bootstrapped 95% confidence intervals.
Figure 3: Models generalize best between nearby years, with degradation increasing more rapidly when generalizing to future data. (Left) Excess loss for Nemotron-CC v1 is smallest near the diagonal and increases with temporal separation; FineWeb-Edu and DCLM-Baseline are in Appendix D . (Right) Averaging by year offset shows the same pattern across all corpora, with larger excess loss when generalizing to future data (positive offsets).
| Dataset | Task Description | Start Year | End Year | Examples |
|---|---|---|---|---|
| BioASQ | Biomedical QA | 2014 | 2025 | 4,067 |
| FinQA | Financial QA | 1999 | 2019 | 8,281 |
| StreamingQA | Evolving knowledge QA | 2007 | 2020 | 16,307 |
| SCIDOCA | Scientific citation prediction | 2012 | 2025 | 70,000 |
| NVD-VSP | CVSS severity scoring | 2019 | 2025 | 116,711 |
| NVD-RCM | CWE root-cause classification | 2017 | 2025 | 132,103 |
Table 1: We evaluate frontier and open-source models on 12 datasets of varying scale, task type, and temporal coverage. We construct all shaded datasets from the relevant source corpus ( Appendix E ).
Figure 4: Recent evaluations better predict future performance across different temporal performance trends. We evaluate GPT-5.6 Sol/Luna on StreamingQA and SCIDOCA. (Left) Depending on the task and model, performance changes over time in distinct ways. (Right) Prediction error increases with evaluation age, so recent evaluations are the best predictors of future performance.
Figure 5: Evaluation staleness smoothly degrades prediction quality at annual and monthly resolutions. We evaluate 19 open-source models on 12 temporal datasets. (Left) Annual prediction error CDFs shift downward as lag increases from 1 to 8 years. (Right) Monthly prediction error rises as lag increases from 1 to 48 months. Error quantiles rise with lag, with the strongest degradation in the upper tail. Each comparison is restricted to periods where all plotted lags exist.
Appendix figures & tables13 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 6: Additional corpus metrics show systematic temporal distribution shift across pretraining corpora. FineWeb-Edu, DCLM-Baseline, and Nemotron-CC v1 exhibit changes in document statistics and token and part-of-speech composition (first three rows) . Annual document-embedding centroids shift over time, while estimated AI-text probability rises sharply in recent years (fourth row) . Dataset-specific quality signals remain relatively stable (fifth row) .
Figure 7: Models generalize best to nearby years across all three pretraining corpora. On FineWeb-Edu, DCLM-Baseline, and Nemotron-CC v1, excess loss is smallest near the diagonal and increases as the pretraining year and evaluation year separate.
Figure 8: LiveCodeBench evaluations become less predictive as they grow stale. (Left) Prediction error increases with monthly lag, so recent evaluations are the best predictors of future performance. Shaded bands are 95% confidence intervals from resampling within model families. (Right) Monthly prediction error CDFs shift downward as lag increases from 1 to 12 months. Each comparison is restricted to periods where all plotted monthly lags exist, giving 336 data points at every lag.
Figure 9: Prediction error distributions smoothly degrade as evaluations grow stale. (Left) The 10th through 90th percentiles of absolute prediction error increase nearly linearly with monthly lag, with the largest increases occurring in the upper tail. (Right) The 1-Wasserstein distance between each lag’s error distribution and the 1-lag error distribution increases steadily with monthly lag.
Figure 10: Time-aware forecasters outperform historical mean pooling, with explicit trajectory modeling providing greater benefits at monthly resolution. We plot prediction error CDFs for one-step-ahead forecasts at (left) annual and (right) monthly resolutions. At annual resolution, persistence performs nearly as well as Chronos-2, while at monthly resolution, explicit time series forecasters like Chronos-2 and SES improve substantially over persistence and perform similarly.
| Method | Annual MAE | Monthly MAE |
|---|---|---|
| Historical mean (baseline) | 0.03083 | 0.02163 |
| Linear trend | 0.02755 | 0.01527 |
| Damped Holt | 0.02360 | 0.01388 |
| ARIMA | 0.02266 | 0.01425 |
| SES | 0.02085 | 0.01373 |
| Persistence (previous obs.) | 0.02001 | 0.01735 |
Table 2: Chronos-2 achieves the lowest MAE at annual and monthly resolutions, although several time-aware forecasters have comparable performance. Mean absolute error (MAE) is for one-step-ahead forecasts of model performance. Lower is better.
| Method | Monthly MAE |
|---|---|
| Historical mean (baseline) | 0.15486 |
| Linear trend | 0.08539 |
| Damped Holt | 0.08717 |
| ARIMA | 0.10298 |
| SES | 0.09071 |
| Persistence (previous obs.) | 0.08506 |
Table 3: Persistence achieves the lowest MAE at monthly resolution on LiveCodeBench, although several time-aware forecasters have comparable performance. Mean absolute error (MAE) is for one-step-ahead forecasts of model performance. Lower is better.
| Sensitivity analysis | Reduction vs. historical mean | Reduction vs. weighted mean |
|---|---|---|
| None | 45.1% | 42.5% |
| Equal-weight families | 47.1% | 44.7% |
| Leave one family out | 45.6–48.2% | 43.1–45.8% |
| Leave one month out | 41.5–49.3% | 38.7–46.7% |
| Post-cutoff months | 49.4% | 47.4% |
| Post-cutoff months & history | 39.0% | 32.9% |
Table 4: The advantage of persistence over the historical mean on LiveCodeBench is not sensitive to model-family composition, target-month choice, or training-cutoff restrictions. Each sensitivity analysis report the reduction in persistence MAE relative to the usual historical mean and to a historical mean weighted by monthly problem counts.
Figure 11: Model rankings can reverse over time, causing temporally pooled scores to preserve outdated orderings. Ranking inversions occur across years for thinking versus non-thinking variants, different model sizes within a family, similarly capable models across families, and frontier versus open-weight models. (Bottom left) On WildTime-arXiv, the historical mean favors Llama-3.1-70B-Instruct even after Qwen3-32B-thinking becomes the better model in most later years.
Figure 12: Chronos-2 improves model selection at both annual and monthly resolutions, while persistence improves only annual selection. As the number of candidate models increases, we plot (left) absolute gains in optimal selection rate over the historical mean and (right) reductions in model selections with regret greater than two percentage points relative to the historical mean. Circles are annual results, triangles are monthly results, and the dashed line is the historical mean baseline.
Figure 13: Recent evaluations better predict future multimodal performance. We evaluate Gemma-3-12B-IT and Gemma-3-27B-IT on Fakeddit, grouping performance by year. (Left) Performance substantially increases over time for both models. (Right) Prediction error increases with evaluation age, so recent evaluations are the best predictors of future performance.
Figure 14: The benefit of data recency in pretraining is domain-dependent. (Left) Recent-window pretraining often yields lower next-year perplexity than cumulative pretraining on FineWeb and PubMed, while arXiv shows no preference. (Right) The most recent yearly target-domain evaluation accurately selects between the two mixtures across all three datasets. Selection accuracy sharply degrades with evaluation age on FineWeb but remains stable on PubMed and arXiv.
Figure 15: Recent supervision improves next-year performance under a fixed annotation budget. For each evaluation year from 2022 to 2025, we finetune Llama-3.1-8B-Instruct on NVD-RCM data uniformly sampled from either all preceding years (All) or the most recent year (Recent). Points show paired Recent-All next-year accuracy differences across five seeds, and lines show means. Recent sampling improves mean accuracy in all 20 year-budget settings, with an average gain of 2.63%.