cs.CLSep 25, 2025
SaveAnticipatory Evaluation of Language Models: Predicting LLM Benchmark Scores from Descriptions
Organizations: Georgia Institute of Technology, Atlanta, Georgia · The Hebrew University of Jerusalem, Jerusalem, Israel
Abstract
Modern ML research increasingly faces an evaluation bottleneck: building benchmarks and running experiments is costly, slowing iteration. We ask how much can be inferred about evaluation outcomes before experiments, given only a natural-language description of the task and setup. We call this setting anticipatory evaluation. We introduce EVALCAST, a literature-grounded resource of 2,290 description-performance pairs from 1,519 arXiv papers, including a test set of papers published after the model's knowledge cutoff. We evaluate two tasks: predicting absolute performance through regression and comparing benchmark difficulty through pairwise prediction. Experiments with frontier LLMs show that accurate regression remains challenging, while pairwise predictions can recover relative performance ordering under matched settings. Comparable performance on post-cutoff and streaming evaluations suggests that the observed signal is not solely attributable to previously reported results. Overall, EVALCAST provides a resource and evaluation framework for studying anticipatory evaluation and its potential use in prioritizing benchmarks.
Figures & tables
| Task Design | |||||
| Work | Text | Task | Reg. | No Feat. | No Runs |
| Xia et al. (2020) | ✗ | ✗ | ✓ | ✗ | ✓ |
| Ye et al. (2021) | ✗ | ✗ | ✓ | ✗ | ✓ |
| Ye et al. (2023) | ✗ | ✗ | ✓ | ✗ | ✓ |
| Maia Polo et al. (2024) | ✗ | ✗ | ✓ | ✓ | ✗ |
| Pacchiardi et al. (2024) | ✗ | ✗ | ✓ | ✗ | ✗ |
Table 1: Comparison of performance-prediction schemas. Text : uses text descriptions as input; Task : predicts performance on unseen tasks; Reg. : predicts a metric value (regression); No Feat. : requires no manual feature engineering; No Runs : does not require running the model on the benchmark.
Figure 1: Overview of the EvalCast curation pipeline. Each record combines a result paper ( Mitchell et al., 2023 ) and a dataset paper ( Moskvichev et al., 2023 ) to describe the task and evaluation setup. The reported score is normalized to – as the prediction target.
| Statistic | 2023–2024 | 2025 |
| Record counts | ||
| total instances | 767 | 1,523 |
| result paper dataset paper | 504 | 549 |
| result paper dataset paper | 263 | 974 |
| Paper counts | ||
| unique result paper | 443 | 579 |
Table 2: EvalCast statistics by collection period and results of the 60-sample human audit.
| 2023–2024 | 2025 | |||
| Model | MAE | MAE | ||
| Baselines | ||||
| Test set average | 21.4 | — | 23.0 | — |
| E5-Mistral + kNN | 19.9 | 0.30 | 21.2 | 0.22 |
| E5-Mistral + XGBoost | 20.3 | 0.30 | 21.4 | 0.20 |
| LLMs | ||||
Table 3: Prediction performance on EvalCast for 2023–2024 (767) and 2025 (1,523 instances). † indicates significantly lower absolute error than the test-set average baseline ( , one-sided binomial sign test ( Hollander and Wolfe, 1999 ) ).
Figure 2: Pairwise accuracy under matched settings across gap thresholds, where each threshold retains only pairs whose realized scores differ by at least that many points. Accuracy measures whether predicted scores correctly preserve the ordering.
Table 4: MAE and Pearson across subsets of EvalCast 2025 by benchmark category, publication status, target model, and dataset novelty. Peer-review status is determined via DBLP; new datasets are introduced in the result paper, and established datasets in prior work. Darker shading indicates better performance.
Figure 3: Prediction errors in the pilot human study on the Accuracy subset ( ).
| Statistic | Value |
| Counterfactual descriptions | 250 |
| Median Spearman | |
| Negative | 50/50 |
| Mean prediction drop | 15.6 pts |
Table 5: Sensitivity of predicted performance to the number of answer options.
| Error Category | Share |
| Task Characteristics | |
| Ambiguous chance baseline | 35% |
| Multi-step compositional reasoning | 27% |
| Adversarial or filtered instances | 13% |
| Other | 25% |
| Predictor Failure Modes | |
Table 6: Error taxonomy for the 100 largest Accuracy prediction errors. Each case is categorized by both task characteristic and predictor failure mode.
Figure 4: MAE (left axis) and coverage (right axis) as a function of confidence thresholds (scores band lower bound) for GPT-5 and Qwen3-32B across different year splits (2023-2024 vs 2025). Higher thresholds retain fewer examples (lower coverage) and generally yield lower MAE. Confidence values are derived from a 10-bin categorical output elicited from the model.
| GPT-5 | Qwen3-32B | |
| Prediction performance ( MAE / ) | ||
| No Search | 14.7 / 0.69 | 19.6 / 0.40 |
| With Search | 14.0 / 0.73 | 20.8 / 0.31 |
| Search-tool usage & token diversity | ||
| Avg. search calls | 3.06 | 0.89 |
| Std. search calls | 0.87 | 0.32 |
Table 7: Search-tool usage, token diversity, and prediction performance for GPT-5 and Qwen3-32B. Token counts are measured with the GPT - 4o tokenizer. (all) aggregates all formulated queries; (random) uses one random query per instance.
Figure 5: Streaming prediction errors for GPT - 5 + Search. Confident denotes predictions with over 50% confidence ( Yoon et al., 2025 ) .
| Model | MAE | |
| Test set average | 25.5 | – |
| GPT - 5 | 15.0 | 0.74 |
| GPT - 5 + Search | 14.1 | 0.78 |
Table 8: Prediction performance in the streaming setting on 79 experimental records.
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 6: Per-metric MAE (↓) for all baselines and LLMs, grouped by evaluation metric (e.g., Accuracy, Exact Match (EM), Recall, F1) and computed separately on the 2023–2024 and 2025 data subsets. The red and orange bars show the per-metric averages for each year split, with their absolute MAE values labeled, and the remaining bars indicate how much each model deviates from the corresponding yearly average.
| Model | MAE | Pearson Correlation |
| E5-Mistral-7B + KNN | 21.2 | 0.22 |
| E5-Mistral-7B + XGBoost | 21.4 | 0.20 |
| Qwen3-8B-Embed + KNN | 20.8 | 0.27 |
| Qwen3-8B-Embed + XGBoost | 21.6 | 0.21 |
| KaLM-Embedding-Gemma3-12B-2511 + KNN | 20.8 | 0.26 |
| KaLM-Embedding-Gemma3-12B-2511 + XGBoost | 21.5 | 0.21 |
Table 9: Results on the EvalCast-2025 dataset with different embedding backbones and regressors. Newer, stronger embedding models provide modest gains, but improvements are relatively small compared to the overall bottleneck from compressing each model–task pair into a fixed-size vector.
| Subset | Gold / Pred. acc. (%) | Abs. error (pp) |
| On-air | 65 / 68 | 3 |
| Off-air | 6 / 14 | 8 |
| Overall | — | 5.5 (MAE) |
Table 10: Gold and predicted GPT-4o accuracies on the on-air and off-air puzzle subsets, along with absolute error in percentage points (pp).
Figure 7: Predicted GPT–4o accuracy as a function of the number of multiple-choice options for a multitask reasoning benchmark. The solid line shows our description-based predictions for 5, 10, 15, 20, and 25 options, and the dashed horizontal line marks the reported ground-truth accuracy of 72.6% for the 10-option configuration used in practice.