Language Model Performance Prediction

Follow this topic Ask about this topic

Latest papers

41 papers23 papers in the last four weeks, against 1 the four weeks before. 0.2% of all new papers.

Predicting Inference-Time Scaling Gains from Labeled Validation-Set Output Statistics
Jun 2, 2026Luyang Zhang, Jingyan LiLLM Evaluation Language Model Performance Prediction
Carnegie Mellon University · Johns Hopkins University
Item Response Scaling Laws: A Measurement Theory Approach for Efficient and Generalizable Neural Scaling Estimation
May 29, 2026Sang Truong, Yuheng Tu, Rylan Schaeffer +1Language Model Performance Prediction Language Model Scaling Laws
1Stanford University · University of California, Los Angeles
Forecasting Downstream Performance of LLMs With Proxy Metrics
May 18, 2026Arkil Patel, Siva Reddy, Marius Mosbach +1Language Model Pretraining LLM Evaluation
Mila – Quebec AI Institute & McGill University Canada · ServiceNow Research · Periodic Labs
Charon: A Unified and Fine-Grained Simulator for Large-Scale LLM Training and Inference
May 16, 2026Mengtian Yang, Zhekun Zhang, Mingheng Wu +3Language Model Performance Prediction LLM Inference
ByteDance Seed · University of Texas at Austin
RPRA: Predicting an LLM-Judge for Efficient but Performant Inference
Apr 14, 2026Dylan R. Ashley, Gaël Le Lan, Changsheng Zhao +7Language Model Cascading Language Model Performance Prediction
Meta Platforms, Inc., Menlo Park, United States of America · The Swiss AI Lab IDSIA (USI-SUPSI), Lugano, Switzerland · Università della Svizzera italiana, Lugano, Switzerland +2
PoC: Performance-oriented Context Compression for Large Language Models via Performance Prediction
Mar 20, 2026Runsong Zhao, Shilei Liu, Jiwei Tang +8Language Model Performance Prediction LLM Context Compression
Northeastern University, China · Future Living Lab of Alibaba · Tsinghua University
Disentangling Geometry, Performance, and Training in Language Models
Feb 24, 2026Atharva Kulkarni, Jacob Mitchell Springer, Arjun Subramonian +1LLM Evaluation Neural Representation Geometry
University of Southern California · 2Carnegie Mellon Univer · University of California, Los Angeles.
Prescriptive Scaling Reveals the Evolution of Language Model Capabilities
Feb 17, 2026Hanlin Zhang, Jikai Jin, Vasilis Syrgkanis +1Language Model Performance Prediction Language Model Scaling Laws
Harvard University · Stanford University
The Lexical Imprint of Pre-training Data
Feb 11, 2026Jeonghoon Kim, Woojin Chung, Woomin Song +3Language Model Pretraining Language Model Performance Prediction
NAVER Cloud · Korea Advanced Institute of Science and Technology (KAIST)
Neural Neural Scaling Laws
Jan 27, 2026Michael Y. Hu, Jane Pan, Ayush Rajesh Jhaveri +2Language Model Performance Prediction Language Model Scaling Laws
New York University
Anticipatory Evaluation of Language Models: Predicting LLM Benchmark Scores from Descriptions
Sep 25, 2025Jungsoo Park, Ethan Mendes, Gabriel Stanovsky +1Language Model Performance Prediction LLM Evaluation
Georgia Institute of Technology, Atlanta, Georgia · The Hebrew University of Jerusalem, Jerusalem, Israel

Jun 2, 2026J/K move · Enter open · S save

Predicting Inference-Time Scaling Gains from Labeled Validation-Set Output Statistics

Luyang Zhang, Jingyan Li

Carnegie Mellon University · Johns Hopkins University

LLM Evaluation Language Model Performance Prediction Inference-Time Scaling Best-of-N Sampling

Open paper AskPDF arXiv

Best-of-

N

inference scaling (drawing

N

candidate answers from a language model and returning the one a reward model ranks highest) improves accuracy by an amount that varies across models, but predicting that amount in advance currently requires running the procedure end-to-end. Prior work links cheap statistics of a model's sampled outputs and validation-set correctness (how often samples agree, how diverse they are, how confident the model is, and where correct samples appear) to model behavior, but does not isolate which of these form a stable, compact predictor of best-of-

N

gain. We fit ridge predictors on features computed from a single labeled validation-set sampling pass, use bootstrap-Lasso as a stability analysis of the candidate feature set, and give a concentration analysis with an explicit linear-approximation residual. Across three base-model families, six post-training methods, and math and reasoning task domains, the stability analysis identifies a strict three-feature core spanning prompt-level agreement spread, label-assisted first-correct-sample position, and completion-length variance; a compact ridge predictor built from this core plus an entropy add-on reaches Spearman

ρ= 0.90

with actual best-of-

N

gain under a reward-model verifier. The intended use is labeled validation-set screening of candidate configurations before paying the full reward-model scoring cost.