Language Model Performance Prediction

Latest papers

41 papers23 papers in the last four weeks, against 1 the four weeks before. 0.2% of all new papers.

Scout Before You Route: Predicting Task Success and Token Count from a Small LLM Prefill
Nov 6, 2026Aristides Milios, Christopher PalLLM Output Length Prediction LLM Routing
Mila – Quebec AI Institute, Université de Montréal · ServiceNow AI Research · Mila – Quebec AI Institute, Polytechnique Montréal +1
RouteAlign: Aligning Task Requirements with Model Profiles for LLM Service Selection
Nov 5, 2026Jihao Zhao, Ding Chen, Yanfang Chen +4LLM Routing Language Model Performance Prediction
MemTensor (Shanghai) Technology Co., Ltd., Shanghai, China · Renmin University of China, Beijing, China · Shanghai Jiao Tong University, Shanghai, China
ACSeL: Adaptive Configuration Search for Hardware-Constrained LLM Inference
Nov 5, 2026Bokun Wang, Hung-Yueh Chiang, Diana MarculescuLLM Inference Efficient Language Model Inference
University of Texas at Austin
PAYWORTHY: Can LLMs Predict Which Freelance Jobs an AI Agent Will Complete?
Nov 3, 2026Mohammad Reza Rezaei, Dingxian Wang, Brett Levert +3Language Model Performance Prediction
Upwork Inc.
ROCm InferaSim: Cut LLM Serving Trial Costs and Deploy the Best Configuration
Nov 2, 2026Anshu Raina, Yao Fu, Zhang Jiejing +11LLM Serving Language Model Performance Prediction
Advanced Micro Devices Inc. (AMD)
Hinted Scaling Laws
Nov 1, 2026Suze van Adrichem, Luke Bailey, Rohith Kuditipudi +6Language Model Performance Prediction Language Model Scaling Laws
Stanford University
Fragility Is Not Reducible to Hardness: Conditional Causal Difficulty for LLMs
Nov 1, 2026Mohammad Reza Rezaei, Dingxian Wang, Nikhil Verma +3Language Model Robustness Item Difficulty Estimation
Upwork Inc.
Comparative Evaluation of LLM-Based Review Paradigms for Academic Paper Quality Assessment
Oct 30, 2026Fan Huang, Hongyu ZhouAutomated Peer Review LLM-as-a-Judge
Indiana University Bloomington, Bloomington, Indiana, USA · School of Information Resource Management, Renmin University of China, Beijing, China · Faculty of Social Sciences, University of Antwerp, Antwerp, Belgium
From the Zoo to the BULLPEN: Sample-Efficient LLM Profiling via Multi-Channel Model Embeddings
Oct 30, 2026Russell Ro, Qiming Cui, Reza Abbasi-AslLanguage Model Performance Prediction Text Embedding Models
University of California, San Francisco · University of California, Berkeley
Predicting Human-Gain Curves under Local Proxy Optimization from Repeated Ratings
Oct 30, 2026Moonwon Choi, Kisung Nam, Seunggeun LeeLanguage Model Performance Prediction Human Preference Evaluation
Graduate School of Data Science, Seoul National University · Massachusetts General Hospital, Harvard Medical School · Broad Institute of MIT and Harvard
CIRCE Divines LLM Task Complexity from Constraint Structure
Oct 30, 2026Chenhao Hong, Jiakai Xu, Kaushiki Singh +2Language Model Performance Prediction Language Model Reasoning Evaluation
Columbia University
The Real Cost of Context: Predicting Compute Time and Energy Consumption in LLM Prefill and Decode Phases
Oct 29, 2026Lars Krupp, Daniel Geißler, Francisco M. Calatrava-Nicolas +3LLM Inference Energy-Aware LLM Inference
RPTU Kaiserslautern-Landau, DFKI · Örebro University · RPTU Kaiserslautern-Landau
GAUSS: A Workload-Aware Analytical and Monte-Carlo Simulator for LLM Inference Serving
Oct 27, 2026Srikant Bharadwaj, Ankur Mallick, Spyridon Mastorakis +3Large Language Model-Based Simulation LLM Serving
Microsoft
Meta Models: Models that Model Learning
Oct 27, 2026Peihang Li, Ziming LiuLearning Curve Prediction Language Model Performance Prediction
MetaCircle · The University of Hong Kong · University of California, Berkeley +2
LADDER: Diagnosing Domain-Level Self-Assessment Under Noisy Feedback
Oct 26, 2026Zhenshuai Yin, Jianshu Liang, Xuyue Xiong +6Language Model Self-Assessment Metacognition in Language Models
School of Science and Engineering, The Chinese University of Hong Kong (Shenzhen), Longgang, Shenzhen, Guangdong, 518172, P.R. China · ByteDance · Shenzhen Research Institute of Big Data, Shenzhen, China
STEP: Soft Trajectory Estimation of Prediction Correctness via Per-Step Probability Scoring
Oct 25, 2026Chenyang Yu, Zifan Zeng, Qunli Zhang +1Confidence Estimation in Language Models Language Model Performance Prediction
Technical University of Munich · RAMS Lab, Huawei Heisenberg Research Center
Predictable performance is suboptimal, and what it means for model routing
Oct 22, 2026Guanhua Zhang, Moritz HardtLLM Routing Language Model Performance Prediction
Max Planck Institute for Intelligent Systems, Tübingen · Tübingen AI Center
Scaling Laws for Safety Probes
Oct 22, 2026Yuchen Wu, Jerry WeiLLM Safety Monitoring Language Model Probing
University of Washington · Anthropic
Bridging Foundational and Agentic Capabilities of LLMs
Oct 21, 2026Wei Guo, Yuanjian Xu, Jianing Hao +4LLM Agent Evaluation AI Agent Benchmarks
Nankai University · The Hong Kong University of Science and Technology (Guangzhou) · Ant Group +2
Foreword: Pre-Generation Prediction and Characterisation of LLM Test Failures using the Jacobian Lens
Oct 20, 2026Somin Kim, Robert Feldt, Shin YooLanguage Model Performance Prediction LLM Evaluation
KAIST Republic of Korea · Chalmers University of Technology Sweden
How Language Models Organize Task Information
Oct 20, 2026Kaiyang Wan, Yuxia WangLanguage Model Probing Language Model Performance Prediction
INSAIT, Sofia University “St. Kliment Ohridski” Sofia, Bulgaria
Predicting Linear Relational Decodability Without the Relation
Oct 19, 2026Jonas Rohweder, Adrian Mülthaler, Faridoun Mehri +2LLM Interpretability Representation Learning
Center for Information and Language Processing (CIS), LMU Munich · Ubiquitous Knowledge Processing Lab (UKP Lab), TU Darmstadt · Munich Center for Machine Learning (MCML) +1
The Geometry of Agent Memory: Forecasting Retrieval-Bound Memory Failures, and Characterizing Its Limits
Oct 19, 2026Iliya Zhechev, Radoslav Ralev, Chris Coleman +4LLM Memory Conversational Memory
Redis
When Planning Problems Become Hard for LLMs: Separating Structural Complexity from Semantic Interpretability
Oct 18, 2026Mayukh Debnath, Adeyinka Adedigba, Rammohan MallipeddiLLM Planning Classical Planning
Robotics and Autonomous Systems (AI), Arizona State University, Tempe, USA · Department of Artificial Intelligence, Kyungpook National University, Daegu, South Korea
Can This LLM Annotate My Data? Estimating Zero-Shot LLM Performance
Oct 16, 2026Tong Lin, Grant Schoenebeck, Sabina TomkinsLanguage Model Performance Prediction LLM-Assisted Annotation
School of Information, University of Michigan
ScoutRoute: Efficient Mixture-of-Experts Routing Using Internal Language Model Signals
Oct 16, 2026Rajdeep Pandhare, Delzaan BharuchaLLM Routing Mixture-of-Experts Inference
Department of Artificial Intelligence and Data Science, Rajiv Gandhi Institute of Technology, Mumbai, Maharashtra, India
Sample, Mix, or Verify? Allocating Test-Time Compute under Imperfect Verifiers
Oct 16, 2026Srinivasan IyengarInference-Time Compute Allocation Language Model Performance Prediction
Microsoft Research India
LLM Evaluation is a Forecasting Problem
Oct 12, 2026Rahul Thomas, Adam Block, Micah GoldblumLanguage Model Performance Prediction LLM Evaluation
Department of Electrical Engineering, Columbia University. · Department of Computer Science, Columbia University.
Target-Independent Micro-Interventions for Predicting Training Response Across Language-Model Families
Sep 8, 2026Zhongxuan Liu, Sicheng Zhou, Hongzhi WangLanguage Model Performance Prediction Causal Interventions in Language Models
Faculty of Computing, Harbin Institute of Technology
TuneAhead: Predicting Fine-tuning Performance Before Full Training Begins
Jun 16, 2026Yuxiang Luo, Haonan Long, Chen Wang +6Fine-Tuning LLM Fine-Tuning
The Hong Kong University of Science and Technology (Guangzhou) · Huawei Technologies Ltd.

Nov 6, 2026J/K move · Enter open · S save

Scout Before You Route: Predicting Task Success and Token Count from a Small LLM Prefill

Aristides Milios, Christopher Pal

Mila – Quebec AI Institute, Université de Montréal · ServiceNow AI Research · Mila – Quebec AI Institute, Polytechnique Montréal+1

LLM Output Length Prediction LLM Routing Cost-Aware Inference Language Model Performance Prediction Language Model Probing

Open paper AskPDF arXiv

Routers typically choose a model per query by trading predicted correctness against cost. Most price each model at a fixed cost, as if every query used the same number of output tokens, but the output length of reasoning models varies by an order of magnitude across queries. When is per-query cost worth predicting? What factors drive cost? How cheaply can predictions be made? We predict both the success probability and the output length of candidate models from the prefill activations of a single small frozen LLM, the scout. With reasoning candidates, perfect per-query cost knowledge would save 29-61% of spending on eight of nine datasets, and much less with non-reasoning candidates on the same problems. On those eight datasets, our predicted costs save 9-33% at matched accuracy relative to median-length pricing, up to 69% of what perfect knowledge would save. They are significantly cheaper than each estimator we reimplemented or adapted from published work, including ZeroRouter and a MixLLM-style predictor, on a majority of datasets. Output length depends mostly on the problem. On most math and code datasets, harder problems are longer, so predicted success alone prices cost nearly as well. On knowledge and mixed-format datasets they are not, and predicted success falls well behind a probe trained on length. The probe keeps most of its saving with 20 labelled problems and transfers without labels to four of seven new benchmarks. On a mixture of all nine benchmarks, it saves 22% beyond pricing each benchmark at its own median length. Savings stop growing at about a 2B scout, so above 4B a larger scout costs more than it adds.