LLM Output Length Prediction

LLM: Large Language Model

Latest papers

8 papers1 paper in the last four weeks, against 1 the four weeks before. 0.0% of all new papers.

Scout Before You Route: Predicting Task Success and Token Count from a Small LLM Prefill
Nov 6, 2026Aristides Milios, Christopher PalLLM Output Length Prediction LLM Routing
Mila – Quebec AI Institute, Université de Montréal · ServiceNow AI Research · Mila – Quebec AI Institute, Polytechnique Montréal +1
LEAP: Efficient LLM Inference Scheduling with Generalizable Length Prediction
Oct 30, 2026Kwan Yan, Jialun Zhong, Yanzeng LiLLM Output Length Prediction LLM Inference Scheduling
Beijing Normal University · Peking University
MAPS: Memory-Aware Predictive Scheduling Framework for Large Language Model Serving
Sep 14, 2026Tiancheng Zhang, Yulin Chen, Yunfeng Zhao +3Disaggregated LLM Serving LLM Output Length Prediction
College of Intelligence and Computing, Tianjin University, Tianjin, China · The International Joint Institute of Tianjin University, Tianjin University, Fuzhou, China · Faculty of Digital Economics and Managements, Tianjin University of Finance and Economics, Tianjin, China
OUTLETS: Output-Length Prediction from Speculative Decoding Backbones
Sep 1, 2026Weihuang Wen, Yingying Liu, Yichuan Liu +5LLM Serving LLM Output Length Prediction
The Chinese University of Hong Kong, Shenzhen · The University of Hong Kong · Huawei Technologies Co., Ltd.
When Entropy Is Not Enough: Reclaiming Lost Semantics in LLM Output Length Prediction
Aug 16, 2026Feiyang Ren, Shengtao Wen, Yu Tian +5LLM Output Length Prediction LLM Serving
MIIT Key Laboratory of Pattern Analysis and Machine Intelligence, College of Computer Science and Technology, Nanjing University of Aeronautics and Astronautics · Tsinghua University · Nanjing University of Information Science and Technology
Robust Length Prediction: A Perspective from Heavy-Tailed Prompt-Conditioned Distributions
Apr 9, 2026Jing Wang, Yu-Yang Qian, Ke Xue +3LLM Output Length Prediction LLM Serving
State Key Laboratory of Novel Software Technology, Nanjing University, Nanjing 210023, China · School of Artificial Intelligence, Nanjing University, Nanjing 210023, China
Scheduling LLM Inference with Uncertainty-Aware Output Length Predictions
Apr 1, 2026Haoyu Zheng, Yongqiang Zhang, Fangcheng Fu +7LLM Output Length Prediction LLM Inference Scheduling
School of Computer Science, Wuhan University, Wuhan, China · Dameng Database Co., Ltd., Wuhan, China · School of Artificial Intelligence, Shanghai Jiao Tong University, Shanghai, China +2
Not Too Short, Not Too Long: How LLM Response Length Shapes People's Critical Thinking in Error Detection
Mar 6, 2026Natalie Friedman, Adelaide Nyanyo, Kevin Weatherwax +4LLM Output Length Prediction Human-AI Decision Making
BTP Innovation, SAP, Palo Alto, CA, USA

Nov 6, 2026J/K move · Enter open · S save

Scout Before You Route: Predicting Task Success and Token Count from a Small LLM Prefill

Aristides Milios, Christopher Pal

Mila – Quebec AI Institute, Université de Montréal · ServiceNow AI Research · Mila – Quebec AI Institute, Polytechnique Montréal+1

LLM Output Length Prediction LLM Routing Cost-Aware Inference Language Model Performance Prediction Language Model Probing

Open paper AskPDF arXiv

Routers typically choose a model per query by trading predicted correctness against cost. Most price each model at a fixed cost, as if every query used the same number of output tokens, but the output length of reasoning models varies by an order of magnitude across queries. When is per-query cost worth predicting? What factors drive cost? How cheaply can predictions be made? We predict both the success probability and the output length of candidate models from the prefill activations of a single small frozen LLM, the scout. With reasoning candidates, perfect per-query cost knowledge would save 29-61% of spending on eight of nine datasets, and much less with non-reasoning candidates on the same problems. On those eight datasets, our predicted costs save 9-33% at matched accuracy relative to median-length pricing, up to 69% of what perfect knowledge would save. They are significantly cheaper than each estimator we reimplemented or adapted from published work, including ZeroRouter and a MixLLM-style predictor, on a majority of datasets. Output length depends mostly on the problem. On most math and code datasets, harder problems are longer, so predicted success alone prices cost nearly as well. On knowledge and mixed-format datasets they are not, and predicted success falls well behind a probe trained on length. The probe keeps most of its saving with 20 labelled problems and transfers without labels to four of seven new benchmarks. On a mixture of all nine benchmarks, it saves 22% beyond pricing each benchmark at its own median length. Savings stop growing at about a 2B scout, so above 4B a larger scout costs more than it adds.