Cost-Aware Inference

Follow this topic Ask about this topic

Momentum

24 papers in the last four weeks, up 50% on the four weeks before. 0.2% of all new papers.

Jul 13Week of Sep 28

Latest papers 183

All topics

Poodle: Seamlessly Scaling Down Large Language Models with Just-in-Time Model Replacement
Dec 5, 2025Nils Strassenburg, Boris Glavic, Tilmann RablCost-Aware Inference Small Language Models
Hasso Plattner Institute, Uni Potsdam · University of Illinois Chicago
PERTINENCE: Input-based Opportunistic Neural Network Dynamic Execution
Jul 2, 2025Omkar Shende, Gayathri Ananthanarayanan, Marcello TraiolaEfficient Neural Network Inference Adaptive Inference
Department of Computer Science and Engineering, Indian Institute of Technology Dharwad, Karnataka, India · University of Rennes, CNRS, Inria, IRISA, Rennes, France
Harnessing the Reasoning Economy: A Survey of Efficient Reasoning for Large Language Models
Mar 31, 2025Rui Wang, Hongru Wang, Boyang Xue +9LLM Inference Efficiency Cost-Aware Inference
University of Edinburgh, Edinburgh, UK · University of Macau, Macau SAR, China · University of Illinois Urbana-Champaign, USA

Dec 5, 2025, cs.DBJ/K move · Enter open · S save

Poodle: Seamlessly Scaling Down Large Language Models with Just-in-Time Model Replacement

Nils Strassenburg, Boris Glavic, Tilmann Rabl

Hasso Plattner Institute, Uni Potsdam · University of Illinois Chicago

Cost-Aware Inference Small Language Models LLM Inference Acceleration LLM Routing Inference-Time Intervention

Open paper AskPDF arXiv

Businesses increasingly rely on large language models (LLMs) to automate simple repetitive tasks instead of developing custom machine learning models. LLMs require few, if any, training examples and can be utilized by users without expertise in model development. However, this comes at the cost of substantially higher resource and energy consumption compared to smaller models, which often achieve similar predictive performance for simple tasks. In this paper, we present our vision for just-in-time model replacement (JITR), where, upon identifying a recurring task in calls to an LLM, the model is replaced transparently with a cheaper alternative that performs well for this specific task. JITR retains the ease of use and low development effort of LLMs, while saving significant cost and energy. We discuss the main challenges in realizing our vision regarding the identification of recurring tasks and the creation of a custom model. Specifically, we argue that model search and transfer learning will play a crucial role in JITR to efficiently identify and fine-tune models for a recurring task. Using our JITR prototype Poodle, we reduce inference time by up to 7.5x compared to a self- hosted LLM and save more than $2,200 per 1M requests compared to a flagship hosted LLM, while achieving accuracy competitive with the LLM baseline.

Momentum

Related topics

Latest papers 183

Poodle: Seamlessly Scaling Down Large Language Models with Just-in-Time Model Replacement

PERTINENCE: Input-based Opportunistic Neural Network Dynamic Execution

Harnessing the Reasoning Economy: A Survey of Efficient Reasoning for Large Language Models