Efficient Language Model Inference

Ask about this topic

Overview

The field studies how to serve language models with less computation, memory, time, and energy while preserving useful outputs. Approaches include quantization and sparsity, faster decoding such as speculative and parallel generation, KV-cache and context compression, and smaller or specialized models routed to tasks they can handle. For reasoning models, early stopping and adaptive compute allocation aim to spend more tokens on difficult questions and fewer on easy ones. The central challenge is that savings depend on the model, workload, hardware, and serving setup, and reducing token counts or latency does not always reduce total cost or energy without harming accuracy or reliability.

Latest papers

CardsList

321 papers64 papers in the last four weeks, up 68% on the four weeks before. 0.4% of all new papers.

  1. VineLM: Trie-Based Fine-Grained Control for Agentic Workflows

    Apr 9, 2026Nikos Pagonas, Matthew Lou, Tianyi Peng +2LLM Agent Workflow OptimizationAgentic Workflows

    Columbia University

  2. Alloc-MoE: Budget-Aware Expert Activation Allocation for Efficient Mixture-of-Experts Inference

    Apr 9, 2026Baihui Liu, Kaiyuan Tian, Wei Wang +3Mixture-of-Experts InferenceMixture of Experts

    National Key Laboratory of Parallel and Distributed Computing, College of Computer Science and Technology, National University of Defense Technology

  3. CROP: Token-Efficient Reasoning in Large Language Models via Regularized Prompt Optimization

    Apr 8, 2026Deep Shah, Sanket Badhe, Nehal Kathrotia +1Efficient Language Model ReasoningAutomatic Prompt Optimization

    Google LLC · Purdue University

  4. SHIELD: A Segmented Hierarchical Memory Architecture for Energy-Efficient LLM Inference on Edge NPUs

    Apr 8, 2026Jintao Zhang, Xuanyao FongOn-Device Language Model InferenceEfficient Language Model Inference

    Department of Electrical and Computer Engineering, National University of Singapore, Singapore

  5. The Detection-Extraction Gap: Models Know the Answer Before They Can Say It

    Apr 8, 2026Hanyang Wang, Mingxuan ZhuEfficient Language Model ReasoningEfficient Language Model Inference

    The University of Chicago · Imperial College London

  6. Just Pass Twice: Efficient Token Classification with LLMs for Zero-Shot NER

    Apr 6, 2026Ahmed Ewais, Ahmed Hashish, Amr AliNamed Entity RecognitionDecoder-Only Language Models

    WitnessAI

  7. Adaptive Cost-Efficient Evaluation for Reliable Patent Claim Generation

    Apr 5, 2026Yongmin Yoo, Qiongkai Xu, Longbing CaoClaim VerificationAutomated Evaluation

    Frontier AI Research Centre, Macquarie University · School of Computing, FSE, Macquarie University

  8. Batched Contextual Reinforcement: A Task-Scaling Law for Efficient Reasoning

    Apr 2, 2026Bangji Yang, Hongbo Ma, Jiajun Fan +1Efficient Language Model ReasoningEfficient Language Model Inference

    University of Illinois Urbana-Champaign · Tsinghua University

  9. Universal YOCO for Efficient Depth Scaling

    Apr 1, 2026Yutao Sun, Li Dong, Tianzhu Ye +3Efficient Language Model InferenceInference-Time Scaling

    Microsoft Research · Tsinghua University

  10. Do Small Language Models Know When They're Wrong? Confidence-Based Cascade Scoring for Educational Assessment

    Mar 29, 2026Tyler BurleighLanguage Model CascadingAutomated Grading

    Khan Academy

  11. Schema on the Inside: A Two-Phase Fine-Tuning Method for High-Efficiency Text-to-SQL at Scale

    Mar 25, 2026Chinmay Soni, Shivam Chourasia, Gaurav Kumar +1Sports AnalyticsText-to-SQL

    Sporta Technologies Private Limited, Mumbai, Maharashtra, India

  12. Task-Specific Efficiency Analysis: When Small Language Models Outperform Large Language Models

    Mar 22, 2026Jinghan Cao, Yu Ma, Xinjin Li +2Small Language ModelsEfficient Language Model Inference

    San Francisco State University - Department of Computer Science 1600 Holloway Ave, San Francisco - USA · Carnegie Mellon University - Department of Computer Science 5000 Forbes Ave, Pittsburgh - USA · Columbia University - Department of Computer Science 116th and Broadway, New York - USA +2

  13. A transformer architecture alteration to incentivise externalised reasoning

    Mar 22, 2026Elizabeth Pavlova, Mariia Koroliuk, Karthik Viswanathan +3Efficient Language Model InferenceLarge Reasoning Models

    MARS · Geodesic Research, equal contribution · University of Amsterdam

  14. Adaptive Domain Models: Bayesian Evolution, Warm Rotation, and Principled Training for Geometric and Neuromorphic AI

    Mar 18, 2026Houston HaynesEfficient Language Model InferenceBayesian Inference

    Braidpoint, Asheville, NC

  15. EngGPT2: Sovereign, Efficient and Open Intelligence

    Mar 17, 2026G. Ciarfaglia, A. Rosanova, S. Cipolla +17Language ModelingOpen-Weight Language Models

  16. SpecSteer: Synergizing Local Context and Global Reasoning for Efficient Personalized Generation

    Mar 17, 2026Hang Lv, Sheng Liang, Hao Wang +6Personalized Language ModelsOn-Device Language Model Inference

    University of Science and Technology of China · Huawei Technologies Co., Ltd.

  17. NeuronSpark: Selective Spiking State Space Models with Adaptive Computation for Language Modeling

    Mar 17, 2026Zhengzheng TangSpiking Language ModelsSpiking Neural Networks

    Boston University

  18. Mamba-3: Improved Sequence Modeling using State Space Principles

    Mar 16, 2026Aakash Lahoti, Kevin Y. Li, Berlin Chen +5State Space Language ModelsSequence Modeling

    Carnegie Mellon University · Princeton University · Together AI +1

  19. FlashHead: Efficient Drop-In Replacement for the Classification Head in Language Model Inference

    Mar 15, 2026Wilhelm Tranheden, Shahnawaz Ahmed, Devdatt Dubhashi +2LLM Inference AccelerationEfficient Language Model Inference

    Embedl Gamla Almedalsvägen 21, 412 63 Gothenburg, Sweden

  20. An Alternative Trajectory for Generative AI

    Mar 14, 2026Margarita Belova, Yuval Kansal, Yihao Liang +2Language ModelingNeuro-Symbolic AI

    Princeton University, USA

  21. Retrieval-Feedback-Driven Distillation and Preference Alignment for Efficient LLM-based Query Expansion

    Mar 14, 2026Minghan Li, Guodong ZhouLanguage Model DistillationEfficient Language Model Inference

    School of Computer Science and Technology, Soochow University, China

  22. TERMINATOR: Learning Optimal Exit Points for Early Stopping in Chain-of-Thought Reasoning

    Mar 13, 2026Alliot Nagle, Jakhongir Saydaliev, Dhia Garbaya +3Early Stopping for Language Model ReasoningEfficient Language Model Inference

    UT Austin · EPFL · ENS Paris-Saclay +1

  23. EnComp: Lightweight Encoder-Only Context Compression for Retrieval-Augmented Question Answering

    Mar 10, 2026Thao Do, Dinh Phu Tran, An Vo +2Efficient Language Model InferenceKnowledge-Intensive QA

    Korea Advanced Institute of Science and Technology (KAIST), Daejeon 34141, South Korea

  24. The Compression Paradox in LLM Inference: Provider-Dependent Energy Effects of Prompt Compression

    Mar 6, 2026Warren JohnsonLLM Inference EfficiencyEnergy-Aware LLM Inference

    Plexor Labs Principal Researcher

  25. Compression Method Matters: Benchmark-Dependent Output Dynamics in LLM Prompt Compression

    Mar 6, 2026Warren JohnsonPrompt CompressionLLM Inference Efficiency

    Plexor Labs Principal Researcher

  26. Prompt Compression in Production Task Orchestration: A Pre-Registered Randomized Trial

    Mar 6, 2026Warren Johnson, Charles LeePrompt CompressionToken Compression

    Plexor Labs, Issaquah, WA, USA · Project Autobots, Seattle, WA, USA

  27. Diffusion Language Models Are Natively Length-Aware

    Mar 6, 2026Vittorio Rossi, Giacomo Cirò, Davide Beltrame +3Adaptive Computation for Diffusion ModelsDiffusion Language Models

    Department of Computing Sciences, Bocconi University, Milan, Italy.

  28. Reasoning Theater: Disentangling Model Beliefs from Chain-of-Thought

    Mar 5, 2026Siddharth Boppana, Annabel Ma, Max Loeffler +5Faithfulness of Language Model ReasoningLanguage Model Reasoning Evaluation

    Goodfire AI · Harvard University, Cambridge, MA

  29. Transducing Language Models

    Mar 5, 2026Vésteinn Snæbjarnarson, Samuel Kiegeland, Tianyu Liu +3Efficient Language Model Inference

    ETH Zürich · University of Copenhagen · CHI-FRO

  30. How Small Can 6G Reason? Scaling Tiny-to-Small Language Models for AI-Native Networks

    Mar 2, 2026Mohamed Amine Ferrag, Abderrahmane Lakas, Merouane Debbah6G NetworksSmall Language Models

    Department of Computer and Network Engineering, United Arab Emirates University, UAE · Research Institute for Digital Future, Khalifa University, UAE