Efficient Language Model Inference

Follow this topic Ask about this topic

The field studies how to serve language models with less computation, memory, time, and energy while preserving useful outputs. Approaches include quantization and sparsity, faster decoding such as speculative and parallel generation, KV-cache and context compression, and smaller or specialized models routed to tasks they can handle. For reasoning models, early stopping and adaptive compute allocation aim to spend more tokens on difficult questions and fewer on easy ones. The central challenge is that savings depend on the model, workload, hardware, and serving setup, and reducing token counts or latency does not always reduce total cost or energy without harming accuracy or reliability.

Latest papers

321 papers64 papers in the last four weeks, up 68% on the four weeks before. 0.4% of all new papers.

Reasoning Compression with Mixed-Policy Distillation
May 9, 2026Han Yang, Mingyan Wu, Bailan He +4Language Model Distillation Efficient Language Model Inference
Technical University of Munich · GESIS – Leibniz Institute for the Social Sciences · Northeastern University +5
Hint Tuning: Less Data Makes Better Reasoners
May 9, 2026Siqi Fan, Minghao Li, Xiaoqian Ma +6Overthinking in Language Models LLM Fine-Tuning
University of Electronic Science and Technology of China · Xiaohongshu Inc. · National University of Singapore
Reliable Chain-of-Thought via Prefix Consistency
May 8, 2026Naoto Iwase, Yuki Ichihara, Mohammad Atif Quamar +1Self-Consistency Decoding Reasoning Consistency in Language Models
Nagoya University · Nara Institute of Science and Technology · Mohamed bin Zayed University of Artificial Intelligence +1
Confidence-Aware Alignment Makes Reasoning LLMs More Reliable
May 8, 2026Kejia Chen, Jiawen Zhang, Yihong Wu +5LLM Alignment Direct Preference Optimization
Zhejiang University · Université de Montréal · Sun Yat-sen University +1
A Few Good Clauses: Comparing LLMs vs Domain-Trained Small Language Models on Structured Contract Extraction
May 7, 2026Nicole Lincoln, Nick Whitehouse, Jaron Mar +1Legal NLP Mixture-of-Experts Language Models
Onit AI Labs, Onit Inc., New Zealand
SCOUT: Active Information Foraging for Long-Text Understanding with Decoupled Epistemic States
May 6, 2026Zhenliang Zhang, Wenqing Wang, Yong Hu +4Document Understanding Long-Context QA
Peking University · WeChat, Tencent · Work done during an internship at WeChat, Tencent
Gated Subspace Inference for Transformer Acceleration
May 4, 2026Stephen J. ThomasTransformer Inference LLM Inference Acceleration
Department of Computer Science and Engineering, Lehigh University, Bethlehem, PA 18015
When Less is Enough: Efficient Inference via Collaborative Reasoning
May 1, 2026Yilei Chen, Sharut Gupta, Yannis Paschalidis +2Efficient Language Model Reasoning Efficient Language Model Inference
Boston University · MIT CSAIL · Biohub
Component-Aware Self-Speculative Decoding in Hybrid Language Models
May 1, 2026Hector Borobia, Elies Seguí-Mas, Guillermina Tormo-CarbóSpeculative Decoding Language Modeling
VRAIN – Valencian Research Institute for Artificial Intelligence, Universitat Politècnica de València, Valencia, Spain · Department of Economics and Social Sciences, Universitat Politècnica de València, Valencia, Spain · Department of Business Organisation, Universitat Politècnica de València, Valencia, Spain
Making Every Verified Token Count: Adaptive Verification for MoE Speculative Decoding
May 1, 2026Lehan Pan, Ziyang Tao, Ruoyu Pang +3Mixture-of-Experts Inference Speculative Decoding
University of Science and Technology of China · Tianyijiaotong Technology Ltd., Suzhou, China
Select to Think: Unlocking SLM Potential with Local Sufficiency
Apr 29, 2026Wenxuan Ye, Yangyang Zhang, Xueli An +2Small Language Models Efficient Language Model Reasoning
1Technical University of Munich · 3Ludwig Maximilian University of Munich · 2Huawei Technologies +2
Dual-Track CoT: Budget-Aware Stepwise Guidance for Small LMs
Apr 27, 2026Sagnik Chatterjee, Atharva Patil, Sricharan RameshCoT Reasoning Small Language Models
Nautile-370M: Spectral Memory Meets Attention in a Small Reasoning Model
Apr 27, 2026Maixent ChenebauxSelf-Attention Small Language Models
Trickstr
Stabilizing Efficient Reasoning with Step-Level Advantage Selection
Apr 27, 2026Han Wang, Xiaodong Yu, Jialian Wu +4RL for Language Model Reasoning Efficient Language Model Reasoning
UNC Chapel Hill · Advanced Micro Devices, Inc.
Thinking Without Words: Efficient Latent Reasoning with Abstract Chain-of-Thought
Apr 24, 2026Keshav Ramji, Tahira Naseem, Ramón Fernandez AstudilloEfficient Language Model Reasoning Implicit Reasoning in Language Models
IBM Research AI
An End-to-End Ukrainian RAG for Local Deployment. Optimized Hybrid Search and Lightweight Generation
Apr 23, 2026Mykola Trokhymovych, Yana Oliinyk, Nazarii NyzhnykRetrieval-Augmented Generation Hybrid Retrieval
Pompeu Fabra University · Independent Researcher
Thinking with Reasoning Skills: Fewer Tokens, More Accuracy
Apr 23, 2026Guangxiang Zhao, Qilong Shi, Xusen Xiao +3Efficient Language Model Reasoning Efficient Language Model Inference
Qiyuan Tech · Tsinghua University · The University of Hong Kong +1
TRACES: Tagging Reasoning Steps for Adaptive Cost-Efficient Early-Stopping
Apr 22, 2026Yannis Belkhiter, Seshu Tirupathi, Giulio Zizzo +1Reasoning Trace Analysis Early Stopping
IBM Research Europe · Trinity College Dublin · Dublin, Ireland +1
Less Languages, Less Tokens: An Efficient Unified Logic Cross-lingual Chain-of-Thought Reasoning Framework
Apr 22, 2026Chenyuan Zhang, Qiguang Chen, Xie Chen +6Cross-Lingual Reasoning in Language Models CoT Reasoning
Harbin Institute of Technology, Shenzhen · Shanghai Innovation Institute · Central South University +3
Neural Garbage Collection: Learning to Forget while Learning to Reason
Apr 20, 2026Michael Y. Li, Jubayer Ibn Hamid, Emily B. Fox +1LLM Compression Memory-Efficient Optimization
Stanford University
WISV: Wireless-Informed Semantic Verification for Distributed Speculative Decoding in Device-Edge LLM Inference
Apr 20, 2026Zixuan Liu, Zhiyong Chen, Nan Xue +4Speculative Decoding Efficient Language Model Inference
Cooperative Medianet Innovation Center and the Department of Electronic Engineering, Shanghai Jiao Tong University, Shanghai 200240, China
Efficient Test-Time Scaling via Temporal Reasoning Aggregation
Apr 19, 2026Jiakun Li, Xingwei He, Kefan Li +3Test-Time Scaling Efficient Language Model Inference
School of Computer Science and Engineering, Beihang University · 3Hangzhou Innovation Institute, Beihang University · 2Qingdao Research Institute, Beihang University
Layer-wise MoE Routing Locality under Shared-Prefix Code Generation: Token-Identity Decomposition and Compile-Equivalent Fork Redundancy
Apr 19, 2026Shun-ichiro Hayashi, Daichi Mukunoki, Tetsuya Hoshino +1Expert Routing Efficient Language Model Inference
Graduate School of Informatics, Nagoya University, Japan · Information Technology Center, Nagoya University, Japan
RankGuide: Tensor-Rank-Guided Routing and Steering for Efficient Reasoning
Apr 17, 2026Jiayi Tian, Yupeng Su, Ryan Solgi +2LLM Routing Efficient Language Model Inference
University of California, Santa Barbara · Intel Labs
SCATR: Simple Calibrated Test-Time Ranking
Apr 16, 2026Divya Shyamal, Marta Knežević, Lan Tran +3Test-Time Scaling Best-of-N Sampling
EPFL · MIT
Towards Faster Language Model Inference Using Mixture-of-Experts Flow Matching
Apr 16, 2026Aihua LiNon-Autoregressive Text Generation Flow Matching
Duke University
TrigReason: Trigger-Based Collaboration between Small and Large Reasoning Models
Apr 16, 2026Yi Zhao, Yajuan Peng, Cam-Tu Nguyen +4Efficient Language Model Reasoning LLM Inference Acceleration
AGI Institute, School of Computer Science, Shanghai Jiao Tong University, Shanghai, China · Shanghai Key Laboratory for Intelligent Information Processing, Fudan University · State Key Laboratory for Novel Software Technology, Nanjing University +2
RPRA: Predicting an LLM-Judge for Efficient but Performant Inference
Apr 14, 2026Dylan R. Ashley, Gaël Le Lan, Changsheng Zhao +7Language Model Cascading Language Model Performance Prediction
Meta Platforms, Inc., Menlo Park, United States of America · The Swiss AI Lab IDSIA (USI-SUPSI), Lugano, Switzerland · Università della Svizzera italiana, Lugano, Switzerland +2
EdgeCIM: A Hardware-Software Co-Design for CIM-Based Acceleration of Small Language Models
Apr 13, 2026Jinane Bazzi, Mariam Rakka, Fadi Kurdahi +2On-Device Language Model Inference Efficient Language Model Inference
King Abdullah University of Science and Technology, Thuwal, Saudi Arabia. · the University of California Irvine, Irvine, CA, USA. · Rain Neuromorphics Inc., San Francisco, CA, USA
SLM Finetuning for Natural Language to Domain Specific Code Generation in Production
Apr 10, 2026Renjini R. Nair, Damian K. Kowalczyk, Marco Gaudesi +1Code Generation Fine-Tuning
Microsoft

May 9, 2026J/K move · Enter open · S save

Reasoning Compression with Mixed-Policy Distillation

Han Yang, Mingyan Wu, Bailan He, Zeyu Cao+3

Technical University of Munich · GESIS – Leibniz Institute for the Social Sciences · Northeastern University+5

Language Model Distillation Efficient Language Model Inference Policy Distillation CoT Compression Large Reasoning Models

Open paper AskPDF arXiv

Reasoning-centric large language models (LLMs) achieve strong performance by generating intermediate reasoning trajectories, but often incur excessive token usage and high inference-time decoding cost. We observe that, when solving the same problems, larger reasoning models can often produce more concise traces, whereas smaller reasoning models tend to generate longer and more redundant trajectories. This is especially problematic in real-world deployment, where memory, latency, and serving-cost constraints often favor smaller models. Our observations suggest that reasoning compression can be transferred from large models to small ones rather than enforced through explicit length constraints. Based on this insight, we propose Mixed-Policy Distillation (MPD), a reasoning compression framework that transfers concise reasoning behavior from a larger-sized teacher to a smaller student by distilling teacher-compressed student trajectories. Unlike on-policy distillation, which aligns the student with teacher distributions over verbose student trajectories, or off-policy distillation, which relies on teacher-generated trajectories and may suffer from distribution mismatch, MPD combines the strengths of both. Given a student-sampled trajectory, the teacher rewrites it into a more concise reasoning trace, and the student is trained via KL-based alignment on the compressed trajectory. This preserves student-policy exploration while injecting teacher-guided compression. Experiments on Qwen3-1.7B show that MPD reduces token usage by up to 27.1% while improving performance across multiple reasoning benchmarks, demonstrating an effective approach to efficient small-model reasoning.