Efficient Language Model Inference

Follow this topic Ask about this topic

The field studies how to serve language models with less computation, memory, time, and energy while preserving useful outputs. Approaches include quantization and sparsity, faster decoding such as speculative and parallel generation, KV-cache and context compression, and smaller or specialized models routed to tasks they can handle. For reasoning models, early stopping and adaptive compute allocation aim to spend more tokens on difficult questions and fewer on easy ones. The central challenge is that savings depend on the model, workload, hardware, and serving setup, and reducing token counts or latency does not always reduce total cost or energy without harming accuracy or reliability.

Latest papers

321 papers64 papers in the last four weeks, up 68% on the four weeks before. 0.4% of all new papers.

TableTune: Measured, Self-Correcting Draft-Length Tables for Dynamic Speculative Decoding
Oct 28, 2026Abhishek Malvankar, Paul Castro, Nick Mitchell +1LLM Inference Acceleration Speculative Decoding
IBM Research.
CopyLoop: Fast Looped Transformer Decoding via Context Copying and Shallow Screening
Oct 28, 2026Wenyu Du, Stephen ChungEfficient Language Model Inference Recurrent Transformers
DualverseAI
SatOps-Decision: Low-Latency Discriminative Decisions for Onboard Satellite Communication Operations
Oct 28, 2026Lifan Mei, Yuxuan Li, Jiaxian Tu +1LLM Decision-Making Efficient Language Model Inference
Academy of AIAT, Xi’an Jiaotong-Liverpool University · Amazon, Inc.
Channel Selection for Mixture-of-Experts Decoding Needs No Calibration Data
Oct 27, 2026Geunhyoung Kim, Sung Ju HwangMixture-of-Experts Inference Efficient Language Model Inference
KAIST
From Generation to Scoring: Scaling LLM-Based Semantic Labeling at LinkedIn
Oct 26, 2026Jian Sheng, Sriram Vasudevan, Sundara Raman Ramachandran +12Efficient Language Model Inference Text Classification
LinkedIn.
Decision Model Factory: Building Specialized Sovereign Decision Models
Oct 26, 2026Yuma IchikawaLLM Decision-Making Efficient Language Model Inference
Fujitsu Limited · RIKEN Center for AIP
ScaleRLT: Exact Log-Probabilities and Fast Deferred Rollouts for Reinforcement Learning on Looped Transformers
Oct 25, 2026Dezhen Lu, Zixiang Ni, Runguo LiRL for Language Models Efficient Language Model Inference
University of Tübingen, Tübingen, Germany. · Xi’an Jiaotong University, Xi’an, China. · University of Illinois Urbana-Champaign, Urbana, IL, USA.
Accordion: Elastic-Depth Inference via Fixed-Prefix Layer Looping
Oct 23, 2026Yudong Zhang, Wei Ding, Kaili Zheng +2Efficient Language Model Inference LLM Inference Acceleration
Tsinghua University · Baidu Inc.
Joint or Per-Option? Compact Encoders for Closed-Set Decisions from 17 Million to 1 Billion Parameters
Oct 22, 2026Tardelli Ronan Coelho StekelInput Order Sensitivity in Language Models LLM Decision-Making
Federal Institute of São Paulo (IFSP), São Paulo, Brazil
RetiJev: Fast and Source-Grounded Ophthalmic Decision Models
Oct 22, 2026Bo Liu, Lianyu Wang, Meng Wang +2Ophthalmology Clinical Decision Support
Centre for Innovation and Precision Eye Health, Yong Loo Lin School of Medicine, National University of Singapore, Singapore 119228, Singapore. · Department of Ophthalmology, Yong Loo Lin School of Medicine, National University of Singapore, Singapore 119228, Singapore. · University of Washington, Seattle, WA, USA. +3
Compiling Autoregressive LLM Control into Policy-Exact Parallel Inference for Hierarchical RAG
Oct 22, 2026Zehao Lin, Qianfeng Shen, Xixuan Hao +8Language Model-Based Control Efficient Language Model Inference
MemTensor (Shanghai) Technology Co., Ltd. · China Haisum Engineering Co., Ltd. · The Hong Kong University of Science and Technology (Guangzhou) +1
Better Reasoning, Not Just Bigger Models: A Structure-Grounded Framework for Efficient and Reliable Inference
Oct 22, 2026Amirhossein Aghaei, Liangyan Li, Shiyi Yang +2LLM Reasoning with Graphs LLM Grounding
Huawei Research Canada, Markham, Canada · Huawei Research Canada, Waterloo, Canada · Huawei Technologies Co., Ltd., Beijing, China +1
Our Own Jev: Typed Decisions in One Forward Pass, from an In-House Judge to Open LoRA Models
Oct 21, 2026Yuchuan Weng, Chi Liu, Xin Chen +1LLM-as-a-Judge Efficient Language Model Inference
Answer Delivery Under a Shared Token Budget: An Audited Study of Stopping and Finalization
Oct 19, 2026Yupeng Chang, Linyuan Gao, Zhiyuan Li +2Efficient Language Model Inference Early Stopping
School of Artificial Intelligence, Jilin University · Engineering Research Center of Knowledge-Driven Human-Machine Intelligence, MOE, China · International Center of Future Science, Jilin University
TimesJev: Thinking Fast on Time Series with System 1 Sufficiency
Oct 19, 2026Chenghao Liu, Wei Fan, Jindong Han +4Time Series Reasoning Temporal Representation Learning
University of Auckland · Shandong University · Northeast Normal University +2
Fine-Grained Units, Flat Decisions: Hierarchical Micro-Expert Pruning for Mixture-of-Experts Models
Oct 19, 2026Zukang Xu, Xinpei Gao, Xing Hu +2Channel Pruning Mixture-of-Experts Pruning
Houmo AI · University of Science and Technology of China · Southeast University
P2N: Reusing Deep Representations for Greater Effective Depth
Oct 18, 2026Yiqin Hao, Boyi Zeng, Zitong Wang +7Efficient Language Model Inference Recurrent Transformers
LUMIA Lab, School of Artificial Intelligence, Shanghai Jiao Tong University · Sun Yat-sen University · Shanghai AI Laboratory
ECHO: Efficient Causal Self-Recursion Beyond Auxiliary Multi-Token Prediction
Oct 17, 2026Yuxuan Zhou, Junyuan Shang, Tianyu Wang +5Multi-Token Prediction Efficient Language Model Inference
ERNIE Team, Baidu · Independent Researcher
System1-Arena: Which Decisions Can Be Left to System 1?
Oct 15, 2026Tianyu Chen, Xinyang Liu, Ruyi An +2LLM Decision-Making LLM Evaluation
The University of Texas at Austin
LEV: Thinking Fast and Deep with Looped Decision Models
Oct 14, 2026Dingjie Song, Zhiling Yan, Yuxuan Zhang +1Efficient Language Model Inference Adaptive Computation
Lehigh University · University of British Columbia · Vector Institute
Verdict: Encode Once, Decide Many
Oct 14, 2026Avdhesh Singh Chouhan, Jack PerryAI-Assisted Decision Making Efficient Language Model Inference
LEC AI
Energy Consumption Analysis of Discrete Diffusion and Autoregressive Language Models
Oct 14, 2026Boris Ruf, Marcin DetynieckiLLM Inference Energy-Aware LLM Inference
AXA, AI Research Paris, France
Towards Efficient Simultaneous Inverse Text Normalization with Pretrained Text-to-Text Language Model and Read-Tag-Write Policy
Oct 14, 2026Kiet Anh Hoang, Khanh Le, Bao Nguyen +6Efficient Language Model Inference Text Generation
UNEY, Switzerland
AgriJev: Efficient Agricultural Candidate Scoring with Multi-Scale Adapted Models
Oct 14, 2026Xuechen LiLanguage Modeling LLM Inference
University of Minnesota Twin Cities
Jeva: System One Models from First Principles
Oct 13, 2026Qi Li, Xinchao WangLLM Decision-Making Efficient Language Model Inference
National University of Singapore
Efficient Safety Guardrails through Latent Dual-Hypothesis Reasoning with On-Demand Trace Reconstruction
Oct 13, 2026Md Asiful Islam, Fahmida Alam, Mihai SurdeanuLLM Guardrails LLM Safety
Department of Computer Science University of Arizona, USA
Diff Diffusion: Predicting Edit Paths with Diffusion Models for Efficient Code Editing
Oct 13, 2026Xiaolong Xu, Hao Jiang, Bo Jiang +5Diffusion Language Models Diffusion Language Model Decoding
ByteDance, Shenzhen, China
Full-duplex Spoken Language Modeling with Discrete Diffusion
Oct 13, 2026Chen Chen, Pin-Jui Ku, Ankita Pasad +3Speech Language Models Full-Duplex Spoken Dialogue Systems
NVIDIA
FunDM: Cheap & Label-Free Decision Models from Frozen Language Models
Oct 12, 2026Ujjawal K. PanchalLLM Decision-Making Efficient Language Model Inference
Fundamental AI
Label-Centric Knowledge Retrieval for Zero-Shot Text Classification
Oct 12, 2026Daoping Wu, Dong Wang, Siyu Liu +5Zero-Shot Text Classification Large Language Model-Based Text Classification
Zhejiang Lab, Hangzhou, China · School of Computer Science and Software Engineering, Harbin Institute of Technology, China · School of Mechanical Engineering, Shanghai Jiao Tong University, China

Oct 28, 2026J/K move · Enter open · S save

TableTune: Measured, Self-Correcting Draft-Length Tables for Dynamic Speculative Decoding

Abhishek Malvankar, Paul Castro, Nick Mitchell, Lionel Villard

IBM Research.

LLM Inference Acceleration Speculative Decoding Efficient Language Model Inference

Open paper AskPDF arXiv

Speculative decoding speeds up large language model (LLM) serving only while the GPU has spare compute, so the best draft length, the number of tokens a small drafter guesses per step, shrinks as the batch grows. vLLM lets operators set the draft length for each batch size with a hand-written table, but nothing measures or corrects that table. We present TableTune. Its tuner times the decoding step for every batch size and draft length on vLLM's own serving path and computes, in minutes, the table that maximizes throughput. Its controller, a vLLM plugin, tracks how many guesses are accepted in live traffic and rebuilds the table while serving. We measure TableTune only against other ways to speculate. On an H100, the tuned table delivers at least 94% of the throughput of the best fixed draft length at every load in four of five drafter settings (88% in the fifth) without the operator having to know which length that is. Against reimplementations of TurboSpec- and Nightjar-style control on the same engine, its worst-case loss is never higher than theirs, and with the EAGLE-3 drafter it is 3% against 9% and 12%. On a mixture-of-experts model, vLLM's documented table loses up to 18% and the tuned table at most 3%. Starting from mistuned tables, the controller raises throughput by up to 58% (up to 45% on vLLM's default runner), and after a workload switch with the DFlash drafter it recovers 96% of a stale table's loss. In llm-d's optimized-baseline deployment as shipped, two Qwen3-32B replicas on four GPUs, the tuned table is up to 32% faster than a fixed draft length of 3 and up to 25% faster than vLLM's documented table at high load, and the controller raises the documented table's throughput by up to 29%.