LLM Inference Efficiency

Follow this topic Ask about this topic

Recent momentum

-5%

61 papers in the last 28 days · 1.0% of indexed attention

Twelve weeks of publication activity for this topic as it is defined today.

Weekly history

Recent digests

What was published in this topic, kept on the site without email delivery.

Period ending 2026-09-21

21 new papers

A weekly snapshot of new work published in LLM Inference Efficiency.

Period ending 2026-09-14

12 new papers

A weekly snapshot of new work published in LLM Inference Efficiency.

Period ending 2026-09-07

16 new papers

A weekly snapshot of new work published in LLM Inference Efficiency.

541 papers

Latest in LLM Inference Efficiency

Browse all topics →

Frontier: Towards Comprehensive and Accurate LLM Inference Simulation
May 20, 2026Yicheng Feng, Xin Tan, Yangtao Deng +3LLM Inference Efficiency Frontiers
Runtime-Certified Bounded-Error Quantized Attention
May 20, 2026Dean CalverKey-Value Cache Quantization Systematic Fp16 Divergence
Llamas on the Web: Memory-Efficient, Performance-Portable, and Multi-Precision LLM Inference with WebGPU
May 20, 2026Reese Levine, Rithik Sharma, Nikhil Jain +5LLM Inference Efficiency Bird
When Do LLMs Reason? A Dynamical Systems View via Entropy Phase Transitions
May 20, 2026Wei Xia, Haoqing Wang, Zhi-Hong Deng +1Large Language Model Reasoning LLM Inference Efficiency
Mix-Quant: Quantized Prefilling, Precise Decoding for Agentic LLMs
May 19, 2026Haiquan Lu, Zigeng Chen, Gongfan Fang +2LLM Inference Efficiency Prefill
Draft Less, Retrieve More: Hybrid Tree Construction for Speculative Decoding
May 19, 2026Yuhao Shen, Tianyu Liu, Xinyi Hu +9Speculative Decoding Heterogeneous Pruning Scores
FlexDraft: Flexible Speculative Decoding via Attention Tuning and Bonus-Guided Calibration
May 19, 2026Yaojie Zhang, Jianuo Huang, Junlong Ke +5Speculative Decoding LLM Inference Efficiency
Block-Sphere Vector Quantization
May 19, 2026Heesang Ann, Joongkyu Lee, Min-hwan OhCalibration-Free Low-Bit Vector Quantization Extreme Low-Bit Quantization
The Silent Hyperparameter: Quantifying the Impact of Inference Backends on LLM Reproducibility
May 19, 2026David Pape, Jonathan Evertz, Lea SchönherrBenchmarking Large Language Models LLM Inference Efficiency
Latent Action Reparameterization for Efficient Agent Inference
May 18, 2026Wenhao Huang, Qingwen Zeng, Qiyue Chen +11Latent Actions LLM Inference Efficiency
Context Memorization for Efficient Long Context Generation
May 18, 2026Yasuyuki Okoshi, Hao Mark Chen, Guanxi Lu +3Efficient Long-Context Inference LLM Inference Efficiency
KVDrive: A Holistic Multi-Tier KV Cache Management System for Long-Context LLM Inference
May 18, 2026Jian Lin, Jiazhi Mi, Zicong Hong +5Key-Value Cache Efficient Long-Context Inference
LLMForge: Multi-Backend Hardware-Aware Neural Architecture Search with Infinite-Head Attention for Edge Language Models
May 17, 2026Xinting Jiang, Junyi Luo, Ruichen Qi +4Neural Architecture Search LLM Inference Efficiency
VeriCache: Turning Lossy KV Cache into Lossless LLM Inference
May 17, 2026Jiayi Yao, Samuel Shen, Kuntai Du +7Key-Value Cache LLM Inference Efficiency
Adaptive Generate-Rank-Verify: Inference-Time Search with Costly Verification
May 17, 2026Shaddin Dughmi, Mahdi Haghifam, Yusuf Hakan KalayciInference-Time Verifier
Delay-Adaptive Speculation Control for Low-Latency Edge-Cloud LLM Inference
May 17, 2026Kangkang Sun, Jianhua Li, Xiuzhen Chen +2Self-Speculative Speculative Decoding
TriAxialKV: Toward Extreme Low-Precision KV-Cache Quantization for Agentic Inference Tasks
May 16, 2026Hanzhang Shen, Haoran Wu, Yiren Zhao +1Key-Value Cache Quantization LLM Inference Efficiency
Full Attention Strikes Back: Transferring Full Attention into Sparse within Hundred Training Steps
May 16, 2026Yanke Zhou, Yiduo Li, Hanlin Tang +6Dynamic Sparse Attention Efficient Long-Context Inference
CompactAttention: Accelerating Chunked Prefill with Block-Union KV Selection
May 16, 2026Jiwon Song, Dongwon Jo, Beomseok Kang +1Prefill Efficient Long-Context Inference
Lever: Speculative LLM Inference on Smartphones
May 16, 2026Tuowei Wang, Fengzu Li, Yanfan Sun +2LLM Inference Efficiency Inference Latency
Property-Guided LLM Program Synthesis for Planning
May 15, 2026André G. Pereira, Augusto B. Corrêa, Jendrik SeippMock-Interface Synthesis Large Language Model Planning
IO-SVD: Input-Output Whitened SVD for Adaptive-Rank LLM Compression
May 15, 2026Ali Abbasi, Chayne Thrash, Haoran Qin +2Large Language Model Compression Singular Value Decomposition
PSD: Pushing the Pareto Frontier of Diffusion LLMs via Parallel Speculative Decoding
May 15, 2026Shengyin Sun, Yiming Li, Renxi Liu +7Diffusion Language Models Speculative Decoding
Measuring Maximum Activations in Open Large Language Models
May 15, 2026Luxuan Chen, Han Tian, Xinran Chen +9Large Language Model Quantization Model Activations
STS: Efficient Sparse Attention with Speculative Token Sparsity
May 15, 2026Ceyu Xu, Jiangnan Yu, Yongji Wu +1Dynamic Sparse Attention LLM Inference Efficiency
Dual-Dimensional Consistency: Balancing Budget and Quality in Adaptive Inference-Time Scaling
May 14, 2026Rongman Xu, Yifei Li, Tianzhe Zhao +3LLM Inference Efficiency Large Language Model Reasoning
GQLA: Group-Query Latent Attention for Hardware-Adaptive Large Language Model Decoding
May 14, 2026Fanxu MengLarge Language Model Decoder LLM Inference Efficiency
Performance-Driven Policy Optimization for Speculative Decoding with Adaptive Windowing
May 14, 2026Jie Jiang, Xing Sun, Ruotian Chen +2Speculative Decoding LLM Inference Efficiency
XFP: Quality-Targeted Adaptive Codebook Quantization with Sparse Outlier Separation for LLM Inference
May 14, 2026Thomas WittExtreme Low-Bit Quantization LLM Inference Efficiency
EnergyLens: Predictive Energy-Aware Exploration for Multi-GPU LLM Inference Optimization
May 14, 2026Zhiye Song, Kyungmi Lee, Eun Kyung Lee +3LLM Inference Efficiency Multi-Gpu Systems
SPHERICAL KV: Angle-Domain Attention and Rate-Distortion Retention for Efficient Long-Context Inference
May 13, 2026Anay Chauhan, Gurucharan Marthi Krishna Kumar, Arion Das +4Key-Value Cache Key-Value
Mistletoe: Stealthy Acceleration-Collapse Attacks on Speculative Decoding
May 13, 2026Shuoyang Sun, Chang Dai, Hao Fang +6Speculative Decoding LLM Inference Efficiency
Attention Once Is All You Need: Efficient Streaming Inference with Stateful Transformers
May 13, 2026Victor NorgrenLLM Inference Efficiency Real-Time Inference
MARLIN: Multi-Agent Game-Theoretic Reinforcement Learning for Sustainable LLM Inference in Cloud Datacenters
May 13, 2026H. Moore, S. Qi, D. Milojicic +2LLM Inference Efficiency Data Centers
Multi-Scale Dequant: Eliminating Dequantization Bottleneck via Activation Decomposition for Efficient LLM Inference
May 13, 2026Lingchao Zheng, Yuwei Fan, Jun Li +5Quantizer Systematic Fp16 Divergence
Orthrus: Memory-Efficient Parallel Token Generation via Dual-View Diffusion
May 12, 2026Chien Van Nguyen, Chaitra Hegde, Van Cuong Pham +3Diffusion Language Models LLM Inference Efficiency
KV-Fold: One-Step KV-Cache Recurrence for Long-Context Inference
May 12, 2026Alireza Nadali, Patrick Cooper, Ashutosh Trivedi +1Efficient Long-Context Inference Key-Value Cache
LIDSA: Cognitive Arbitration for Signal-Free Autonomous Intersection Management
May 12, 2026Abderrahmane Lakas, Mohamed Amine Ferrag, Merouane DebbahAdaptive Traffic Signal Control Autonomous Driving
CR^2: Cost-Aware Risk-Controlled Routing for Wireless Device-Edge LLM Inference
May 12, 2026Nan Xue, Shengkang Chen, Zhiyong Chen +4Large Language Model Routing Heterogeneous Edge Devices
The Illusion of Power Capping in LLM Decode: A Phase-Aware Energy Characterisation Across Attention Architectures
May 12, 2026Bole Ma, Ayesha Afzal, Jan Eitzinger +1Consumer Graphics Processing Units Throughput
From Token to Token Pair: Efficient Prompt Compression for Large Language Models in Clinical Prediction
May 12, 2026Mingcheng Zhu, Zhiyao Luo, Yu Liu +1Next-Token Prediction LLM Inference Efficiency
Training-Inference Consistent Segmented Execution for Long-Context LLMs
May 12, 2026Xianpeng Shang, Jiang Li, Zehua Duo +2Efficient Long-Context Inference LLM Inference Efficiency
GAR: Carbon-Aware Routing for LLM Inference via Constrained Optimization
May 12, 2026Disha Sheshanarayana, Rajat Subhra Pal, Manjira Sinha +1Large Language Model Routing LLM Inference Efficiency
D-PACE: Dynamic Position-Aware Cross-Entropy for Parallel Speculative Drafting
May 12, 2026Tianyu Wu, Yu Yao, Zhenting Qi +7Speculative Decoding Parallel
Efficient LLM-based Advertising via Model Compression and Parallel Verification
May 12, 2026Wenxin Dong, Chang Gao, Guanghui Yu +9Ads LLM Inference Efficiency
Ada-MK: Adaptive MegaKernel Optimization via Automated DAG-based Search for LLM Inference
May 12, 2026Wenxin Dong, Mingqing Hu, Guanghui Yu +7LLM Inference Efficiency Graphics Processing Unit Kernels
Fast MoE Inference via Predictive Prefetching and Expert Replication
May 12, 2026Ankit Jyothish, Ali Jannesari, Aishwarya Sarkar +1Mixture-Of-Expert Inference LLM Inference Efficiency
Measuring Five-Nines Reliability: Sample-Efficient LLM Evaluation in Saturated Benchmarks
May 11, 2026Eungyeup Kim, Chenchen Gu, Vashisth Tiwari +1Large Language Model Reliability Large Language Model Evaluation
Continuous Discovery of Vulnerabilities in LLM Serving Systems with Fuzzing
May 11, 2026Yunze Zhao, Yibo Zhao, Yuchen Zhang +2Vulnerability Detection & Repair Large Language Model Serving
CATS: Cascaded Adaptive Tree Speculation for Memory-Limited LLM Inference Acceleration
May 11, 2026Yuning Han, Yangchenchen Jin, Dylan Zhao +1Speculative Decoding LLM Inference Efficiency
Enabling Performant and Flexible Model-Internal Observability for LLM Inference
May 11, 2026Nengneng Yu, Sixian Xiong, Yibo Zhao +2LLM Inference Efficiency Inference Workloads
EnergyLens: Interpretable Closed-Form Energy Models for Multimodal LLM Inference Serving
May 11, 2026Vittorio Palladino, Gianluca Palermo, Michael E. Papka +1LLM Inference Efficiency Accelerator
SlimSpec: Low-Rank Draft LM-Head for Accelerated Speculative Decoding
May 11, 2026Anton Plaksin, Sergei Krutikov, Sergei Skvortsov +1Speculative Decoding LLM Inference Efficiency
Theory-optimal Quantization Based on Flatness
May 11, 2026Xiusheng Huang, Zhe Li, Xuanwu Yin +5Large Language Model Quantization Quantization
GELATO: Generative Entropy- and Lyapunov-based Adaptive Token Offloading for Device-Edge Speculative LLM Inference
May 11, 2026Zengzipeng Tang, Yuxuan Sun, Wei Chen +2LLM Inference Efficiency Speculative Decoding
Attention Drift: What Autoregressive Speculative Decoding Models Learn
May 11, 2026Doğaç Eldenk, Payal Mohapatra, Yigitcan Comlek +3Speculative Decoding Drafter
Merlin: Deterministic Byte-Exact Deduplication for Lossless Context Optimization in Large Language Model Inference
May 11, 2026Sietse SchelpeSemantic Deduplication LLM Inference Efficiency
PruneTIR: Inference-Time Tool Call Pruning for Effective yet Efficient Tool-Integrated Reasoning
May 11, 2026Luan Zhang, Dandan Song, Zhijing Wu +8Large Language Model Reasoning Fewer Tool-Call Errors
Test-Time Speculation
May 10, 2026Avinash Kumar, Sujay Sanghavi, Poulami DasSpeculative Decoding Self-Speculative
ProxyKV: Cross-Model Proxy Pruning for Efficient Long-Context LLM Inference
May 9, 2026Junjie Li, Jiong Lou, Jie LiLLM Inference Efficiency Efficient Long-Context Inference

May 20, 2026 · cs.DCJ/K move · Enter open · S save

Frontier: Towards Comprehensive and Accurate LLM Inference Simulation

Yicheng Feng, Xin Tan, Yangtao Deng, Yimin Jiang+2

The Chinese University of Hong Kong · Anuttacon · StepFun

Modern LLM serving is no longer homogeneous or monolithic. Production systems now combine disaggregated execution, complex parallelism, runtime optimizations, and stateful workloads such as reasoning, agents, and RL rollouts. Simulation is attractive for exploring this growing design space, yet existing simulators lack the architectural completeness and decision-grade fidelity it demands. Their monolithic-replica abstractions are ill-suited to disaggregated serving, while average-case analytical proxies can distort SLA predictions and even reverse optimization conclusions. We present Frontier, a discrete-event simulator for modern LLM inference serving. Frontier features a disaggregated abstraction. It captures the structure and dynamics of modern serving systems by modeling co-location, Prefill-Decode Disaggregation (PDD), and Attention-FFN Disaggregation (AFD) with role-specific cluster workers, incorporating key runtime optimizations (e.g., CUDA Graphs, speculative decoding) within the scheduler-batch-engine loop, and supporting stateful requests for emerging workloads. It further provides accurate and generalizable predictions of computation, communication, and memory costs across diverse serving scenarios with complex workload compositions. On 16-H800 GPU testbed, Frontier achieves an average throughput error below 4%. Compared with state-of-the-art simulators, it reduces end-to-end latency error from 44.9% to 6.4% under co-location and from 51.7% to 2.6% under disaggregation. It scales to over 1K GPUs on commodity CPUs and enables new use cases such as SLA-dependent Pareto frontier exploration, heterogeneous disaggregated allocation, agentic reasoning scheduling validation, and RL post-training reconfiguration. We release Frontier at https://github.com/NetX-lab/Frontier.

LLM Inference Efficiency Frontiers Autoregressive Rollout Pareto Frontier Replication

Open paper Ask about it PDF