LLM Inference Acceleration

LLM: Large Language Model

Follow this topic Ask about this topic

Momentum

108 papers in the last four weeks, up 120% on the four weeks before. 1.1% of all new papers.

Jul 13Week of Sep 28

Latest papers 677

All topics

Threshold-Based Exclusive Batching for LLM Inference
May 30, 2026Weifang Zhang, Yuzhou Nie, Bowen Pang +2GPU Acceleration LLM Inference
The Hong Kong Polytechnic University · UC Santa Barbara · Technology and Engineering Center for Space Utilization, Chinese Academy of Sciences +1
AdaptR1: Reinforcement Learning Based Adaptive Interleaved Thinking in Multi-hop Question Answering
May 29, 2026Yuxin Wang, Jiahao Lu, Qifeng Wu +5Multi-Hop QA RL for Language Model Reasoning
1Computer Science, Fudan University · Institute of Modern Languages and Linguistics, Fudan University · 3Shanghai Innovation Institute +1
Speculative Pipeline Decoding: Higher-Accruacy and Zero-Bubble Speculation via Pipeline Parallelism
May 29, 2026Yijiong Yu, Huazheng Wang, Shuai Yuan +2Speculative Decoding LLM Inference Acceleration
aOregon State University · bDeepSolution
CoMem: Context Management with A Decoupled Long-Context Model
May 29, 2026Yuwei Zhang, Chengyu Dong, Shuowei Jin +11LLM Inference LLM Inference Acceleration
Halıcıoğlu Data Science Institute, University of California, San Diego · Amazon
Reducing the GPU Memory Bottleneck with Lossless Compression for ML -- Extended
May 29, 2026Aditya K Kamath, Arvind Krishnamurthy, Marco Canini +1Efficient Neural Network Inference GPU Kernel Optimization
University of Washington Seattle, WA, USA · KAUST Thuwal, Saudi Arabia
Speculative Decoding Across Languages
May 28, 2026Nirajan Paudel, Michael Ginn, Luc De Nardi +1Multilingual Language Models Speculative Decoding
University of Colorado
Unlocking the Working Memory of Large Language Models for Latent Reasoning
May 28, 2026Lukas Aichberger, Sepp HochreiterMemory-Augmented Language Models Efficient Language Model Reasoning
ELLIS Unit Linz and LIT AI Lab, Institute for Machine Learning, Johannes Kepler University Linz, Austria · NXAI GmbH, Linz, Austria
Accelerating Constrained Decoding with Token Space Compression
May 28, 2026Michael Sullivan, Alexander KollerConstrained Decoding LLM Inference Acceleration
Department of Language Science and Technology Saarland Informatics Campus Saarland University, Saarbrücken, Germany
Domino: Decoupling Causal Modeling from Autoregressive Drafting in Speculative Decoding
May 28, 2026Jianuo Huang, Yaojie Zhang, Qituan Zhang +3Speculative Decoding LLM Inference Acceleration
1EPIC Lab, Shanghai Jiao Tong University · School of Software Engineering, HUST · 3UESTC +2
Draft-OPD: On-Policy Distillation for Speculative Draft Models
May 28, 2026Haodi Lei, Yafu Li, Haoran Zhang +8Speculative Decoding LLM Inference Acceleration
Shanghai Jiao Tong University · Shanghai AI Laboratory · The Chinese University of Hong Kong +3
Parallax: Parameterized Local Linear Attention for Language Modeling
May 27, 2026Yifei Zuo, Dhruv Pai, Zhichen Zeng +3Language Model Pretraining Self-Attention
Northwestern University · Tilde Research · University of Washington
From Human Guidance to Autonomy: Agent Skill System for End-to-End LLM Deployment on Spatial NPUs
May 27, 2026Jiajie Li, Erwei Wang, Zhiru Zhang +1AI Accelerator Inference LLM Agent Orchestration
Cornell University · AMD
Inference-Native Zeroth-Order Optimization for LLMs
May 27, 2026Zelin Li, Caiwen DingLLM Serving Zeroth-Order Optimization
University of Minnesota
Augmenting Attention with Exponentially Decaying Memory Improves Query-Aware KV Sparsity
May 27, 2026Xiuying Wei, Caglar GulcehreMemory-Augmented Language Models Self-Attention
EPFL, CLAIRE Lausanne, Switzerland
RW-TTT: Batched Serving for Request-Owned Test-Time Training State
May 27, 2026Jian Yang, Zhizhuo Kou, Yao Tian +4LLM Serving Test-Time Optimization
1The Hong Kong University of Science and Technology · 2The Chinese University of Hong Kong · 3National University of Singapore
Locality-Aware Redundancy Pruning for LLM Depth Compression
May 27, 2026Vincent-Daniel Yun, Youngrae Kim, Woosang Lim +3LLM Pruning LLM Compression
University of Southern California, United States · Neural Superintelligence Lab, MODULABS, Republic of Korea · Seoul National University, Republic of Korea +1
UNIQUE: Universal Top-k Sparse Attention for Training-free Inference and Sparsity-aware Training
May 26, 2026Keqi Deng, Shaoshi Ling, Ruchao Fan +1Self-Attention Long-Context Language Model Inference
Microsoft, USA
Pair-In, Pair-Out: Latent Multi-Token Prediction for Efficient LLMs
May 26, 2026Wenhui Tan, Minghao Li, Xiaoqian Ma +5Multi-Token Prediction LLM Compression
Gaoling School of Artificial Intelligence, Renmin University of China · AI Platform, Xiaohongshu Inc. · University of Electronic Science and Technology of China +1
NestedKV: Nested Memory Routing for Long-Context KV Cache Compression
May 26, 2026Hong Chen, Xiang Liu, Yubo Gao +5KV Caching Long-Context Language Modeling
The Hong Kong University of Science and Technology (Guangzhou) · Jimei University
Do Language Models Need Sleep? Offline Recurrence for Improved Online Inference
May 25, 2026Sangyun Lee, Sean McLeish, Tom Goldstein +1Memory-Augmented Language Models Long-Context Language Model Inference
Carnegie Mellon University · University of Maryland
Selective Latent Thinking: Adaptive Compression of LLM Reasoning Chains
May 25, 2026Hui Xie, Jie Liu, Ziyue Qiao +1Efficient Language Model Reasoning LLM Inference Acceleration
Eindhoven University of Technology, Netherlands · School of Computing and Information Technology, Great Bay University, China
Neural Router: Semantic Content Matching for Agentic AI
May 25, 2026Lauri Lovén, Abhishek Kumar, Alexander Engelhardt +5Semantic Textual Similarity LLM Inference Acceleration
Future Computing Group, University of Oulu, Finland · University of Jyväskylä, Finland · Department of Computer Science, University of Helsinki, Finland +1
Fine-Tuning and Serving Gemma 4 31B on Google Cloud TPU: A Technical Comparison with GPU Baselines
May 25, 2026Jatin Kishnani, Mayank Goel, Amit Singh +2AI Accelerator Inference LLM Serving
IndexMem: Learned KV-Cache Eviction with Latent Memory for Long-Context LLM Inference
May 25, 2026Xintong Yang, Hao Gu, Binxing Xu +6Memory-Augmented Language Models Self-Attention
The Hong Kong University of Science and Technology · Zhejiang University
H $^{2}$ MT: Semantic Hierarchy-Aware Hierarchical Memory Transformer
May 24, 2026Maryam Haghifam, Zifan He, Jason Cong +1Memory-Augmented Language Models Long-Context Language Model Inference
University of California, Los Angeles USA
Beyond the Target: From Imitation to Collaboration in Speculative Decoding
May 24, 2026Jinze Li, Yixing Xu, Guanchen Li +7Speculative Decoding LLM Inference Acceleration
Advanced Micro Devices, Inc., Beijing, China · The University of Hong Kong · University of North Texas
CONF-KV: Confidence-Aware KV Cache Eviction with Mixed-Precision Storage for Long-Horizon LLM
May 24, 2026Yubo Li, Yidi MiaoKV Caching Long-Context Language Model Inference
Carnegie Mellon University
EVA: Accelerating LLM Decoding via an Efficient Vector Quantization Architecture
May 22, 2026Bowen Duan, Cong Guo, Chiyue Wei +8Vector Quantization LLM Inference Acceleration
Duke University, USA
When NPUs Are Not Always Faster: A Stage-Level Analysis of Mobile LLM Inference
May 22, 2026Pu Li, Jiawen Qi, Qinyu ChenAI Accelerator Inference On-Device Language Model Inference
Leiden Institute of Advanced Computer Science (LIACS), Leiden University, The Netherlands.
A Simple Plug-in for Improving Eviction-Based KV Cache Compression
May 22, 2026Yuping Lin, Jiayuan Ding, Yue Xing +3KV Caching Memory-Efficient Optimization
1Michigan State University · 2Hippocratic AI

May 30, 2026, cs.AIJ/K move · Enter open · S save

Threshold-Based Exclusive Batching for LLM Inference

Weifang Zhang, Yuzhou Nie, Bowen Pang, Guangrui Ma+1

The Hong Kong Polytechnic University · UC Santa Barbara · Technology and Engineering Center for Space Utilization, Chinese Academy of Sciences+1

GPU Acceleration LLM Inference LLM Inference Scheduling LLM Inference Acceleration

Open paper AskPDF arXiv

Mixed batching (MB)--interleaving prefill and decode in a single batch--has become the standard scheduling strategy for large language model (LLM) inference due to its efficiency in maximizing compute and memory utilization. However, through controlled experiments, we find that prefill-decode interference inflates MB's per-step marginal cost above that of pure decode. On the high-bandwidth H200 (4.8 TB/s), this occurs only when decode tokens exceed 80% of the batch; however, on the bandwidth-constrained RTX PRO 6000 (1.792 TB/s), this threshold plummets to just 20%. Consequently, the optimal choice between MB and exclusive batching (EB) fundamentally depends on GPU memory bandwidth, model size, and workload composition. We derive a closed-form condition for this EB-MB performance crossover, along with asymptotically optimal phase-switching thresholds and memory-safe batch sizing for EB. Optimized EB achieves up to 41.9% higher throughput on bandwidth-constrained GPUs, while MB retains its advantage on high-bandwidth hardware with larger models. Our hybrid scheduler EB+ applies this condition online to dynamically switch between EB and MB without manual intervention. Under non-stationary traffic with distribution or concurrency shifts, EB+ attains the highest or near-highest throughput in every setting, outperforming MB by up to 36.4%.

Momentum

Related topics

Latest papers 677