LLM Inference Acceleration

LLM: Large Language Model

Follow this topic Ask about this topic

Momentum

108 papers in the last four weeks, up 120% on the four weeks before. 1.1% of all new papers.

Jul 13Week of Sep 28

Latest papers 677

All topics

DeInfer: Efficient Parallel Inferencing for Decomposed Large Language Models
Apr 20, 2026You-Liang Huang, Xinhao Huang, Chengxi Liao +1LLM Inference LLM Inference Acceleration
Boston University · Boston, Massachusetts, USA · HKUST (Guangzhou) +2
Predictive Multi-Tier Memory Management for KV Cache in Large-Scale GPU Inference
Apr 19, 2026Sanjeev Rao GanjihalKV-Cache Offloading KV Caching
Independent Researcher
Hive: A Multi-Agent Infrastructure for Algorithm- and Task-Level Scaling
Apr 19, 2026Zizhang Luo, Yuhao Luo, Youwei Xiao +3LLM Inference Efficiency Multi-Agent LLM Systems
Peking University Beijing, China
CRISP: Compressing Redundancy in Chain-of-Thought via Intrinsic Saliency Pruning
Apr 19, 2026Yangsong Lan, Hongliang Dai, Piji LiLLM Pruning LLM Compression
College of Artificial Intelligence, Nanjing University of Aeronautics and Astronautics, Nanjing, China · The Key Laboratory of Brain-Machine Intelligence Technology, Ministry of Education, Nanjing, China.
Probabilistic Programs of Thought
Apr 19, 2026Poorva Garg, Renato Lui Geh, Daniel Israel +3LLM-Based Program Synthesis Efficient Language Model Reasoning
University of California Los Angeles · Allen Institute for AI
UCCL-Zip: Lossless Compression Supercharged GPU Communication
Apr 19, 2026Shuang Ma, Chon Lam Lao, Zhiying Xu +8GPU Kernel Optimization GPU Acceleration
UC Davis · Harvard University · Amazon Web Services +1
Open-TQ-Metal: Fused Compressed-Domain Attention for Long-Context LLM Inference on Apple Silicon
Apr 18, 2026Sai VegasenaAI Accelerator Inference Self-Attention
Ensue
Step-GRPO: Internalizing Dynamic Early Exit for Efficient Reasoning
Apr 18, 2026Benteng Chen, Weida Wang, Shufei Zhang +2RL for Language Model Reasoning LLM Inference Acceleration
East China Normal University · Shanghai AI Laboratory · Fudan University +1
SinkRouter: Sink-Aware Routing for Efficient Long-Context Decoding in Large Language and Multimodal Models
Apr 18, 2026Junnan Liu, Xinyan Liu, Peifeng Gao +4GPU Acceleration Self-Attention
Harbin Institute of Technology (Weihai) · University of Hong Kong, Hong Kong, China · City University of Hong Kong, Hong Kong, China
Cut Your Losses! Learning to Prune Paths Early for Efficient Parallel Reasoning
Apr 17, 2026Jiaxi Bi, Tongxu Luo, Wenyu Du +2LLM Inference Acceleration Large Reasoning Models
The Chinese University of Hong Kong, Shenzhen · Shenzhen Loop Area Institute · DualityRL
Sparse Prefix Caching for Hybrid and Recurrent LLM Serving
Apr 17, 2026Mikhail Shirokikh, Sergey NikolenkoLLM Serving KV-Cache Management
EvoSpec: Evolving Speculative Decoding via Real-Time Vocabulary and Parameter Adaptation
Apr 17, 2026Shuyu Zhang, Lingfeng Pan, Qicheng Wang +6Speculative Decoding LLM Inference Acceleration
School of Computer Science and Technology Xidian University Xi’an, China
Faster LLM Inference via Sequential Monte Carlo
Apr 17, 2026Yahya Emara, Mauricio Barba da Costa, Chi-Chih Chang +4Speculative Decoding LLM Inference Acceleration
Cornell University · Makora · MIT +1
Ragged Paged Attention: A High-Performance and Flexible LLM Inference Kernel for TPU
Apr 16, 2026Jevin Jiang, Ying Chen, Blake A. Hechtman +2Efficient Transformer Inference AI Accelerator Inference
Google
From Tokens to Steps: Verification-Aware Speculative Decoding for Efficient Multi-Step Reasoning
Apr 16, 2026Kiran Purohit, Ramasuri Narayanam, Soumyabrata PalLLM Answer Verification Speculative Decoding
IIT Kharagpur · Adobe Research
ARGUS: Agentic GPU Optimization Guided by Data-Flow Invariants
Apr 16, 2026Haohui Mai, Xiaoyan Guo, Xiangyun Ding +7GPU Kernel Optimization GPU Acceleration
UC Riverside5 · UCAS4 · Tsinghua University2 +2
MemoSight: Unifying Context Compression and Multi Token Prediction for Reasoning Acceleration
Apr 16, 2026Xinyu Liu, Xin Liu, Bo Jin +8Multi-Token Prediction KV Caching
School of Computer Science and Engineering, Northeastern University, China · Meituan Inc. · NiuTrans Research, Shenyang, China
RACER: Retrieval-Augmented Contextual Rapid Speculative Decoding
Apr 16, 2026Zihong Zhang, Zuchao Li, Lefei Zhang +2Speculative Decoding LLM Inference Acceleration
School of Artiﬁcial Intelligence, Wuhan University, Wuhan, China · School of Computer Science, Wuhan University, Wuhan, China · School of Information Management, Wuhan University, Wuhan, China +1
TrigReason: Trigger-Based Collaboration between Small and Large Reasoning Models
Apr 16, 2026Yi Zhao, Yajuan Peng, Cam-Tu Nguyen +4Efficient Language Model Reasoning LLM Inference Acceleration
AGI Institute, School of Computer Science, Shanghai Jiao Tong University, Shanghai, China · Shanghai Key Laboratory for Intelligent Information Processing, Fudan University · State Key Laboratory for Novel Software Technology, Nanjing University +2
ELMoE-3D: Leveraging Intrinsic Elasticity of MoE for Hybrid-Bonding-Enabled Self-Speculative Decoding in On-Premises Serving
Apr 16, 2026Yuseon Choi, Jingu Lee, Jungjun Oh +5LLM Serving Mixture-of-Experts Inference
KAIST · Samsung Electronics · GIST
ConfLayers: Adaptive Confidence-based Layer Skipping for Self-Speculative Decoding
Apr 16, 2026Walaa Amer, Uday das, Fadi KurdahiSpeculative Decoding LLM Inference Acceleration
Department of Electrical Engineering and Computer Sciences, University of California Irvine, Irvine, USA · Advanced Micro Devices, San Jose, USA
Tessera: Unlocking Heterogeneous GPUs through Kernel-Granularity Disaggregation
Apr 11, 2026Tiancheng Hu, Jin Qin, Zheng Wang +10GPU Acceleration LLM Inference Acceleration
SCS, Peking University, Beijing, China · University of Chinese Academy of Sciences, Beijing, China · University of Leeds, West Yorkshire, England +4
SMART: When is it Actually Worth Expanding a Speculative Tree?
Apr 9, 2026Lifu Wang, Pan ZhouInference-Time Optimization GPU Acceleration
Singapore Management University, Singapore
Flux Attention: Context-Aware Hybrid Attention for Efficient LLMs Inference
Apr 8, 2026Quantong Qiu, Zhiyi Hong, Yi Yang +5Self-Attention Long-Context Language Model Inference
School of Computer Science and Technology, Soochow University · Baidu Inc, China
Screening Is Enough
Apr 1, 2026Ken M. NakanishiSelf-Attention Long-Context Language Modeling
Center for Emergent Matter Science (CEMS), RIKEN · Graduate School of Science, The University of Tokyo
Executing as You Generate: Hiding Execution Latency in LLM Code Interpreters
Apr 1, 2026Zhensu Sun, Zhihao Lin, Zhi Chen +4LLM Inference Acceleration Pipeline Parallelism
Singapore Management University · Beihang University
HISA: Efficient Hierarchical Indexing for Fine-Grained Sparse Attention
Mar 30, 2026Yufei Xu, Fanxu Meng, Fan Jiang +11Self-Attention LLM Inference Acceleration
Xiaohongshu
SpecEyes: Accelerating Agentic Multimodal LLMs via Speculative Perception and Planning
Mar 24, 2026Haoyu Huang, Jinfa Huang, Zhongwei Wan +3Efficient Multimodal Inference LLM Inference Acceleration
Xiamen University, Xiamen, Fujian, China · University of Rochester, Rochester, NY, USA · The Ohio State University, Columbus, OH, USA
Learning When to Attend: Conditional Memory Access for Long-Context LLMs
Mar 18, 2026Sakshi Choudhary, Aditya Chattopadhyay, Luca Zancato +4Self-Attention Long-Context Language Modeling
Department of Electrical and Computer Engineering, Purdue University, USA · AWS Agentic AI, USA
FlexServe: A Fast and Secure LLM Serving System for Mobile Devices with Flexible Resource Isolation
Mar 10, 2026Yinpeng Wu, Yitong Chen, Lixiang Wang +3LLM Serving On-Device Language Model Inference
Institute of Parallel and Distributed Systems, Shanghai Jiao Tong University

Apr 20, 2026, cs.CLJ/K move · Enter open · S save

DeInfer: Efficient Parallel Inferencing for Decomposed Large Language Models

You-Liang Huang, Xinhao Huang, Chengxi Liao, Zeyi Wen

Boston University · Boston, Massachusetts, USA · HKUST (Guangzhou)+2

LLM Inference LLM Inference Acceleration

Open paper AskPDF arXiv

Existing works on large language model (LLM) decomposition mainly focus on improving performance on downstream tasks, but they ignore the poor parallel inference performance when trying to scale up the model size. To mitigate this important performance issue, this paper introduces DeInfer, a high-performance inference system dedicated to parallel inference of decomposed LLMs. It consists of multiple optimizations to maximize performance and be compatible with state-of-the-art optimization techniques. Extensive experiments are carried out to evaluate DeInfer's performance, where the results demonstrate its superiority, suggesting it can greatly facilitate the parallel inference of decomposed LLMs.

Momentum

Related topics

Latest papers 677