LLM Inference Acceleration

LLM: Large Language Model

Follow this topic Ask about this topic

Momentum

108 papers in the last four weeks, up 120% on the four weeks before. 1.1% of all new papers.

Jul 13Week of Sep 28

Latest papers 683

All topics

LayerRoute: Input-Conditioned Adaptive Layer Skipping via LoRA Fine-Tuning for Agentic Language Models
Jun 1, 2026Prateek Kumar SikdarLLM Inference Acceleration LLM Agents
Accenture Bengaluru, India
LongAttnComp: Cross-Family Context Compression for Long-Context Reasoning
May 31, 2026Mengmeng Ji, Ravi Shanker Raju, Jonathan Lingjie Li +1LLM Compression LLM Fine-Tuning
SambaNova Systems, Inc. San Jose, CA, USA
Soft-NBCE: Entropy-Weighted Chunk Fusion for Long-Context
May 31, 2026Shihao Ji, Mingyu Li, Zihui SongLong-Context Language Model Inference LLM Inference Acceleration
Chunjiang Intelligence · Beijing Normal University
Hybrid Verified Decoding: Learning to Allocate Verification in Speculative Decoding
May 31, 2026Xin Su, Dawid Majchrowski, Fangyuan Yu +5Speculative Decoding LLM Inference Acceleration
Thoughtworks · Nvidia
ViBE: Co-Optimizing Workload Skew and Hardware Variability for MoE Serving
May 30, 2026Seokjin Go, Marko Scrbak, Ephrem Wu +2Expert Load Balancing Mixture-of-Experts Inference
Georgia Institute of Technology Atlanta, GA, USA · Advanced Micro Devices, Inc. Austin, TX, USA · Advanced Micro Devices, Inc. Santa Clara, CA, USA +1
ProactiveLLM: Learning Active Interaction for Streaming Large Language Models
May 30, 2026Junlong Tong, Yao Zhang, Anhao Zhao +3LLM Inference LLM Inference Acceleration
Eastern Institute of Technology, Ningbo · Shanghai Jiao Tong University · Hong Kong Polytechnic University +1
Threshold-Based Exclusive Batching for LLM Inference
May 30, 2026Weifang Zhang, Yuzhou Nie, Bowen Pang +2GPU Acceleration LLM Inference
The Hong Kong Polytechnic University · UC Santa Barbara · Technology and Engineering Center for Space Utilization, Chinese Academy of Sciences +1
AdaptR1: Reinforcement Learning Based Adaptive Interleaved Thinking in Multi-hop Question Answering
May 29, 2026Yuxin Wang, Jiahao Lu, Qifeng Wu +5Multi-Hop QA RL for Language Model Reasoning
1Computer Science, Fudan University · Institute of Modern Languages and Linguistics, Fudan University · 3Shanghai Innovation Institute +1
Speculative Pipeline Decoding: Higher-Accruacy and Zero-Bubble Speculation via Pipeline Parallelism
May 29, 2026Yijiong Yu, Huazheng Wang, Shuai Yuan +2Speculative Decoding LLM Inference Acceleration
aOregon State University · bDeepSolution
CoMem: Context Management with A Decoupled Long-Context Model
May 29, 2026Yuwei Zhang, Chengyu Dong, Shuowei Jin +11LLM Inference LLM Inference Acceleration
Halıcıoğlu Data Science Institute, University of California, San Diego · Amazon
Reducing the GPU Memory Bottleneck with Lossless Compression for ML -- Extended
May 29, 2026Aditya K Kamath, Arvind Krishnamurthy, Marco Canini +1Efficient Neural Network Inference GPU Kernel Optimization
University of Washington Seattle, WA, USA · KAUST Thuwal, Saudi Arabia
Speculative Decoding Across Languages
May 28, 2026Nirajan Paudel, Michael Ginn, Luc De Nardi +1Multilingual Language Models Speculative Decoding
University of Colorado
Unlocking the Working Memory of Large Language Models for Latent Reasoning
May 28, 2026Lukas Aichberger, Sepp HochreiterMemory-Augmented Language Models Efficient Language Model Reasoning
ELLIS Unit Linz and LIT AI Lab, Institute for Machine Learning, Johannes Kepler University Linz, Austria · NXAI GmbH, Linz, Austria
Accelerating Constrained Decoding with Token Space Compression
May 28, 2026Michael Sullivan, Alexander KollerConstrained Decoding LLM Inference Acceleration
Department of Language Science and Technology Saarland Informatics Campus Saarland University, Saarbrücken, Germany
Domino: Decoupling Causal Modeling from Autoregressive Drafting in Speculative Decoding
May 28, 2026Jianuo Huang, Yaojie Zhang, Qituan Zhang +3Speculative Decoding LLM Inference Acceleration
1EPIC Lab, Shanghai Jiao Tong University · School of Software Engineering, HUST · 3UESTC +2
Draft-OPD: On-Policy Distillation for Speculative Draft Models
May 28, 2026Haodi Lei, Yafu Li, Haoran Zhang +8Speculative Decoding LLM Inference Acceleration
Shanghai Jiao Tong University · Shanghai AI Laboratory · The Chinese University of Hong Kong +3
Parallax: Parameterized Local Linear Attention for Language Modeling
May 27, 2026Yifei Zuo, Dhruv Pai, Zhichen Zeng +3Language Model Pretraining Self-Attention
Northwestern University · Tilde Research · University of Washington
From Human Guidance to Autonomy: Agent Skill System for End-to-End LLM Deployment on Spatial NPUs
May 27, 2026Jiajie Li, Erwei Wang, Zhiru Zhang +1AI Accelerator Inference LLM Agent Orchestration
Cornell University · AMD
Inference-Native Zeroth-Order Optimization for LLMs
May 27, 2026Zelin Li, Caiwen DingLLM Serving Zeroth-Order Optimization
University of Minnesota
Augmenting Attention with Exponentially Decaying Memory Improves Query-Aware KV Sparsity
May 27, 2026Xiuying Wei, Caglar GulcehreMemory-Augmented Language Models Self-Attention
EPFL, CLAIRE Lausanne, Switzerland
RW-TTT: Batched Serving for Request-Owned Test-Time Training State
May 27, 2026Jian Yang, Zhizhuo Kou, Yao Tian +4LLM Serving Test-Time Optimization
1The Hong Kong University of Science and Technology · 2The Chinese University of Hong Kong · 3National University of Singapore
Locality-Aware Redundancy Pruning for LLM Depth Compression
May 27, 2026Vincent-Daniel Yun, Youngrae Kim, Woosang Lim +3LLM Pruning LLM Compression
University of Southern California, United States · Neural Superintelligence Lab, MODULABS, Republic of Korea · Seoul National University, Republic of Korea +1
UNIQUE: Universal Top-k Sparse Attention for Training-free Inference and Sparsity-aware Training
May 26, 2026Keqi Deng, Shaoshi Ling, Ruchao Fan +1Self-Attention Long-Context Language Model Inference
Microsoft, USA
Pair-In, Pair-Out: Latent Multi-Token Prediction for Efficient LLMs
May 26, 2026Wenhui Tan, Minghao Li, Xiaoqian Ma +5Multi-Token Prediction LLM Compression
Gaoling School of Artificial Intelligence, Renmin University of China · AI Platform, Xiaohongshu Inc. · University of Electronic Science and Technology of China +1
NestedKV: Nested Memory Routing for Long-Context KV Cache Compression
May 26, 2026Hong Chen, Xiang Liu, Yubo Gao +5KV Caching Long-Context Language Modeling
The Hong Kong University of Science and Technology (Guangzhou) · Jimei University
Do Language Models Need Sleep? Offline Recurrence for Improved Online Inference
May 25, 2026Sangyun Lee, Sean McLeish, Tom Goldstein +1Memory-Augmented Language Models Long-Context Language Model Inference
Carnegie Mellon University · University of Maryland
Selective Latent Thinking: Adaptive Compression of LLM Reasoning Chains
May 25, 2026Hui Xie, Jie Liu, Ziyue Qiao +1Efficient Language Model Reasoning LLM Inference Acceleration
Eindhoven University of Technology, Netherlands · School of Computing and Information Technology, Great Bay University, China
Neural Router: Semantic Content Matching for Agentic AI
May 25, 2026Lauri Lovén, Abhishek Kumar, Alexander Engelhardt +5Semantic Textual Similarity LLM Inference Acceleration
Future Computing Group, University of Oulu, Finland · University of Jyväskylä, Finland · Department of Computer Science, University of Helsinki, Finland +1
Fine-Tuning and Serving Gemma 4 31B on Google Cloud TPU: A Technical Comparison with GPU Baselines
May 25, 2026Jatin Kishnani, Mayank Goel, Amit Singh +2AI Accelerator Inference LLM Serving
IndexMem: Learned KV-Cache Eviction with Latent Memory for Long-Context LLM Inference
May 25, 2026Xintong Yang, Hao Gu, Binxing Xu +6Memory-Augmented Language Models Self-Attention
The Hong Kong University of Science and Technology · Zhejiang University

Jun 1, 2026, cs.CLJ/K move · Enter open · S save

LayerRoute: Input-Conditioned Adaptive Layer Skipping via LoRA Fine-Tuning for Agentic Language Models

Prateek Kumar Sikdar

Accenture Bengaluru, India

LLM Inference Acceleration LLM Agents LLM Routing

Open paper AskPDF arXiv

Agentic language model systems alternate between two structurally distinct step types: structured tool calls (short, deterministic, low perplexity) and open-ended planning/reasoning steps (long, complex, high perplexity). Despite this heterogeneity, current inference systems apply identical compute to every step. We introduce LayerRoute, a lightweight adapter that learns to selectively skip transformer blocks on a per-input basis. LayerRoute augments each of the 24 transformer blocks in Qwen2.5-0.5B-Instruct with: (1) a per-layer router (~897 parameters, Linear(896,1)) that outputs a hard binary gate via the straight-through estimator, and (2) LoRA adapters (rank 8, ~1.08M parameters) on the Q/K/V/O attention projections. The backbone weights remain frozen. A single end-to-end training pass on agentic data (Hermes, Glaive, GSM8K, Turing) with a gate regularisation term forces the system to discover which blocks are skippable per input type. After 3,000 steps (6.4 minutes on an A100 40GB), LayerRoute achieves a 12.91% skip differential: tool calls skip 15.25% of FLOPs while planning steps skip only 2.34%, using only 1.10M trainable parameters (0.22% of the 494M backbone). Quality improves over the base model due to LoRA adaptation, with perplexity delta of -1.29 on tool calls and -1.30 on planning.

Momentum

Related topics

Latest papers 683