LLM Inference Acceleration

LLM: Large Language Model

Follow this topic Ask about this topic

Momentum

108 papers in the last four weeks, up 120% on the four weeks before. 1.1% of all new papers.

Jul 13Week of Sep 28

Latest papers 677

All topics

CARVE: Verified Expansion for Variable-Length Generation in Diffusion Language Models
Aug 31, 2026Wail Bouhedja, Amr Mohamed, Guokan ShangMasked Diffusion Models LLM Inference Acceleration
1MBZUAI · 2Sorbonne Université · 3Ecole Polytechnique
Deploying DeepSeek 175B Locally on a Single Consumer-Grade RTX 4060 Laptop with 32GB RAM for 200k-Scale Protein-Ligand Virtual Screening
Aug 31, 2026Rui Xiao, Yili XuProtein-Ligand Binding Affinity Prediction Virtual Screening
Hangzhou Tsingxin quantum Co., Ltd.
TopoCompress: Long Context Compression via Graph-Wired Semantic Trajectories
Aug 31, 2026Daniel Agyei Asante, Yang LiLLM Compression Long-Context QA
1Iowa State University, United States
Tensor Methods for Language Models: From Token Representation to Training, Adaptation, Inference, Compression, and Interpretability
Aug 31, 2026Matvei Tarasov, Salman Ahmadi-Asl, Andre L. F. de Almeida +1LLM Compression LLM Inference Acceleration
Innopolis University, Innopolis 420500, Russia · Department of Teleinformatics Engineering, Federal University of Cear´a, Fortaleza 60455-760, Brazil · Systems Research Institute of Polish Academy of Science and Warsaw University of Technology, Poland
Event-Driven Language Models with Sparse Neural Activity for Neuromorphic Hardware
Aug 31, 2026Simon Richter, Ruhai Lin, Jason Yik +4Activation Sparsity LLM Inference Acceleration
Department of Electrical and Computer Engineering, Aarhus University, Aarhus, Denmark · Department of Computer Science and Engineering, University of California, Santa Cruz, Santa Cruz, USA · School of Engineering and Applied Sciences, Harvard University, Cambridge, USA +1
DASC: Decay-Aware State Compression for Hybrid Linear-Attention Serving
Aug 31, 2026Yanqi Yu, Pingwei Sun, Jianchao Tan +4KV Caching Memory-Efficient Inference
East China Normal University · Meituan · South China University of Technology
Tail-Replay: Escaping the Curse of Linear Attention in Prefix Caching for Hybrid LLMs
Aug 31, 2026Yirui Liu, Ruoling Qi, Xuaner Wu +2LLM Inference Acceleration Prefix Caching
Institute of Artificial Intelligence, China Telecom (TeleAI) · Shanghai Jiao Tong University
CateKV: On Sequential Consistency for Long-Context LLM Inference Acceleration
Aug 31, 2026Haoyun Jiang, Haolin Li, Jianwei Zhang +7Long-Context Language Model Inference LLM Inference Acceleration
1CMIC, Shanghai Jiao Tong University · 2Alibaba Group · 3Fudan University
Strong Drafts Need Compact Memories: Long-Context Speculative Decoding with Compressed KV Cache
Aug 31, 2026Tong Yuan, Chengxi Liao, Zeyi WenMemory-Augmented Language Models KV Caching
Data Science and Analytics Thrust, Information Hub, The Hong Kong University of Science and Technology (Guangzhou)
Verification-Aware Training for Speculative Decoding
Aug 31, 2026Geonmo Gu, Byeongho Heo, HeeJae Jun +4Speculative Decoding LLM Inference Acceleration
NAVER AI Lab · NAVER AI Search Platform · Korea University
AutoCRAT: Within-trajectory Joint Control of Stochasticity and Compute for LLM Reasoning
Aug 30, 2026Hanjun Luo, Qiushi Liu, Jingya Zhang +8Adaptive Inference LLM Inference Acceleration
New York University · New York University Abu Dhabi · University of Washington Seattle +4
ReTrace: Rejected-Trajectory Conditioning for Speculative Decoding
Aug 30, 2026Luxi Lin, Zhanpeng Zeng, Shuang Peng +2Speculative Decoding LLM Inference Acceleration
1Xiamen University, 2ByteDance
Sliding-window beats linear attention
Aug 28, 2026Alexia Jolicoeur-Martineau, Rhea Sanjay Sukthanker, Pashmina Cameron +1KV Caching LLM Inference Acceleration
Microsoft Applied Sciences Group (ASG) · Independent
HyQuant: Hybrid-Precision Quantization for LLM Attention
Aug 28, 2026Jiatong Ding, Bingxin Xing, Yu Zhang +9Mixed-Precision Quantization LLM Quantization
Shanghai Jiao Tong University · Xi’an Jiaotong University · Xiamen University +1
Beyond Parallel Blindness: Information Floors and Model Gaps in Block Drafting
Aug 27, 2026Xinwei Qiang, Xiang Fang, Chang Chen +2Speculative Decoding LLM Inference Acceleration
University of California San Diego
DAMP: Decay-Aware Mixed-Precision Recurrent-State Quantization
Aug 27, 2026Tao Zhang, Jianchao Tan, Pingwei Sun +6Mixed-Precision Quantization LLM Quantization
South China University of Technology · Meituan · East China Normal University
FAMPWQ: Fisher Information-based Adaptive Mixed Precision Weight Quantization for Effective LLM Inference
Aug 24, 2026Gongwei Lee, Ji Liu, Juncheng Jia +1Mixed-Precision Quantization LLM Quantization
School of Computer Science and Technology, Soochow University, Suzhou, China · Hithink Research, Hangzhou, China · Electronic Engineering, Tsinghua University, Beijing, China
How Weight Encoding Affects Language Model Placement and Performance on the Apple Neural Engine
Aug 22, 2026Shahir M ALLM Quantization AI Accelerator Inference
Independent researcher
LiLiCorr: Lightweight Likelihood Correlation of Parallel Drafts for Speculative Decoding
Aug 20, 2026Matan Rusanovsky, Yoav Miron, Roy Uziel +5Speculative Decoding LLM Inference Acceleration
NVIDIA
Self-Indexing Attention for Compression-Compatible Sparse Long-Context LLM Inference
Aug 16, 2026Xu Yang, Jiapeng Zhang, Yuxin Chen +4Long-Context Language Model Inference LLM Inference Acceleration
Hunan University · Tencent
Reduced Matrix Multiplication: Input-Adaptive Matrix-Product Reduction for LLM Inference
Aug 13, 2026Zixuan Lan, Yanhong Li, Jiawei ZhouInference-Time Optimization LLM Inference Acceleration
University of Chicago · Independent Researcher · Stony Brook University
vToken: Token-Level Virtualization for Reclaimable KV Caches
Aug 13, 2026Yuanhang Gao, Xiangrui Yang, Yuanfeng Chen +4KV Caching KV-Cache Eviction
National University of Defense Technology · Changsha, China · Peking University +1
SPADE: Speculative Decoding for Precise and Low Cost Distributed Edge Cloud Inference
Aug 13, 2026Divya Jyoti Bajpai, Kishan Kumar Upadhyay, Manjesh Kumar HanawalEdge Computing Cost-Aware Inference
MLiONS, Department of IEOR, IIT Bombay Mumbai, Maharashtra-400076, India
Decoupled Contrastive Decoding via Expert-Aligned Drafting
Aug 13, 2026Zhixuan Liu, Zhichen Dong, Yuanfu Wang +1Speculative Decoding LLM Inference Acceleration
Shanghai Jiao Tong University · Shanghai Artificial Intelligence Laboratory
Trie Automata for Constrained Decoding over Large Finite Sets
Aug 12, 2026Xingzi Xu, Karim BouyarmaneConstrained Decoding LLM Inference Acceleration
Amazon
Matryoshka Language Model Suites
Aug 10, 2026Nathan Godey, Yoav ArtziEfficient Language Model Training LLM Inference Acceleration
Cornell University
Depth-adaptive Inference of Looped Language Models via Continuous Depth Batching
Aug 10, 2026Kristian Schwethelm, Daniel Rueckert, Georgios KaissisLLM Inference LLM Inference Scheduling
Chair for AI in Healthcare and Medicine, Technical University of Munich (TUM), Germany · Department of Computing, Imperial College London, UK · Munich Center for Machine Learning (MCML), Germany +1
SwiftQK: Fast and Communication-Efficient Tensor Parallelism for Query-Key Normalization
Aug 10, 2026Gyudong Kim, Wonjun Han, Young Geun KimGPU Kernel Optimization Tensor Parallelism
Department of Computer Science and Engineering, Korea University, Seoul 02855, Republic of Korea
Tied Trit-Planes: Constraining PTQTP to a Uniform Nine-Level Quantizer, with a Persistent Folded Format for Disk-Streamed Mixture-of-Experts Serving
Aug 9, 2026Matteo GrellaTernary Quantization Mixture-of-Experts Inference
Crisis24
DistillCache: KL-Guided Adaptive KV-Cache Eviction for Memory-Efficient LLM Inference
Aug 9, 2026Asaad AlthoubiKV Caching Long-Context Language Model Inference
Oklahoma State University Stillwater, OK USA

Aug 31, 2026, cs.AIJ/K move · Enter open · S save

CARVE: Verified Expansion for Variable-Length Generation in Diffusion Language Models

Wail Bouhedja, Amr Mohamed, Guokan Shang

1MBZUAI · 2Sorbonne Université · 3Ecole Polytechnique

Masked Diffusion Models LLM Inference Acceleration Diffusion Language Model Decoding Diffusion Language Models LLM Code Generation LLM Reasoning

Open paper AskPDF arXiv

Masked diffusion language models predict tokens from a partially observed response canvas, enabling bidirectional conditioning and parallel token refinement. Yet standard masked-diffusion decoders use a rigid inference interface: the number of masked positions allocated to the answer is fixed before generation begins. Choosing this length is difficult. A short canvas can truncate reasoning or code, while a long canvas wastes computation and can perturb denoising. We introduce CARVE (Counterfactual-Aware Reveal with Verified Expansion), a training-free variable-length algorithm for masked diffusion LMs. Starting from a shorter canvas, CARVE can grow the response during decoding by inserting additional [MASK] positions. Rather than keeping every insertion, CARVE tests a candidate expanded canvas and asks a counterfactual question: would the model make similar predictions for the unresolved positions in the original canvas if the extra masked space were present? The inserted masks are kept only when they induce low Jensen-Shannon (JS) divergence on aligned unresolved positions. This makes length growth a verified stability decision rather than a pure confidence heuristic. CARVE applies without retraining to both full-canvas and blockwise diffusion decoders. Across code generation and mathematical reasoning benchmarks, CARVE consistently improves average performance over fixed-length baselines across all evaluated model families. Crucially, CARVE achieves these accuracy gains while reducing inference cost, reaching half the FLOPs of fixed-length decoding in some settings.

Momentum

Related topics

Latest papers 677