LLM Inference Acceleration

LLM: Large Language Model

Follow this topic Ask about this topic

Momentum

108 papers in the last four weeks, up 120% on the four weeks before. 1.1% of all new papers.

Jul 13Week of Sep 28

Latest papers 677

All topics

Learning When to Sample: Confidence-Aware Selective Sampling for Efficient Chain-of-Thought Reasoning
Mar 9, 2026Juming Xiong, Kevin Guo, Congning Ni +7CoT Reasoning Efficient Language Model Reasoning
1Vanderbilt University · 2Vanderbilt University Medical Center · 3Intuit AI Research
InfoFlow KV: Information-Flow-Aware KV Recomputation for Long Context
Mar 5, 2026Xin Teng, Canyu Zhang, Shaoyi Zheng +3Retrieval-Augmented Generation KV Caching
New York University · New York University Shanghai · Duke University +1
Vectorizing the Trie: Efficient Constrained Decoding for LLM-based Generative Retrieval on Accelerators
Feb 26, 2026Zhengyang Su, Isay Katsman, Yueqi Wang +10Constrained Decoding Generative Retrieval
Google Mountain View, CA, USA · Yale University New Haven, CT, USA
MineDraft: A Framework for Batch Parallel Speculative Decoding
Feb 24, 2026Zhenwei Tang, Arun Verma, Zijian Zhou +4Speculative Decoding LLM Inference Acceleration
College of Computing and Data Science, Nanyang Technological University, Republic of Singapore · Singapore-MIT Alliance for Research and Technology Centre · Department of Computer Science, National University of Singapore, Republic of Singapore +1
Mobility-Aware Cache Framework for Scalable LLM-Based Human Mobility Simulation
Feb 17, 2026Hua Yan, Heng Tan, Yingxue Zhang +1Human Behavior Simulation LLM Inference Acceleration
Lehigh University Bethlehem, USA · State University of New York at Binghamton Binghamton, USA
QTALE: Quantization-Robust Token-Adaptive Layer Execution for LLMs
Feb 11, 2026Kanghyun Noh, Jinheon Choi, Yulhwa KimLLM Quantization LLM Compression
Department of Semiconductor Convergence Engineering, Sungkyunkwan University · Department of Electrical and Computer Engineering, Sungkyunkwan University · Department of Semiconductor Systems Engineering, Sungkyunkwan University
Near-Oracle KV Selection via Pre-hoc Sparsity for Long-Context Inference
Feb 9, 2026Yifei Gao, Lei Wang, Rong-Cheng Tu +3Self-Attention KV Caching
University College London, London, UK · Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences (CAS), Shenzhen 518055, China · The Chinese University of Hong Kong +1
ResidualKV: Residual-Based KV Cache Compression for Efficient Long-Context Inference
Feb 8, 2026Jitai Hao, Qiang Huang, Yaowei Wang +2Self-Attention KV Caching
Harbin Institute of Technology (Shenzhen), China
OmniMoE: An Efficient MoE by Orchestrating Atomic Experts at Scale
Feb 5, 2026Jingze Shi, Zhangyang Peng, Yizhang Zhu +3Mixture-of-Experts Inference LLM Inference Acceleration
The Hong Kong University of Science and Technology (Guangzhou) · Beijing Academy of Artificial Intelligence
TIDE: Temporal Incremental Draft Engine for Self-Improving LLM Inference
Feb 5, 2026Jiyoung Park, Hankyu Jang, Changseok Song +1Adaptive Inference LLM Inference Serving
Moreh, Inc.
Beyond Tokens: Semantic-Aware Speculative Decoding for Efficient Inference by Probing Internal States
Feb 3, 2026Ximing Dong, Shaowei Wang, Dayi Lin +2Speculative Decoding LLM Inference Acceleration
Centre for Software Excellence, Huawei, Canada · Department of Computer Science, University of Manitoba, Canada · School of Computing, Queen’s University, Canada
MatGPTQ: Efficient and Accurate Inference over Nested Quantized Models
Feb 3, 2026Maximilian Kleinegger, Elvir Crnčević, Dan AlistarhLLM Quantization LLM Inference Acceleration
1Vienna University of Technology, Vienna, Austria · Institute of Science & Technology Austria (ISTA), Vienna, Austria · 3Red Hat AI, Boston, USA
STILL: Selecting Tokens for Intra-Layer Hybrid Attention to Linearize LLMs
Feb 2, 2026Weikang Meng, Liangyu Huo, Yadan Luo +4LLM Inference Acceleration Linear Attention
SMULL Group, Harbin Institute of Technology, Shenzhen · Pengcheng Laboratory · UQMM Lab, University of Queensland +1
ConPress: Learning Efficient Reasoning from Multi-Question Contextual Pressure
Feb 1, 2026Jie Deng, Shining Liang, Jun Li +2Efficient Language Model Reasoning LLM Inference Acceleration
University of Chinese Academy of Sciences, Beijing, China · 2Microsoft, Beijing, China · 3Tongji University, Shanghai, China
Divide-and-Conquer CoT: RL for Reducing Latency via Parallel Reasoning
Jan 30, 2026Arvind Mahankali, Kaiyue Wen, Tengyu MaMulti-Agent LLM Systems CoT Reasoning
Stanford University
ARCQuant: Boosting NVFP4 Quantization with Augmented Residual Channels for LLMs
Jan 12, 2026Haoqian Meng, Yilun Luo, Yafei Zhao +3LLM Quantization 4-Bit Quantization
School of Computer Science and Technology, Tianjin University
JuDi: Revisiting Judge Decoding from First Principles via Training-Free Distributional Divergence
Jan 8, 2026Shengyin Sun, Yiming Li, Renxi Liu +5Speculative Decoding LLM Inference Acceleration
City University of Hong Kong · Huawei Technologies
Entropy-Aware Token Rejection for Improving Speculative Decoding
Dec 29, 2025Tiancheng Su, Meicong Zhang, Guoxiu HeSpeculative Decoding LLM Inference Acceleration
School of Economics and Management East China Normal University
Kascade: A Practical Sparse Attention Method for Long-Context LLM Inference
Dec 18, 2025Dhruv Deshmukh, Saurabh Goyal, Nipun Kwatra +1Self-Attention Long-Context Language Model Inference
Microsoft Research India.
xGR: Efficient Generative Recommendation Serving at Scale
Dec 12, 2025Qingxiao Sun, Tongxuan Liu, Shen Zhang +13LLM Serving LLM Inference Acceleration
Beihang University · JD Company · University of Science and Technology Beijing +1
Block Sparse Flash Attention
Dec 7, 2025Daniel Ohayon, Itay Lamprecht, Itay Hubara +3GPU Acceleration Self-Attention
Technion – Israel Institute of Technology, Haifa, Israel · Intel - Habana Labs · Intel – Habana Labs, Tel Aviv, Israel
Poodle: Seamlessly Scaling Down Large Language Models with Just-in-Time Model Replacement
Dec 5, 2025Nils Strassenburg, Boris Glavic, Tilmann RablCost-Aware Inference Small Language Models
Hasso Plattner Institute, Uni Potsdam · University of Illinois Chicago
Data-Free Pruning of Self-Attention Layers in LLMs
Dec 3, 2025Dhananjay Saikumar, Blesson VargheseLLM Pruning LLM Compression
School of Computer Science University of St Andrews United Kingdom
ThreadWeaver: Adaptive Threading for Efficient Parallel Reasoning in Language Models
Nov 24, 2025Long Lian, Sida Wang, Felix Juefei-Xu +7RL for Language Model Reasoning Inference-Time Scaling
Meta Superintelligence Labs (MSL), Menlo Park, CA, United States · UC Berkeley, Berkeley, CA, United States · UCSF, San Francisco, CA, United States
DCC: Data-Centric Compilation of Machine Learning Kernels for Processing-In-Memory Architectures
Nov 19, 2025Peiming Yang, Sankeerth Durvasula, Ivan Fernandez +4Compute-in-Memory LLM Inference Acceleration
University of Toronto & Vector Institute · Barcelona Supercomputing Center · ETH Zürich +2
CoT-X: An Adaptive Framework for Cross-Model Chain-of-Thought Transfer and Optimization
Nov 7, 2025Ziqian Bi, Yinzhi Wang, Tianyang Wang +6Efficient Language Model Reasoning LLM Inference Acceleration
1Purdue University, USA · 2Baruch College, City University of New York, USA · 3The Ohio State University, USA +6
BudgetMem: Training-Free Selective Memory for Cost-Efficient Long-Context Processing in Language Models
Nov 7, 2025Chandra Vamsi Krishna Alla, Harish Naidu Gaddam, Manohar Kommi +1LLM Compression Long-Context Language Model Inference
University of Texas at Arlington
NOSA: Native and Offloadable Sparse Attention
Oct 15, 2025Yuxiang Huang, Pengjie Wang, Jicheng Han +9KV-Cache Offloading GPU Acceleration
NLP Group, DCST, IAI, BNRIST, Tsinghua University, Beijing, China. · OpenBMB, China. · BUPT, Beijing, China. +1
Accelerating Attention with Basis Decomposition
Oct 2, 2025Jialin ZhaoSelf-Attention Transformer Attention
Department of Computer Science Tsinghua University
Think Right: Learning to Mitigate Under-Over Thinking via Adaptive, Attentive Compression
Oct 2, 2025Joykirat Singh, Justin Chih-Yao Chen, Archiki Prasad +3RL for Language Model Reasoning LLM Inference Acceleration
UNC Chapel Hill · The University of Texas at Austin · Microsoft Research

Mar 9, 2026, cs.CLJ/K move · Enter open · S save

Learning When to Sample: Confidence-Aware Selective Sampling for Efficient Chain-of-Thought Reasoning

Juming Xiong, Kevin Guo, Congning Ni, Weixin Liu+6

1Vanderbilt University · 2Vanderbilt University Medical Center · 3Intuit AI Research

CoT Reasoning Efficient Language Model Reasoning LLM Inference Acceleration Adaptive Sampling LLM Uncertainty Estimation LLM Reasoning+1

Open paper AskPDF arXiv

Large language models (LLMs) can achieve strong reasoning performance through chain-of-thought (CoT) reasoning, yet they often generate unnecessarily long reasoning paths that incur high inference cost. Self-consistency-based approaches push accuracy higher still, but they require sampling and aggregating multiple reasoning trajectories, leading to substantial computational overhead. In this paper, we introduce a confidence-aware selective sampling framework that, at inference time, analyzes a single reasoning trajectory to adaptively determine whether to rely on that trajectory alone or trigger multi-path sampling. The framework uses trajectory-level numeric features and sentence-level linguistic features extracted from reasoning states to guide selective multi-path reasoning. We train it on MedQA and evaluate it in-domain on MedQA and under calibration-only transfer on MathQA, MedMCQA, and MMLU, without further fine-tuning. Experimental results show that the proposed framework maintains comparable performance to full and efficient multi-path reasoning baselines, with accuracy changes of

-0.41 \pm 0.58

and

-0.31 \pm 0.58

percentage points, respectively, while reducing token usage by

71.7 \pm 5.0%

and

36.6 \pm 9.1%

. These findings demonstrate that reasoning trajectories contain rich signals for uncertainty estimation, enabling a simple, transferable mechanism to balance accuracy and efficiency in LLM reasoning.

Momentum

Related topics

Latest papers 677