LLM Inference Acceleration

LLM: Large Language Model

Follow this topic Ask about this topic

Momentum

108 papers in the last four weeks, up 120% on the four weeks before. 1.1% of all new papers.

Jul 13Week of Sep 28

Latest papers 677

All topics

FusionML: Prefill, Not Decode - Mechanism and Boundaries of CPU+GPU Co-Execution on Unified-Memory Apple Silicon
Jul 24, 2026Om MohiteLLM Serving LLM Inference Acceleration
Independent Researcher, Mumbai, India
Unified Static-Dynamic Pruning for Efficient LLM Inference
Jul 24, 2026Jinhyeok Kim, Yejoon Lee, Jaeyoung DoEfficient Neural Network Inference LLM Pruning
Seoul National University Seoul, South Korea
RIS-Kernel: A Model-Agnostic Architecture for Long-Context LLM Inference via Sparse Attention
Jul 24, 2026Anderson R. SantosLong-Context Language Model Inference Memory-Efficient Inference
Faculty of Computing, Federal University of Uberlândia (UFU), Brazil.
Windowed-MTP: Removing the Full-Context Draft-KV Tax at Million-Token Context
Jul 23, 2026Alagappan ValliappanMulti-Token Prediction Long-Context Language Model Inference
NVIDIA
Adaptive Depth Sparse Framework: Similarity-Driven Resource Allocation for Pre-Trained LLMs
Jul 23, 2026Yidu Wu, Xiang Wang, Kejie Zhao +3LLM Pruning LLM Inference Acceleration
Department of Electronic and Electrical Engineering, Southern University of Science and Technology, Shenzhen, China · ACS Lab, Huawei Technologies Co., Ltd., Shenzhen, China · Department of Computer Science and Engineering, Southern University of Science and Technology, Shenzhen, China
How Fast Can Reward Models Score? A Systems Study of C++ and PyTorch Inference Runtimes for RLHF
Jul 22, 2026Venkata Naga Sai Vishnu Rohit Pulipaka, Anish Katta, Deva Rohit Reddy PeddireddyEfficient Neural Network Inference Reward Modeling
Efficient Clustering with Provable Guardrails for LLM Inference at Scale
Jul 22, 2026Longshaokan Wang, Wai Tsang Keung, Punit Ghodasara +3Clustering LLM Inference Acceleration
Amazon
CausalGate: Causal Importance Distillation for Transformer Module Pruning
Jul 21, 2026Kiran Nair, Smriti Regmi, Rodrigue RizkEfficient Transformer Inference LLM Pruning
USD Artificial Intelligence Research Lab, Department of Computer Science University of South Dakota, Vermillion, SD 57069, USA
AdaFlash: Adaptive Speculative Decoding via On-Policy Distilled Diffusion Drafters
Jul 21, 2026Yu-Yang Qian, Hao-Cong Wu, Chen Chen +4Speculative Decoding for Diffusion Language Models Speculative Decoding
State Key Laboratory for Novel Software Technology, Nanjing University · School of Artificial Intelligence, Nanjing University · Huawei Foundation Model Dept
BaseRT: Advancing Best-in-Class LLM Inference with Apple M5 Neural Accelerators
Jul 21, 2026Fabian Waschkowski, Prabod Rathnayaka, Lukas WesemannAI Accelerator Inference On-Device Language Model Inference
Base Compute, Melbourne, Australia
FlashRT: Agent Harness for Guiding Agents to Deploy Real-Time Multimodal Applications
Jul 20, 2026Krish Agarwal, Zhuoming Chen, Yanyuan Qin +3Efficient Multimodal Inference LLM Inference Acceleration
Carnegie Mellon University · AMD · University at Buffalo
SelectInfer: Selective Neuron Loading and Computation for On-Device LLMs
Jul 20, 2026Huzaifa Shaaban Kabakibo, Eric Schniedermeyer, Artem Burchanow +1On-Device Language Model Inference Memory-Efficient Inference
Department of Computer Networks, Paderborn University, Paderborn, Germany
MXSens: Sensitivity-Aware Mixed-Precision Quantization for Efficient LLM Inference
Jul 20, 2026Simla Burcu Harma, Danila Mishin, Zhengyuan Su +7Mixed-Precision Quantization LLM Quantization
EcoCloud, EPFL · Tsinghua University · MangoBoost Inc. +2
C $^2$ KV: Compressed and Composable KV Cache Reuse for Efficient LLM Inference
Jul 20, 2026Chuheng Du, Junyi Chen, Hanlin Tang +7KV Caching LLM Inference Acceleration
School of Computer Science Shanghai Jiao Tong University Shanghai, China · Alibaba Group Hangzhou, China
WAR: Workload-Aware Rollouts for Synchronous Agentic Reinforcement Learning
Jul 19, 2026Ryan Xu, Atlas Zhao, David Bao +1Agentic RL Reinforcement Learning
Lossless but Not Free: An Empirical Anatomy of Speculative Decoding on Consumer Hardware
Jul 19, 2026Param ChordiyaOn-Device Language Model Inference Speculative Decoding
University of California, San Diego
ThAME: 3D Memory-Enabled Heterogeneous Accelerator for LLM Mixture of Experts
Jul 19, 2026Pratyush Dhingra, Pramit Kumar Pal, Janardhan Rao Doppa +1AI Accelerator Inference Mixture-of-Experts Inference
Department of Electrical Engineering and Computer Science, Washington State University, Pullman, WA 99163 USA
SpecLA: Efficient Speculative Decoding for Linear-Attention Models
Jul 18, 2026Zhibin Wang, Xuying Han, Zhaohua Yang +5Speculative Decoding LLM Inference Acceleration
State Key Laboratory of Novel Software Technology Nanjing University Nanjing, China · Alibaba Group Hangzhou, China
PagedWeight: Efficient MoE LLM Serving with Dynamic Quality-Aware Weight Quantization
Jul 17, 2026Yuchen Yang, Yifan Zhao, Anisha Dasgupta +1LLM Quantization Mixture-of-Experts Language Models
University of Illinois Urbana-Champaign
An MLIR-Based Compilation Method for Large Language Models
Jul 17, 2026Pengchao Hu, Zhibin Xin, Yifan Chen +3LLM Serving LLM Inference Scheduling
Sophgo Inc.
D-cut: Adaptive Verification Depth Pruning for Batched Speculative Decoding
Jul 16, 2026Tianyu Liu, Yuhao Shen, Rui Cen +7Speculative Decoding LLM Inference Acceleration
1Tencent · 2Independent Researcher
PolyQ: Codesigning End-to-End Quantization Framework for Scalable Edge CPU LLM Inference
Jul 16, 2026Hyunwoo Oh, Suyeon Jang, Hanning Chen +4Mixed-Precision Quantization LLM Quantization
University of California, Irvine
Smarter and Cheaper at Once: Byte-Exact KV-Cache Grafting Turns a Frozen Small Model into a Verified-Knowledge Flywheel
Jul 15, 2026Sietse SchelpeMemory-Augmented Language Models Knowledge Augmentation for Language Models
Corbenic AI
PReM: Learning What to Preserve and When to Refresh for Context Compression
Jul 15, 2026Bohan Yu, Lei Shen, Chenxi Zhou +5Memory-Augmented Language Models Long-Context Language Model Inference
Alibaba Group
Accepted Prefixes Are Not All You Need: A Negative Result on PEFT-Based Block-Diffusion Drafting
Jul 14, 2026Abdurrahman Javat, Allan KazakovSpeculative Decoding LLM Inference Acceleration
Bahçeşehir University, Department of Artificial Intelligence
Full-Pipeline Inference Optimization for MiMo-V2.5 Series: Pushing Hybrid SWA Efficiency to the Limit
Jul 14, 2026Xiaomi MiMo Team, Anqi Liu, Aoxin Ma +28Efficient Multimodal Inference KV Caching
MiMo Team, Xiaomi
Speculate with Memory: Lossless Acceleration for LLM Agents
Jul 14, 2026Yu Li, Qinyuan Ye, Prafulla Kumar Choubey +2Next-Action Prediction LLM Agent Memory
Salesforce Research
Decomposing Runtime, Kernel, and Quantization Speedups via a Matched FP16 Intermediate: A Hardware-Conditioned Case Study on Four NVIDIA RTX A5000 GPUs
Jul 13, 2026Weijia Han, Lisha QuLLM Quantization GPU Kernel Optimization
University of Washington
FastTPS: An Optimized Method for LLM Token Phase for AI accelerators
Jul 13, 2026Wenzong Yang, Danyang Zhang, Kun Cao +17AI Accelerator Inference LLM Inference
Advanced Micro Devices, Inc., Santa Clara, California, USA.
LiteTopK: Exploiting the Curse of Dimensionality for a Fused Indexer-TopK Kernel in Long-Context Sparse Attention
Jul 13, 2026Ziqi Yin, Jianyang Gao, Peiqi Yin +2GPU Kernel Optimization LLM Inference Acceleration
Nanyang Technological University · ETH Zurich · The Chinese University of Hong Kong

Jul 24, 2026, cs.ARJ/K move · Enter open · S save

FusionML: Prefill, Not Decode - Mechanism and Boundaries of CPU+GPU Co-Execution on Unified-Memory Apple Silicon

Om Mohite

Independent Researcher, Mumbai, India

LLM Serving LLM Inference Acceleration Heterogeneous Computing

Open paper AskPDF arXiv

Apple-Silicon SoCs share CPU, GPU, and Neural Engine over one unified memory system, raising the question of whether transformer inference can be accelerated by splitting single operators across units. Prior attempts, including our own, failed or produced precision-confounded wins. We identify the cause: MLX's lazy-graph scheduler \emph{serializes} cross-stream work whenever a CPU-stream operation consumes an unmaterialized GPU result inside one evaluation graph, so a row-split matmul that runs \x{1.38} faster with materialized inputs runs \x{0.66} slower than GPU-only inside a lazy graph; an eager materialization boundary restores concurrency (\x{1.34}). \sys{} implements a per-layer, contention-aware CPU+GPU row split for transformer prefill built on this fix. Evaluated across five chips and three Apple-Silicon generations, community-replicated, the split accelerates Llama-shaped decoder-block prefill by \x{1.15}--\x{1.38}, unchanged at full 32-block depth, and reaches \x{1.18}--\x{1.25} faster time-to-first-token on a real Qwen2.5-7B checkpoint served through stock MLX-LM, with token-identical outputs and unchanged decode throughput. We characterize the boundaries equally carefully: decode cannot benefit, bound by shared bandwidth co-execution does not add; precision-matched training loses \x{0.86}--\x{0.97} on all five chips; ANE dispatch overhead excludes it at layer granularity; and a no-regression runtime gate becomes self-defeating under memory pressure, where probing an alternative mode evicts the active mode's working set. Code, raw results, and generation transcripts are released.

Momentum

Related topics

Latest papers 677