LLM Inference Acceleration

LLM: Large Language Model

Follow this topic Ask about this topic

Momentum

108 papers in the last four weeks, up 120% on the four weeks before. 1.1% of all new papers.

Jul 13Week of Sep 28

Latest papers 677

All topics

PATCH: Learnable Tile-level Hybrid Sparsity for LLMs
Sep 27, 2025Younes Hourri, Mohammad Mozaffari, Maryam Mehri DehnaviLLM Pruning LLM Compression
Department of Computer Science University of Toronto
PiERN: Token-Level Routing for Integrating High-Precision Computation and Reasoning
Sep 17, 2025Jingyuan Fan, Purui Liu, Hengbo Xiao +4Numerical Reasoning in Language Models LLM Inference Acceleration
Peking University · Tsinghua University
FSA: An Alternative Efficient Implementation of Native Sparse Attention Kernel
Aug 25, 2025Ran Yan, Youhe Jiang, Zhuoming Chen +3GPU Kernel Optimization GPU Acceleration
1. The Hong Kong University of Science and Technology · 2. Carnegie Mellon University
From "Aha Moments" to Controllable Thinking: Toward Meta-Cognitive Reasoning in Large Reasoning Models via Decoupled Reasoning and Control
Aug 6, 2025Rui Ha, Rui Pu, Chaozhuo Li +2RL for Language Model Reasoning LLM Inference Acceleration
Beijing University of Posts and Telecommunications, China · 2Chongqing University of Posts and Telecommunications, China
Any-Order GPT as Masked Diffusion Model: Decoupling Formulation and Architecture
Jun 24, 2025Shuchen Xue, Tianyu Xie, Tianyang Hu +5Masked Diffusion Models Decoder-Only Language Models
University of Chinese Academy of Sciences · Academy of Mathematics and Systems Science, Chinese Academy of Sciences · School of Mathematical Sciences, Peking University +2
Less Is More: Reducing Token Counts Without Compromising Performance
Jun 18, 2025Gyeongje Cho, Yeonkyoung So, Sangmin Lee +1LLM Inference Acceleration Token Efficiency
Graduate School of Data Science, Seoul National University · Department of Computer Science, Seoul National University
FPTQuant: Function-Preserving Transforms for LLM Quantization
Jun 5, 2025Boris van Breugel, Yelysei Bondarenko, Paul Whatmough +1LLM Quantization LLM Inference Acceleration
Qualcomm AI Research, Amsterdam, Netherlands
NSNQuant: A Double Normalization Approach for Calibration-Free Low-Bit Vector Quantization of KV Cache
May 23, 2025Donghyun Son, Euntae Choi, Sungjoo YooKV Caching Vector Quantization
Seoul National University
Dimple: Discrete Diffusion Multimodal Large Language Model with Parallel Decoding
May 22, 2025Runpeng Yu, Xinyin Ma, Xinchao WangMultimodal Large Language Models LLM Inference Acceleration
National University of Singapore
Adaptive GoGI-Skip: Coupling Goal-Gradient Importance with Dynamic Uncertainty for Efficient Reasoning
May 13, 2025Ren ZhuangGradient-Based Attribution LLM Pruning
School of Information Science and Technology, Hangzhou Normal University
The Sparse Frontier: Sparse Attention Trade-offs in Transformer LLMs
Apr 24, 2025Piotr Nawrot, Robert Li, Renjie Huang +3Transformer Inference LLM Evaluation
University of Edinburgh · Cohere · Meta
Prima.cpp: Fast 30-70B LLM Inference on Heterogeneous and Low-Resource Home Clusters
Apr 7, 2025Zonghang Li, Tao Li, Wenjiao Feng +8On-Device Language Model Inference LLM Inference
MBZUAI · UESTC · City University of Hong Kong +1
FlashBack: Efficient Retrieval-Augmented Language Modeling for Fast Inference
May 7, 2024Runheng Liu, Xingchen Xiao, Heyan Huang +2Knowledge Augmentation for Language Models LLM Inference Acceleration
School of Computer Science and Technology, Beijing Institute of Technology
Dynamic Expert Quantization for Scalable Mixture-of-Experts Inference
Date pendingKexin Chu, Dawei Xiang, Zixu Shen +3Mixed-Precision Quantization GPU Acceleration
University of Connecticut · University of California, Santa Cruz · Independent Researcher
FluxMoE: Decoupling Expert Residency for High-Performance MoE Serving
Date pendingQingxiu Liu, Yongchao He, Runhan Jiang +4Expert Offloading GPU Acceleration
CUHK · Unaffiliated · SCITIX +1
UltraQuant: 4-bit KV Caching for Context-Heavy Agents
Date pendingInesh Chakrabarti, David Limpus, Aditi Ghai Rana +4LLM Inference Efficiency GPU Kernel Optimization
Advanced Micro Devices · University of California, Los Angeles · Purdue University
FastE: Readout-Triggered Token Compression for LLM Embedding Inference
Date pendingJinsong Shu, Jinyong Wen, Baokun Wang +4Token Pruning LLM Inference Acceleration
Zhejiang University · Hangzhou High-Tech Zone (Binjiang) Institute of Blockchain and Data Security · Ant Group +1

Sep 27, 2025, cs.LGJ/K move · Enter open · S save

PATCH: Learnable Tile-level Hybrid Sparsity for LLMs

Younes Hourri, Mohammad Mozaffari, Maryam Mehri Dehnavi

Department of Computer Science University of Toronto

LLM Pruning LLM Compression Structured Sparsity LLM Inference Acceleration Neural Network Pruning

Open paper AskPDF arXiv

Large language models (LLMs) deliver impressive performance but incur prohibitive memory and compute costs at deployment. Model pruning is an effective way to reduce these overheads, yet existing approaches face challenges: unstructured sparsity, where nonzeros can appear anywhere, preserves accuracy but yields irregular access patterns that prevent GPU acceleration, while semi-structured 2:4 sparsity is hardware-friendly but enforces a rigid 50% pattern that degrades model quality. To bridge this gap, we introduce PATCH, a hybrid sparsity framework that enables a continuous sparsity ratio between 0% and 50%. PATCH partitions weight matrices into tiles, assigning each tile to be either dense or 2:4 sparse via a learnable mask selection mechanism. This design provides fine-grained control over accuracy-acceleration tradeoffs and supports non-uniform sparsity across layers, leading to superior overall quality. Across models from 0.5B to 13B parameters, PATCH consistently narrows the gap to dense accuracy while delivering practical speedups. For instance, on LLaMA-2 7B with an A6000 GPU, PATCH achieves 1.18x-1.38x end-to-end speedup over dense baselines while improving accuracy by 0.37%-2.96% compared to the state-of-the-art 2:4 pruning method, MaskLLM.

Momentum

Related topics

Latest papers 677