LLM Inference Acceleration

LLM: Large Language Model

Follow this topic Ask about this topic

Momentum

108 papers in the last four weeks, up 120% on the four weeks before. 1.1% of all new papers.

Jul 13Week of Sep 28

Latest papers 677

All topics

Why Is Video Still So Expensive? A Survey of Inference-Efficiency Mechanisms in Video and Audiovisual LLMs
Sep 9, 2026Killian Steunou, Yannis Tevissen, Mounîm A. El YacoubiAudio-Visual Understanding Efficient VLM Inference
SAMOVAR Laboratory, T´el´ecom SudParis, Institut Polytechnique de Paris, Palaiseau, France · Moments Lab, Paris, France
KVShareArena: KV-Cache Reuse Across Contexts and Model Checkpoints
Sep 9, 2026Xi Shi, Mengxin Zheng, Qian LouKV Caching LLM Inference Acceleration
University of Central Florida
Maverick: Private and Verifiable LLM Inference Made Practical via Matrix-Vector Multiplication Delegation
Sep 9, 2026Ben Merbaum, Mohammad Amin Raeisi, Wenhao Wang +3Privacy-Preserving Language Models LLM Inference
Yale University, IC3
Forward-Free LLM Depth Pruning via Weight Redundancy
Sep 9, 2026Vincent-Daniel Yun, Woosang LimLLM Pruning LLM Inference Acceleration
University of Southern California, United States · 2Seoul National University, Republic of Korea
OMP-MoE: Efficient Expert Pruning for Mixture-of-Experts LLMs via Orthogonal Matching Pursuit
Sep 9, 2026Dezhi Li, Lujun Li, Qiyuan Zhu +4Mixture-of-Experts Pruning Mixture-of-Experts Language Models
The Hong Kong University of Science and Technology
PELM: Power Efficient On-Device LLM Inference with Speculative Decoding and Dynamic Voltage Frequency Scaling
Sep 9, 2026Weisi Yang, Stephen XiaLLM Inference Efficiency On-Device Language Model Inference
Northwestern University Evanston, Illinois, USA
Osprey: Target-agnostic Pre-training Makes Stronger Drafters in Speculative Decoding
Sep 8, 2026Fengxiang Bie, Yuqing Jian, Yifan Yu +7Language Model Pretraining Speculative Decoding
Together AI · The University of Sydney, Australia · University of Illinois Urbana-Champaign, USA +1
Do Dynamic Routers Need Memory? HeRo: History-Aware Routing for Efficient LLM Inference
Sep 8, 2026Hongjin Lin, Wentao Wan, Keze WangLLM Inference Acceleration LLM Routing
Sun Yat-sen University
RedKnot-MLA: Multi-Head Offline-Online Reuse for DeepSeek-V4 Long-Context Serving
Sep 7, 2026Yang Liu, Zhaokai Luo, Huayi Jin +10KV Caching LLM Inference Acceleration
Xiaohongshu Inc., China · Huawei Cloud · Shanghai Jiao Tong University +2
Compression Beyond the Uncompressed: A Two-Stage Training Recipe for Soft Context Compression in RAG
Sep 4, 2026Shuyu Guo, Shuo Zhang, Zhaochun RenRetrieval-Augmented Generation LLM Inference Acceleration
Shandong University Qingdao, China · Bloomberg London, United Kingdom · Leiden University Leiden, The Netherlands
Unlocking Lossless Speedups in LLMs via Discrete Diffusion
Sep 3, 2026Subham Sekhar Sahoo, Lingjie Chen, Khiem Pham +14Diffusion Language Model Inference LLM Inference Acceleration
Institue of Foundation Models · University of Illinois Urbana-Champaign · Cornell Tech +2
VestigeKV: The NoPE-MLA KV Cache Carries Its Own Sparse-Attention Signal in a Vestigial Branch
Sep 3, 2026WenJie FanKV Caching KV-Cache Management
Yotta Labs
What Matters for Aggressive Decoding-Time KV Eviction? Temporal Aggregation and Ranking Preservation
Sep 3, 2026Bo Zeng, Yu Zhao, Yefeng Liu +3KV Caching KV-Cache Eviction
Ant Group · Alibaba Group · Universität Hamburg
GrowPage: On-Demand KV Budgeting for Efficient LLM Reasoning Serving
Sep 3, 2026Qiankun Ma, Yanjiang Zhou, Zinan Xiong +5LLM Inference KV Caching
Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences · Pengcheng Laboratory · University of Chinese Academy of Sciences
Random Attention: Rethinking KV Cache Eviction for Efficient Reasoning
Sep 3, 2026Heng Wang, Jielin Qiu, Wenting Zhao +8KV Caching Memory-Efficient Optimization
Salesforce AI Research · University of Illinois Urbana-Champaign
RecurTrace: Adaptive Latent Reasoning with Loop-Time Memory
Sep 3, 2026Yuxiang Wang, Kunyu Feng, Yingda Shen +3Memory-Augmented Language Models Recurrent Transformers
The Chinese University of Hong Kong, Shenzhen · The Chinese University of Hong Kong · Tianjin University +2
SGD-KV: Summarization Guided KV Cache Compression
Sep 3, 2026Zeyu Liu, Woomin Song, Xuandi Fu +5KV Caching Long-Context Language Model Inference
USC, USA · KAIST, Korea · Amazon AGI, USA
HeadWiseKV: Budgeted Per-Head Cache Residency for Hybrid Long-Context Language Models
Sep 2, 2026Renjie Xie, Juncheng Yang, Aoting Hu +4KV Caching Long-Context Language Model Inference
Nanjing University of Posts and Telecommunications · Tylogi AI Lab / TAIL · Anhui University of Technology +2
CRISP: Cliff-awaRe Input-adaptive Sparse Prefilling with Structural-Mass-Motivated Routing
Sep 1, 2026Huu Huy Nguyen, Chien Van Nguyen, Franck Dernoncourt +4Long-Context Language Model Inference LLM Inference Acceleration
University of Oregon · Adobe Research · Hanoi University of Science and Technology
hLLM: Single Pass Decoding for Generative Reranking
Sep 1, 2026Emil Laftchiev, Prachi Agrawal, Moe Kayali +7Language Model Decoding LLM Reranking
Meta Platforms, Inc.
LatentPress: Context Compression Beyond Text and Vision
Sep 1, 2026Zhengze Zhou, Hejian SangLLM Inference Acceleration LLM Context Management
1Cornell University · 2Iowa State University
mzCache: On-Device LLM Memory Management under Multitasking
Sep 1, 2026Hongseung Yu, Minsung Kim, Jongseok Park +1On-Device Language Model Inference Memory-Efficient Inference
Seoul National University · UC Berkeley
PCoMoE: Shifting MoE Inference from Monolithic Expert Selection to Fine-Grained Path Composition
Sep 1, 2026Ziyan Gan, Fangxin Liu, Chenyang Guan +10Mixture-of-Experts Pruning Mixture-of-Experts Inference
1Shanghai Jiao Tong University, China · 2Alibaba Group, China
CacheBridge: Efficient Cross-Model KV Cache Transfer
Sep 1, 2026Xingyu Qu, Siyuan Lu, Zhiyu Chen +2Transformer Inference KV Caching
Westlake University. · †Work conducted during an internship at Westlake University. · Amazon. +1
SFAD: Speculative Factuality-Aware Decoding
Sep 1, 2026Guanqiao Chen, Di Wang, Lijie HuLLM Hallucination Mitigation Speculative Decoding
1MBZUAI · University of Science and Technology of China · 3Provable Responsible AI and Data Analytics (PRADA) Lab +1
HBQ: Hierarchical Scaling Block Quantization with Hardware-Efficiency-Aware Design for Accurate LLM Inference
Aug 31, 2026Chun-Ting Chen, Dongmin Han, Hangyeol Mun +6LLM Quantization LLM Inference Acceleration
Cornell University New York, NY, USA · Intel Corporation Santa Clara, CA, USA · Intel Corporation Hillsboro, OR, USA
QTEA: Ternary LLMs with Sparse Residual Salient Weight and By-Column Optimization
Aug 31, 2026Yipin Guo, Arun M George, Jie Fu +3LLM Quantization Ternary Quantization
University of Notre Dame
A Model with No Head and Many Thoughts
Aug 31, 2026Nikita Koriagin, Yaroslav Aksenov, George Bredis +3Efficient Language Model Reasoning LLM Inference Acceleration
1Yandex · 2T-Tech
A Universal Context-Reuse Layer for Cross-Model KV Sharing
Aug 31, 2026Yi Li, Dongming Jiang, Yi Zhao +1LLM Inference KV Caching
University of Texas at Dallas
Faster Than Flash: Exploiting Attention Sparsity for Efficient Long-Context Decoding
Aug 31, 2026Zhigeng Liu, Zhiyuan Ning, Ruixiao Li +5Long-Context Language Model Inference LLM Inference Acceleration
Fudan University, Shanghai, China · Shanghai Innovation Institute, Shanghai, China · Harbin Institute of Technology, Harbin, China
CARVE: Verified Expansion for Variable-Length Generation in Diffusion Language Models
Aug 31, 2026Wail Bouhedja, Amr Mohamed, Guokan ShangMasked Diffusion Models LLM Inference Acceleration
1MBZUAI · 2Sorbonne Université · 3Ecole Polytechnique
Deploying DeepSeek 175B Locally on a Single Consumer-Grade RTX 4060 Laptop with 32GB RAM for 200k-Scale Protein-Ligand Virtual Screening
Aug 31, 2026Rui Xiao, Yili XuProtein-Ligand Binding Affinity Prediction Virtual Screening
Hangzhou Tsingxin quantum Co., Ltd.
TopoCompress: Long Context Compression via Graph-Wired Semantic Trajectories
Aug 31, 2026Daniel Agyei Asante, Yang LiLLM Compression Long-Context QA
1Iowa State University, United States
Tensor Methods for Language Models: From Token Representation to Training, Adaptation, Inference, Compression, and Interpretability
Aug 31, 2026Matvei Tarasov, Salman Ahmadi-Asl, Andre L. F. de Almeida +1LLM Compression LLM Inference Acceleration
Innopolis University, Innopolis 420500, Russia · Department of Teleinformatics Engineering, Federal University of Cear´a, Fortaleza 60455-760, Brazil · Systems Research Institute of Polish Academy of Science and Warsaw University of Technology, Poland
Event-Driven Language Models with Sparse Neural Activity for Neuromorphic Hardware
Aug 31, 2026Simon Richter, Ruhai Lin, Jason Yik +4Activation Sparsity LLM Inference Acceleration
Department of Electrical and Computer Engineering, Aarhus University, Aarhus, Denmark · Department of Computer Science and Engineering, University of California, Santa Cruz, Santa Cruz, USA · School of Engineering and Applied Sciences, Harvard University, Cambridge, USA +1
DASC: Decay-Aware State Compression for Hybrid Linear-Attention Serving
Aug 31, 2026Yanqi Yu, Pingwei Sun, Jianchao Tan +4KV Caching Memory-Efficient Inference
East China Normal University · Meituan · South China University of Technology
Tail-Replay: Escaping the Curse of Linear Attention in Prefix Caching for Hybrid LLMs
Aug 31, 2026Yirui Liu, Ruoling Qi, Xuaner Wu +2LLM Inference Acceleration Prefix Caching
Institute of Artificial Intelligence, China Telecom (TeleAI) · Shanghai Jiao Tong University
CateKV: On Sequential Consistency for Long-Context LLM Inference Acceleration
Aug 31, 2026Haoyun Jiang, Haolin Li, Jianwei Zhang +7Long-Context Language Model Inference LLM Inference Acceleration
1CMIC, Shanghai Jiao Tong University · 2Alibaba Group · 3Fudan University
Strong Drafts Need Compact Memories: Long-Context Speculative Decoding with Compressed KV Cache
Aug 31, 2026Tong Yuan, Chengxi Liao, Zeyi WenMemory-Augmented Language Models KV Caching
Data Science and Analytics Thrust, Information Hub, The Hong Kong University of Science and Technology (Guangzhou)
Verification-Aware Training for Speculative Decoding
Aug 31, 2026Geonmo Gu, Byeongho Heo, HeeJae Jun +4Speculative Decoding LLM Inference Acceleration
NAVER AI Lab · NAVER AI Search Platform · Korea University
AutoCRAT: Within-trajectory Joint Control of Stochasticity and Compute for LLM Reasoning
Aug 30, 2026Hanjun Luo, Qiushi Liu, Jingya Zhang +8Adaptive Inference LLM Inference Acceleration
New York University · New York University Abu Dhabi · University of Washington Seattle +4
ReTrace: Rejected-Trajectory Conditioning for Speculative Decoding
Aug 30, 2026Luxi Lin, Zhanpeng Zeng, Shuang Peng +2Speculative Decoding LLM Inference Acceleration
1Xiamen University, 2ByteDance
Sliding-window beats linear attention
Aug 28, 2026Alexia Jolicoeur-Martineau, Rhea Sanjay Sukthanker, Pashmina Cameron +1KV Caching LLM Inference Acceleration
Microsoft Applied Sciences Group (ASG) · Independent
HyQuant: Hybrid-Precision Quantization for LLM Attention
Aug 28, 2026Jiatong Ding, Bingxin Xing, Yu Zhang +9Mixed-Precision Quantization LLM Quantization
Shanghai Jiao Tong University · Xi’an Jiaotong University · Xiamen University +1
Beyond Parallel Blindness: Information Floors and Model Gaps in Block Drafting
Aug 27, 2026Xinwei Qiang, Xiang Fang, Chang Chen +2Speculative Decoding LLM Inference Acceleration
University of California San Diego
DAMP: Decay-Aware Mixed-Precision Recurrent-State Quantization
Aug 27, 2026Tao Zhang, Jianchao Tan, Pingwei Sun +6Mixed-Precision Quantization LLM Quantization
South China University of Technology · Meituan · East China Normal University
FAMPWQ: Fisher Information-based Adaptive Mixed Precision Weight Quantization for Effective LLM Inference
Aug 24, 2026Gongwei Lee, Ji Liu, Juncheng Jia +1Mixed-Precision Quantization LLM Quantization
School of Computer Science and Technology, Soochow University, Suzhou, China · Hithink Research, Hangzhou, China · Electronic Engineering, Tsinghua University, Beijing, China
How Weight Encoding Affects Language Model Placement and Performance on the Apple Neural Engine
Aug 22, 2026Shahir M ALLM Quantization AI Accelerator Inference
Independent researcher
LiLiCorr: Lightweight Likelihood Correlation of Parallel Drafts for Speculative Decoding
Aug 20, 2026Matan Rusanovsky, Yoav Miron, Roy Uziel +5Speculative Decoding LLM Inference Acceleration
NVIDIA
Self-Indexing Attention for Compression-Compatible Sparse Long-Context LLM Inference
Aug 16, 2026Xu Yang, Jiapeng Zhang, Yuxin Chen +4Long-Context Language Model Inference LLM Inference Acceleration
Hunan University · Tencent
Reduced Matrix Multiplication: Input-Adaptive Matrix-Product Reduction for LLM Inference
Aug 13, 2026Zixuan Lan, Yanhong Li, Jiawei ZhouInference-Time Optimization LLM Inference Acceleration
University of Chicago · Independent Researcher · Stony Brook University
vToken: Token-Level Virtualization for Reclaimable KV Caches
Aug 13, 2026Yuanhang Gao, Xiangrui Yang, Yuanfeng Chen +4KV Caching KV-Cache Eviction
National University of Defense Technology · Changsha, China · Peking University +1
SPADE: Speculative Decoding for Precise and Low Cost Distributed Edge Cloud Inference
Aug 13, 2026Divya Jyoti Bajpai, Kishan Kumar Upadhyay, Manjesh Kumar HanawalEdge Computing Cost-Aware Inference
MLiONS, Department of IEOR, IIT Bombay Mumbai, Maharashtra-400076, India
Decoupled Contrastive Decoding via Expert-Aligned Drafting
Aug 13, 2026Zhixuan Liu, Zhichen Dong, Yuanfu Wang +1Speculative Decoding LLM Inference Acceleration
Shanghai Jiao Tong University · Shanghai Artificial Intelligence Laboratory
Trie Automata for Constrained Decoding over Large Finite Sets
Aug 12, 2026Xingzi Xu, Karim BouyarmaneConstrained Decoding LLM Inference Acceleration
Amazon
Matryoshka Language Model Suites
Aug 10, 2026Nathan Godey, Yoav ArtziEfficient Language Model Training LLM Inference Acceleration
Cornell University
Depth-adaptive Inference of Looped Language Models via Continuous Depth Batching
Aug 10, 2026Kristian Schwethelm, Daniel Rueckert, Georgios KaissisLLM Inference LLM Inference Scheduling
Chair for AI in Healthcare and Medicine, Technical University of Munich (TUM), Germany · Department of Computing, Imperial College London, UK · Munich Center for Machine Learning (MCML), Germany +1
SwiftQK: Fast and Communication-Efficient Tensor Parallelism for Query-Key Normalization
Aug 10, 2026Gyudong Kim, Wonjun Han, Young Geun KimGPU Kernel Optimization Tensor Parallelism
Department of Computer Science and Engineering, Korea University, Seoul 02855, Republic of Korea
Tied Trit-Planes: Constraining PTQTP to a Uniform Nine-Level Quantizer, with a Persistent Folded Format for Disk-Streamed Mixture-of-Experts Serving
Aug 9, 2026Matteo GrellaTernary Quantization Mixture-of-Experts Inference
Crisis24
DistillCache: KL-Guided Adaptive KV-Cache Eviction for Memory-Efficient LLM Inference
Aug 9, 2026Asaad AlthoubiKV Caching Long-Context Language Model Inference
Oklahoma State University Stillwater, OK USA

Sep 9, 2026, cs.CVJ/K move · Enter open · S save

Why Is Video Still So Expensive? A Survey of Inference-Efficiency Mechanisms in Video and Audiovisual LLMs

Killian Steunou, Yannis Tevissen, Mounîm A. El Yacoubi

SAMOVAR Laboratory, T´el´ecom SudParis, Institut Polytechnique de Paris, Palaiseau, France · Moments Lab, Paris, France

Audio-Visual Understanding Efficient VLM Inference LLM Inference Acceleration Video-Language Models

Open paper AskPDF arXiv

Video understanding has rapidly evolved toward video large language models (VideoLLMs): systems that couple video representations with pretrained large language models and condition generation on a textual prompt. Their strong performance on captioning, question answering, retrieval and temporal grounding comes at a computation and memory cost that grows with frame count and context length, limiting deployment in real-time, mobile and resource-constrained settings. This survey covers inference-efficiency mechanisms for visual and audiovisual VideoLLMs that report concrete reductions in parameter count, FLOPs per input, latency, memory, or visual and audio token count. We analyze bottlenecks across frame sampling, modality encoding, connector-level token reduction, and LLM prefilling and decoding. We organize methods by the pipeline stage at which they act, covering VideoLLMs developed since late 2022 together with earlier frame-sampling and vision-encoder mechanisms that remain components of current pipelines. We assemble literature-reported accuracy--cost comparisons under shared host models and input protocols wherever available, distinguish them from heterogeneous cross-paper evidence, and identify gaps in audiovisual efficiency and standardized evaluation. We maintain a repository at https://github.com/momentslab/awesome-efficient-videollm.

Momentum

Related topics

Latest papers 677