LLM Inference Acceleration

LLM: Large Language Model

Follow this topic Ask about this topic

Momentum

108 papers in the last four weeks, up 120% on the four weeks before. 1.1% of all new papers.

Jul 13Week of Sep 28

Latest papers 677

All topics

Why Is Video Still So Expensive? A Survey of Inference-Efficiency Mechanisms in Video and Audiovisual LLMs
Sep 9, 2026Killian Steunou, Yannis Tevissen, Mounîm A. El YacoubiAudio-Visual Understanding Efficient VLM Inference
SAMOVAR Laboratory, T´el´ecom SudParis, Institut Polytechnique de Paris, Palaiseau, France · Moments Lab, Paris, France
KVShareArena: KV-Cache Reuse Across Contexts and Model Checkpoints
Sep 9, 2026Xi Shi, Mengxin Zheng, Qian LouKV Caching LLM Inference Acceleration
University of Central Florida
Maverick: Private and Verifiable LLM Inference Made Practical via Matrix-Vector Multiplication Delegation
Sep 9, 2026Ben Merbaum, Mohammad Amin Raeisi, Wenhao Wang +3Privacy-Preserving Language Models LLM Inference
Yale University, IC3
Forward-Free LLM Depth Pruning via Weight Redundancy
Sep 9, 2026Vincent-Daniel Yun, Woosang LimLLM Pruning LLM Inference Acceleration
University of Southern California, United States · 2Seoul National University, Republic of Korea
OMP-MoE: Efficient Expert Pruning for Mixture-of-Experts LLMs via Orthogonal Matching Pursuit
Sep 9, 2026Dezhi Li, Lujun Li, Qiyuan Zhu +4Mixture-of-Experts Pruning Mixture-of-Experts Language Models
The Hong Kong University of Science and Technology
PELM: Power Efficient On-Device LLM Inference with Speculative Decoding and Dynamic Voltage Frequency Scaling
Sep 9, 2026Weisi Yang, Stephen XiaLLM Inference Efficiency On-Device Language Model Inference
Northwestern University Evanston, Illinois, USA
Osprey: Target-agnostic Pre-training Makes Stronger Drafters in Speculative Decoding
Sep 8, 2026Fengxiang Bie, Yuqing Jian, Yifan Yu +7Language Model Pretraining Speculative Decoding
Together AI · The University of Sydney, Australia · University of Illinois Urbana-Champaign, USA +1
Do Dynamic Routers Need Memory? HeRo: History-Aware Routing for Efficient LLM Inference
Sep 8, 2026Hongjin Lin, Wentao Wan, Keze WangLLM Inference Acceleration LLM Routing
Sun Yat-sen University
RedKnot-MLA: Multi-Head Offline-Online Reuse for DeepSeek-V4 Long-Context Serving
Sep 7, 2026Yang Liu, Zhaokai Luo, Huayi Jin +10KV Caching LLM Inference Acceleration
Xiaohongshu Inc., China · Huawei Cloud · Shanghai Jiao Tong University +2
Compression Beyond the Uncompressed: A Two-Stage Training Recipe for Soft Context Compression in RAG
Sep 4, 2026Shuyu Guo, Shuo Zhang, Zhaochun RenRetrieval-Augmented Generation LLM Inference Acceleration
Shandong University Qingdao, China · Bloomberg London, United Kingdom · Leiden University Leiden, The Netherlands
Unlocking Lossless Speedups in LLMs via Discrete Diffusion
Sep 3, 2026Subham Sekhar Sahoo, Lingjie Chen, Khiem Pham +14Diffusion Language Model Inference LLM Inference Acceleration
Institue of Foundation Models · University of Illinois Urbana-Champaign · Cornell Tech +2
VestigeKV: The NoPE-MLA KV Cache Carries Its Own Sparse-Attention Signal in a Vestigial Branch
Sep 3, 2026WenJie FanKV Caching KV-Cache Management
Yotta Labs
What Matters for Aggressive Decoding-Time KV Eviction? Temporal Aggregation and Ranking Preservation
Sep 3, 2026Bo Zeng, Yu Zhao, Yefeng Liu +3KV Caching KV-Cache Eviction
Ant Group · Alibaba Group · Universität Hamburg
GrowPage: On-Demand KV Budgeting for Efficient LLM Reasoning Serving
Sep 3, 2026Qiankun Ma, Yanjiang Zhou, Zinan Xiong +5LLM Inference KV Caching
Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences · Pengcheng Laboratory · University of Chinese Academy of Sciences
Random Attention: Rethinking KV Cache Eviction for Efficient Reasoning
Sep 3, 2026Heng Wang, Jielin Qiu, Wenting Zhao +8KV Caching Memory-Efficient Optimization
Salesforce AI Research · University of Illinois Urbana-Champaign
RecurTrace: Adaptive Latent Reasoning with Loop-Time Memory
Sep 3, 2026Yuxiang Wang, Kunyu Feng, Yingda Shen +3Memory-Augmented Language Models Recurrent Transformers
The Chinese University of Hong Kong, Shenzhen · The Chinese University of Hong Kong · Tianjin University +2
SGD-KV: Summarization Guided KV Cache Compression
Sep 3, 2026Zeyu Liu, Woomin Song, Xuandi Fu +5KV Caching Long-Context Language Model Inference
USC, USA · KAIST, Korea · Amazon AGI, USA
HeadWiseKV: Budgeted Per-Head Cache Residency for Hybrid Long-Context Language Models
Sep 2, 2026Renjie Xie, Juncheng Yang, Aoting Hu +4KV Caching Long-Context Language Model Inference
Nanjing University of Posts and Telecommunications · Tylogi AI Lab / TAIL · Anhui University of Technology +2
CRISP: Cliff-awaRe Input-adaptive Sparse Prefilling with Structural-Mass-Motivated Routing
Sep 1, 2026Huu Huy Nguyen, Chien Van Nguyen, Franck Dernoncourt +4Long-Context Language Model Inference LLM Inference Acceleration
University of Oregon · Adobe Research · Hanoi University of Science and Technology
hLLM: Single Pass Decoding for Generative Reranking
Sep 1, 2026Emil Laftchiev, Prachi Agrawal, Moe Kayali +7Language Model Decoding LLM Reranking
Meta Platforms, Inc.
LatentPress: Context Compression Beyond Text and Vision
Sep 1, 2026Zhengze Zhou, Hejian SangLLM Inference Acceleration LLM Context Management
1Cornell University · 2Iowa State University
mzCache: On-Device LLM Memory Management under Multitasking
Sep 1, 2026Hongseung Yu, Minsung Kim, Jongseok Park +1On-Device Language Model Inference Memory-Efficient Inference
Seoul National University · UC Berkeley
PCoMoE: Shifting MoE Inference from Monolithic Expert Selection to Fine-Grained Path Composition
Sep 1, 2026Ziyan Gan, Fangxin Liu, Chenyang Guan +10Mixture-of-Experts Pruning Mixture-of-Experts Inference
1Shanghai Jiao Tong University, China · 2Alibaba Group, China
CacheBridge: Efficient Cross-Model KV Cache Transfer
Sep 1, 2026Xingyu Qu, Siyuan Lu, Zhiyu Chen +2Transformer Inference KV Caching
Westlake University. · †Work conducted during an internship at Westlake University. · Amazon. +1
SFAD: Speculative Factuality-Aware Decoding
Sep 1, 2026Guanqiao Chen, Di Wang, Lijie HuLLM Hallucination Mitigation Speculative Decoding
1MBZUAI · University of Science and Technology of China · 3Provable Responsible AI and Data Analytics (PRADA) Lab +1
HBQ: Hierarchical Scaling Block Quantization with Hardware-Efficiency-Aware Design for Accurate LLM Inference
Aug 31, 2026Chun-Ting Chen, Dongmin Han, Hangyeol Mun +6LLM Quantization LLM Inference Acceleration
Cornell University New York, NY, USA · Intel Corporation Santa Clara, CA, USA · Intel Corporation Hillsboro, OR, USA
QTEA: Ternary LLMs with Sparse Residual Salient Weight and By-Column Optimization
Aug 31, 2026Yipin Guo, Arun M George, Jie Fu +3LLM Quantization Ternary Quantization
University of Notre Dame
A Model with No Head and Many Thoughts
Aug 31, 2026Nikita Koriagin, Yaroslav Aksenov, George Bredis +3Efficient Language Model Reasoning LLM Inference Acceleration
1Yandex · 2T-Tech
A Universal Context-Reuse Layer for Cross-Model KV Sharing
Aug 31, 2026Yi Li, Dongming Jiang, Yi Zhao +1LLM Inference KV Caching
University of Texas at Dallas
Faster Than Flash: Exploiting Attention Sparsity for Efficient Long-Context Decoding
Aug 31, 2026Zhigeng Liu, Zhiyuan Ning, Ruixiao Li +5Long-Context Language Model Inference LLM Inference Acceleration
Fudan University, Shanghai, China · Shanghai Innovation Institute, Shanghai, China · Harbin Institute of Technology, Harbin, China

Sep 9, 2026, cs.CVJ/K move · Enter open · S save

Why Is Video Still So Expensive? A Survey of Inference-Efficiency Mechanisms in Video and Audiovisual LLMs

Killian Steunou, Yannis Tevissen, Mounîm A. El Yacoubi

SAMOVAR Laboratory, T´el´ecom SudParis, Institut Polytechnique de Paris, Palaiseau, France · Moments Lab, Paris, France

Audio-Visual Understanding Efficient VLM Inference LLM Inference Acceleration Video-Language Models

Open paper AskPDF arXiv

Video understanding has rapidly evolved toward video large language models (VideoLLMs): systems that couple video representations with pretrained large language models and condition generation on a textual prompt. Their strong performance on captioning, question answering, retrieval and temporal grounding comes at a computation and memory cost that grows with frame count and context length, limiting deployment in real-time, mobile and resource-constrained settings. This survey covers inference-efficiency mechanisms for visual and audiovisual VideoLLMs that report concrete reductions in parameter count, FLOPs per input, latency, memory, or visual and audio token count. We analyze bottlenecks across frame sampling, modality encoding, connector-level token reduction, and LLM prefilling and decoding. We organize methods by the pipeline stage at which they act, covering VideoLLMs developed since late 2022 together with earlier frame-sampling and vision-encoder mechanisms that remain components of current pipelines. We assemble literature-reported accuracy--cost comparisons under shared host models and input protocols wherever available, distinguish them from heterogeneous cross-paper evidence, and identify gaps in audiovisual efficiency and standardized evaluation. We maintain a repository at https://github.com/momentslab/awesome-efficient-videollm.

Momentum

Related topics

Latest papers 677