LLM Inference Efficiency

Follow this topic Ask about this topic

Recent momentum

-5%

61 papers in the last 28 days · 1.0% of indexed attention

Twelve weeks of publication activity for this topic as it is defined today.

Weekly history

Recent digests

What was published in this topic, kept on the site without email delivery.

Period ending 2026-09-21

21 new papers

A weekly snapshot of new work published in LLM Inference Efficiency.

Period ending 2026-09-14

12 new papers

A weekly snapshot of new work published in LLM Inference Efficiency.

Period ending 2026-09-07

16 new papers

A weekly snapshot of new work published in LLM Inference Efficiency.

541 papers

Latest in LLM Inference Efficiency

Browse all topics →

ProbeScale: Probing Analysis to Optimize Neural Scaling Laws for Efficient Small Language Model Inference
Jun 1, 2026Sourav DasSmall Language Models LLM Inference Efficiency
Improving Visual Token Reduction via Rectifying Distortions for Efficient Multimodal LLM Inference
Jun 1, 2026Hyeonwoo Cho, Donghyeon Baek, Yewon Kim +1Vision-Language Distortion
MomentKV: Closing the Directional Gap in KV Cache Eviction for Long-Context Inference
Jun 1, 2026Yu Li, Binxu Li, Tian LanKey-Value Cache Eviction Key-Value Cache
Move the Query, Not the Cache: Characterizing Cross-Instance Latent Attention Redistribution Across GPU Fabrics
May 31, 2026Bole Ma, Jan Eitzinger, Harald Köstler +1Dynamic Sparse Attention Kv-Cache Blocks
LongAttnComp: Cross-Family Context Compression for Long-Context Reasoning
May 31, 2026Mengmeng Ji, Ravi Shanker Raju, Jonathan Lingjie Li +1Efficient Long-Context Inference Context Length
Inference Cost Attacks for Retrieval-Augmented Large Language Models
May 31, 2026Chengliang Liu, Liangbo Ning, Yujuan Ding +1Retrieval-Augmented Generation Systems Language Model Evasion Attacks
Hybrid Verified Decoding: Learning to Allocate Verification in Speculative Decoding
May 31, 2026Xin Su, Dawid Majchrowski, Fangyuan Yu +5Speculative Decoding LLM Inference Efficiency
Lodestar: An Online-Learning LLM Inference Router
May 31, 2026Gangmuk Lim, Wanyu Zhao, Brighten Godfrey +3LLM Inference Efficiency Per-Token Latency
Threshold-Based Exclusive Batching for LLM Inference
May 30, 2026Weifang Zhang, Yuzhou Nie, Bowen Pang +2Well-Tuned Static Batching Policy LLM Inference Efficiency
Bit-Exact AI Inference Verification Without Performance Tradeoffs
May 29, 2026Naci CankayaOnline Inference Verifier
GPU Forecasters: Language Models as Selective Surrogates for Kernel Runtime Optimization
May 29, 2026Zaid Khan, Justin Chih-Yao Chen, Jaemin Cho +2Graphics Processing Unit Kernels LLM Inference Efficiency
Divergence Decoding: Inference-Time Unlearning via Auxiliary Models
May 29, 2026Humzah Merchant, Bradford LevyLarge Language Model Unlearning Unlearning Aims
UniScale: Adaptive Unified Inference Scaling via Online Joint Optimization of Model Routing and Test-Time Scaling
May 29, 2026Kaiyu Huang, Xingyu Wang, Mingze Kong +6LLM Inference Efficiency Test-Time Scaling
dMoE: dLLMs with Learnable Block Experts
May 29, 2026Sicheng Feng, Zigeng Chen, Gongfan Fang +2Mixture-of-Experts Models Diffusion Language Models
Speculative Pipeline Decoding: Higher-Accruacy and Zero-Bubble Speculation via Pipeline Parallelism
May 29, 2026Yijiong Yu, Huazheng Wang, Shuai Yuan +2Speculative Decoding Self-Speculative
Efficient Diffusion LLMs via Temporal-Spatial Parallel Decoding and Confidence Extrapolation
May 29, 2026Zekai Li, Ji Liu, Yiqing Huang +3Diffusion Language Models LLM Inference Efficiency
Reducing the GPU Memory Bottleneck with Lossless Compression for ML -- Extended
May 29, 2026Aditya K Kamath, Arvind Krishnamurthy, Marco Canini +1Large Language Model Compression Context Compression
Memory-Bound but Not Bandwidth-Limited: The Physical AI Inference Gap in Batch-1 LLM Decode
May 28, 2026Josef ChenLLM Inference Efficiency Inference Workloads
MarginGate: Sparse Margin-Triggered Verification for Batch-Invariant LLM Inference
May 28, 2026Kexin Chu, Yang Zhou, Wei ZhangLLM Inference Efficiency Token-Level Uncertainty
Accelerating Constrained Decoding with Token Space Compression
May 28, 2026Michael Sullivan, Alexander KollerToken Compression Context-Free Grammars
Fingerprinting Inference Systems of Large Language Models
May 28, 2026Anna Wimbauer, Jonas Möller, Erik Imgrund +1LLM Inference Efficiency Fingerprint
Moment-KV: Momentum-Based Decode-Time KV Cache Compression for Long Generation
May 28, 2026Soumyadeep Jana, Sagar Nishad, Sanasam Ranbir SinghKey-Value Cache Compression Efficient Long-Context Inference
LFQ: Logit-aware Final-block Quantization for Boosting the Generation Quality of Low-Bit Quantized LLMs
May 28, 2026Jung Hyun Lee, June Yong Yang, Jungwook Choi +1Large Language Model Quantization Weight-Only Quantization
Revisiting Observation Reduction for Web Agents: Comprehensive Evaluation with a Lightweight Framework
May 28, 2026Masafumi Enomoto, Ryoma Obara, Haochen Zhang +1Web Agents Unified Evaluation
Draft-OPD: On-Policy Distillation for Speculative Draft Models
May 28, 2026Haodi Lei, Yafu Li, Haoran Zhang +8Speculative Decoding Self-Speculative
BlockBatch: Multi-Scale Consensus Decoding for Efficient Diffusion Language Model Inference
May 28, 2026Xiaoyou Wu, Cheng-Jhih Shih, Binfei Ji +2Diffusion Language Models LLM Inference Efficiency
Inference-Native Zeroth-Order Optimization
May 27, 2026Zelin Li, Caiwen DingLarge Language Model Fine-Tuning LLM Inference Efficiency
Thinking as Compression: Your Reasoning Model is Secretly a Context Compressor
May 27, 2026Guoxin Ma, Yibing Liu, Chengzhengxu Li +7Efficient Long-Context Inference Context Compression
Understanding Generalization and Forgetting in In-Context Continual Learning
May 27, 2026Guangyu Li, Meng Ding, Lijie HuIn-Context Learning Efficient Continual Learning
Efficient Post-training of LLMs for Code Generation With Offline Reinforcement Learning
May 27, 2026Mingze Wu, Abhinav Anand, Shweta Verma +1Code Generation LLM Post-Training Methods
PrunePath: Towards Highly Structured Sparse Language Models
May 27, 2026Zhexuan Gu, Zixun Fu, Yancheng YuanPruning Sparsity
Context Distillation as Latent Memory Management
May 27, 2026Ziyang Zheng, Zeju Li, Xiangyu Wen +5Chain-Of-Thought Distillation LLM Inference Efficiency
Dynamic-dLLM: Dynamic Cache-Budget and Adaptive Parallel Decoding for Training-Free Acceleration of Diffusion LLM
May 27, 2026Tianyi Wu, Xiaoxi Sun, Yanhua Jiao +5Diffusion Language Models LLM Inference Efficiency
UNIQUE: Universal Top-k Sparse Attention for Training-free Inference and Sparsity-aware Training
May 26, 2026Keqi Deng, Shaoshi Ling, Ruchao Fan +1Dynamic Sparse Attention LLM Inference Efficiency
LaneRoPE: Positional Encoding for Collaborative Parallel Reasoning and Generation
May 26, 2026Gabriele Cesa, Thomas Hehn, Aleix Torres-Camps +4Parallel LLM Inference Efficiency
Pair-In, Pair-Out: Latent Multi-Token Prediction for Efficient LLMs
May 26, 2026Wenhui Tan, Minghao Li, Xiaoqian Ma +5Next-Token Prediction LLM Inference Efficiency
The Coverage Illusion: From Pre-retrieval Routing Failure to Post-retrieval Cascades in a Production RAG System
May 26, 2026Zafar Hussain, Kristoffer NielboRetrieval-Augmented Generation Systems LLM Inference Efficiency
ReMoE: Boosting Expert Reuse through Router Fine-Tuning in Memory-Constrained MoE LLM Inference
May 26, 2026Xiongwei Zhu, Xiaojian Liao, Tianyang Jiang +3Mixture-Of-Expert Inference LLM Inference Efficiency
WINDQuant: Weight-Informed Neural Decision-Making for Global Mixed-Precision LLM Quantization
May 26, 2026Phong Nam Huu Nguyen, Khoi M. Le, Cong-Duy T Nguyen +3Large Language Model Quantization Quantization
Do Language Models Need Sleep? Offline Recurrence for Improved Online Inference
May 25, 2026Sangyun Lee, Sean McLeish, Tom Goldstein +1LLM Inference Efficiency Sleep
IndexMem: Learned KV-Cache Eviction with Latent Memory for Long-Context LLM Inference
May 25, 2026Xintong Yang, Hao Gu, Binxing Xu +6Efficient Long-Context Inference Key-Value Cache Eviction
InfoQuant: Shaping Activation Distributions for Low-Bit LLM Quantization
May 25, 2026Ke Li, Dong An, Xiaoling Zang +6Large Language Model Quantization Quantization
Influence-Inspired Spectral Rotations for Extreme Low-Bit LLM Quantization
May 24, 2026Gorgi PavlovLarge Language Model Quantization LLM Inference Efficiency
H $^{2}$ MT: Semantic Hierarchy-Aware Hierarchical Memory Transformer
May 24, 2026Maryam Haghifam, Zifan He, Jason Cong +1LLM Inference Efficiency Transformer Architectures
Divide-and-Conquer Inference for Large-Scale Visual Recognition with Multimodal Large Language Models
May 24, 2026Zhipeng Ye, Jiaqi Huang, Feng Jiang +5LLM Inference Efficiency Imagenet
Beyond the Target: From Imitation to Collaboration in Speculative Decoding
May 24, 2026Jinze Li, Yixing Xu, Guanchen Li +7Speculative Decoding LLM Inference Efficiency
CONF-KV: Confidence-Aware KV Cache Eviction with Mixed-Precision Storage for Long-Horizon LLM
May 24, 2026Yubo Li, Yidi MiaoKv-Cache Blocks Key-Value Cache Eviction
Beyond Inference-Only Deployment: Comparing Weight-Based Consolidation Against Cascading Compaction
May 23, 2026Simon Dennis, Kevin Shabahang, Hao Guo +1LLM Inference Efficiency Consolidation
Identifying and Mitigating Systemic Measurement Bias in Production LLM Inference Benchmarks
May 22, 2026Ashok Chandrasekar, Jason KrambergerPer-Token Latency LLM Inference Efficiency
Inference Time Context Sparsity: Illusion or Opportunity?
May 22, 2026Sahil Joshi, Prithvi Dixit, Agniva Chowdhury +5LLM Inference Efficiency Dynamic Sparse Attention
EVA: Accelerating LLM Decoding via an Efficient Vector Quantization Architecture
May 22, 2026Bowen Duan, Cong Guo, Chiyue Wei +8Calibration-Free Low-Bit Vector Quantization Extreme Low-Bit Quantization
When NPUs Are Not Always Faster: A Stage-Level Analysis of Mobile LLM Inference
May 22, 2026Pu Li, Jiawen Qi, Qinyu ChenNeural Processing Units LLM Inference Efficiency
XWind: A Cross-site Router for Large Language Model Inference Serving at Renewable Energy Farms
May 22, 2026Tella Rajashekhar Reddy, Atharva Deshmukh, Liangcheng Yu +7Artificial Intelligence Data Centers Data Centers
Adaptive Mass-Segmented KV Compression for Long-Context Reasoning
May 22, 2026Junzhe Yang, Xiaoyu ShenKey-Value Cache Compression Key-Value Cache
ThriftAttention: Selective Mixed Precision for Long-Context FP4 Attention
May 21, 2026Joe SharrattEfficient Long-Context Inference Quantizer
GEMQ: Global Expert-Level Mixed-Precision Quantization for MoE LLMs
May 21, 2026Jianing Deng, Song Wang, Dongwei Wang +4Mixed-Precision Quantization Mixture-Of-Expert Language Models
ModeSwitch-LLM: A Lightweight Phase-Aware Controller for Cross-Mode LLM Inference on a Single GPU
May 21, 2026Aman Sunesh, Ali Alshehhi, Hivansh DhakneLLM Inference Efficiency Large Language Models(Llms
LLM Code Smells: A Taxonomy and Detection Approach
May 21, 2026Zacharie Chenail-Larcher, Brahim Mahmoudi, Naouel Moha +2Code Quality LLM Inference Efficiency
Asymmetric Virtual Memory Paging for Hybrid Mamba-Transformer Inference
May 21, 2026An Xuan NguyenInference Workloads LLM Inference Efficiency
PALS: Power-Aware LLM Serving for Mixture-of-Experts Models
May 20, 2026Can Hankendi, Rana Shahout, Minlan Yu +1LLM Inference Efficiency Single Graphics Processing Unit

Jun 1, 2026 · cs.CLJ/K move · Enter open · S save

ProbeScale: Probing Analysis to Optimize Neural Scaling Laws for Efficient Small Language Model Inference

Sourav Das

Department of Computer Science and Engineering Indian Institution of Information Technology Kalyani Kalyani, West Bengal, India

Small Language Models (SLMs) offer a balance between capability and computational feasibility. Neural scaling laws inform their optimal training, suggesting that they possess rich internal representations that scale with their size. However, deploying even these SLMs can be challenging under strict resource constraints. Language model probing provides methods for analyzing the linguistic knowledge encoded in a model's internals. We propose ProbScale, a framework that unifies insights from scaling laws and probing to identify parameter-efficient subnetworks within pre-trained SLMs. ProbScale utilizes the high-quality representations of well-scaled SLMs and uses task-specific probes to mathematically quantify the relevance of each layer for target downstream capabilities. This allows selecting subnetworks that optimally trade off performance against parameter size. We formulate the subnetwork selection as finding a layer subset maximizing aggregated, task-weighted probe performance under a parameter budget. Experiments on representative SLMs such as RoBERTa-Large and T5-Base demonstrate that ProbScale identifies subnetworks achieving significant parameter reduction, from 5 to 10 times, while maintaining high performance (95% to 98% of the original SLMs) on targeted tasks, outperforming heuristic baselines.

Small Language Models LLM Inference Efficiency Scaling Laws

Open paper Ask about it PDF