LLM Inference

LLM: Large Language Model

Large language model inference is the process of turning prompts into outputs, and its speed, cost, reliability, and privacy shape what applications can use models in practice. Systems work optimizes prefill and token-by-token decoding through KV-cache management, batching and scheduling, parallelism, quantization, and specialized GPU, mobile, or edge kernels. Another strand changes generation at test time using speculative decoding, early exit, retrieval, or repeated reasoning paths to trade extra computation for better accuracy or confidence. The central challenge is that these trade-offs depend heavily on workload, hardware, and inference backend, making reproducible benchmarks and methods that preserve quality while reducing latency, memory, and energy ongoing priorities.

Latest papers

285 papers44 papers in the last four weeks, up 63% on the four weeks before. 0.3% of all new papers.

Tile-Level Activation Overlap for Efficient LLM Inference
May 5, 2026Abhinav Jangda, Tyler Sorensen, Sebastian Burckhardt +3LLM Inference Efficiency GPU Kernel Optimization
Microsoft Research · Microsoft
Two Calls, Two Moments, and the Vote-Accuracy Curve of Repeated LLM Inference
May 5, 2026Yi LiuLLM Inference Test-Time Scaling
York University
Reasoning emerges from constrained inference manifolds in large language models
May 2, 2026Yanbiao Ma, Fei Luo, Linfeng Zhang +10LLM Inference LLM Interpretability
Renmin University of China · Tsinghua University · Xiaomi EV +1
Position: LLM Serving Needs Mathematical Optimization and Algorithmic Foundations, Not Just Heuristics
May 2, 2026Zijie ZhouLLM Serving LLM Inference
Department of Industrial Engineering and Decision Analytics, HKUST
Belief-Guided Inference Control for Large Language Model Services via Verifiable Observations
Apr 30, 2026Wenhao Yuan, Chenchen Lin, Jian Chen +3LLM Inference Cost-Aware Inference
The University of Hong Kong · Sun Yat-sen University
Length Value Model: Scalable Value Pretraining for Token-Level Length Modeling
Apr 29, 2026Zhen Zhang, Changyi Yang, Zijie Xia +11LLM Inference Controllable Text Generation
University of California, Santa Barbara · Carnegie Mellon University · LMSYS Org +2
DUAL-BLADE: Dual-Path NVMe-Direct KV-Cache Offloading for Edge LLM Inference
Apr 29, 2026Bodon Jeong, Hongsu Byun, Youngjae Kim +4On-Device Language Model Inference KV-Cache Offloading
1Sogang University, Seoul, Republic of Korea · 2Florida State University, FL, USA · 3Samsung Electronics Co.
Entropy Centroids as Intrinsic Rewards for Test-Time Scaling
Apr 28, 2026Wenshuo Zhao, Qi Zhu, Xingshan Zeng +4LLM Inference Test-Time Scaling
May · HKUST · ZJU +1
Hybrid JIT-CUDA Graph Optimization for Low-Latency Large Language Model Inference
Apr 25, 2026Divakar Kumar Yadav, Tian ZhaoLLM Inference LLM Inference Acceleration
Department of Computer Science University of Wisconsin-Milwaukee Milwaukee, WI, USA
Cloud to Edge: Benchmarking LLM Inference On Hardware-Accelerated Single-Board Computers
Apr 24, 2026Harri Renney, Fouad Trad, Michael Mattarock +1AI Accelerator Inference On-Device Language Model Inference
Kaze Technologies Kaze Consulting Bath, BA1 2HN, UK · Electrical and Computer Engineering Lebanese American University Byblos, Lebanon · Carnegie Mellon University Pittsburgh Pennsylvania USA +1
Introducing Background Temperature to Characterise Hidden Randomness in Large Language Models
Apr 24, 2026Alberto Messina, Stefano ScottaLLM Inference ML Reproducibility
1RAI - Radiotelevisione Italiana, Centre for Research, Technological Innovation and Experimentation (CRITS)
Latent Phase-Shift Rollback: Inference-Time Error Correction via Residual Stream Monitoring and KV-Cache Steering
Apr 20, 2026Manan Gupta, Dhruv KumarLLM Self-Correction Language Model Error Detection
1BITS Pilani, Pilani Campus, India
Benchmarking System Dynamics AI Assistants: Cloud Versus Local LLMs on CLD Extraction and Discussion
Apr 20, 2026Terry LeitchLLM Quantization Dynamical Systems
Ruxton AI
Unlocking the Edge deployment and ondevice acceleration of multi-LoRA enabled one-for-all foundational LLM
Apr 20, 2026Sravanth Kodavanti, Sowmya Vajrala, Srinivas Miriyala +13On-Device Language Model Inference LLM Inference
Samsung Research Institute Bangalore, India · Samsung Electronics, Suwon, South Korea
PoQ-Judge: A Multi-Architecture Evaluation Framework for Cost-Aware Proof-of-Quality in Decentralized LLM Inference
Apr 20, 2026Arther Tian, Alex Ding, Frank Chen +2LLM Evaluation LLM-as-a-Judge
DGrid AI
DeInfer: Efficient Parallel Inferencing for Decomposed Large Language Models
Apr 20, 2026You-Liang Huang, Xinhao Huang, Chengxi Liao +1LLM Inference Parallel LLM Inference
Boston University · Boston, Massachusetts, USA · HKUST (Guangzhou) +2
MoE-nD: Per-Layer Mixture-of-Experts Routing for Multi-Axis KV Cache Compression
Apr 20, 2026Libo Sun, Peixiong He, Po-Wei Harn +1LLM Inference KV Caching
Department of Computer Science and Software Engineering, Auburn University, Auburn, AL, USA · Department of Information Management, National Central University, Taoyuan, Taiwan
Configuration Over Selection: Hyperparameter Sensitivity Exceeds Model Differences in Open-Source LLMs for RTL Generation
Apr 18, 2026Minghao Shao, Zeng Wang, Weimin Fu +5LLM Evaluation Language Model Generation Evaluation
NYU Tandon School of Engineering, USA · Kansas State University, USA · NYU Abu Dhabi, UAE
LACE: Lattice Attention for Cross-thread Exploration
Apr 16, 2026Yang Li, Zirui Zhang, Yang Liu +1Parallel Reasoning in Language Models LLM Inference
Department of Computer Science Rutgers University · Amazon AGI Labs
Scepsy: Serving Agentic Workflows Using Aggregate LLM Pipelines
Apr 16, 2026Otto White, Marcel Wagenländer, Britannio Jarrett +7LLM Inference LLM Inference Scheduling
Imperial College London · Independent Researcher
The Illusion of Equivalence: Systematic FP16 Divergence in KV-Cached Autoregressive Inference
Apr 16, 2026Ranjith Chodavarapu, Lei XuLLM Inference
Department of Computer Science Kent State University, USA
Fully Homomorphic Encryption on Llama 3 model for privacy preserving LLM inference
Apr 14, 2026Anes Abdennebi, Nadjia Kara, Laaziz LahlouPrivacy-Preserving Language Models LLM Inference
Software and IT Engineering Department, École de Technologie Supérieure, Montreal, Quebec, Canada.
ProbeLogits: Kernel-Level LLM Inference Primitives for AI-Native Operating Systems
Apr 13, 2026Daeyeon SonAI Safety LLM Guardrails
Independent Researcher Republic of Korea
DeCoVec: Building Decoding Space based Task Vector for Large Language Models via In-Context Learning
Apr 13, 2026Feiyang Li, Yile WangLanguage Model Steering Task Vectors
College of Computer Science and Software Engineering, Shenzhen University
Dead Weights, Live Signals: Feedforward Graphs of Frozen Language Models
Apr 9, 2026Marcus Armstrong, Navid Ayoobi, Arjun MukherjeeLLM Inference Language Model Ensembles
Department of Computer Science University of Houston Houston, TX 77204
Blink: CPU-Free LLM Inference by Delegating the Serving Stack to GPU and SmartNIC
Apr 8, 2026Mohammad Siavashi, Mariano Scazzariello, Gerald Q. Maguire +2LLM Serving LLM Inference
KTH Royal Institute of Technology, Stockholm, Sweden · RISE, Stockholm, Sweden
Comparative Characterization of KV Cache Management Strategies for LLM Inference
Apr 6, 2026Oteo Mamo, Olga Kogiou, Hyunjin Yi +1KV-Cache Management LLM Inference Efficiency
Florida State University, Tallahassee, FL, USA
The Energy Footprint of LLM-Based Environmental Analysis: LLMs and Domain Products
Mar 31, 2026Alicia Bao, Jiamian He, Angel Hsu +3LLM Inference Efficiency LLM Inference
James · Department of Computer Science, University of North Carolina at Chapel Hill, Chapel Hill, NC 27516 · Arboretica, Rotterdam, The Netherlands +2
Understand and Accelerate Memory Processing Pipeline for Large Language Model Inference
Mar 30, 2026Zifan He, Rui Ma, Yizhou Sun +1LLM Inference LLM Inference Acceleration
Department of Computer Science, University of California, Los Angeles · Microsoft Research Asia
Stream2LLM: Overlap Context Streaming and Prefill for Reduced Time-to-First-Token (TTFT)
Mar 29, 2026Rajveer Bachkaniwala, Chengqi Luo, Richard So +2LLM Inference LLM Inference Serving
Georgia Tech, Atlanta, USA

May 5, 2026J/K move · Enter open · S save

Tile-Level Activation Overlap for Efficient LLM Inference

Abhinav Jangda, Tyler Sorensen, Sebastian Burckhardt, Jianlan YE+2

Microsoft Research · Microsoft

LLM Inference Efficiency GPU Kernel Optimization GPU Acceleration LLM Inference High-Performance Computing

Open paper AskPDF arXiv

SwiGLU is the dominant MLP activation in modern large language models, yet its intermediate tensor materialization costs 9-37% of MLP execution time. We present two complementary CUTLASS-based SM90 kernels that fuse SwiGLU into GeMM at the tile level. Kernel-1 overlaps Swish computation on the Gate accumulator with Up-tile loading using the Pingpong warp-specialized schedule; Kernel-2 interleaves SwiGLU with tile stores via a custom Epilogue Visitor Tree. Evaluated on Qwen-2.5 models (0.5B-72B) on NVIDIA H100, our kernels achieve up to 2.47x speedup over PyTorch, shifting workloads from memory-bound to compute-bound and reaching 79.5% peak BF16 utilization. We demonstrate that torch.compile cannot replicate this fusion (3-7x slower than our kernels), validating the need for hand-crafted tile-level design. Our fused kernels are also numerically superior, achieving zero mismatches compared to 4.5-11% for cuBLAS.