LLM Inference

LLM: Large Language Model

Ask about this topic

Overview

Large language model inference is the process of turning prompts into outputs, and its speed, cost, reliability, and privacy shape what applications can use models in practice. Systems work optimizes prefill and token-by-token decoding through KV-cache management, batching and scheduling, parallelism, quantization, and specialized GPU, mobile, or edge kernels. Another strand changes generation at test time using speculative decoding, early exit, retrieval, or repeated reasoning paths to trade extra computation for better accuracy or confidence. The central challenge is that these trade-offs depend heavily on workload, hardware, and inference backend, making reproducible benchmarks and methods that preserve quality while reducing latency, memory, and energy ongoing priorities.

Latest papers

CardsList

285 papers44 papers in the last four weeks, up 63% on the four weeks before. 0.3% of all new papers.

  1. The Real Cost of Context: Predicting Compute Time and Energy Consumption in LLM Prefill and Decode Phases

    Oct 29, 2026Lars Krupp, Daniel Geißler, Francisco M. Calatrava-Nicolas +3LLM InferenceEnergy-Aware LLM Inference

    RPTU Kaiserslautern-Landau, DFKI · Örebro University · RPTU Kaiserslautern-Landau

  2. LLM-PowerTrace: A Dataset of Time-Resolved GPU Power Traces for AI Compute

    Oct 28, 2026Jiansheng Wang, Jaskeerat Singh, Haochen Xu +6LLM Inference

    New York University, New York, NY, USA · University of Alberta, Edmonton, AB, Canada · Massachusetts Institute of Technology, Cambridge, MA, USA

  3. Answer-Only Disclosure: Private-Context LLM Assistance over Web APIs with a Leakage Ledger

    Oct 28, 2026Liang Yuan, Xiaohu Du, Ruikun Luo +6Privacy Leakage in Language ModelsPrivacy-Preserving Language Models

    Australian Catholic University North Sydney, NSW, Australia · Ant Group China · University of Macau Macau, China +5

  4. Satellite Intelligence: Distributed Training, Collaborative Inference,and LLM Adaptation in LEO Constellation

    Oct 26, 2026Zheng Lin, Zhanwei Wang, Yuxin Zhang +9LEO Satellite NetworksLLM Inference

    Interdisciplinary Centre for Security, Reliability and Trust (SnT), University of Luxembourg, L-1855 Luxembourg, Luxembourg · Department of Computer and Electronic Engineering, University of Hong Kong, Pok Fu Lam, Hong Kong, China · Institute of Space Internet, Fudan University, Shanghai, China +5

  5. BellowsKV: One Latent for Many Layers, Expanded on Demand

    Oct 26, 2026Jiamu Zhang, Hanjie ChenKV-Cache CompressionLLM Inference

    Rice University

  6. One Model, Two Paths: Balancing Single-Pass and Looped Performance in Language Models

    Oct 24, 2026Jiarun Shen, Lei Zhao, Xiao LiLLM InferenceFine-Tuning Language Models

    The Chinese University of Hong Kong, Shenzhen · Shanghai Jiao Tong University

  7. DualPace: Decoupled Scheduling and Cooperative GPU--CPU Decoding for LLM Serving

    Oct 24, 2026YuXuan Lin, KeQiang Wu, Kingsum ChowLLM ServingLLM Inference

    Ningbo Global Innovation Center, Zhejiang University, Ningbo, China

  8. LLMs in Cloud: A Discussion on Architecture, Orchestration, and Inference

    Oct 24, 2026Jannatun Noor, Athar Noor Mohammad Rafee, Md. Sadman Haque +3LLM ServingLLM Inference

    Computing for Sustainability and Social Good (C2SG) Research Group, Department of Computer Science and Engineering, United International University, Dhaka, Dhaka 1212, Bangladesh. · Department of Computer Science and Engineering, School of Data & Computational Sciences, BRAC University, Dhaka, Dhaka 1212, Bangladesh.

  9. InferSlice: A Reference Architecture for SLO-driven LLM Inference over a Telco Wide-Area Network

    Oct 23, 2026Juan Parra-Ullauri, Anthony Chiu, Hari Madhukumar +6LLM InferenceLLM Inference Serving

    BT Group · University of Bristol

  10. Speculative Diffusion is a Tale of Two Tradeoffs

    Oct 22, 2026Aaron Zorel, Guy Van den Broeck, Aditya Grover +1LLM InferenceSpeculative Decoding

    Independent Researcher · University of California, Los Angeles

  11. Evaluating the Applicability of LLM Inference Optimizations on Apple Silicon

    Oct 22, 2026Kyrylo Yemets, Andrii Yaroshevych, Victor Muryn +5LLM InferenceOn-Device Language Model Inference

    MacPaw AI&Research, Kyiv, Ukraine · Department of Artificial Intelligence, Lviv Polytechnic National University, Lviv, Ukraine · Institute of Mathematics of NAS of Ukraine, Kyiv, Ukraine

  12. Low-Latency MoE Inference in Space Data Centers: A Wireless-Power-Transfer Perspective

    Oct 22, 2026Haotian Zheng, Zhanwei Wang, Kaibin HuangEnergy-Aware LLM InferenceLLM Inference

    Department of Electrical and Computer Engineering, The University of Hong Kong, Hong Kong SAR, China

  13. The Token-Energy Law of LLM Decode: Active Bytes and Realized Time Account for Joules per Token

    Oct 20, 2026Chenxu Niu, Batuhan Sencer, Wei Zhang +1Energy-Aware LLM InferenceLLM Inference

    NVIDIA Corporation, USA · Texas Tech University, USA · Texas Advanced Computing Center, USA

  14. Local Inference of LLMs on Long Contexts: A Data Systems Perspective

    Oct 20, 2026Mattias Dürrmeier, Alberto LernerLLM InferenceLong-Context Inference

    University of Fribourg, Fribourg, Switzerland · Computing Flows GmbH, Fribourg, Switzerland

  15. Privacy-preserving inference on hosted models: a learned input-embedding transform for frozen Qwen3.8-27B

    Oct 20, 2026Mahendra Roopa, Srikanth RoopaPrivacy-Preserving Language ModelsLLM Inference

    Spinal

  16. Making every stage of medical device document assessment with large language models deterministic

    Oct 15, 2026Ryosuke SuzukiLLM ReliabilityLLM Inference

    GFLOPS Co., Ltd., Tokyo, Japan.

  17. Energy Consumption Analysis of Discrete Diffusion and Autoregressive Language Models

    Oct 14, 2026Boris Ruf, Marcin DetynieckiLLM InferenceEnergy-Aware LLM Inference

    AXA, AI Research Paris, France

  18. Selective Reasoning for Typed Decision Models: When to Start, Read, and Stop

    Oct 14, 2026Bin Zheng, Zheng ChenEfficient Language Model ReasoningLLM Decision-Making

    University of Electronic Science and Technology of China

  19. AgriJev: Efficient Agricultural Candidate Scoring with Multi-Scale Adapted Models

    Oct 14, 2026Xuechen LiLanguage ModelingLLM Inference

    University of Minnesota Twin Cities

  20. Scaling LLM Inference via Parallelism: Throughput-Latency Trade-off

    Oct 14, 2026Javad Mirzaei, Jeebak MitraParallel LLM InferenceLLM Inference

    Office of CTO Dell Technologies Inc. Canada

  21. Whose turn is it? Context-guided model routing for stateful agents

    Oct 13, 2026Anikesh Parashar, Avik Dutta, Vashu Chauhan +6LLM RoutingLLM Agent Routing

    Microsoft

  22. Stable Token-level Scheduling for Serverless LLM Inference

    Oct 13, 2026Hardi Ashishbhai Desai, Balaji N K, Sourav Kanti AddyaLLM Inference SchedulingLLM Inference Serving

    Cloud & Smart System Services Lab, Department of Computer Science and Engineering, National Institute of Technology Karnataka, Surathkal, India

  23. Towards Green AI: Evaluating Energy Efficiency for Quantized LLM Inference

    Oct 13, 2026Petar BastaLLM Inference EfficiencyLLM Quantization

    University of Waterloo Computer Science Waterloo, Ontario, Canada

  24. Can a System-One LLM Perform Knowledge Tracing When Few or No Learners Are Logged?

    Oct 8, 2026Unggi Lee, Haeun ParkKnowledge TracingLLM Inference

    Korea University Sejong Campus · Korea University

  25. SemanticFold: Latent Sequence Compression SeparatesLanguage Modeling, Decodability, and Reasoning

    Oct 7, 2026Mingyan Liu, Min HuangLLM Context CompressionLLM Compression

  26. Reproducible LLM Inference Benchmarking: A Sequential Isolation Protocol for Regression Testing

    Oct 7, 2026Arnold Olympio, Juan Manuel Servera Bondroit, Wael Abdelmalek +2Benchmark ReproducibilityLLM Inference

    Lucerne University of Applied Sciences and Arts, Switzerland · Microsoft, Switzerland · Uthereal AG, Switzerland +1

  27. Decoupling Logic from Persona: Structural Immunity of Edge LLM Agents to Context Pollution

    Oct 7, 2026Masaaki Nakatsu, Reno WangLLM InferenceLLM Context Management

    AO, Inc. / OrbLabs AG · AO, Inc.

  28. DySCo: Dynamic Sharding for Collaborative Edge-Cloud LLM Inference with Depth-Synchronized Batching

    Oct 6, 2026Jingpo Xu, Paul Joe Maliakel, Ivona Brandic +1LLM InferenceEdge-Cloud Collaborative Inference

    University of Amsterdam Amsterdam, Netherlands · TU Wien Vienna, Austria

  29. Cascadia: Resident 975B MoE Inference on Eleven AI PCs

    Oct 5, 2026Tate Berenbaum, Matias Parij, Muthaiah VenkatachalamLLM InferenceMixture-of-Experts Inference

    Not Community Labs Inc. · Intel Corporation

  30. Shared Stopping Decisions Change Answers in HQQ Cache Quantization

    Oct 5, 2026Seunghui Jwa, Minsu Oh, Chanjun Park +1LLM InferenceKV-Cache Quantization

    Department of Artificial Intelligence, Jeju National University · School of Software, Soongsil University