LLM Inference

LLM: Large Language Model

Overview

Large language model inference is the process of turning prompts into outputs, and its speed, cost, reliability, and privacy shape what applications can use models in practice. Systems work optimizes prefill and token-by-token decoding through KV-cache management, batching and scheduling, parallelism, quantization, and specialized GPU, mobile, or edge kernels. Another strand changes generation at test time using speculative decoding, early exit, retrieval, or repeated reasoning paths to trade extra computation for better accuracy or confidence. The central challenge is that these trade-offs depend heavily on workload, hardware, and inference backend, making reproducible benchmarks and methods that preserve quality while reducing latency, memory, and energy ongoing priorities.

Latest papers

CardsList

321 papers43 papers in the last four weeks, up 8% on the four weeks before. 0.3% of all new papers.

  1. Local AI Agents in Power Electronics Simulation: Deployment, Measurements and Costs

    Nov 18, 2026Yuzhuo Li, Pasan Gunawardena, Yunwei LiLLM Agent EfficiencyLLM Agent Evaluation

  2. Harvesting Expert-Parallel Bubbles for Fine-Tuning

    Nov 18, 2026Oana Balmau, Saurav Banka, Jiaxuan Chen +5LLM Inference SchedulingMixture-of-Experts Fine-Tuning

  3. Learning Stratified Policies for Inference-Time Objectives

    Nov 17, 2026Mridul Mahajan, Xuezhou Zhang, Aldo PacchianoLLM InferenceInference-Time Optimization

  4. Benchmarking LLM Inference on Desktop-Scale NVIDIA Grace Blackwell Clusters

    Nov 15, 2026Kevin Bhimani, Christian Escobar, Daniel Mihovch +2LLM InferenceParallel LLM Inference

  5. How Much Determinism Can Fine-Tuning Buy? Margin Regularisation Against Batch-Induced Token Flips

    Nov 14, 2026Charan Teja Nagisettygari, Siva Raja SindiramuttyLLM InferenceError Propagation in LLM Inference

  6. From Bytes to Behavior: Request-Level Attribution from Live GPU Hidden States

    Nov 13, 2026Jonathan Pan, Yidi Yuan, Xin Wei Chia +3LLM AuditingLLM Inference

  7. Evaluating Large Language Models for Scientific Paper Search: Retrieval Success and Inference Latency on Mac Studio and NVIDIA A100

    Nov 13, 2026Frederic Andriantseheno Rakoto, Yannis Bendi-Ouis, Xavier HinautLLM EvaluationScientific Document Retrieval

  8. Superfluid: Local AI Server for Distributed LLM Inference

    Nov 13, 2026Prabod Rathnayaka, Isura Manchanayake, Srikandabala Kogul +3Local LLM DeploymentLLM Serving

  9. EngineAgent: Agentizing On-Device LLM Inference Engines with Inference Stack Hierarchy Awareness and Hardware Abilities

    Nov 10, 2026Zhengxiang Huang, Yujie Sun, Shengheng Chen +6On-Device Language Model InferenceLLM Inference

  10. Decode Token Parallelism: Memory-Efficient Distributed Attention Parallelism for Scalable LLM Decoding

    Nov 9, 2026Susav Shrestha, Brad Settlemyer, Narasimha ReddyParallel LLM InferenceLLM Inference

  11. One Temperature to Rule Them All?

    Nov 9, 2026Aviv Orly, Ori Shem-Ur, Yaron OzLLM InferenceDiversity-Aware Sampling

  12. BridgeKV: Large-Model Prefill and Small-Model Decoding for LLM Inference via KV Cache Translation

    Nov 9, 2026Payman Behnam, Debopam Sanyal, Pranav Ponnusamy +6LLM InferenceLLM Inference Acceleration

  13. Enhancing Mathematical Reasoning in Turkish Large Language Models with Search-Based Inference and Model Merging

    Nov 7, 2026Ahmed Zeer, Osama Shbib, Muzaffer Kaan Yuce +2LLM ReasoningLLM Fine-Tuning

  14. Detent: Batch-Invariant LLM Serving and RL Rollouts at Near-Stock Speed

    Nov 7, 2026Peixu Xin, Runchuan Feng, Jiazhen Guo +2LLM ServingLLM Inference

  15. Efficient LLM Adaptation with Forward-Only Passes

    Nov 7, 2026Baichuan Huang, Ananth Balashankar, Amir AminifarInference-Time Adaptation of Language ModelsLLM Fine-Tuning

  16. StrataFlux: Memory-Hierarchy-Native Low-Bit LLM Inference

    Nov 5, 2026Jake Hyun, Ziyu Zhao, Jae-sun Seo +1LLM InferenceLLM Inference Efficiency

  17. Create Your Own Work Stack: Deploying Self-hosted AI and Tooling in a Tech Startup

    Nov 5, 2026Saad Ahmed Bazaz, Talha Amjad, Abdurrehman Subhani +2Local LLM DeploymentCost-Aware Inference

  18. ACSeL: Adaptive Configuration Search for Hardware-Constrained LLM Inference

    Nov 5, 2026Bokun Wang, Hung-Yueh Chiang, Diana MarculescuLLM InferenceEfficient Language Model Inference

  19. BlockLock: Reproducible LLM Inference with Fixed Batch Geometry

    Nov 4, 2026Xiuyuan Qi, Pedro Reviriego, Javier Conde +1LLM InferenceLLM Inference Scheduling

  20. Dory1.0-4B: A Looped Normalized-GPT

    Nov 2, 2026Maxim Fishman, Simeng Sun, Shantanu Acharya +6LLM InferenceRecurrent Language Models

  21. The SIMPLE Law for Energy Cost of LLM Inference

    Nov 2, 2026Hiari Pizzini Cavagna, Andrea Proia, Daniele Cesarini +2LLM InferenceEnergy-Aware LLM Inference