LLM Inference

LLM: Large Language Model

Ask about this topic

Overview

Large language model inference is the process of turning prompts into outputs, and its speed, cost, reliability, and privacy shape what applications can use models in practice. Systems work optimizes prefill and token-by-token decoding through KV-cache management, batching and scheduling, parallelism, quantization, and specialized GPU, mobile, or edge kernels. Another strand changes generation at test time using speculative decoding, early exit, retrieval, or repeated reasoning paths to trade extra computation for better accuracy or confidence. The central challenge is that these trade-offs depend heavily on workload, hardware, and inference backend, making reproducible benchmarks and methods that preserve quality while reducing latency, memory, and energy ongoing priorities.

Latest papers

CardsList

285 papers44 papers in the last four weeks, up 63% on the four weeks before. 0.3% of all new papers.

  1. Accelerating PayPal's Commerce Agent with Speculative Decoding: An Empirical Study on EAGLE3 with Fine-Tuned Nemotron Models

    Mar 27, 2026Ally Qin, Jian Wan, Sarat Mudunuri +1LLM InferenceLLM Inference Acceleration

  2. GhostServe: A Lightweight Checkpointing System in the Shadow for Fault-Tolerant LLM Serving

    Mar 26, 2026Shakya Jayakody, Youpeng Zhao, Chinmay Dhanraj Nehate +1LLM ServingLLM Inference

    University of Central Florida, Orlando, FL, USA.

  3. Secure On-Premise Deployment of Open-Weights Large Language Models in Radiology: An Isolation-First Architecture with Prospective Pilot Evaluation

    Mar 25, 2026Sebastian Nowak, Jann-Frederick Laß, Narine Mesropyan +8LLM ServingLLM Security

    Department of Diagnostic and Interventional Radiology, University Hospital Bonn · Department of Research-IT, University Hospital Bonn

  4. Energy Efficient Software Hardware CoDesign for Machine Learning: From TinyML to Large Language Models

    Mar 24, 2026Mohammad Saleh Vahdatpour, Yanqing ZhangPower-Efficient InferenceEnergy-Efficient ML

    Department of Computer Science Georgia State University Atlanta, GA, USA

  5. TIDE: Token-Informed Depth Execution for Per-Token Early Exit in LLM Inference

    Mar 22, 2026Jaber Jaber, Osama JaberLLM InferenceLLM Inference Acceleration

    RightNow AI

  6. Enhancing reasoning accuracy in large language models during inference time

    Mar 22, 2026Vinay Sharma, Manish JainLLM InferenceLLM Reasoning

  7. NanoZK: Privacy-Preserving Verifiable Inference for Large Language Models via Layerwise Zero-Knowledge Proofs

    Mar 17, 2026Zhaohui WangZero-Knowledge MLPrivacy-Preserving ML

    USC Viterbi School of Engineering, Los Angeles, CA, USA

  8. This Is Taking Too Long -- Investigating Time as a Proxy for Energy Consumption of LLMs

    Mar 16, 2026Lars Krupp, Daniel Geißler, Francisco M. Calatrava-Nicolas +3Energy-Aware LLM InferenceLLM Inference

    Embedded Intelligence DFKI and RPTU Kaiserslautern, Germany · Centre for Applied Autonomous Sensor Systems (AASS) Örebro University Örebro, Sweden

  9. Clinician input steers AI toward accurate and harmful recommendations

    Mar 14, 2026Ivan Lopez, Selin S. Everett, Bryan J. Bunning +10Medical LLMsClinical Language Model Evaluation

    Department of Biomedical Data Science, Stanford University, Stanford, CA, USA · Stanford University School of Medicine, Stanford, CA, USA · Quantitative Sciences Unit, Department of Medicine, Stanford University School of Medicine, Stanford, CA, USA +14

  10. Domain-Specialized Tree of Thought through Plug-and-Play Predictors

    Mar 14, 2026Xuanqi Gao, Haoyu Wang, Jun Sun +2LLM ReasoningTree of Thoughts

    Xi’an Jiaotong University, China · Singapore Management University, Singapore · University of Massachusetts at Amherst, United States

  11. Intelligent Materials Modelling: Large Language Models Versus Partial Least Squares Regression for Predicting Polysulfone Membrane Mechanical Performance

    Mar 14, 2026Dingding Cao, Mieow Kee Chan, Wan Sieng Yeo +2Materials Property PredictionPolymer Property Prediction

    School of Big Data, Baoshan University, Baoshan, 678000, Yunnan Province, China. · Centre for Water Research, Faculty of Engineering, Built Environment and Information Technology, SEGi University. Jalan Teknologi, Kota Damansara, 47810 Petaling Jaya, Selangor Darul Ehsan, Malaysia. · School of Energy and Chemical Engineering, Xiamen University Malaysia, Jalan Sunsuria, Bandar Sunsuria, 43900 Sepang, Selangor, Malaysia. +3

  12. TaxBreak: Unmasking the Hidden Costs of LLM Inference Through Overhead Decomposition

    Mar 12, 2026Prabhu Vellaisamy, Shreesh Tripathi, Vignesh Natarajan +3LLM Inference EfficiencyLLM Inference

    NeuroAI Computer Architecture Lab (NCAL), Electrical and Computer Engineering Department, Carnegie Mellon University

  13. Characterizing Performance-Energy Trade-offs of Large Language Models in Multi-Request Workflows

    Mar 12, 2026Md. Monzurul Amin Ifath, Israat HaqueParallel LLM InferenceLLM Inference

    Dalhousie University, Halifax, NS, Canada

  14. BLooP: Zero-Shot Abstractive Summarization using Large Language Models with Bigram Lookahead Promotion

    Mar 12, 2026Varun Iyer, Cornelia CarageaText SummarizationLanguage Model Decoding

    University of Illinois Chicago

  15. Markovian Generation Chains in Large Language Models

    Mar 11, 2026Mingmeng Geng, Amr Mohamed, Guokan Shang +2LLM InferenceDiverse Text Generation

    ENS-PSL & CNRS-Lattice · MBZUAI & Ecole Polytechnique · MBZUAI

  16. Summarize Before You Speak with ARACH: A Training-Free Inference-Time Plug-In for Enhancing LLMs via Global Attention Reallocation

    Mar 10, 2026Jingtao Wang, Yucong Wang, Jun Ding +2LLM InferenceInference-Time Adaptation of Language Models

    Meakins-Christe Laboratories, Research Institute of McGill University Health Centre, 1001 Decarie Blvd, Montreal, H4A 3J1, Quebec, Canada · Department of Medicine, Division of Experimental Medicine, McGill University, 1001 Decarie Blvd, Montreal, H4A 3J1, Quebec, Canada · The College of Computer Science and Technology, Zhejiang Gongshang University, Hangzhou, China +4

  17. Can LLMs Perceive Time? An Empirical Investigation

    Mar 9, 2026Aniketh GarikaparthiTemporal Reasoning in Language ModelsLLM Inference

    TCS Research

  18. Reject, Resample, Repeat: Understanding Parallel Reasoning in Language Model Inference

    Mar 9, 2026Noah Golowich, Fan Chen, Dhruv Rohatgi +4Parallel Reasoning in Language ModelsLLM Inference

    Microsoft Research · MIT · NYU

  19. Trusting What You Cannot See: Auditable Fine-Tuning and Inference for Proprietary AI

    Mar 8, 2026Heng Jin, Chaoyu Zhang, Hexuan Yu +4Verifiable Outsourced ComputationLLM Auditing

    Virginia Tech · Washington University in St. Louis

  20. M2K: Making the Model-Kernel Interface Explicit for Reliable CUDA Kernel Verification

    Mar 6, 2026Mengting He, Shihao Xia, Haomin Jia +2GPU Kernel Correctness VerificationLLM Inference

    Penn State, USA · SKLP, ICT, CAS, China · Peking University, China

  21. Parallelization Strategies for Dense LLM Deployment: Navigating Through Application-Specific Tradeoffs and Bottlenecks

    Mar 5, 2026Burak Topcu, Musa Oguzhan Cim, Poovaiah Palangappa +2Parallel LLM InferenceLLM Inference

    The Pennsylvania State University, State College, Pennsylvania, USA · Advanced Micro Devices, San Jose, California, USA · Advanced Micro Devices, Portland, Oregon, USA

  22. From Static Inference to Dynamic Interaction: A Survey of Streaming Large Language Models

    Mar 4, 2026Junlong Tong, Zilong Wang, YuJie Ren +4LLM Inference

    Shanghai Jiao Tong University · Institute of Digital Twin, Eastern Institute of Technology, Ningbo

  23. A Multi-Dimensional Quality Scoring Framework for Decentralized LLM Inference with Proof of Quality

    Mar 4, 2026Arther Tian, Alex Ding, Frank Chen +2Language Model Generation EvaluationLLM Evaluation

    DGrid AI

  24. SEALing the Gap: A Reference Framework for LLM Inference Carbon Estimation via Multi-Benchmark Driven Embodiment

    Mar 3, 2026Priyavanshi Pathania, Rohit Mehra, Vibhu Saujanya Sharma +4LLM InferenceEnergy-Efficient ML

    Accenture Labs, India · Accenture, USA · Accenture, India

  25. Eliciting Numerical Predictive Distributions of LLMs Without Autoregression

    Mar 3, 2026Julianna Piskorz, Katarzyna Kobalczyk, Mihaela van der SchaarLLM Uncertainty EstimationLLM Inference

    Department of Applied Mathematics and Theoretical Physics, University of Cambridge

  26. Towards Privacy-Preserving LLM Inference via Covariant Obfuscation (Technical Report)

    Mar 2, 2026Yu Lin, Qizhi Zhang, Wenqiang Ruan +6Privacy-Preserving Language ModelsLLM Inference

    ByteDance · Nanjing University

  27. Suffix-Constrained Greedy Search Algorithms for Causal Language Models

    Mar 1, 2026Ayoub Hammal, Pierre Zweigenbaum, Caio CorroConstrained Generative ModelingConstrained Decoding

    1Université Paris-Saclay, CNRS, LISN · 2INSA Rennes, IRISA, CNRS, Université de Rennes

  28. Architecture-Aware LLM Inference Optimization on AMD Instinct GPUs: A Comprehensive Benchmark and Deployment Study

    Feb 27, 2026Athos GeorgiouLLM Inference EfficiencyLLM Inference

  29. PREBA: Surgical Duration Prediction via PCA-Weighted Retrieval-Augmented LLMs and Bayesian Averaging Aggregation

    Feb 27, 2026Wanyin Wu, Kanxue Li, Baosheng Yu +4Bayesian Model AveragingClinical Prediction

    Wanyin Wu, Haoyun Zhao, and Dapeng Tao are with the School of Information Science and Engineering, Yunnan University, Kunming 650500, China. · Kanxue Li and Yibing Zhan are with the School of Computer Science, Wuhan University, Wuhan 430072, China. · Baosheng Yu is with the Lee Kong Chian School of Medicine, Nanyang Technological University, Singapore 639798. +1

  30. SLA-Aware Distributed LLM Inference Across Device-RAN-Cloud

    Feb 27, 2026Hariz Yet, Nguyen Thanh Tam, Mao V. Ngo +5LLM Inference ServingLLM Inference

    Singapore University of Technology and Design (SUTD)