LLM Inference

LLM: Large Language Model

Large language model inference is the process of turning prompts into outputs, and its speed, cost, reliability, and privacy shape what applications can use models in practice. Systems work optimizes prefill and token-by-token decoding through KV-cache management, batching and scheduling, parallelism, quantization, and specialized GPU, mobile, or edge kernels. Another strand changes generation at test time using speculative decoding, early exit, retrieval, or repeated reasoning paths to trade extra computation for better accuracy or confidence. The central challenge is that these trade-offs depend heavily on workload, hardware, and inference backend, making reproducible benchmarks and methods that preserve quality while reducing latency, memory, and energy ongoing priorities.

Latest papers

285 papers44 papers in the last four weeks, up 63% on the four weeks before. 0.3% of all new papers.

Empirical Characterization of Inference-Time Elicited Probability Transformations in Large Language Models
Feb 26, 2026Mike Farmer, Abhinav Kochar, Yugyung LeeLLM Inference Belief Updating in LLMs
Bloch School of Management, Regnier Institute for Entrepreneurship & Innovation, University of Missouri–Kansas City. · Department of Computer Science, School of Science and Engineering, University of Missouri–Kansas City.
Sustainable LLM Inference using Context-Aware Model Switching
Feb 25, 2026Yuvarani, Akashdeep Singh, Zahra Fathanah +3LLM Inference Efficiency LLM Inference
Faculty of Computer Science and Information Technology, Universiti Malaya, 50603 Kuala Lumpur, Malaysia
RPU -- A Reasoning Processing Unit
Feb 20, 2026Matthew Adiletta, Gu-Yeon Wei, David BrooksAI Accelerator Inference LLM Inference Efficiency
Harvard University
Privacy-Preserving Mechanisms Enable Cheap Verifiable Inference of LLMs
Feb 19, 2026Arka Pal, Louai Zahran, William Gvozdjak +2Verifiable Outsourced Computation LLM Inference
Ritual · MIT · Columbia University
TensorCommitments: A Lightweight Verifiable Inference for Large Language Models
Feb 13, 2026Oguzhan Baser, Elahe Sadeghi, Eric Wang +5LLM Inference LLM Security
Electrical and Computer Engineering, The University of Texas at Austin · Theseus AI Labs · Electrical and Computer Engineering, McGill University +1
The Hidden Cost of Structured Generation in LLMs: Draft-Conditioned Constrained Decoding
Feb 8, 2026Avinash Reddy, Thayne T. Walker, James S. Ide +1Constrained Generative Modeling Constrained Decoding
Department of Computer Science, University of Central Florida, FL, US · Lockheed Martin AI Center, CT, US
Test-Time Compute Games
Jan 29, 2026Ander Artola Velasco, Dimitrios Rontogiannis, Stratis Tsirtsis +1LLM Serving Cost-Aware Inference
Max Planck Institute for Software Systems, Kaiserslautern, Germany · Hasso Plattner Institute, Potsdam, Germany
Large Language Models Naively Recover Ethnicity from Individual Records
Jan 29, 2026Noah DasanaikeLLM Inference Demographic Bias in Language Models
PhD Candidate, Department of Government, Harvard University
Scalable Synthesis of distributed LLM workloads through Symbolic Tensor Graphs
Nov 13, 2025Changhai Man, Joongun Park, Hanjiang Wu +3LLM Inference Distributed Training
Georgia Institute of Technology · NVIDIA Inc.
Best-of-Majority: Minimax-Optimal Strategy for Pass@ $k$ Inference Scaling
Oct 3, 2025Qiwei Di, Kaixuan Ji, Xuheng Li +2LLM Inference
Department of Computer Science, University of California, Los Angeles, CA 90095, USA
On the Effect of Uncertainty on Layer-wise Inference Dynamics
Jul 9, 2025Sunwoo Kim, Haneul Yoo, Alice OhLLM Inference LLM Interpretability
School of Computing, KAIST, Daejeon, South Korea.
Prima.cpp: Fast 30-70B LLM Inference on Heterogeneous and Low-Resource Home Clusters
Apr 7, 2025Zonghang Li, Tao Li, Wenjiao Feng +8On-Device Language Model Inference LLM Inference
MBZUAI · UESTC · City University of Hong Kong +1
Foundations of Large Language Models
Jan 16, 2025Tong Xiao, Jingbo ZhuLanguage Model Pretraining LLM Alignment
NLP Lab, Northeastern University & NiuTrans Research
Saving GPU Hours in LLM Inference System Development and Online Workloads with Simulation and DBMS-Inspired Cache Replacement Policies
Nov 12, 2024Kyoungmin Kim, Jiacheng Li, Kijae Hong +3LLM Inference Efficiency LLM Inference
EPFL, Switzerland · NUS, Singapore · POSTECH, South Korea
Few-Shot Fairness: Unveiling LLM's Potential for Fairness-Aware Classification
Feb 28, 2024Garima Chhikara, Bhaskara Hanuma Vedula, Anurag Sharma +3Algorithmic Fairness Tabular Classification
Indian Institute of Technology Delhi · Delhi Technological University · Indian Institute of Science Education and Research Kolkata, Mohanpur, India

Feb 26, 2026J/K move · Enter open · S save

Empirical Characterization of Inference-Time Elicited Probability Transformations in Large Language Models

Mike Farmer, Abhinav Kochar, Yugyung Lee

Bloch School of Management, Regnier Institute for Entrepreneurship & Innovation, University of Missouri–Kansas City. · Department of Computer Science, School of Science and Engineering, University of Missouri–Kansas City.

LLM Inference Belief Updating in LLMs Confidence Estimation in Language Models Language Model Calibration

Open paper AskPDF arXiv

Large language models increasingly rely on inference-time procedures such as chain-of-thought reasoning, self-refinement, retrieval augmentation, and verifier-guided revision, yet the structure of elicited probability transformations under these procedures remains poorly understood. We study externally elicited probability assignments over candidate answers and observe recurring approximate log-ratio relationships:

\log \tilde q_t(i) = α_t \left( \log q_t(i) + \log b_t(i) \right) + c_t,

where

q_t

and

\tilde q_t

are pre- and post-elicitation probabilities,

b_t

is an externally constructed evidence signal, and

α_t

is an empirical descriptor of the prompting configuration. Across 4,975 reasoning problems from GPQA Diamond, TheoremQA, MMLU-Pro, and ARC-Challenge, evaluated on multiple instruction-tuned model families, we observe approximate log-ratio relationships with mean

R^2 \approx 0.76

over about

1.3 \times 10^5

candidate-level observations. Coefficients vary across elicitation settings, but qualitatively similar relationships persist across evaluated conditions. Robustness analyses using alternative statistical representations, prompting configurations, held-out evaluation, and token-level log-probabilities suggest that the observed structure is not tied to one prompting procedure or probability estimation method. The main contribution is not the algebraic form itself, which is related to generalized Bayesian updating and probability-transformation frameworks, but the empirical observation that diverse inference-time prompting pipelines repeatedly exhibit reproducible log-ratio structure under controlled conditions. The framework provides a protocol-sensitive perspective for analyzing calibration, evidence amplification, uncertainty propagation, and interaction sensitivity in inference-time LLM pipelines.