LLM Inference

LLM: Large Language Model

Follow this topic Ask about this topic

Momentum

30 papers in the last four weeks, up 233% on the four weeks before. 0.3% of all new papers.

Jul 13Week of Sep 28

Latest papers 187

All topics

Towards Distributed Inference of LLMs on a P2P Network
May 7, 2026Shabari S Nair, Krishanu SainiLLM Inference KV Caching
Department of Computer Science, The University of Texas at Austin.
Quantum-enhanced Large Language Models on Quantum Hardware via Cayley Unitary Adapters
May 7, 2026Borja Aizpurua, Sukhbinder Singh, Augustine Kshetrimayum +2LLM Inference LLM Fine-Tuning
Multiverse Computing, Parque Científico y Tecnológico de Gipuzkoa, Paseo May 7 de Miramón 170, 20014, Donostia / San Sebastián, Spain · Department of Basic Sciences, Tecnun – University of Navarra, San Sebastián, Spain · Multiverse Computing, Centre for Social Innovation, 192 Spadina Avenue, ON M5T 2C2, Toronto, Canada +2
LLMSpace: Carbon Footprint Modeling for Large Language Model Inference on LEO Satellites
May 7, 2026Lei Jiang, Adrian Ildefonso, Daniel Loveless +1Energy-Efficient ML LLM Inference
Indiana University
BitCal-TTS: Bit-Calibrated Test-Time Scaling for Quantized Reasoning Models
May 7, 2026Sai Babu Patarlapalli, Surya Teja AvvaruLLM Inference 4-Bit Quantization
NoisyCoconut: Counterfactual Consensus via Latent Space Reasoning
May 6, 2026Michael Jerge, David EvansSelective Prediction LLM Inference
University of Virginia
A Queueing-Theoretic Framework for Stability Analysis of LLM Inference with KV Cache Memory Constraints
May 6, 2026Chengyi Nie, Nian Si, Zijie ZhouLLM Serving LLM Inference
Department of Electrical and Computer Engineering, Stony Brook University · Department of Industrial Engineering and Decision Analytics, HKUST
Tile-Level Activation Overlap for Efficient LLM Inference
May 5, 2026Abhinav Jangda, Tyler Sorensen, Sebastian Burckhardt +3LLM Inference Efficiency GPU Kernel Optimization
Microsoft Research · Microsoft
Two Calls, Two Moments, and the Vote-Accuracy Curve of Repeated LLM Inference
May 5, 2026Yi LiuLLM Inference Test-Time Scaling
York University
Reasoning emerges from constrained inference manifolds in large language models
May 2, 2026Yanbiao Ma, Fei Luo, Linfeng Zhang +10LLM Inference LLM Interpretability
Renmin University of China · Tsinghua University · Xiaomi EV +1
Position: LLM Serving Needs Mathematical Optimization and Algorithmic Foundations, Not Just Heuristics
May 2, 2026Zijie ZhouLLM Serving LLM Inference
Department of Industrial Engineering and Decision Analytics, HKUST
Belief-Guided Inference Control for Large Language Model Services via Verifiable Observations
Apr 30, 2026Wenhao Yuan, Chenchen Lin, Jian Chen +3LLM Inference Cost-Aware Inference
The University of Hong Kong · Sun Yat-sen University
Length Value Model: Scalable Value Pretraining for Token-Level Length Modeling
Apr 29, 2026Zhen Zhang, Changyi Yang, Zijie Xia +11LLM Inference Controllable Text Generation
University of California, Santa Barbara · Carnegie Mellon University · LMSYS Org +2
DUAL-BLADE: Dual-Path NVMe-Direct KV-Cache Offloading for Edge LLM Inference
Apr 29, 2026Bodon Jeong, Hongsu Byun, Youngjae Kim +4On-Device Language Model Inference KV-Cache Offloading
1Sogang University, Seoul, Republic of Korea · 2Florida State University, FL, USA · 3Samsung Electronics Co.
Entropy Centroids as Intrinsic Rewards for Test-Time Scaling
Apr 28, 2026Wenshuo Zhao, Qi Zhu, Xingshan Zeng +4LLM Inference Test-Time Scaling
May · HKUST · ZJU +1
Hybrid JIT-CUDA Graph Optimization for Low-Latency Large Language Model Inference
Apr 25, 2026Divakar Kumar Yadav, Tian ZhaoLLM Inference LLM Inference Acceleration
Department of Computer Science University of Wisconsin-Milwaukee Milwaukee, WI, USA
Cloud to Edge: Benchmarking LLM Inference On Hardware-Accelerated Single-Board Computers
Apr 24, 2026Harri Renney, Fouad Trad, Michael Mattarock +1AI Accelerator Inference On-Device Language Model Inference
Kaze Technologies Kaze Consulting Bath, BA1 2HN, UK · Electrical and Computer Engineering Lebanese American University Byblos, Lebanon · Carnegie Mellon University Pittsburgh Pennsylvania USA +1
Introducing Background Temperature to Characterise Hidden Randomness in Large Language Models
Apr 24, 2026Alberto Messina, Stefano ScottaLLM Inference ML Reproducibility
1RAI - Radiotelevisione Italiana, Centre for Research, Technological Innovation and Experimentation (CRITS)
Latent Phase-Shift Rollback: Inference-Time Error Correction via Residual Stream Monitoring and KV-Cache Steering
Apr 20, 2026Manan Gupta, Dhruv KumarLLM Self-Correction Language Model Error Detection
1BITS Pilani, Pilani Campus, India
Benchmarking System Dynamics AI Assistants: Cloud Versus Local LLMs on CLD Extraction and Discussion
Apr 20, 2026Terry LeitchLLM Quantization Dynamical Systems
Ruxton AI
Unlocking the Edge deployment and ondevice acceleration of multi-LoRA enabled one-for-all foundational LLM
Apr 20, 2026Sravanth Kodavanti, Sowmya Vajrala, Srinivas Miriyala +13On-Device Language Model Inference LLM Inference
Samsung Research Institute Bangalore, India · Samsung Electronics, Suwon, South Korea
PoQ-Judge: A Multi-Architecture Evaluation Framework for Cost-Aware Proof-of-Quality in Decentralized LLM Inference
Apr 20, 2026Arther Tian, Alex Ding, Frank Chen +2LLM Evaluation LLM-as-a-Judge
DGrid AI
DeInfer: Efficient Parallel Inferencing for Decomposed Large Language Models
Apr 20, 2026You-Liang Huang, Xinhao Huang, Chengxi Liao +1LLM Inference LLM Inference Acceleration
Boston University · Boston, Massachusetts, USA · HKUST (Guangzhou) +2
MoE-nD: Per-Layer Mixture-of-Experts Routing for Multi-Axis KV Cache Compression
Apr 20, 2026Libo Sun, Peixiong He, Po-Wei Harn +1LLM Inference KV Caching
Department of Computer Science and Software Engineering, Auburn University, Auburn, AL, USA · Department of Information Management, National Central University, Taoyuan, Taiwan
Configuration Over Selection: Hyperparameter Sensitivity Exceeds Model Differences in Open-Source LLMs for RTL Generation
Apr 18, 2026Minghao Shao, Zeng Wang, Weimin Fu +5LLM Evaluation Language Model Generation Evaluation
NYU Tandon School of Engineering, USA · Kansas State University, USA · NYU Abu Dhabi, UAE
LACE: Lattice Attention for Cross-thread Exploration
Apr 16, 2026Yang Li, Zirui Zhang, Yang Liu +1LLM Inference Attention Mechanisms
Department of Computer Science Rutgers University · Amazon AGI Labs
Scepsy: Serving Agentic Workflows Using Aggregate LLM Pipelines
Apr 16, 2026Otto White, Marcel Wagenländer, Britannio Jarrett +7LLM Inference LLM Inference Scheduling
Imperial College London · Independent Researcher
The Illusion of Equivalence: Systematic FP16 Divergence in KV-Cached Autoregressive Inference
Apr 16, 2026Ranjith Chodavarapu, Lei XuLLM Inference
Department of Computer Science Kent State University, USA
ProbeLogits: Kernel-Level LLM Inference Primitives for AI-Native Operating Systems
Apr 13, 2026Daeyeon SonLLM Inference LLM Guardrails
Independent Researcher Republic of Korea
Suffix-Constrained Greedy Search Algorithms for Causal Language Models
Mar 1, 2026Ayoub Hammal, Pierre Zweigenbaum, Caio CorroConstrained Generative Modeling Constrained Decoding
1Université Paris-Saclay, CNRS, LISN · 2INSA Rennes, IRISA, CNRS, Université de Rennes
TensorCommitments: A Lightweight Verifiable Inference for Language Models
Feb 13, 2026Oguzhan Baser, Elahe Sadeghi, Eric Wang +5LLM Inference LLM Security
Electrical and Computer Engineering, The University of Texas at Austin · Theseus AI Labs · Electrical and Computer Engineering, McGill University +1

May 7, 2026, cs.DCJ/K move · Enter open · S save

Towards Distributed Inference of LLMs on a P2P Network

Shabari S Nair, Krishanu Saini

Department of Computer Science, The University of Texas at Austin.

LLM Inference KV Caching LLM Inference Serving Prefix Caching LLM Routing Distributed Inference

Open paper AskPDF arXiv

Prefix caching can reduce LLM inference latency by reusing KV caches across requests with shared prompts, but cluster-scale reuse is challenging because caches are partitioned across nodes. We propose a decentralized, prefix-cache-aware routing scheme for peer-to-peer LLM serving. Each node maintains a local radix tree of its own cached prefixes and asynchronously refreshed estimates of peer caches using periodic anti-entropy. Requests are routed to the node with the longest estimated prefix match, without centralized coordination or KV-cache transfer. Stale metadata only causes cache misses, not incorrect outputs, making weak consistency sufficient for correctness. Evaluation on simulated MMLU workloads show that decentralized routing improves latency under low communication delay and skewed prefix distributions, while high network latency and affinity-induced hotspots limit its benefits.

Momentum

Related topics

Latest papers 187