LLM Inference

LLM: Large Language Model

Follow this topic Ask about this topic

Momentum

30 papers in the last four weeks, up 233% on the four weeks before. 0.3% of all new papers.

Jul 13Week of Sep 28

Latest papers 187

All topics

Prefill/Decode-Aware Evaluation of LLM Inference on Emerging AI Accelerators
Jun 14, 2026Shun Usami, Venkatram Vishwanath, E. Wes BethelAI Accelerator Inference LLM Inference
Department of Computer Science San Francisco State University San Francisco, CA, 94132 · Argonne National Laboratory Lemont, IL, 60439 · Lawrence Berkeley National Laboratory Berkeley, CA, 94720
STREAM: Multi-Tier LLM Inference Middleware with Dual-Channel HPC Token Streaming
Jun 11, 2026Anas Nassar, Steve Mohr, Leonard Apanasevich +1LLM Serving LLM Inference
Advanced Cyberinfrastructure for Education and Research (ACER) University of Illinois Chicago Chicago, Illinois, USA
Real-Time Language Model Jamming: A Case Study for Live Music Accompaniment Generation
Jun 10, 2026Bowen Zheng, Andrew H. Yang, Jiaqi Ruan +5LLM Inference
University of Wisconsin–Madison, USA · University of California, San Diego, USA · University of Science and Technology of China, China +3
TileFuse: A Fused Mixed-Precision Kernel Library for Efficient Quantized LLM Inference on AMD NPUs
Jun 9, 2026Wesley Pang, Gregory Hyegang Jun, Feiyang Liu +1Mixed-Precision Quantization LLM Quantization
University of Illinois Urbana-Champaign Department of Electrical and Computer Engineering USA
Dep-LLM: Training-Free Depression Diagnosis via Evidence-Guided Structured Multi-factor with Reliable LLM Reasoning
Jun 9, 2026Yiqing Lyu, Xianbing Zhao, Buzhou Tang +1LLM Inference Depression Detection
Steganography Without Modification: Hidden Communication via LLM Seeds
Jun 8, 2026Felix Mächtle, Jonas Sander, Sebastian Berndt +3Language Model Decoding LLM Inference
Institute for IT Security, University of Lübeck, Lübeck, Germany · Technische Hochschule Lübeck, Lübeck, Germany
Inference-Time Conformal Reasoning with Valid Factuality Control for Large Language Models
Jun 7, 2026Ting Wang, Yuanjie Shi, Yan Yan +1LLM Inference LLM Reasoning with Graphs
University of Illinois Urbana-Champaign · School of EECS, Washington State University.
Cross-LLM Consistency in Inference: Evidence from Shared Interactions
Jun 6, 2026Siyu Lou, Yao Yan, Yuntian Chen +1LLM Inference LLM Interpretability
School of Computer Science Shanghai Jiao Tong University Ningbo Key Laboratory of Advanced Manufacturing Simulation Eastern Institute of Technology, Ningbo · College of Computer and Information Science Chongqing Normal University · Ningbo Key Laboratory of Advanced Manufacturing Simulation Eastern Institute of Technology, Ningbo
When Does Delegation Beat Majority? A Delegation-Based Aggregator for Multi-Sample LLM Inference
Jun 6, 2026Yasushi Sakai, Allen Song, Kent LarsonLLM Inference Language Model Ensembles
1MIT Media Lab, Cambridge, MA
Breaking the Ice: Analyzing Cold Start Latency in vLLM
Jun 5, 2026Huzaifa Shaaban Kabakibo, Animesh Trivedi, Lin WangLLM Inference LLM Inference Serving
Paderborn University, Paderborn, Germany · IBM Research Europe, Zurich, Switzerland
Skip a Layer or Loop It? Learning Program-of-Layers in LLMs
Jun 4, 2026Ziyue Li, Yang Li, Tianyi ZhouLLM Inference LLM Inference Acceleration
University of Maryland, College Park, MD, USA · 2MBZUAI, Abu Dhabi, UAE.
Closing the Loop on Latent Reasoning via Test-Time Reconstruction
Jun 4, 2026Xiaopeng Yuan, Haibo Jin, Ye Yu +4LLM Inference Test-Time Training
University of Illinois Urbana-Champaign · Google · Florida State University
Generative Criticality in Large Language Model Temperature Scaling
Jun 4, 2026Huajian Ruan, Jinyang Li, Xingyu Guo +1LLM Inference Phase Transitions
State Key Laboratory of Nuclear Physics and Technology, Institute of Quantum Matter, South China Normal University, Guangzhou 510006, China · Key Laboratory of Atomic and Subatomic Structure and Quantum Control (MOE), Guangdong-Hong Kong Joint Laboratory of Quantum Matter, Guangzhou 510006, China · Guangdong Basic Research Center of Excellence for Structure and Fundamental Interactions of Matter, Guangdong Provincial Key Laboratory of Nuclear Science, Guangzhou 510006, China +4
IR3DE: A Linear Router for Large Language Models
Jun 4, 2026Eros Fanì, Oğuzhan ErsoyLLM Inference LLM Routing
Gensyn
MolE-RAG: Molecular Structure-Enhanced Retrieval-Augmented Generation for Chemistry
Jun 4, 2026Joey Chan, Wonbin Kweon, Ashley Shin +4Retrieval-Augmented Generation LLM Inference
University of Illinois Urbana-Champaign · University of California, San Diego
SharedRequest: Privacy-Preserving Model-Agnostic Inference for Large Language Models
Jun 3, 2026Peihua Mai, Xuanrong Gao, Youlong Ding +3Privacy-Preserving Language Models LLM Inference
1National University of Singapore (Chongqing) Research Institute · 2Chongqing Key Laboratory of Trusted Perception and Interaction Technology for Intelligent and Connected Vehicles, Chongqing, China · 3National University of Singapore +3
NetKV: Network-Aware Decode Instance Selection for Disaggregated LLM Inference
Jun 2, 2026Mubarak Adetunji OjewaleLLM Inference KV Caching
Cloud Competency Centre, National College of Ireland
Building Reliable Long-Form Generation via Hallucination Rejection Sampling
Jun 2, 2026Lin Li, Georgia Channing, Suhaas M Bhat +2Long-Form Document Generation LLM Hallucination Mitigation
DriftSched: Adaptive QoS-Aware Scheduling under Runtime Token Drift for Multi-Tenant GPU Inference
Jun 2, 2026Kathiravan PalaniappanLLM Inference LLM Inference Serving
Independent Researcher (University of Colorado Colorado Springs Alumni)
Multi-Segment Attention: Enabling Efficient KV-Cache Management for Faster Large Language Model Serving
Jun 1, 2026Chunan Shi, Yilei Chen, Yilin Chen +2KV-Cache Offloading LLM Inference
Peking University Beijing, China
Not All Errors Are Equal: A Systematic Study of Error Propagation in Large Language Model Inference
Jun 1, 2026Yafan Huang, Sheng Di, Guanpeng LiLLM Inference Language Model Robustness
University of Iowa · Argonne National Laboratory · University of Florida
Off-the-Shelf LLMs as Process Scorers: Training-Free Alternative to PRMs for Mathematical Reasoning
Jun 1, 2026Atoosa Chegini, Soheil FeiziNumerical Reasoning in Language Models Inference-Time Search
Department of Computer Science, University of Maryland
Move the Query, Not the Cache: Characterizing Cross-Instance Latent Attention Redistribution Across GPU Fabrics
May 31, 2026Bole Ma, Jan Eitzinger, Harald Köstler +1LLM Inference KV Caching
Benchmarking Local LLMs for Natural-Language-to-SQL Querying in Biopharmaceutical Manufacturing: An Empirical Benchmark on Consumer-Grade Hardware
May 31, 2026Sagar Bhetwal, Rajan Bastakoti, Nirajan Acharya +2LLM Evaluation LLM Inference
Department of Computer Science, University of the Cumberlands, Kentucky, United States · Department of Computer Science, DePaul University, Chicago, IL, United States · Youngstown State University, Youngstown, Ohio, United States +1
ProactiveLLM: Learning Active Interaction for Streaming Large Language Models
May 30, 2026Junlong Tong, Yao Zhang, Anhao Zhao +3LLM Inference LLM Inference Acceleration
Eastern Institute of Technology, Ningbo · Shanghai Jiao Tong University · Hong Kong Polytechnic University +1
Threshold-Based Exclusive Batching for LLM Inference
May 30, 2026Weifang Zhang, Yuzhou Nie, Bowen Pang +2GPU Acceleration LLM Inference
The Hong Kong Polytechnic University · UC Santa Barbara · Technology and Engineering Center for Space Utilization, Chinese Academy of Sciences +1
Model-Native Computing Architecture: Envisioning Future System Architecture Through the Lens of Computer Architecture
May 29, 2026Hai Lin, Hoilam Pao, Shaoxiong Zhan +1LLM Inference Persistent Memory for Language Models
Shenzhen International Graduate School, Tsinghua University · Pengcheng Laboratory, Shenzhen, China
Bit-Exact AI Inference Verification Without Performance Tradeoffs
May 29, 2026Naci CankayaNeural Network Verification LLM Inference
Machine Intelligence Research Institute
CoMem: Context Management with A Decoupled Long-Context Model
May 29, 2026Yuwei Zhang, Chengyu Dong, Shuowei Jin +11LLM Inference LLM Inference Acceleration
Halıcıoğlu Data Science Institute, University of California, San Diego · Amazon
Memory-Bound but Not Bandwidth-Limited: The Physical AI Inference Gap in Batch-1 LLM Decode
May 28, 2026Josef ChenGPU Kernel Optimization GPU Acceleration
KAIKAKU

Jun 14, 2026, cs.ARJ/K move · Enter open · S save

Prefill/Decode-Aware Evaluation of LLM Inference on Emerging AI Accelerators

Shun Usami, Venkatram Vishwanath, E. Wes Bethel

Department of Computer Science San Francisco State University San Francisco, CA, 94132 · Argonne National Laboratory Lemont, IL, 60439 · Lawrence Berkeley National Laboratory Berkeley, CA, 94720

AI Accelerator Inference LLM Inference LLM Inference Serving

Open paper AskPDF arXiv

As large language models (LLMs) are increasingly deployed in latency- and cost-sensitive settings, inference efficiency has become a central systems challenge. While GPUs dominate current deployments, a growing number of AI accelerators claim advantages for LLM inference, yet it remains unclear under which conditions such accelerators outperform GPUs in practice. Recent inference systems decompose execution into Prefill and Decode phases, which exhibit distinct computational characteristics and latency metrics, commonly captured by time to first token (TTFT) and time per output token (TPOT). This paper presents a phase-aware evaluation of LLM inference performance across GPUs and emerging AI accelerators using a common model, Llama2-7B. By separately measuring Prefill and Decode performance, we reveal that accelerator advantages differ by phase and metric. Our results show that GPUs consistently excel in the compute-intensive Prefill phase, while GroqRack achieves significantly lower TPOT during Decode (batching not currently supported). However, GPUs regain an advantage in Decode throughput as batch size increases. These findings demonstrate that each platform exhibits distinct phase-dependent strengths. We further analyze heterogeneous Prefill/Decode disaggregation across different accelerator platforms, identifying performance gains and the workload and network conditions under which such gains are realized.

Momentum

Related topics

Latest papers 187