Parallel LLM Inference

LLM: Large Language Model

Follow this topic Ask about this topic

Latest papers

16 papers2 papers in the last four weeks, against 2 the four weeks before. 0.0% of all new papers.

Scaling LLM Inference via Parallelism: Throughput-Latency Trade-off
Oct 14, 2026Javad Mirzaei, Jeebak MitraParallel LLM Inference LLM Inference
Office of CTO Dell Technologies Inc. Canada
TriFleetRCA: On-Premise LLM Root Cause Analysis for Kubernetes
Sep 20, 2026Rohit Patel, Susil Kumar Mohanty, Jeenal ChaudharyLLM Inference Parallel LLM Inference
Department of Computer Science and Engineering Indian Institute of Technology Jodhpur, Jodhpur, India
Sample Count Is Not Enough: Candidate-Generation Strategy Shapes the Energy and Performance of LLM Test-Time Scaling
Sep 16, 2026Mobina Kashaniyan, Ali JannesariLLM Inference Efficiency LLM Inference
Department of Computer Science Iowa State University, USA
Self-Orchestrating Language Models: Leveraging Semantic Dependence for Efficient Inference
Sep 13, 2026Tian JinAutoregressive Decoding Efficient Language Model Reasoning
Department of Electrical Engineering and Computer Science, MASSACHUSETTS INSTITUTE OF TECHNOLOGY
NeuroPrefetcher: Storage-Aware Sparse LLM Inference via Delta Prefetching
Aug 23, 2026Nobel Dhar, Md Romyull Islam, Xuechen Zhang +4Parallel LLM Inference LLM Inference
Kennesaw State University Marietta, USA · Memory Solutions Lab, Samsung Semiconductor, Inc. San Jose, California, USA
Celty: SpMspV GPU Kernel and SIMT Co-Design for Efficient Dual-Sparse LLM Inference
Aug 2, 2026Ruokai Yin, Priyadarshini PandaGPU Kernel Optimization LLM Inference
Yale University New Haven, USA · University of Southern California Los Angeles, USA
When Does Delegation Beat Majority? A Delegation-Based Aggregator for Multi-Sample LLM Inference
Jun 6, 2026Yasushi Sakai, Allen Song, Kent LarsonLLM Inference Parallel LLM Inference
1MIT Media Lab, Cambridge, MA
Speculative Decoding Across Languages
May 28, 2026Nirajan Paudel, Michael Ginn, Luc De Nardi +1Multilingual Language Models Speculative Decoding
University of Colorado
Memory-Bound but Not Bandwidth-Limited: The Physical AI Inference Gap in Batch-1 LLM Decode
May 28, 2026Josef ChenGPU Kernel Optimization GPU Acceleration
KAIKAKU
BlockBatch: Multi-Scale Consensus Decoding for Efficient Diffusion Language Model Inference
May 28, 2026Xiaoyou Wu, Cheng-Jhih Shih, Binfei Ji +2Blockwise Diffusion Language Models Diffusion Language Model Inference
Georgia Institute of Technology
PSD: Pushing the Pareto Frontier of Diffusion LLMs via Parallel Speculative Decoding
May 15, 2026Shengyin Sun, Yiming Li, Renxi Liu +7Parallel Speculative Decoding Masked Diffusion Models
Department of Computer Science City University of Hong Kong · Huawei Technologies · City University of Hong Kong
DeInfer: Efficient Parallel Inferencing for Decomposed Large Language Models
Apr 20, 2026You-Liang Huang, Xinhao Huang, Chengxi Liao +1LLM Inference Parallel LLM Inference
Boston University · Boston, Massachusetts, USA · HKUST (Guangzhou) +2
Characterizing Performance-Energy Trade-offs of Large Language Models in Multi-Request Workflows
Mar 12, 2026Md. Monzurul Amin Ifath, Israat HaqueParallel LLM Inference LLM Inference
Dalhousie University, Halifax, NS, Canada
Reject, Resample, Repeat: Understanding Parallel Reasoning in Language Model Inference
Mar 9, 2026Noah Golowich, Fan Chen, Dhruv Rohatgi +4Parallel Reasoning in Language Models LLM Inference
Microsoft Research · MIT · NYU
Parallelization Strategies for Dense LLM Deployment: Navigating Through Application-Specific Tradeoffs and Bottlenecks
Mar 5, 2026Burak Topcu, Musa Oguzhan Cim, Poovaiah Palangappa +2Parallel LLM Inference LLM Inference
The Pennsylvania State University, State College, Pennsylvania, USA · Advanced Micro Devices, San Jose, California, USA · Advanced Micro Devices, Portland, Oregon, USA
MineDraft: A Framework for Batch Parallel Speculative Decoding
Feb 24, 2026Zhenwei Tang, Arun Verma, Zijian Zhou +4Speculative Decoding LLM Inference Acceleration
College of Computing and Data Science, Nanyang Technological University, Republic of Singapore · Singapore-MIT Alliance for Research and Technology Centre · Department of Computer Science, National University of Singapore, Republic of Singapore +1

Oct 14, 2026J/K move · Enter open · S save

Scaling LLM Inference via Parallelism: Throughput-Latency Trade-off

Javad Mirzaei, Jeebak Mitra

Office of CTO Dell Technologies Inc. Canada

Parallel LLM Inference LLM Inference LLM Inference Scheduling Tensor Parallelism Pipeline Parallelism Long-Context Language Model Inference

Open paper AskPDF arXiv

Large Language Model (LLM) serving systems must balance latency and throughput while efficiently utilizing distributed GPU resources. This challenge arises because inference consists of two fundamentally different phases: prefill, which determines time-to-first-token (TTFT), and decode, which determines inter-token latency (ITL). The use of tensor parallelism (TP), pipeline parallelism (PP), and hybrid configurations exposes different trade-offs between these latency dimensions, making the optimal parallelism configuration highly dependent on workload characteristics, latency requirements, and context length. Moreover, despite the widespread adoption of distributed inference, there is still limited understanding of how parallelism configurations should be selected under varying workload conditions and service-level objectives (SLO). In this paper, we develop a systematic framework for characterizing LLM inference parallelism via throughput-latency trade-offs, SLO-aware serving, and long-context scalability. We introduce the notion of parallelism operating regions by constructing TTFT-ITL feasible sets and Pareto frontiers across different workload compositions, thereby revealing a systematic transition from TP-dominant to PP-dominant configurations as workloads become increasingly prefill-heavy. We further formulate SLO-aware parallelism selection using goodput, showing that stringent TTFT requirements favor PP-dominated configurations, whereas relaxed latency constraints progressively shift the optimality toward TP-dominated configurations with higher serving capacity. Finally, we demonstrate that increasing context length fundamentally changes the relative advantages of parallelism configurations, with PP-dominant configurations exhibiting more favorable scalability and SLO attainment in the long-context regime.