Tensor Parallelism

Follow this topic Ask about this topic

Tensor parallelism divides operations such as attention and feed-forward layers across GPUs, allowing them to share the work and model memory during training or inference. Devices exchange intermediate results through collectives such as all-reduce, all-gather, and reduce-scatter, so communication overhead can limit the gains. Current approaches optimize sharding layouts and communication, overlap transfers with computation, or compress exchanged tensors, often alongside pipeline or sequence parallelism. The central tradeoff is between lower latency and greater model capacity on one hand, and communication cost, hardware topology, and workload-specific throughput needs on the other.

Latest papers

20 papers4 papers in the last four weeks, against 2 the four weeks before. 0.0% of all new papers.

Scaling LLM Inference via Parallelism: Throughput-Latency Trade-off
Oct 14, 2026Javad Mirzaei, Jeebak MitraParallel LLM Inference LLM Inference
Office of CTO Dell Technologies Inc. Canada
Echo: Error-Compensated 4-Bit Tensor-Parallel Communication for LLM Training and Inference
Oct 9, 2026Xingyu Xie, Xiaoyong Xie, Zhouchen Lin +1Error Feedback Communication-Efficient Distributed Training
Department of Mathematics, National University of Singapore · State Key Laboratory of General Artificial Intelligence, School of Intelligence Science and Technology, Peking University
Characterizing Parallelism Strategies in LLM Inference: Fundamental Compute-Communication Trade-offs
Oct 4, 2026Javad Mirzaei, Jeebak MitraLLM Inference Tensor Parallelism
Office of CTO Dell Technologies Inc. Canada
SPLASH: Switching Parallel Layouts of Attention with Seamless Handoff for LLM Serving
Sep 29, 2026Chuan Liu, Shuoming Zhang, Zhicheng Li +6LLM Serving Tensor Parallelism
State Key Laboratory of Processors, Institute of Computing Technology, Chinese Academy of Sciences, Beijing, China · University of Chinese Academy of Sciences, Beijing, China
vidax: A Unified JAX Framework for Video Generative Models on Accelerator Meshes
Sep 16, 2026Congyue DengVideo Diffusion Models Tensor Parallelism
Massachusetts Institute of Technology
Affinity-Aware Sharding for Delayed Tensor Parallelism
Sep 12, 2026Eloi de ReynalTensor Parallelism Distributed Training
Kog Team
More GPUs or a Smaller Cache? Tensor Parallelism versus KV Compression for Memory-Bound LLM Serving
Aug 25, 2026Srikanta Datta Tumkur, Mehar Simhadri, Anshu Bansal +5LLM Serving LLM Inference Efficiency
MIT and Vizuara
FleetSieve: Decision-Critical Profiling for SLO-Aware LLM Fleet Configuration
Aug 20, 2026Huang Cheng, Scott Zhang, Aubert LiLLM Serving LLM Inference Scheduling
Meta, Menlo Park, California, USA
TERRA: A Hierarchical Parallel Training and Memory Orchestration Framework for High-Resolution AI-based Earth Modeling
Aug 15, 2026Ruohan Wu, Ziqi Zhu, Yang Zhao +6Distributed Training Memory-Efficient Fine-Tuning
School of Computer Science and Technology, University of Science and Technology of China, Hefei 230026, China · School of Artificial Intelligence and Data Science, University of Science and Technology of China, Hefei 230026, China · Department of Ocean Big Data and Artificial Intelligence, Laoshan Laboratory, Qingdao, China +1
SwiftQK: Fast and Communication-Efficient Tensor Parallelism for Query-Key Normalization
Aug 10, 2026Gyudong Kim, Wonjun Han, Young Geun KimGPU Kernel Optimization Tensor Parallelism
Department of Computer Science and Engineering, Korea University, Seoul 02855, Republic of Korea
Scaling Neural Network Verification with Tensor Parallelism and Fully Sharded Data Parallelism
Jun 8, 2026Sergei Vorobyov, Eugene IlyushinNeural Network Verification High-Performance Computing
Lomonosov Moscow State University · Central University
Accelerating Long-Tail Generation in Synchronous RLHF Training via Adaptive Tensor Parallelism
May 3, 2026Long Zhao, Qinghe Wang, Jiaan Zhu +5Tensor Parallelism LLM Inference Acceleration
Anhui University · Institute of Artificial Intelligence, Hefei Comprehensive National Science Center · University of Science and Technology of China +2
Folding Tensor and Sequence Parallelism for Memory-Efficient Transformer Training & Inference
Apr 29, 2026Vasu Shyam, Anna Golubeva, Quentin AnthonyTransformer Inference Transformer
Zyphra
GLM-5 Serving Parameter Tuning for OpenClaw: Single-Deployment MaaS Inference Optimization for Long-Context Agent Workloads
Apr 27, 2026Minjie Hua, Ning Wang, Peijun Yang +2LLM Inference Efficiency LLM Serving
China Unicom
TACO: Efficient Communication Compression of Intermediate Tensors for Scalable Tensor-Parallel LLM Training
Apr 27, 2026Man Liu, Xingchen Liu, Xingjian Tian +8Communication-Efficient Distributed Training Tensor Parallelism
Hangzhou Institute for Advanced Study, University of Chinese Academy of Sciences · Institute of Computing Technology, Chinese Academy of Sciences · University of Science and Technology of China
CommFuse: Hiding Tail Latency via Communication Decomposition and Fusion for Distributed LLM Training
Apr 27, 2026Rezaul Karim, Austin Wen, Wang Zongzuo +3Communication-Efficient Distributed Training Distributed Optimization
Toronto Ascend Team
AEGIS: Scaling Long-Sequence Homomorphic Encrypted Transformer Inference via Hybrid Parallelism on Multi-GPU Systems
Apr 3, 2026Zhaoting Gong, Ran Ran, Fan Yao +1Efficient Transformer Inference Homomorphic Encryption
North Carolina State University, Raleigh, NC, USA · University of Central Florida, Orlando, FL, USA
Parallelization Strategies for Dense LLM Deployment: Navigating Through Application-Specific Tradeoffs and Bottlenecks
Mar 5, 2026Burak Topcu, Musa Oguzhan Cim, Poovaiah Palangappa +2Parallel LLM Inference LLM Inference
The Pennsylvania State University, State College, Pennsylvania, USA · Advanced Micro Devices, San Jose, California, USA · Advanced Micro Devices, Portland, Oregon, USA
Scaling State-Space Models on Multiple GPUs with Tensor Parallelism
Feb 24, 2026Anurag Dutt, Nimit Shah, Hazem Masarani +1LLM Inference Acceleration Tensor Parallelism
Stony Brook University
Systematic Exploration of Multi-core Architectures for Efficient LLM Serving using WaferAI-SIM
Oct 7, 2025Tianhao Zhu, Dahu Feng, Erhu Feng +1AI Accelerator Inference LLM Serving
Institute of Parallel and Distributed Systems, Shanghai Jiao Tong University · Department of Precision Instrument, Tsinghua University

Oct 14, 2026J/K move · Enter open · S save

Scaling LLM Inference via Parallelism: Throughput-Latency Trade-off

Javad Mirzaei, Jeebak Mitra

Office of CTO Dell Technologies Inc. Canada

Parallel LLM Inference LLM Inference LLM Inference Scheduling Tensor Parallelism Pipeline Parallelism Long-Context Language Model Inference

Open paper AskPDF arXiv

Large Language Model (LLM) serving systems must balance latency and throughput while efficiently utilizing distributed GPU resources. This challenge arises because inference consists of two fundamentally different phases: prefill, which determines time-to-first-token (TTFT), and decode, which determines inter-token latency (ITL). The use of tensor parallelism (TP), pipeline parallelism (PP), and hybrid configurations exposes different trade-offs between these latency dimensions, making the optimal parallelism configuration highly dependent on workload characteristics, latency requirements, and context length. Moreover, despite the widespread adoption of distributed inference, there is still limited understanding of how parallelism configurations should be selected under varying workload conditions and service-level objectives (SLO). In this paper, we develop a systematic framework for characterizing LLM inference parallelism via throughput-latency trade-offs, SLO-aware serving, and long-context scalability. We introduce the notion of parallelism operating regions by constructing TTFT-ITL feasible sets and Pareto frontiers across different workload compositions, thereby revealing a systematic transition from TP-dominant to PP-dominant configurations as workloads become increasingly prefill-heavy. We further formulate SLO-aware parallelism selection using goodput, showing that stringent TTFT requirements favor PP-dominated configurations, whereas relaxed latency constraints progressively shift the optimality toward TP-dominated configurations with higher serving capacity. Finally, we demonstrate that increasing context length fundamentally changes the relative advantages of parallelism configurations, with PP-dominant configurations exhibiting more favorable scalability and SLO attainment in the long-context regime.