Overview

Tensor parallelism divides operations such as attention and feed-forward layers across GPUs, allowing them to share the work and model memory during training or inference. Devices exchange intermediate results through collectives such as all-reduce, all-gather, and reduce-scatter, so communication overhead can limit the gains. Current approaches optimize sharding layouts and communication, overlap transfers with computation, or compress exchanged tensors, often alongside pipeline or sequence parallelism. The central tradeoff is between lower latency and greater model capacity on one hand, and communication cost, hardware topology, and workload-specific throughput needs on the other.

Latest papers

CardsList

22 papers2 papers in the last four weeks, down 50% on the four weeks before. 0.0% of all new papers.

  1. Benchmarking LLM Inference on Desktop-Scale NVIDIA Grace Blackwell Clusters

    Nov 15, 2026Kevin Bhimani, Christian Escobar, Daniel Mihovch +2LLM InferenceParallel LLM Inference

  2. KOSMOS: Self-scheduling Persistent Kernels for Communication-Compute Overlap

    Nov 5, 2026Alexander Magro, Aristotle Martin, Ye Wang +1GPU Kernel OptimizationDistributed Training

  3. Scaling LLM Inference via Parallelism: Throughput-Latency Trade-off

    Oct 14, 2026Javad Mirzaei, Jeebak MitraParallel LLM InferenceLLM Inference

  4. Echo: Error-Compensated 4-Bit Tensor-Parallel Communication for LLM Training and Inference

    Oct 9, 2026Xingyu Xie, Xiaoyong Xie, Zhouchen Lin +1Error FeedbackCommunication-Efficient Distributed Training

  5. SPLASH: Switching Parallel Layouts of Attention with Seamless Handoff for LLM Serving

    Sep 29, 2026Chuan Liu, Shuoming Zhang, Zhicheng Li +6LLM ServingTensor Parallelism

  6. FleetSieve: Decision-Critical Profiling for SLO-Aware LLM Fleet Configuration

    Aug 20, 2026Huang Cheng, Scott Zhang, Aubert LiLLM ServingLLM Inference Scheduling

  7. TERRA: A Hierarchical Parallel Training and Memory Orchestration Framework for High-Resolution AI-based Earth Modeling

    Aug 15, 2026Ruohan Wu, Ziqi Zhu, Yang Zhao +6Distributed TrainingMemory-Efficient Fine-Tuning

  8. SwiftQK: Fast and Communication-Efficient Tensor Parallelism for Query-Key Normalization

    Aug 10, 2026Gyudong Kim, Wonjun Han, Young Geun KimGPU Kernel OptimizationTensor Parallelism

  9. Scaling Neural Network Verification with Tensor Parallelism and Fully Sharded Data Parallelism

    Jun 8, 2026Sergei Vorobyov, Eugene IlyushinNeural Network VerificationHigh-Performance Computing

  10. Accelerating Long-Tail Generation in Synchronous RLHF Training via Adaptive Tensor Parallelism

    May 3, 2026Long Zhao, Qinghe Wang, Jiaan Zhu +5Tensor ParallelismLLM Inference Acceleration

  11. TACO: Efficient Communication Compression of Intermediate Tensors for Scalable Tensor-Parallel LLM Training

    Apr 27, 2026Man Liu, Xingchen Liu, Xingjian Tian +8Communication-Efficient Distributed TrainingTensor Parallelism

  12. AEGIS: Scaling Long-Sequence Homomorphic Encrypted Transformer Inference via Hybrid Parallelism on Multi-GPU Systems

    Apr 3, 2026Zhaoting Gong, Ran Ran, Fan Yao +1Efficient Transformer InferenceHomomorphic Encryption

  13. Parallelization Strategies for Dense LLM Deployment: Navigating Through Application-Specific Tradeoffs and Bottlenecks

    Mar 5, 2026Burak Topcu, Musa Oguzhan Cim, Poovaiah Palangappa +2Parallel LLM InferenceLLM Inference

  14. Scaling State-Space Models on Multiple GPUs with Tensor Parallelism

    Feb 24, 2026Anurag Dutt, Nimit Shah, Hazem Masarani +1LLM Inference AccelerationTensor Parallelism

  15. Systematic Exploration of Multi-core Architectures for Efficient LLM Serving using WaferAI-SIM

    Oct 7, 2025Tianhao Zhu, Dahu Feng, Erhu Feng +1AI Accelerator InferenceLLM Serving