cs.DCOct 28, 2026

MCCL: Training Pytorch models across a fleet of Macs at Skywalker Sound

Authors: Kevin Sasso, Chris Fiegel, Francis Aitken, Nicolas Tsingos, Nicholas Tong

Organizations: Skywalker Sound, Nicasio, California, USA.

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Skywalker Sound, a premier audio post-production facility, operates more than 200 M4 Max Mac Studios on a 10Gb Ethernet network. When not in use by artists, these machines sit idle, representing massive untapped unified memory and compute capacity. While these fleets are excellent candidates for distributed machine learning alongside existing NVIDIA GPU clusters, PyTorch lacks an accelerated distributed backend for Apple's Metal Performance Shaders (MPS). To bridge this gap, we introduce the Metal Communication Collective Library (MCCL), the first open-source MPS backend for multi-node PyTorch training over switched Ethernet. We evaluate its performance and scalability across 24 nodes, demonstrating that large all-reduce operations reach 98-99.7% of TCP goodput. Training a 113.5M-parameter Vision Transformer (ViT) scales from 198 to 4,568 samples/s, achieving 95.9% parallel efficiency. Benchmarked against an NVIDIA A100 GPU, one M4 Max Mac equals roughly 1.7x the throughput of FP32 matrix multiplication, and 5.5 Macs match an A100 using TF32. Harnessing 100 idle Macs for 12 hours nightly offsets approximately $200,000 annually in equivalent cloud compute costs. Our results demonstrate that existing Mac infrastructure can be seamlessly repurposed for high performance distributed training without abandoning the standard PyTorch codebase.

Figures & tables

Figure 1: One MCCL training step on a Mac Studio node: audio batches stream from ZFS storage, forward and backward run on the MPS GPU in unified memory, and MCCL averages each finished gradient bucket in place across peers while the GPU keeps working.
BackendDeviceHow gradients get averaged
NCCL / RCCLCUDA / ROCmGPU kernels & DMA over NVLink, RDMA, or TCP
XCCL / HCCLXPU / NPU / HPUvendor c10d::Backend plugins (Intel, Ascend, Gaudi)
Gloo / MPICPU RAMCPU-only; stock PyTorch fails on MPS models, and manual staging stalls the GPU
MLXmx.arrayseparate framework; Ethernet TCP/MPI paths are CPU-bound, not PyTorch MPS
MCCL (Ours)Apple MPSaverages in place in unified memory; Metal shared events order GPU and CPU
Table 1: How distributed backends handle gradients across accelerators.
Figure 2: How one gradient bucket bkb_{k} changes hands without stopping the GPU. A ticket signal queued right after the kernels that write bkb_{k} tells the CPU it may start (O1); a second ticket raised when the ring finishes gates the optimizer’s read (O2); only PyTorch’s thread ever commits the GPU stream (O3).
Figure 3: Scaling from 4 to 24 Macs. Left and center: for training-sized buckets (16–256 MB), all-reduce bus bandwidth stays flat at 98–99.7% of the 10 GbE TCP ceiling through 24 nodes. Right: in a bucketed training step, MCCL hides 94% of communication behind GPU backward ; blitting through a Metal staging pool and waiting for completion ( MCCL_UNIFIED_COLLECTIVE=0 ) eliminates that overlap.
Figure 5
Figure 6
More in this topic

Distributed Training

Distributed training coordinates computation across multiple devices or machines to train models that exceed single-device limits.

See all 133 papers

Explore similar work

CardsList
Oct 9, 2026

Minions: A Dynamic and Distributed Virtual GPU Cloud Built on Idle Apple Silicon Devices

The increasing demand for GPU-accelerated machine learning inference has exposed limitations in centralized GPU infrastructures, including high cost, limited elasticity, and long wait times under resource constraints. At the same time, modern Apple Silicon-based macOS devices are equipped with powerful GPUs that remain largely idle during normal usage. This paper presents Minions, a dynamic and distributed virtual GPU cloud that aggregates idle Apple Silicon GPUs across macOS devices to execute large-scale machine learning inference workloads. Minions enables teams to leverage existing hardware without requiring additional GPU procurement. The system provides flexible GPU inference task execution, fault tolerance, and efficient resource utilization. Our experience shows that idle macOS devices can be transformed into a practical distributed GPU inference platform in environments where centralized GPU resources are constrained.

Jianhua Li, Yun Xing, Girisha De Silva +3

Apple USA

Distributed InferenceGPU AccelerationEdge Computing
AskPDF
Jul 21, 2026

BaseRT: Advancing Best-in-Class LLM Inference with Apple M5 Neural Accelerators

Apple's M5 generation introduces a redesigned GPU architecture in which every core carries a dedicated Neural Accelerator: on-die matrix units exposed through the Metal4 tensor API. We show that BaseRT, our native Metal inference runtime for large language models on Apple Silicon, exploits these units to push inference throughput on Apple hardware substantially beyond both llama.cpp and MLX. Building on BaseRT's framework-free design, we add a family of hand-written Metal4 tensor-core kernels (including dense and mixture-of-experts GEMM and flash-attention prefill kernels) that route the compute-bound matrix multiplications of inference through the M5 Neural Accelerators while leaving the memory-bound decode path on our existing specialised kernels. On an Apple M5 Pro, across fifteen model configurations spanning the Qwen3, Qwen3.5/3.6, Llama3.2, and Gemma4 families from sub-1B to 35B parameters, BaseRT delivers up to 6.4×6.4\times higher prompt-processing throughput than llama.cpp and 3.9×3.9\times higher than MLX, with the largest margins on the mixture-of-experts models where matrix multiplication dominates, while maintaining its lead on decode of up to 1.75×1.75\times over llama.cpp and 1.33×1.33\times over MLX. These results establish a new performance ceiling for on-device LLM inference and show that the M5's tensor cores are the decisive lever for prompt processing on Apple Silicon. BaseRT is publicly available at https://github.com/basecompute/baseRT.

Fabian Waschkowski, Prabod Rathnayaka, Lukas Wesemann

Base Compute, Melbourne, Australia

AI Accelerator InferenceOn-Device Language Model InferenceLLM Inference AccelerationNeural Network Hardware Acceleration
AskPDF
Aug 24, 2026

ShardMeter: Sharded and Geo-Distributed Training Without the Guesswork

Training large-scale AI models often outgrows a single data center, demanding sharded, multi-cluster, and decentralized training. However, the huge space of resource allocations makes exhaustive benchmarking and manual tuning impractical, while performance depends on tightly coupled factors like model size, GPU memory, batch size, bandwidth, and sharding strategy. We introduce ShardMeter, a lightweight analytical performance model that predicts the end-to-end runtime of transformer-based workloads across arbitrary sharded, distributed, and even decentralized training. Given a model's characteristics and a target hardware topology, ShardMeter estimates per-GPU and per-island throughput, training cost, total wall-clock time, and identifies performance bottlenecks. Our analysis reveals diminishing-return regimes as island size increases, quantifies transitions between compute- and communication-bound scaling, evaluates hyperparameter trade-offs, and models cost-throughput for large-scale decentralized training. ShardMeter exposes these insights to quickly explore the configuration space, choose near-optimal deployment plans, and avoid costly trial and error.

Tim Beringer, Patrick Diem, Felix Wolf +1

Technical University of Darmstadt · PanocularAI

Distributed TrainingDecentralized Learning
AskPDF