Kafila: Serving Large Language Models on a Trusted Set of Heterogeneous Commodity Machines

Murtaza Rangwala; Richard O. Sinnott; Rajkumar Buyya

cs.DCSep 28, 2026

Kafila: Serving Large Language Models on a Trusted Set of Heterogeneous Commodity Machines

Authors: Murtaza Rangwala, Richard O. Sinnott, Rajkumar Buyya

Organizations: Quantum Cloud Computing and Distributed Systems (qCLOUDS) Lab, School of Computing and Information Systems, The University of Melbourne, Australia

Abstract

Between them, the members of a research group or a circle of friends own several consumer computers, none large enough to run a capable large language model. Existing systems pool such capacity across open swarms anyone may join, which a group admitting only trusted machines cannot use. Bounding membership removes what they depend on: a swarm holds each part of the model on several peers and routes around a slow one. A bounded session must use every device it admits. Its pipeline advances at the pace of whichever device received a share it cannot serve quickly, so the division has to be right before serving begins. We propose Kafila, whose protocol assembles a ring from behind NATs, preferring direct paths and relaying where traversal fails, while its planner measures each device's memory bandwidth, capacity and reachability, divides the model exactly for a fixed ring order, and places the head, which holds the embedding and output projection, together with that division rather than beforehand. On machines with different capabilities across three fleets, from a shared LAN to five devices spanning two continents, Kafila shortens the slowest pipeline stage by up to $5.2\times$ against the even split of pipeline parallelism, as in GPipe, and up to $3\times$ against the memory-proportional split of personal-device inference, as in exo, keeps 75 to 87 per cent of the committed hardware doing work where those divisions fall below half, and serves a model no uniform split can place on the fleet at all. What that is worth to a user depends on how much of a token is computation rather than network. Where the members share a network the same division returns $1.56\times$ the throughput of a uniform split and $1.25\times$ of a memory-proportional one, and under four concurrent users that lead compounds to $3.2\times$ rather than fading, each user served at almost the rate of one.

Figures & tables

System	Workload	A slow or absent participant	Connectivity graph	Per-stage cost obtained from	Ordering by reachability	Head placed with the division
Petals [ 6 ]	LLM inference	Routed around at request time	Complete	Throughput measured at runtime	$\times$	$\times$
SWARM [ 7 ] / Parallax [ 8 ]	LLM inference	Rescheduled at runtime	Complete, unequal bandwidth	Throughput measured at runtime	$\times$	$\times$
GPipe [ 11 ]	Training	Does not arise; deployment is provisioned	Complete, uniform speed	Layer count	$\times$	$\times$
exo [ 12 ]	LLM inference	Slows the ring; not addressed	Complete, on one network	Device memory	$\times$	$\times$
PipeEdge [ 13 ] / PipePar [ 14 ]	Encoder inference; training	Does not arise; deployment is provisioned	Complete, unequal speed	A profile taken with the weights resident	$\times$	$\times$
Kafila (this work)	LLM inference	Cannot be avoided, so it is planned for	Incomplete and unequal	A bandwidth probe, before any weight is fetched	$\checkmark$	$\checkmark$

TABLE I: Comparison with related systems

Fig. 1: A four-member session model, arranged as a ring from the host through three members and back.

Fleet	Device	Mem	Bw (GB/s)	Site
LAN	L40S	24	512–529	Melbourne, AU
(4 dev)	L40	24	367–453	Melbourne, AU
	A40	12	377–439	Melbourne, AU
	M3 Pro	18	89–124	Melbourne, AU
US Central	A100	80	915–959	Des Moines, US
(5 dev)	RTX 5090	32	849–1047	Chicago, US

TABLE II: The three evaluation fleets

Initiator	Responder	Rule predicts	Traversal achieves
permissive	permissive	direct	direct
permissive	ordinary	direct	direct
permissive	restrictive	direct	direct
ordinary	permissive	direct	direct
ordinary	ordinary	direct	direct
ordinary	restrictive	relay	direct

TABLE III: NAT class-pair traversal outcomes

LAN , 12% network overhead
Model	Allocation	Imbal.	Bneck	Util.	tok/s
		max/min	ms
Qwen3-8b	Kafila	1.46	11.3	86%	21.5 $\pm$ 0.2
	Mem-prop. (exo)	4.01	26.0	48%	17.2 $\pm$ 3.8
	Uniform (GPipe)	5.14	36.2	41%	13.7 $\pm$ 1.3
US Central , 38 to 62% network overhead

TABLE IV: Kafila against baselines, best bold

Explore similar work

Apr 7, 2025cs.DC

Prima.cpp: Fast 30-70B LLM Inference on Heterogeneous and Low-Resource Home Clusters

On-device inference offers privacy, offline use, and instant response, but consumer hardware restricts large language models (LLMs) to low throughput and capability. To overcome this challenge, we present prima.cpp, a distributed on-device inference system that runs 30-70B LLMs on consumer home clusters with mixed CPUs/GPUs, insufficient RAM/VRAM, slow disks, Wi-Fi links, and heterogeneous OSs. We introduce pipelined-ring parallelism (PRP) to overlap disk I/O with compute and communication, and address the prefetch-release conflict in mmap-based offloading. We further propose Halda, a heterogeneity-aware scheduler that co-optimizes per-device CPU/GPU workloads and device selection under RAM/VRAM constraints. On four consumer home devices, a 70B model reaches 674 ms/token TPOT with <6% memory pressure, and a 32B model with speculative decoding achieves 26 tokens/s. Compared with llama.cpp, exo, and dllama, our proposed prima.cpp achieves 5-17 times lower TPOT, supports fine-grained model sizes from 8B to 70B, ensures broader cross-OS and quantization compatibility, and remains OOM-free, while also being Wi-Fi tolerant, privacy-preserving, and hardware-independent. The code is available at https://github.com/OpenCPIL/prima.cpp.

Zonghang Li, Tao Li, Wenjiao Feng +8

MBZUAI · UESTC · City University of Hong Kong +1

LLM Inference Optimization Offloading

Ask PDF

Jul 14, 2026cs.LG

Agora: Collective and Permissionless Internet-Scale Pretraining of Large Language Models

Training large language models at the multi-billion to trillion parameter scale is confined to datacenters, where data-parallel (DP) and model-parallel (MP) techniques presume homogeneous accelerators, high-speed interconnects, and a single orchestrating entity. Frontier model development is thereby concentrated among the few groups able to assemble such clusters. Meanwhile, an enormous pool of compute remains unusable for training: consumer and professional GPUs that are heterogeneous, preemptible, individually owned, and connected only by the internet. We present Agora, a system that makes efficient use of this compute. Agora combines bandwidth-efficient pipeline-parallel model sharding over internet-grade links with multi-party, fault-tolerant collective operations. Each participant holds only one stage of the model, and no single party ever possesses the full weights. We term this setup Protocol Learning: it enables collectively trained, collectively owned models, opening a path to open-source frontier training with economic sustainability. This report presents the outcome of a research effort spanning communication-efficient parallelism, asynchronous optimization, and fault-tolerant systems design. It culminates in the first demonstration of its kind: Pluralis-8B, an open, permissionless pretraining run of an 8.6B-parameter model on 500B tokens of FineWeb-Edu. The model was trained over 40 days by 330 contributor nodes, predominantly consumer GPUs on internet connections, joining and leaving throughout. The run sustained ~170k tokens/s and 4.2 tokens per TFLOP of pooled compute, 63% of the efficiency of a centralized H100 baseline, and converged to within a small margin of a centralized reference run.

Gil Avraham, Violetta Shevchenko, Hadi Mohaghegh Dolatabadi +9

Pluralis Research

Large Language Model Training Pretraining Fault Tolerance Collective Behaviors

Ask PDF

Sep 30, 2026cs.DC

Cascadia: A Control-Plane-Free Alternative to Hyperconverged AI Infrastructure

We present Cascadia, a system for serving large language models on fleets of commodity Intel AIPCs using their CPU, integrated-GPU, and NPU resources. Every node embeds ingress, scheduling, and execution; inference requests require no dedicated routing control plane. Nodes join a libp2p QUIC mesh using CA-issued ed25519 admission certificates, gossip signed capabilities, exchange live load over direct peer streams, and route OpenAI-compatible requests to eligible peers. An operator-run certificate authority handles admission and fleet management outside the inference path. Three serving modes share one interface: whole-model execution on one node, load-balanced replicas, and pipeline-sharded chains using the compilation and speculative decoding mechanism of our companion paper. Optional KV-cache mobility reuses compatible conversation prefixes after a routing move, with cold recomputation on a miss. Signed response receipts and hash-chained logs support provenance and audit. A three-node Phi-3.5-mini NPU testbed delivered 3.10x the response throughput of its one-node configuration under ten concurrent requests; a separate four-node deployment recorded 4.06x the throughput of direct single-node serving. Paired latency observations, runtime measurements, and internal functional checks characterize the tested configurations. We compare Cascadia with IBM, Nutanix, VMware, and HPE platforms on deployment footprint, hardware requirements, scheduling, scaling, licensing, and trust, using vendor documentation. The paper repository provides benchmark scripts, curated measurements, and a claim-to-evidence map.

Matias Parij, Pawan Paudel, Tate Berenbaum +1

Not Community Labs Inc. · Intel Corporation

Artificial Intelligence Infrastructure Load Balancing Single Pass

Ask PDF