LoCoJAX: Host-Local Coordination for Small-Model RL on TPUs
Authors: Loki Chen, Gregory B. Rehm, Yang Song, Hongye Xie, Ming Lei, Chongyang Bai, Bob Kamma, Mingfu Liang, Kavosh Asadi, Yufei Li, Xi Liu, Jay Xu, Rui Liu, Maziar Sanjabi, Luke Simon, Hamed Firooz
Reinforcement learning (RL) increasingly targets small language models, and executing these workloads efficiently across multiple accelerators requires effective coordination. On TPUs, Pathways-based RL uses a single controller to coordinate computation across hosts. This execution model is effective for large models, but its coordination computational costs are harder to amortize when small models repeatedly invoke short computations. To characterize these computational costs, we sweep dense models from 0.6B to 405B parameters on TPU v7x under a decode-heavy workload and measure how host and network computational overhead vary with model size. At 0.6B, host orchestration occupies 36% of step time and network communication 2%, whereas for large models, estimated network computational costs dominate the non-compute time. Motivated by these findings, we present Local Coordination JAX (LoCoJAX), which implements local rollout and reward processing on each TPU host with synchronous global training. In controlled comparisons against a tuned and optimized Pathways baseline on Qwen3-0.6B across three workloads and 8-64 TPU v7x chips, its colocated mode (LoCoJAX-CO) is 1.3-3.6x faster per step; on GSM8K, it completes a step faster on 8 chips than Pathways does on 64. Finally, a Ray-managed disaggregated mode (LoCoJAX-DA) overlaps rollout and training on separate hosts, and placement experiments establish when this overlap compensates for dividing devices and transferring weights: only when rollout dominates the step and receives most of the hosts. On NuminaMath-CoT at 256 chips, giving rollout 75% of hosts cuts step time by about 15% compared with LoCoJAX-CO.
Figures & tables
Figure 1: Where the Group Relative Policy Optimization (GRPO) step goes as model size grows. For each dense model, we report the host-orchestration and network shares of step time at its fastest tensor-parallel configuration on 128 TPU v7x chips under a decode-heavy workload with 8,192 forced output tokens (Table 2 ). Host orchestration is the largest non-compute component for small models and network communication for large ones.
Figure 2: System architectures of Pathways, LoCoJAX-CO, and LoCoJAX-DA. (a) Our evaluated baseline follows Pathways’ documented single-controller SPMD execution model: one CPU-head controller drives a globally distributed rollout engine and sharded training across TPU hosts. (b) LoCoJAX-CO adopts synchronous, colocated execution with independent host-local JAX controllers and rollout engines. (c) LoCoJAX-DA uses Ray-managed disaggregation to separate rollout and training hosts, enabling asynchronous execution and overlap between generation and training.
Dataset
Character
Prompts × generations
Prompt cap
Response cap
Thinking
GSM8K
balanced prefill and decode
128 × 16
512
2,048
off
NuminaMath-CoT
decode-heavy
256 × 8
4,096
8,192
on
SWE-agent trajectories
prefill-heavy
256 × 8
15,360
1,024
off
Table 1: Stack-comparison workloads (Qwen3-0.6B): prompts × generations per step, prompt and response caps (tokens), and Qwen3 thinking mode.
Figure 3: Step-time breakdown.
Model
Fastest TP
Step (s)
Computational Overhead
Host
Network
Qwen3-0.6B
TP1 ≈ TP2
68.6
40.6%
36.4%
2.2%
Qwen3-1.7B
TP1 ≈ TP2
79.6
29.8%
24.3%
3.5%
Qwen3-4B
TP2
108.5
32.2%
17.6%
12.2%
Qwen3-8B
TP4 ≈ TP2
134.1
39.6%
18.3%
18.5%
Qwen3-14B
TP4
168.9
34.2%
14.8%
17.4%
Qwen3-32B
TP4 ≈ TP8
301.3
29.2%
8.3%
18.6%
Table 2: Step time and its breakdown at each model’s fastest TP, 128 TPU v7x chips, decode-heavy workload (8,192 forced output tokens). Computational overhead is the share of the step outside compute; host and network are its two largest parts. ≈ marks a tie within run-to-run spread; values are for the first-named TP. Medians over 3 runs ( ∼ 100B: 2; 405B: 1, a workaround extension); network shares are upper-end estimates.
Figure 6Figure 7
Figure 8: Step time vs TPU v7x chips (log–log), LoCoJAX-CO vs LoCoJAX-DA at three trainer/rollout host splits, Qwen3-0.6B; medians over launches. Hollow markers: provisional cells. Missing LoCoJAX-DA splits are ruled out by host count, our Ray launcher or trainer memory (Appendix F ). The LoCoJAX-CO point at NuminaMath 16 chips comes from Figure 6 . Median step times per split: Table F1 .
Appendix figures & tables18 assets
Supplementary material from the paper’s appendix.
Appendix
Lever
What it changes
Code it acts on
Applies to
O1
Exports metrics off the per-host barrier
LoCoJAX per-host learner
LoCoJAX only
O2
Freezes Python garbage collection after warmup
LoCoJAX per-host learner
LoCoJAX only
O3
Balances prompts across hosts by length
LoCoJAX per-host learner
LoCoJAX only
O4
Publishes all weights to the local engine in one call
LoCoJAX weight publish
LoCoJAX only
Appendix
Table B1: LoCoJAX optimization levers. Outside this list, both stacks also used LoCoJAX’s sampler, which does not compute the top-k and top-p masks (unused here); LoCoJAX keeps it with its levers off.
Figure 9: The two recurring controller computational costs of Section 4.3 , GSM8K, Qwen3-0.6B, 64 chips. (a) Request admission once per step: the Pathways head admits all 2,048 requests in about 1.0 s, while each LoCoJAX host admits its own 128 requests in under 0.06 s. (b) Ten of about 1,800 decode iterations: each Pathways iteration takes 2.15 ms against 0.90 ms for LoCoJAX, adding about 2.25 s per step.
Family
PRs
Bottleneck
Effect
Active in
A. Attention block sizes
#3102; #3395
Block-size settings ignored on the MaxText path
#3395: step 267.9 → 218.6 s [PR]
Pathways; LoCoJAX uses own tiles
B. EOS-check control
#3174, #3509
Cross-DP EOS all-reduce in every decode iteration, even with the check off
#3509: 216 μ s of a 5.4 ms iteration [PR]
Both
C. Reward parallelism
#1904; #5255, #5235
Serial host scoring of 2,048 completions
#1904: reward 29.6 → 3.5 s [PR]; no improvement on our workload [Ours]
Pathways
D. GC lifecycle
#2086; #5236
Full gc.collect() after each weight sync
∼ 1.2 s per step [PR]
Both
E. Sequence packing
#5100, #2309; covered by #2076
76% of training slots were padding [Hist.]
#5100: training 15.2 → 6.7 s [PR]
Pathways; LoCoJAX uses own packer
F. KV-cache lifecycle
#2231; #5289
KV pool freed and reallocated at every weight sync
Table D1: Upstream changes by mechanism. Effects come from the PR on its author’s workload [PR], our logs [Ours] or earlier measurements not re-verified [Hist.], and are not additive. Active in: Table 3 images in which the change runs. Per-PR detail: Table D2 .
PR
Short title
tpu-inference #3101
Fix leaking mutable default metadata
tpu-inference #3102
Env overrides for attention block sizes
tpu-inference #3174
EOS-check interval from env
tpu-inference #3374
Continue decode on flax_nnx; keep compile options
tpu-inference #3395
Forward block-size overrides on MaxText path
tpu-inference #3466
Never pad past the token budget
Appendix
Table D2: Per-PR inventory. All merged except #3511, closed in favor of #3490, and #2087, superseded by upstream #2076. #5270 (CI repair) is excluded from all claims.
Qwen3
TP
Pathways (s)
LoCoJAX (s)
Ratio [95% CI]
NuminaMath
1.7B
1
51.1
26.7
1.88 [1.86, 1.90] a
4B
1
70.3
46.5
1.51 [1.49, 1.53]
8B
1
89.4
68.9
1.30 [1.28, 1.32]
14B
2
99.2
78.5
1.26 [1.22, 1.30] †
32B
8 b
154.1
149.7
1.03 [0.95, 1.10] †‡
Appendix
Table E1: Model size at 128 chips: frozen Pathways recipe vs LoCoJAX optimized (descriptive). Times: geometric means of launch median step times; TP: rollout TP of both stacks. Ratio: Pathways/LoCoJAX with Welch 95% interval; at 32B, the per-launch interval, which includes 1. a Value of record; times from a confirmatory rerun (1.91). † Post-result. ‡ LoCoJAX adopts Pathways’ activation offload, which slows LoCoJAX only. b TP2 and TP4 not measured. c Pathways times on a basis that favours Pathways.
Qwen3
TP
Pathways (s)
LoCoJAX (s)
Ratio [95% CI]
1.7B
1
11.73
3.60
3.25 [3.16, 3.36]
4B
1
14.79
6.48
2.28 [2.17, 2.40] †a
8B
1
18.24
10.57
1.72 [1.71, 1.73]
14B
2
21.34
11.32
1.88 [1.66, 2.14]
32B
4
31.10
19.67
1.58 [1.49, 1.67]
Appendix
Table E2: GSM8K size ladder at 64 chips: frozen Pathways recipe vs LoCoJAX optimized (descriptive). Times: means of launch mean step times; TP: rollout TP of both stacks. Ratio: geometric mean over launches with Welch 95% interval. † Post-result, pooled across windows. a On launch medians: 2.47 [2.18, 2.80].
Figure 10: Pathways/LoCoJAX step-time ratio and the model R=1+Δ/TLoCo . (a) Against model size: NuminaMath and SWE at 128 chips and the GSM8K size ladder at 64 chips (frozen Pathways recipe vs LoCoJAX optimized). Lines: the model at each block’s mean Δ ; shaded: range of Δ ; hollow: outside the fit (32B, SWE 8B). (b) The same ratios against LoCoJAX step time, plus GSM8K Qwen3-0.6B at 4–128 chips (Pathways vs LoCoJAX base). Dotted blue: NuminaMath fit with a free slope ( a=0.92 ).
Cell
TPW
TLoCo
Δ
Measured
Model
NuminaMath, 128 chips
1.7B
51.12
26.71
24.42
1.91 [1.83, 2.01]
1.81
4B
70.34
46.53
23.81
1.51 [1.49, 1.53]
1.47
8B
89.43
68.95
20.48
1.30 [1.28, 1.32]
1.33
14B (TP2)
99.20
78.54
20.66
1.26 [1.22, 1.30]
1.29
32B (TP8) ∗
154.08
149.67
4.42
1.03 [1.02, 1.04]
1.15
Appendix
Table E3: Inputs and held-out checks of the model (Appendix E.1 ). Times: geometric means over three launches per arm (s). Measured: TPW/TLoCo with Welch 95% interval (not computed for GSM8K). Model: 1+Δ/TLoCo with Δ fitted on the block’s other cells; ∗ outside the fit. NuminaMath and SWE: frozen Pathways recipe vs LoCoJAX optimized; NuminaMath 1.7B from the confirmatory rerun. GSM8K: an earlier reading; the reading of record differs most at 16 chips (Pathways 8.25 s). ‡ Two generation waves. § Additional cell, same head group.
LoCoJAX-DA
Dataset
Chips
LoCoJAX-CO
50%
25%
12.5%
GSM8K
8
5.08
8.20
–
–
16
3.33
4.32
3.75
–
32
2.38
2.87
2.63
–
64
2.02
1.98
1.94
2.56
128
1.73
1.56
1.88
2.90
Appendix
Table F1: Median step time (s), LoCoJAX-CO vs LoCoJAX-DA at three trainer-host shares, Qwen3-0.6B. Fastest per row in bold; –: not run; † provisional; ‡ added post hoc, in a later window; ∗ split chosen after the cell’s main result.
LoCoJAX
Model, chips
Google
CO
DA
Qwen3-1.7B, 4
s/step
9.14
1.71
2.62
(Tunix)
speedup
1.00 ×
5.34 ×
3.48 ×
Gemma4-26B, 64
s/step
298.1
139.1
269.2 a
(MaxText)
speedup
1.00 ×
2.14 ×
1.10 ×
Appendix
Table G1: Google’s published TPU RL recipes vs LoCoJAX (GRPO, TPU v7x, same chips; steady-state seconds per step, three runs per arm). Qwen3-1.7B: steps 4–150, mean of run medians. Gemma4-26B: median of run medians over the steady steps both systems share, within steps 4–19. Speedups per generated token (Qwen3): 5.06 × and 3.49 × .
Share of the step (%)
TP
DP
m
Waves
n
Step (s)
Range (s)
Computational Overhead
Host
Network
TP coll.
Other
Compute
Fastest
Qwen3-0.6B
1
256
4
2
3
68.6
67.8–69.3
40.6
36.4
2.2
0.0
2.0
59.4
fastest
2
128
8
2
3
71.4
67.4–72.2
45.8
28.7
14.8
5.3
2.4
54.2
≈ (tie)
4
64
16
2
3
74.9
73.9–77.0
44.0
22.6
17.7
12.1
3.7
56.0
8
32
32
2
1
100.4
–
45.6
18.6
24.8
21.6
2.2
54.4
Appendix
Table H1: Full accounting of the scaling study (Section 4.1 , Table 2 ): every model of the study at every TP measured with the study recipe on 128 TPU v7x chips (controls and recipe deviations are listed separately), decode-heavy workload (8,192 forced output tokens), random initialization. Each block starts at the smallest TP that fits ( ∼ 100B: TP16 only). Bold : the model’s fastest TP (or its only TP), the row shown in Table 2 ; ≈ : tied with it within run-to-run spread by Table 2 ’s rule (the runner-up’s and the fastest cell’s run ranges overlap, or their median gap is below the larger range). TP, DP: the sampler’s tensor and data parallelism (TP × DP = 256 TensorCores); m : sequences per replica ( m×DP=1,024 , so the 2,048 completions take 2 decode waves; 405B: see its block). n : valid runs. Step: median over runs of each run’s steady-step median (2 runs: mean); range: fastest and slowest run, given when n≥2 . Shares of the profiled step, each the median over runs (2 runs: mean): computational overhead = share outside compute; host = host orchestration; network = upper-end estimate, including waits for TP partners and at the DP check; TP coll. = TP collectives, part of network; other = computational overhead − host − network (unassigned decode residual and launch-skew idle, not host orchestration); compute = 100 − computational overhead. Rounded parts may differ from their sum by 0.1.
Share of the step (%)
Run set
TP
DP
m
Waves
n
Step (s)
Range (s)
Computational Overhead
Host
Network
TP coll.
Other
Compute
Note
(a) Synthetic ∼ 200B controls for the 405B rows (128 chips)
Wave control ( m= 16)
16
16
16
8
1
1,668.8
–
40.9
1.7
39.1
32.8
0.1
59.1
8 waves, as 405B TP16
TP16 package without remat
16
16
64
2
1 ∗
742.6 ∗
–
–
–
–
–
–
–
+1.9% vs grid, same steps
TP32 q/k/v remat
32
8
128
2
1 ∗
962.3 ∗
–
–
–
–
–
–
–
− 0.2% vs grid, same steps
TP16 weight pinning (PIN)
16
16
64
2
0/2
–
–
–
–
–
–
–
–
refused on Pathways
Appendix
Table H2: Controls and diagnostics of the scaling study, kept out of Table H1 and never pooled with it; columns and statistics as there. (a) Wave control: Synthetic ∼ 200B at TP16 with 16 sequences per replica, so 8 decode waves as at 405B TP16 (grid row: m=64 , 2 waves). Workaround controls: the 405B memory workarounds applied to Synthetic ∼ 200B (TP16: the compiler flag and freeing the sampler’s weight copy and the KV cache, without remat; TP32: q/k/v remat), compared with steps 1–3 of the grid runs of the same TP (728.4 s at TP16, 963.8 s at TP32). ∗ Four-step control run, no steady readout by design: median over steps 1–3 (after the warm-up step 0) of the step plus the following weight sync (the readout’s basis); not profiled, so no breakdown. Weight pinning (with freeing): both attempts failed in the first weight sync, refused on Pathways. (b) Llama-3.1-8B architecture, randomly initialized, compared with Qwen3-8B at TP16 in Table H1 . (c) Qwen3-14B at TP1 with a different attention-tile setting (a recipe deviation); with the default tiles 14B runs from TP2. (d) 16 replicas of TP8 on 64 chips; chip count and per-replica work differ from the 128-chip grid, so they are not comparable with Table H1 . GB200 runs (another platform, qualitative only) are not shown.
Dataset
Chips
LoCoJAX
Pathways
Ratio [interval]
GSM8K
4
23.3
29.6
1.27 m
85% (T0.7)
8
28.3
43.0
1.52 [1.42, 1.63]
Δ+ 0.65 pp
16
48.0 a
77.6
1.62 [1.49, 1.75]
32
64.7
160.7
2.49 [2.30, 2.64]
64
121.0
282.1
2.33 [2.14, 2.48]
128
228.4
633.0
2.77 [2.49, 3.00]
Appendix
Table I1: TPU v7x chip-hours to the frozen held-out target, summed over 3 seeds, LoCoJAX base and Pathways with the same shared optimizations. Under the dataset: target and Δ , held-out quality of Pathways minus LoCoJAX (Table I6 ). Ratio: Pathways over LoCoJAX base [95% seed bootstrap]; bar 1.25 (pre-declared). m On the bar (interval 1.245–1.289). a LoCoJAX charged at update 325, Pathways at 275 (same checkpoint: 1.84).
Dataset
Chips
LoCoJAX base (pair)
Pathways (pair)
Ratio [interval]
Same checkpoint
Steady
Status
GSM8K
4
23.326 (275/300)
29.550 (275/300)
1.267 [1.245, 1.289]
1.267 / 1.264
1.232 i
on the bar
8
28.300 (275/300)
42.967 (275/300)
1.518 [1.424, 1.629]
1.518 / 1.509
1.493
supported
16
48.013 (325/348)
77.591 (275/300)
1.616 [1.492, 1.754]
1.838 / 1.841
1.757
supported
32
64.667 (275/300)
160.711 (275/300)
2.485 [2.301, 2.640]
2.485 / 2.499
2.394
supported
64
120.996 (275/300)
282.116 (275/300)
2.332 [2.138, 2.477]
2.332 / 2.368
2.578
supported
128
228.373 (275/300)
632.996 (275/300)
2.772 [2.495, 2.999]
2.772 / 2.811
3.014
supported
Appendix
Table I2: Full grid, LoCoJAX base against Pathways with the same shared optimizations. Chip-hours to the target (3 seeds summed) with the charged checkpoint pair; ratio Pathways over LoCoJAX base [95% seed bootstrap]. Same checkpoint: both stacks charged at each update of the earlier stack’s pair. Steady: chip-hours from update 25 to 275. i Interval 1.201–1.263. Status: against the pre-declared 1.25 bar.
Dataset
Training data (revision)
P × G
Caps
Updates
Evaluation
Target
GSM8K
openai/gsm8k @740312ad: first 7,424 training rows, 6 epochs
128 × 16
256 / 7,936
348
test split (1,319 rows), T 0.7
3-seed sum ≥ 3,364 of 3,957 (85%) at updates 275–348
Appendix
Table I3: Computational cost-to-learn recipe (Qwen3-0.6B, GRPO, 3 seeds per cell), identical on both stacks. P × G: prompts × generations per update; caps in tokens. Scored checkpoints: updates 275, 300, 325 and 348.
Sensitivity
Effect
Verdict change
Slow first weight transfer replaced (raised limit) or removed
Table I4: Sensitivities of the grid, hypothetical on measured inputs. Values are Pathways over LoCoJAX base; chip counts in parentheses. The template resource-manager CPU limit slowed Pathways’ first weight transfer to about 8–11 min in the 16- and 32-chip GSM8K seeds.
Dataset
Chips
Fair
Own best
Frozen / Pathways
Base / optimized
Frozen / optimized (not fair)
GSM8K
4
1.267
1.281
1.385
1.011
1.774
8
1.518
1.554
1.325
1.023
2.058
16
1.616
1.711
1.233
1.059
2.110
32
2.485
2.618
1.162
1.053
3.043
64
2.332
2.590
1.133
1.111
2.934
128
2.772
3.064
1.028
1.106
3.150
Appendix
Table I5: Other pairings of the same runs: chip-hour ratios to the target per cell (3 seeds summed); only the first column is a fair comparison. Fair: Pathways over LoCoJAX base. Own best: Pathways over LoCoJAX optimized. Frozen / Pathways: what the shared optimizations save Pathways. Base / optimized: what O1, O2 and O4 save LoCoJAX. Frozen / optimized: the product of the Fair, Frozen / Pathways and Base / optimized columns.
Dataset (score)
Pathways − LoCoJAX
Floor
GSM8K (T 0.7, mean of updates 275 and 300)
+ 0.65 pp [ + 0.28, + 1.02] (18 v 18)
∼ 1 pp
Appendix
Table I6: Held-out quality, Pathways minus LoCoJAX (Welch 95% CI; runs per stack). Pooled over chip counts, which share seeds and data order, so the interval is optimistic. Floor: evaluation noise of one checkpoint.
Training large language models is generally done on clusters containing thousands of accelerators, communicating over a high-bandwidth interconnect. Scaling up these clusters is expensive and can become impractical, imposing limits on the size of models that can be trained. Several recent studies have proposed training methods that are less communication intensive, avoiding the need for compute clusters with extremely high interconnect speeds. These low communication training methods still employ a global synchronization step for model parameters, which can be too costly with a high number of participants, as the communication cost scales quadratically with group size. In this work, we propose a novel optimization method, NoLoCo, that does not explicitly synchronize all model parameters during training and does not require any collective communication. NoLoCo implicitly synchronizes model weights via a novel variant of the Nesterov momentum optimizer by partially averaging model weights within randomly selected subgroups. We provide both a theoretical convergence analysis of our optimizer and empirical results from language model training. Our method requires significantly less communication than fully sharded data parallel training and DiLoCo, a widely used low-communication baseline. Moreover, our method avoids global blocking communication, thereby reducing accelerator idle time. Our experiments show that NoLoCo is more communication-efficient than DiLoCo, improving final perplexity by up to 4% and converging up to 4× faster in wall-clock time across a range of worker counts, model sizes, and communication bandwidths.
Reinforcement learning (RL) post-training for large language models (LLMs) coordinates multiple models across generation, inference, and training on GPU clusters. Several factors may change during a run, including resource availability, sequence length, memory pressure, and stage bottlenecks. As a consequence, an execution plan that was initially suitable can then become slow or even infeasible over time. However, adapting a job whose models share GPUs entails significant challenges: deciding whether a new plan is worth the transition cost, reusing the job's distributed state, and coordinating GPU transfers across models and stages. Nereus targets these challenges as a cost-aware runtime that adapts RL post-training jobs into efficient execution plans. Its low-overhead controller selects a memory-feasible global plan and admits the transition using a cost model calibrated against the running job. To estimate and execute a transition, Nereus represents the distributed state of each replica of a model-stage (one model in one stage) as an Elastic Model Unit. It then employs a global transition graph to order the transformations and GPU transfers of these units. In a trace built from real data, online TP/PP adaptation reduces average step latency by 27.7% relative to the initial fixed TP/PP layout with DP scaling. In a 1,000-step run reaching 1,024 GPUs, six transitions consume 0.079% of total run time. Nereus improves end-to-end 8B PPO throughput by 2.14--7.27× over OpenRLHF and by 1.10--1.47× over Verl across diverse clusters.
Songlin Jiang, Tuo Shi, Sitong Zhang +3
Aalto University · Shenzhen University of Advanced Technology · Zhejiang University
Modern LLM reinforcement learning (RL) workloads require a high-performance weight transfer system to scale training across heterogeneous compute resources. However, efficiently transferring terabyte-scale model weights across thousands of GPUs remains challenging because the system must accommodate clusters that dynamically scale up and down while keeping coordination, data movement, and storage overhead low. We introduce Reference-Oriented Storage (ROS), a new storage abstraction for RL weight transfer that exploits highly replicated model weights in place. ROS presents the illusion that certain versions of the model weights are stored and can be fetched on demand. Underneath, ROS does not physically store any copies of the weights; instead, it tracks the workers that hold these weights on GPUs for inference. Upon request, ROS directly uses them to serve reads. We build TensorHub, a production-quality system that instantiates the ROS idea with topology-aware transfer, model-parallel consistency, and fault tolerance. Evaluation shows that TensorHub saturates RDMA bandwidth and adapts to three distinct rollout workloads with minimal engineering effort. Specifically, TensorHub reduces total GPU stall time by up to 6.7x for standalone rollouts, accelerates weight updates for elastic rollouts by up to 4.8x, and cuts cross-datacenter rollout stall time by up to 19x. TensorHub has been deployed in ByteDance production to support cutting-edge RL training.