Workload-Static Optical Circuit Switching For Scalable LLM Inference Decode

Maximilian Thürmer; Nicolas Scotto di Perto; Eloise Stein; Natanijel Vasic; Mikaeil Bagherli; Zhenguo Wu; Benjamin Klenk; Holger Fröning

cs.DCNov 3, 2026

Workload-Static Optical Circuit Switching For Scalable LLM Inference Decode

Authors: Maximilian Thürmer, Nicolas Scotto di Perto, Eloise Stein, Natanijel Vasic, Mikaeil Bagherli, Zhenguo Wu, Benjamin Klenk, Holger Fröning,

Organizations: Salience Labs Ltd, 100 Park Drive, Oxfordshire, OX14 4RY, United Kingdom · NVIDIA Corporation, Santa Clara, CA, USA · Hardware and Artificial Intelligence Lab, Heidelberg University, Germany

Abstract

LLM inference decode executes fine-grained collectives whose latency directly limits per-user interactivity, yet scale-up fabrics are still built from electronic packet switches designed for training. We observe that, for a given model deployment and parallelization, the communication structure of decode is fixed, and propose workload-static optical circuit switching (OCS): circuits are configured once per deployment, co-designed with model parallelism and with topology-aware collective schedules that turn spare accelerator ports into fewer communication phases or higher bandwidth. We evaluate this design with GXNET, a graph-based workload emulator coupled to analytical and flit-level network models grounded in hardware measurements. In a 576-accelerator scale-up domain serving 5T-parameter MoE models with up to 5M tokens of context, all-OCS fabrics improve interactivity by up to 65% and energy efficiency by up to 119% over 4:1 oversubscribed packet-switched fabrics.

Figures & tables

Figure 1: Schematic two-level scale-up fabrics connecting a highlighted collective group (ranks 0, 2, 4, and 6). EPS forwards packets independently at runtime; OCS circuits are planned once for the deployment’s collective demands and reused throughout decode. Gray connections denote other physical links. Paths are illustrative rather than a bandwidth-matched comparison.

				Payload per call
Axis	Coll.	$G\times R$	Calls	$B{=}1$	$B{=}4$	$B{=}16$
Attention
world	AR	$1\times 512$	128	4 KiB	16 KiB	64 KiB
KVP	A2A	$128\times 4$	128	66 B	264 B	1056 B
TPA	none	$4\times 128$	0	–	–	–
FFN

Table 1: Deployment-time communication demand per decode step for Megatron 5T (128 layers) on 512 accelerators, using Helix parallelization

(\mathrm{KVP},\mathrm{TPA},\mathrm{EP},\mathrm{TPE})=(4,128,64,8)

and FP4 (AR: all-reduce, AG: all-gather, A2A: all-to-all). Region, axis, collective, groups

G

R

ranks, and calls per group are fixed for deployment; batch size

B

only changes the payload per call.

Panel of Figure 2 — Figure 2: Workload-static OCS topology design example with base generator set $G=(g_{3},\,g_{2},\,g_{1},\,g_{0})$ inducing a $Q_{4}$ host graph under a shared per-accelerator port budget $L$ . The same physical host graph supports multiple logical communication structures (a). One additional port relative to the $Q_{4}$ base can be allocated either to shortcut connectivity for lower collective depth (b) or to parallel circuits for higher effective bandwidth (c).

Figure 3: Latency decomposition of a 512-accelerator all-reduce (parallel RS+AG schedule) on a single-level fabric (36 ports per accelerator, 400 Gb/s links). Total completion time is shown above each bar. Simulations were run using optimized mode (each message is sent as a single flit; see Appendix C.5 ).

Panel of Figure 4 — Figure 4: Per-user interactivity vs. total system token throughput Pareto curves for two large LLMs provisioned for a context of 5 million tokens. The models are spread across 512 Rubin-like (RX00) accelerators, keeping 64 accelerators (eight per rack) in cold spare. Marker color distinguishes the physical topologies (Appendix C.7 ), and labels on the all-OCS curve give the batch size. Each point describes a separately provisioned and planned point in the deployment design space.

Figure 5: Decode-step time over batch size for the fabric-specific Pareto-optimal configurations of Figure 4 , all-EPS (left bar) and all-OCS (right bar) of each pair. For Megatron, attention time grows with batch and dominates from batch 4 on, so the constant OCS saving in communication shrinks in relative terms; for GPT-OSS-5.3-like, communication remains the dominant term. Note the different vertical scales.

Figure 6: System size (world size) scaling of Megatron 5T MoE at a fixed context of 64K tokens for batch sizes 1, 4, and 16. Interactivity is shown relative to the best all-EPS result at each batch size (dotted line).

Figure 7: Energy efficiency vs. interactivity across models of different size and context length, illustrating the benefits of workload-static, all-OCS switching in large-scale inference decode.

Appendix figures & tables32 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 8: One layer of a decode step under Helix parallelism (simplified); the full decode step repeats it for every layer, followed by one all-gather of the logits (not shown). Each region arranges the same accelerators into its own device mesh; every collective runs over one axis of its region’s mesh.

notation	meaning
Hamming weight $\|g\|$	number of 1-bits of a mask; the Hamming distance of two labels $x,y$ is $\|x\oplus y\|$ , the number of bit positions in which they differ
$Q_{n}$	hypercube on $n$ bits; its generators are the $n$ unit masks, i.e., the masks of Hamming weight 1
$Q_{n}{+}g$	$Q_{n}$ with the additional generator $g$ , a fold
$FQ_{n}$	folded cube $Q_{n}{+}1\dots 1$ , in which every node is joined to its antipode
$K_{s}$	complete graph on $s$ nodes
$G\,\square\,H$ , $K_{s}^{\ell}$	Cartesian product: two nodes are adjacent if they agree in one factor and are adjacent in the other; $K_{s}^{\ell}=K_{s}\,\square\,\cdots\,\square\,K_{s}$ with $\ell$ factors

Appendix

Table 2: Notation for topologies and folds used in this appendix. Example: the block fold

Q_{5}[3,2]=K_{8}\,\square\,K_{4}

has degree

7+3=10

and diameter 2.

degree $d$	topology	group	diameter
$d=a$	hypercube $Q_{a}$	$(\mathbb{Z}_{2})^{a}$	$a$
$d>a$	folded cube	$(\mathbb{Z}_{2})^{a}$	$<a$
$2<d<a$	circulant	$\mathbb{Z}_{2^{a}}$	intermediate
$d=2$	ring	$\mathbb{Z}_{2^{a}}$	$2^{a-1}$

Appendix

Table 3: Per-axis degree ladder for an axis of size

2^{a}

at degree

d

Figure 9: Folding a hypercube. Left:

Q_{3}

with generators

\{001,010,100\}

has degree 3 and diameter 3;

000\to 111

takes three hops. Right: the fold generator

g=111

adds one port per node and joins every node to its antipode, so

000\to 111

takes one hop and the diameter drops to 2.

Panel of Figure 10 — Figure 10: Folds shared by two meshes on one host: $N=16$ , $Q_{4}$ with the folds $0011$ and $1100$ , under the partitions of Figure 2 (a). Each panel shows one collective group per axis; every other group of the axis is a translate of it. Solid lines are cube edges, dashed lines folds whose support lies inside the axis. The mask $0011$ folds axis B of both meshes; the mask $1100$ folds axis A of the $4\times 4$ mesh and is idle for the $2\times 8$ mesh, whose partition splits bits 3 and 2.

Figure 11: Diameter frontier and schedulable frontier for axis sizes 16, 32 and 64: the fewest phases a folded cube of a given degree allows, for any schedule (diameter) and for the coset-factored schedules (coset layers). Both are staircases with plateaus that lengthen with axis size. On the axis of size 64, diameter 2 is reached at degree 13, but two coset layers only at degree 14. Circles mark the folded axes of the worked plan (Table 6 ).

Figure 12: Fabric planning. Top: the five stages, colored by the component that runs them; the router selects schedules and realizes the circuits, the planner builds each axis’s fold ladder and searches partitions, folds and multiplicities. Bottom: the worked plan of Table 6 (

N=256

L=18

), which is colored onto ports at the full budget on the first attempt.

topology (deg, diam)	min. $h$	rec. doubling	one-shot	coset-fact.	coset-fact. par.
$Q_{3}$ (3, 3)	3	3 / $3\sigma$	3 / $12\sigma$	3 / $3\sigma$	3 / $\sigma$
$FQ_{3}$ (4, 2)	2	3 / $3\sigma$	2 / $6\sigma$	2 / $2\sigma$	2 / $\sigma$
$Q_{4}$ (4, 4)	4	4 / $4\sigma$	4 / $32\sigma$	4 / $4\sigma$	4 / $\sigma$
$Q_{4}{+}1100$ (5, 3)	3	4 / $4\sigma$	3 / $20\sigma$	3 / $3\sigma$	3 / $\sigma$
$Q_{4}{+}1111$ (5, 2)	2	4 / $4\sigma$	2 / $7\sigma$	3 / $3\sigma$	3 / $\sigma$

Appendix

Table 4: Serialized phases

h

and bottleneck volume

v

(cell:

h

v

, with

v

in units of the buffer size

\sigma

) of each all-reduce schedule, as returned by its closed-form cost. min.

h

: the lower bound on the phase count, i.e., the larger of the diameter and

\lceil\log_{d+1}N\rceil

. Bold marks the schedules that attain it.

schedule	phases	bottleneck vol.	regime
recursive doubling	$\log_{2}N$	$\log_{2}N\cdot\sigma$	dominated
coset-factored	$\ell$	$\ell\sigma$	latency
coset-factored parallel	$\ell$	$\sigma$	latency
RS+AG	$2\ell$	Eq. ( 4 )	bandwidth
RS+AG parallel	$2\ell$	Eq. ( 4 ) $/\ell$	bandwidth

Appendix

Table 5: The all-reduce operating points on a chain of

\ell

coset layers. The volume of parallel RS+AG holds for uniform folds.

mesh	axis	size	topology	deg	$h$	$p$	axis $T$
(4, 64)	0	4	$K_{4}$	3	1	1	$0.60\,\mu$ s
(4, 64)	1	64	$Q_{6}[3,3]=K_{8}\,\square\,K_{8}$	14	2	1	$1.70\,\mu$ s
(16, 16)	0	16	$Q_{4}[2,2]=K_{4}\,\square\,K_{4}$	6	2	1	$1.12\,\mu$ s
(16, 16)	1	16	$Q_{4}[3,1]=K_{8}\,\square\,K_{2}$	8	2	1	$1.12\,\mu$ s

Appendix

Table 6: Resulting plan for the latency-bound multi-mesh example; for the notation, see Table 2 .

schedule	case	topology	deg	$p$	axis $T$
ring (planning only)	multi-mesh $[(2,32),(8,8)]$	$Q_{5}$ sub-cube	5	7	$3660\,\mu$ s
ring (planning only)	single mesh $(2,32)$	ring	2	17	$1526\,\mu$ s
parallel RS+AG	multi-mesh $[(2,32),(8,8)]$	$Q_{5}$ sub-cube	5	7	$731\,\mu$ s
parallel RS+AG	single mesh $(2,32)$	$Q_{5}$	5	7	$731\,\mu$ s

Appendix

Table 7: The same bandwidth-bound axis, multi-mesh versus single-mesh, under the ring schedule and under parallel RS+AG:

N=64

L=36

R=5\times 10^{10}

B/s,

\lambda=0.52\,\mu

s, an axis of size 32 carrying 655 MB (640 000 KiB).

Figure 13: Completion time of a single all-reduce (left) and all-to-all (right) spanning 512 accelerators in 16 racks, 36 ports per accelerator, against the buffer size of the collective. All-EPS: two levels, 4:1 oversubscribed towards the second level, dynamic routing. All-OCS: non-blocking, with the topology planned for every buffer size. Lines show the individual schedules, shaded bands the best schedule per fabric and buffer size, labeled with the schedule that attains it.

Figure 14: All-to-all completion time against buffer size, 512 accelerators in 16 racks of 32 (36 ports per accelerator, 400 Gb/s per link). Four configurations, each differing from the next in one respect: EPS 4:1 with its native schedules; EPS 4:1 with parallel Bruck carried as packets; EPS 1:1 with parallel Bruck on static circuits, a reference that exceeds the power envelope; OCS 1:1 with parallel Bruck on the planned fold. The native EPS schedule is one-shot at every buffer size; the step ratios at 250 MiB are given in the upper left.

Figure 15: 2-D mesh (

8\times 64

), 8 kB all-reduce, 512 accelerators: completion time against accelerator radix, stacked-link allocation vs. shared-host allocation.

Figure 16: HOP-B sensitivity at batch 16 and 5M context, simulated with Simlette , each fabric at its parallelization of Figure 4 . One microbatch is the execution without HOP-B used in Section 6 . Top: decode-step time with HOP-B over 2, 4, and 8 microbatches, relative to the step without HOP-B. Note that results for Megatron 5T coincide at approximately 1.0, so red and orange curves are not visible. Bottom: change against the step without HOP-B in communication time before overlap (red), in communication hidden behind compute (green), and in decode-step time (diamonds).

Figure 17: Completion time from Eqs. ( 1 ) and ( 10 ) for an all-reduce on 512 accelerators connected as a hypercube without extra generators, with

L=9

ports,

p=1

\lambda=0.52\,\mu

s, and

R=400

Gb/s.

Figure 18: Topology-schedule interaction at network level. Completion time of one all-reduce among 64 accelerators on a single-level OCS fabric (400 Gb/s per link), as a function of the buffer size

\sigma

of the all-reduce. Base topology

Q_{6}

+

fold (fewer phases

h

, larger degree

d

), and

+

multiplicity (

p=2

parallel links per logical edge). The dashed line marks the simulated crossover between

+

fold and

+

multiplicity (159 kB), close to the buffer size

\sigma^{\star}=c\,m^{\star}\approx 156

kB predicted by Eq. ( 3 ).

Figure 19: One-shot all-reduce on 32 accelerators with OCS: analytical model against Simlette with an unlimited and a limited buffer per port. Colors show the number of ports per accelerator.

Figure 20: One-shot all-reduce on 32 accelerators with EPS: analytical model against Simlette with an unlimited and a limited buffer per port.

Figure 21: Two-shot all-reduce on 32 accelerators with EPS and an unlimited buffer: analytical model against Simlette in flit-by-flit and optimized modes.

Figure 22: Left: simulation runtime for flit-by-flit vs. optimized mode, as a function of accelerator count, on a fixed-radix EPS single-level topology (8 ports per accelerator) with a 40 KiB all-reduce buffer. Right: the resulting speedup of optimized mode over flit-by-flit. Runtimes measured on a single core of an AMD Ryzen 9 9950X desktop processor.

Figure 23: Measured durations of the ping-pong kernel over many iterations; left: flag-flipping implementation (direct and atomic variants), right: direct-remote-write-local-read implementation.

Figure 24: Cumulative latency from the SM of GPU0 to each memory scope, all arrows sharing one origin (B300, GPU global timer, 30,000-sample average). (1) shared memory, 14 ns; (2) L1, 17 ns; (3) L2, 140 ns; (4) remote access to the L2 of GPU1 through two NVLink ports, two cables and one NVSwitch stage, 1610 ns end-to-end (ping-pong, direct-remote-write-local-read). The on-chip stages account for under 10 % of the round trip; the remainder is fabric traversal and remote-access handling.

	RTX4090	A100	H800	H100	B300
	Luo et al. (2025)	Luo et al. (2025)	Luo et al. (2025)	(ours)	(ours)
L1 cache	32.0	33.0	32.0	41.1	34.1
Shared memory	30.1	29.0	29.0	29.0	28.6
L2 cache	273.0	202.8–408	264.5–502	276.2	276.8

Appendix

Table 8: Pointer-chasing hit latency in clock cycles. Published values from Table 3 of Luo et al. (2025) ; H100 and B300 measured with the same benchmark binaries.

Case	Switch	Switch	MMU	GPU	Link	Total
	total	core	+ TLB	MMU	TLB
Low	200	100	350	217	133	810
Nominal	300	200	250	155	95	810
High	500	400	50	31	19	810

Appendix

Table 9: Remote-access latency budget (ns) under low, nominal and high switch-latency assumptions.

Panel of Figure 25 — Figure 25: Physical scale-up fabrics of the 576-accelerator system: 36 ports per accelerator at 400 Gb/s, switching devices of radix 72. The all-EPS fabric has the topology of (a) with EPS on both levels.

	Parameters				Attention		Experts
Model	Total	Active	Layers	Hidden	Type	Q / KV heads $\times$ dim	Total / top- $k$	Width	Vocab	KV cache per token
Megatron 235B	0.23 T	17 B	96	1536	GQA, full	$64/16\times 96$	128 / 8	4096	51,200	144 KiB
Megatron 1T	1.10 T	38 B	128	2048	GQA, full	$128/64\times 32$	256 / 8	5440	51,200	256 KiB
Megatron 5T	5.00 T	207 B	128	8192	GQA, full	$256/128\times 64$	256 / 8	6144	51,200	1,024 KiB
GPT-OSS-120B	0.12 T	5.7 B	36	2880	GQA, SW / full	$64/8\times 64$	128 / 4	2880	201,088	9 KiB
GPT-OSS-1T-like	1.05 T	21 B	64	4608	GQA, SW / full	$72/8\times 64$	256 / 4	4608	201,088	16 KiB

Appendix

Table 10: Model properties as used in the analyses of this paper. Active parameters include the embedding and LM head. Attention: full, or interleaved sliding-window (SW, 128-token window) and full layers. KV cache per token counts only layers whose cache grows with context.

Panel of Figure 26 — Figure 26: One decoder layer of each model, with the tensor dimensions of its projections. Red markers show where Helix parallelism issues each collective of Table 1 : the KV-parallel all-to-all that exchanges the partial attention outputs and log-sum-exp statistics each rank computes over its KV shard, before they are rescaled into the exact softmax result; the all-reduce over all accelerators after the output projection; the FFN tensor-parallel all-reduce followed by the expert-parallel all-gather, between the experts and their weighted combination; and one all-gather of the vocabulary-sharded logits per decode step.

Figure 27: Energy efficiency vs. interactivity across models of the GPT-OSS family, analogous to Figure 7 .

Accelerator	FP4 peak	HBM capacity	HBM bandwidth	Scale-up ports	Per-port bandwidth (unidirectional)	Kernel launch	In-graph launch
NVIDIA B200	20 PFLOP/s	186 GB	8 TB/s	18	400 Gb/s	2.5 $\mu$ s	0.8 $\mu$ s
NVIDIA RX00 (Rubin-like)	50 PFLOP/s	288 GB	22 TB/s	36	400 Gb/s	2.5 $\mu$ s	0.8 $\mu$ s

Appendix

Table 11: Accelerator properties as used in the analyses of this paper. FP4 is the tensor-core peak that the roofline model scales by a shape-dependent efficiency. Every execution window between collectives additionally pays the in-graph launch overhead once (Appendix B.3 ).

Component	Variant	Value	Unit	Derivation / source
XPU (incl. host share)	B200 (GB200 NVL72)	$1454$	W	$(P_{\text{rack}}-9P_{\text{sw}})/72$ , $P_{\text{rack}}=$ 120\text{,}\mathrm{k}\mathrm{W} $NVIDIA () ; NVIDIA () , $P_{\text{sw}}=$1.7\text{\,}\mathrm{k}\mathrm{W}$
	B300 (GB300 NVL72)	$1663$	W	as above, $P_{\text{rack}}=$ 135\text{,}\mathrm{k}\mathrm{W}$$ NVIDIA () ; Lenovo Press (Product Guide)
	Rubin (VR200 NVL72)	$2631$	W	as above, $P_{\text{rack}}=$ 220\text{,}\mathrm{k}\mathrm{W} $(Max-P) SemiAnalysis () ; Supermicro () , $P_{\text{sw}}=$3.4\text{\,}\mathrm{k}\mathrm{W}$ (4 ASICs/tray NVIDIA () )
EPS (scale-up), per port	$400\text{\,}\mathrm{G}$	$11.81$	W	switch tray: $P_{\text{sw}}/(2\times 72)$ ; cross-check Broadcom (2025) ; TechInsights () ; IP Infusion ()
	$800\text{\,}\mathrm{G}$	$15.35$	W	extrapolated: $2\times$ rate, $-35\,\%$ energy/bit Arista Networks () ; Arista Networks ()
	$1600\text{\,}\mathrm{G}$	$19.95$	W	extrapolated: $2\times$ rate, $-35\,\%$ energy/bit

Appendix

Table 12: Component power assumptions of the power model (TDP basis, not load-aware). Data rates are unidirectional; every port is bidirectional and carries one transceiver module.

Figure 28: Sensitivity of the advantage of hybrid EPS/OCS and all-OCS fabrics over all-EPS for GPT-OSS-5.3-like at batch 1 and 5M context on 512 of 576 accelerators. Solid: interactivity; dashed: energy efficiency (tokens/s/MW). Bold tick labels mark the baseline configuration of Section 6 ; the parallelization is re-optimized for every point.

LLM Inference Acceleration

LLM inference acceleration develops algorithms and systems that reduce language-model latency, memory use, and serving cost while preserving output quality.

See all 1,171 papers

Scheduling Parallel Optical Circuit Switches for AI Training
Mar 7, 2026Kevin Liang, Litao Qiao, Isaac Keslassy +1
UC San Diego · the Technion and UC Berkeley
Characterizing Parallelism Strategies in LLM Inference: Fundamental Compute-Communication Trade-offs
Oct 4, 2026Javad Mirzaei, Jeebak MitraLLM Inference Tensor Parallelism
Office of CTO Dell Technologies Inc. Canada
An Information-Geometric Framework for OCS-Reconfigurable Interconnect Topology in Large-Scale AI Clusters
Oct 18, 2026Shintaro YoshizawaCommunication Topology Optimization Dynamic Regret Minimization
Nagoya Mathematical and Information Science Research

Mar 7, 2026J/K move · Enter open · S save

Scheduling Parallel Optical Circuit Switches for AI Training

Kevin Liang, Litao Qiao, Isaac Keslassy, Bill Lin

UC San Diego · the Technion and UC Berkeley

Open paper AskPDF arXiv

The rapid growth of AI training has dramatically increased datacenter traffic demand and energy consumption, which has motivated renewed interest in optical circuit switches (OCSes) as a high-bandwidth, energy-efficient alternative for AI fabrics. Deploying multiple parallel OCSes is a leading alternative. However, efficiently scheduling time-varying traffic matrices across parallel optical switches with non-negligible reconfiguration delays remains an open challenge. We consider the problem of scheduling a single AI traffic demand matrix

D

over

s

parallel OCSes while minimizing the makespan under reconfiguration delay

δ

. Our algorithm Spectra relies on a three-step approach: Decompose

D

into a minimal set of weighted permutations; Schedule these permutations across parallel switches using load-aware assignment; then Equalize the imbalanced loads on the switches via controlled permutation splitting. Evaluated on realistic AI training workloads (GPT model and Qwen MoE expert routing) as well as standard benchmarks, Spectra vastly outperforms a baseline based on state-of-the-art algorithms, reducing schedule makespan by an average factor of

1.4\times

on GPT AI workloads,

2.7\times

on MoE AI workloads, and

2.4\times

on standard benchmarks. Further, the makespans achieved by Spectra consistently approach newly derived lower bounds.

Abstract

LLM Inference Acceleration

Explore similar work

Scheduling Parallel Optical Circuit Switches for AI Training

Characterizing Parallelism Strategies in LLM Inference: Fundamental Compute-Communication Trade-offs

An Information-Geometric Framework for OCS-Reconfigurable Interconnect Topology in Large-Scale AI Clusters