OptINC: Optical In-Network-Computing for Scalable Distributed Learning

Sijie Fei; Grace Li Zhang; Bing Li; Ulf Schlichtmann

cs.LGMar 30, 2026

OptINC: Optical In-Network-Computing for Scalable Distributed Learning

Authors: Sijie Fei, Grace Li Zhang, Bing Li, Ulf Schlichtmann

Organizations: Technical University of Munich, Munich, Germany · Technical University of Darmstadt, Darmstadt, Germany · Technical University of Ilmenau, Ilmenau, Germany

Abstract

Distributed learning is widely used for training large models on large datasets by distributing parts of the model or dataset across multiple devices and aggregating the computed results for subsequent computations or parameter updates. Existing communication algorithms for distributed learning such as ring all-reduce result in heavy communication overhead between servers. Since communication in large-scale systems uses optical fibers, we propose an Optical In-Network-Computing (OptINC) architecture to offload the computation in servers onto the optical interconnects. To execute gradient averaging and quantization in the optical domain, we incorporate optical devices such as Mach-Zehnder-Interferometers (MZIs) into the interconnects. Such a de facto optical neural network (ONN) can effectively reduce the communication overhead in existing distributed training solutions. To reduce dataset complexity for training this neural network, a preprocessing algorithm implemented in the optical domain is also proposed. Hardware cost is lowered by approximating the weight matrices of the optical neural network with unitary and diagonal matrices, while the accuracy is maintained by a proposed hardware-aware training algorithm. The proposed solution was evaluated on real distributed learning tasks, including ResNet50 on CIFAR-100, and a LLaMA-based network on Wikipedia-1B. In both cases, the proposed framework can achieve comparable training accuracy to the ring all-reduce baseline, while eliminating communication overhead.

Figures & tables

Fig. 1: The ring all-reduce algorithm in distributed training with four servers connected to a switch, forming a logical ring topology.

Fig. 2: The interleaving MZI array for a 4

\times

4 unitary matrix (adopted from [ 21 ] ) where an MZI consists of two DCs and two PSs.

Fig. 3: The proposed OptINC architecture connecting

N

servers,

S_{1}

S_{N}

, each with

M

full-duplex optical transceivers. The system consists of three components: a preprocessing unit

\mathbf{P}

, an ONN

\boldsymbol{f}_{\theta}

, and a splitting unit

\mathbf{T}

Fig. 4: Weight matrix

W

can be partitioned to square submatrices

W_{s}

in two ways, horizontally or vertically.

Fig. 5: The cascading topology with OptINCs in two levels, supporting up to

N^{2}

servers.

Bit Width	#Servers	ONN Structure	Layers With Matrix Approximation	Area Ratio	ONN Accuracy
8	4	4-64-128-256-128-64-4	None	100%	100%
8	4	4-64-128-256-128-64-4	All layers	39.3%	100%
8	8	4-64-128-256-512- 256-128-64-4	None	100%	100%
8	8	4-64-128-256-512- 256-128-64-4	Layers 2–7	40.9%	100%
8	16	4-64-128-256-512-1024- 512-256-128-64-4	None	100%	100%
8	16	4-64-128-256-512-1024- 512-256-128-64-4	Layers 2–9	40.4%	100%

TABLE I: Experimental results under different scenarios with various bit widths and server counts.

Fig. 6: The communication data normalized by the amount of data to be computed, for the ring all-reduce algorithm and OptINC, when 4, 8, 16 servers participate in the distributed learning.

Layers	ONN Acc. (%)	Error Values (Rel. Ratios %)	Norm. Area
4, 5, 6	100	None	49.3%
4, 5, 6, 7	99.99986	$\pm 1$ (90), $-64$ (10)	47.9%
4, 5, 6, 7, 8	99.99999	1024 (100)	47.4%
3, 4, 5, 6	99.98891	$\pm 1$ (99), $\pm 1024$ (0.9),
$-4$ (0.1)	43.7%
3, 4, 5, 6, 7	99.99936	$\pm 4$ (79.5), $-16$ (17),

TABLE II: Selected layers for matrix approximation and the training accuracy in the fourth scenario.

Fig. 7: (a) The trained accuracy and loss, and (b) the overall latency breakdown for one epoch or step of ResNet50 on CIFAR-100 and the LLaMA-based network on Wikipedia-1B.

Gradient Quantization

See all 8 papers

May 22, 2026

Sparse In-Network Learning via Shortest-Path Backpropagation and Finite-Rate Gating

In-network learning (INL) trains distributed neural modules by exchanging latent activations and backpropagated errors over a communication graph. This letter proposes Dijkstra-pruned INL (D-INL), which removes non-tree links by retaining a capacity-aware shortest-path tree rooted at the fusion node. To balance sparsity and predictive information, local routing (or aggregation) is modeled as a finite-rate stochastic gate with rate

R_g=I(Z; T)

. We derive a rate-distortion-generalization bound and validate the method on a reproducible distributed-classification experiment, where D-INL reduces training exchange by

70.4\%

while preserving accuracy within the standard deviation of dense INL. Adding finite-rate regularization further reduces the estimated latent rate by

45.7\%

relative to unregularized Dijkstra INL.

Mohammad Reza Deylam Salehi

Nice, France

Communication-Efficient Distributed Training Rate-Distortion Theory Graph Sparsification

Ask PDF

Aug 26, 2026

Distributed Training using an Intelligent Network

Distributed training across a wide area network (WAN) is challenging, as continuous parameter exchange by islands of compute is constrained by limited bandwidth, high latency, and uneven topology. We propose making the network an active participant in training. On the systems side, such networks should leverage (i) multicast technology to replicate outbound traffic and (ii) in-line FPGAs to aggregate inbound traffic, to ease egress and ingress bottlenecks. These technologies are used for training across workers within a data center, but this paper extends them to the WAN. On the algorithms side, we develop an optimization framework that produces rich synchronization schedules (namely, rotating cliques of islands) around the underlying network topology and these technologies, to maximize information exchange. Finally, we illustrate this on a nine-city topology modeled on the DoubleZero network, a live programmable WAN equipped with both technologies, and show how the optimal schedules shift with the network's capabilities. Together, these can narrow the gap to the gold standard of colocated training.

Nihar Shah, Ben Blier

DoubleZero Foundation

Distributed Training Communication-Efficient Distributed Training Distributed Optimization

Ask PDF

May 4, 2026

SignMuon: Communication-Efficient Distributed Muon Optimization

Distributed training of large neural networks is bottlenecked by full-precision gradient communication and by coordinatewise optimizers that ignore the matrix structure of weight tensors. We propose Sign-Muon, a 1-bit, matrix-aware optimizer that combines majority-vote sign aggregation from signSGD with the polar-step framework of Muon. Each worker forms a Muon-style direction by taking the polar factor of its momentum via a Newton--Schulz iteration, transmits only the entrywise signs, and aggregates by majority vote; an optional local polar step further enforces orthogonality at no extra communication cost. Under spectral-norm smoothness and bounded-variance stochastic gradients, the spectral-norm normalized sign step yields an

\mathcal{O}(1/\sqrt{T})

nonconvex rate for an

\ell_1

-based stationarity measure. With unimodal symmetric noise, majority vote across

M

workers cuts the stochastic term by

1/\sqrt{M}

, matching signSGD. In the

α

β

model, distributed Sign-Muon needs only one integer sum-allreduce per iteration; all orthogonalization is local, giving a

32\times

bandwidth reduction over float32 (

4\times

for int8). Across 330 CIFAR-10/ResNet-50 configurations Sign-Muon attains the best validation accuracy (92.15%); its 4-GPU majority-vote variant reaches 92.02% with 37% less training time at matched effective batch. On nanoGPT, Sign-Muon achieves lower perplexity and better anytime performance than other sign-based baselines, with favorable weak-scaling up to 16 GPUs.

Neel Mishra, Kushagara Trivedi, Pawan Kumar

1Microsoft · 2IIIT Hyderabad

Sign-Based Optimization Communication-Efficient Distributed Training Distributed Optimization Gradient Compression Muon Optimizer

Ask PDF

Abstract

Gradient Quantization

Explore similar work

Sparse In-Network Learning via Shortest-Path Backpropagation and Finite-Rate Gating

Distributed Training using an Intelligent Network

SignMuon: Communication-Efficient Distributed Muon Optimization