Time-multiplexed layer reuse for physical neural networks

Kohei Tsuchiyama; Andre Roehm; Takatomo Mihana; Ryoichi Horisaki

cs.LGOct 28, 2025

Time-multiplexed layer reuse for physical neural networks

Authors: Kohei Tsuchiyama, Andre Roehm, Takatomo Mihana, Ryoichi Horisaki

Organizations: Graduate School of Information Science and Technology, The University of Tokyo, 7-3-1 Hongo, Bunkyo-ku, 113-8656, Tokyo, Japan.

Abstract

Physical neural networks (PNNs) are promising candidates for next-generation computing, but existing demonstrations remain several orders of magnitude smaller than modern digital neural networks, whose recent advances have been driven by rapid growth in trainable parameters. This situation resembles the constraints of early digital neural networks, which led to ideas around parameter reuse. We investigate what similarly efficient hardware architectures may look like, focusing specifically on the common bottleneck of slow re-adjustment of the weights in PNNs. We propose the Time-Indexed Deep Alternating Layers Network (TIDAL-Net), which occupies an intermediate regime between recurrent and deep neural networks, specifically aimed at the scales and restrictions of common PNN prototypes. TIDAL-Net leverages the timescale separation found in many PNNs between fast forward dynamics and slowly trainable weights and biases, using layer-by-layer time multiplexing to increase effective depth while limiting implementation cost. Numerical experiments on image classification and natural language processing tasks show that TIDAL-Net improves performance with only minor modifications to conventional PNNs.

Figures & tables

Figure 1: Representative reported model scales of neural networks implemented on conventional digital electronic computers and of physical neural networks (PNNs). Reference numbers are shown next to the corresponding data points, and marker colors and shapes indicate the implementation platforms. For digital models, the plotted value is the number of trainable scalar parameters reported by the original authors or developers. For PNNs, however, a unique parameter count that is independent of the architecture and physical platform is generally not available. We therefore use the reported or estimated number of trainable or programmable degrees of freedom when such a quantity can be identified. When no direct parameter count is defined, we use the closest platform-specific model-size proxy that can be derived from the original publication, such as the number of controllable physical elements, physical nodes, or virtual nodes. The exact quantity, source, and calculation used for each data point are provided in Supplemental Information Section S8. Consequently, the plotted quantity should not be interpreted as an architecture- or platform-independent measure of expressivity, representational capacity, task performance, or hardware efficiency. The figure is intended only as a qualitative comparison of the reported implementation scales of digital and physical neural-network systems.

Figure 2: Concepts of “time-multiplexing”.

\mathbf{a}

: While the state

h_{l}

in a conventional network evolves when passing through distinct layers

l

, in a time-multiplexed system the state

h[t]

evolves over several time steps

t

(see Eq. ( 3 )).

\mathbf{b}

: By reusing the same nonlinear element, but reconfiguring the network weights

W_{\rm{hh}}[t]

(and biases

b[t]

) at each step

t

, we can reproduce the dynamics of a DNN (Eq. ( 3 )).

\mathbf{c}

: If the matrix

W_{\rm{hh}}

(and bias) is fixed (independent of

t

), the system implements a RNN (Eq. ( 5 )).

\mathbf{d}

: The network weight

W_{\rm{hh}}

and biases

b[t]

are switched periodically at every time step for the TIDAL-Net architecture proposed in Methods.

Figure 3: Time-Indexed Deep Alternating Layers Network (TIDAL-Net). a : The forward propagation switches between separate sets of hidden layers with different parameters

\theta=(W_{\rm{hh}},b_{\rm{h}})

, see Eq. ( 6 ). This enables the hardware-efficient reuse of many components while only requiring high-speed switches (parameter reconfiguration can happen at much slower speeds). b : If we consider how layers are chained temporally, this forms a deep neural network. The total depth is given by the number of repetitions

L_{\rm{T}}

. c, d, e : Examples of proposed implementations for PNN components: c : Micro-Ring Resonator (MRR), Mach-Zehnder Interferometer (MZI), and Magnetic Tunnel Junction (MTJ) crossbar arrays for trainable parameters; d : Digital Mirror Devices (DMDs) Hornbeck (1997) , fiber switches Friberg et al. (1988) , and current switches for switching; e : photodetectors or electrical processing for the nonlinear activation function.

Figure 4: Examples of TIDAL-Net. From left to right, the cases are:

(L_{\rm{W}},L_{\rm{T}})=(1,2),(2,4),(4,7)

. The gray solid line represents the hidden states, while the colored arrows inbetween represent the parameters of the transformations (weights and biases). Shared colors indicate that parameters were reused, whereas differing colors indicate unique parameters.

L_{\rm{T}}

corresponds to the number of layers and

L_{\rm{W}}

corresponds to the number of distinct colors of the arrows, i.e., parameter sets.

Figure 5: TIDAL-Net performance on the image classification task. the number of distinct parameter banks $L_{\mathrm{W}}$ is changed while keeping the number of layers $L_{\mathrm{T}}=12$ fixed. $\mathbf{a}$ : Error Rate and Parameter Footprint when varying $L_{\rm{W}}$ . Blue dots and lines show the error rate, and orange bars indicate the hidden-layer parameter footprint defined in Methods. The dashed line is the performance of the stateless RNN limit for $L_{\rm{W}}=1$ . $\mathbf{b}$ : Illustration of TIDAL-Net architectures. Gray bars represent the hidden layers, and arrows indicate hidden weights. (Left) Reusing the same hidden weight. ( $L_{\mathrm{W}}=1$ ) (Middle) Repeating two weights. ( $L_{\mathrm{W}}=2$ ) (Right) Using a distinct hidden parameter bank at every time step ( $L_{\mathrm{W}}=12$ ) TIDAL-Net $L_{\mathrm{W}}\geq 2$ cases are better than RNN $L_{\mathrm{W}}=1$ cases. Especially, with a few additional layers, TIDAL-Net can outperform RNNs.

Figure 6: TIDAL-Net performance on a natural language processing task. the number of distinct parameter banks

L_{\rm{W}}

is changed while keeping the number of layers

L_{\rm{T}}=12

constant. Blue dots and lines show the Error Rate, and orange bars show the number of trainable parameters, corresponding to the hardware footprint. The gray dashed line is the performance of the stateless RNN limit for

L_{\rm{W}}=1

. Especially, with a few additional layers, TIDAL-Net can outperform RNNs.

Figure 7: TIDAL-Net performance heatmap on the SVHN classification when the width of hidden layers is fixed. a : Error. b : Loss. The number of distinct layers $L_{\mathrm{W}}$ is varied along the y-axis, while the number of time steps $L_{\mathrm{T}}$ is varied along the x-axis. By definition, $L_{\mathrm{T}}<L_{\mathrm{W}}$ is not a valid setting, and gray cells indicate such impossible configurations. c : Extracted results from a . The number of time steps $L_{\mathrm{T}}$ is varied along the x-axis while keeping the number of distinct parameter banks $L_{\mathrm{W}}$ fixed. d : Also extracted results from a . the number of distinct parameter banks $L_{\mathrm{W}}$ is varied along the x-axis while keeping the number of time steps $L_{\mathrm{T}}$ fixed. Generally speaking, large $L_{\mathrm{W}}$ results in better performance. The best performance for a fixed $L_{\mathrm{W}}$ is, in general, not the case when $L_{\mathrm{W}}=L_{\mathrm{T}}$ . This means that repetitive usage can enhance the performance.

Figure 8: Physical inference pipeline for TIDAL-Net. A finite set of $L_{\mathrm{W}}$ physical parameter banks $\theta_{k}=\{W_{\mathrm{hh}}^{(k)},b_{\mathrm{h}}^{(k)}\}$ is programmed before inference and reused across logical layers. At time step $t$ , the current bank $\theta_{\sigma(t)}$ is used for the physical matrix–vector multiplication, with latency $T_{\mathrm{MM}}$ . After the matrix–vector multiplication, selection or preparation of the parameter bank for the next step, $\theta_{\sigma(t+1)}$ , proceeds in parallel with the remaining state-update operations. The state-update block includes detection and optional analog-to-digital conversion, nonlinear activation, signal remodulation, residual/state combination, and buffering or delay, with a total latency $T_{\mathrm{state}}$ . The next logical layer can start once both the next parameter bank and the updated hidden state are ready. Under this coarse-grained scheduling model, the per-step latency is therefore $T_{\mathrm{step}}=T_{\mathrm{MM}}+\max(T_{\mathrm{swt}},T_{\mathrm{state}})$ , where $T_{\mathrm{swt}}$ denotes the parameter-bank switching or selection time. ADC/DAC stages are implementation dependent and may be omitted in fully analog implementations. The input projection and other auxiliary components are omitted from the schematic for clarity.

Platform	Weight-bank scale / representation	Weight update time $T_{\mathrm{upd}}$	Bank switching $T_{\mathrm{swt}}$	MVM / propagation $T_{\mathrm{MM}}$	State overhead $T_{\mathrm{state}}$
MZI mesh	$4\times 4$ academic ONN; $64\times 64$ industrial prototype; continuous phase / amplitude Shen et al. (2017) ; Ramey (2020)	$\sim 4.7\,\mathrm{\mu s}$ thermal tuning time constant for a representative thermo-optic element Duan et al. (2021)	External switching / routing among preconfigured banks	$\sim 200\,\mathrm{ps}$ MVM Ramey (2020)	OEO conversion (PD + MRM): $1.49\,\mathrm{GHz}$ , up to $4\,\mathrm{Gb/s}$ ; ADC/DAC not required in this implementation Arahata et al. (2026)
Microring resonator array	16 tunable MRR weights in a 4-node chip; analog wavelength-dependent weighting Tait et al. (2017)	$\sim 4\,\mathrm{\mu s}$ thermal tuning in a representative microring-microheater implementation Atabak and others (2010)	External optical / electrical bank selection	$\sim 48\,\mathrm{ps}$ optical-path delay; GHz-class signal processing Tait et al. (2017)	OEO conversion (Ge PD + MRM): $1.49\,\mathrm{GHz}$ , up to $4\,\mathrm{Gb/s}$ ; ADC/DAC not required in this implementation Arahata et al. (2026)
Spatial light modulator (SLM)	$\sim 2\times 10^{5}$ trainable parameters in a reconfigurable free-space ONN; analog phase / amplitude Xia et al. (2023)	$\sim 17\,\mathrm{ms}$ update of the modulation pattern encoding the weights (60 Hz LCOS-SLM) Xue and others (2024)	No separate selection; operator change requires pattern update	$\sim 1.3\,\mathrm{ns}$ optical propagation Xue and others (2024)	$42\,\mathrm{Hz}$ CMOS detection ( $\sim 24\,\mathrm{ms/frame}$ ), 8-bit readout Xue and others (2024)
Digital micromirror device (DMD)	$\sim 1.31\times 10^{6}$ binary elements ( $1280\times 1024$ DMD); binary amplitude Hornbeck (1997)	Up to $32\,\mathrm{kHz}$ ( $\sim 31\,\mathrm{\mu s}$ per binary modulation pattern) Ayoub and Psaltis (2021)	No separate selection; operator change requires pattern update	$\sim 2\,\mathrm{ns}$ free-space propagation in a representative DMD-based system Blanche and others (2014)	InGaAs photodiode + TIA + 12-bit ADC (up to $2\,\mathrm{MSample/s}$ ) Cox et al. (2019)
Nonvolatile photonic weights	$16\times 16$ fabricated PCM matrix; analog or multilevel nonvolatile weights Feldmann and others (2021)	$\sim 282\,\mathrm{ns}$ write; $\sim 556\,\mathrm{ns}$ erase for an electrically programmed GST cell Zhou and others (2023)	Passive WDM / external routing	$>14\,\mathrm{GHz}$ photonic-compute bandwidth; passive WDM routing Feldmann and others (2021) ; Zhou and others (2023)	Photodetection + $1.25\,\mathrm{MSample/s}$ DAQ; ReLU implemented digitally Zhou and others (2023)

Table 1: — Comparison of representative physical platforms for implementing reusable parameter banks. The listed quantities correspond to the main operations in the inference schedule of Fig. 8 , including physical bank scale and representation, programming/tuning, bank selection, Matrix-Vector-Multiplication (MVM)/propagation, and conversion/nonlinear-processing overhead. The values are drawn from representative experimental demonstrations and are used to identify characteristic platform trade-offs rather than to specify a single integrated system. Benchmark-specific mapping is discussed in the main text, with further timescale details in Supplemental Information Section S10.

Appendix figures & tables12 assets

Supplementary material from the paper’s appendix.

Appendix

Supplementary Fig. S9: —Performance of weight mixing on TIDAL-Net. $\mathbf{a}$ : Error Rate on SVHN image classification tasks. Both the mixing model (solid blue plot) and the periodic switching model (dot blue plot) results are similar. Orange bars show the number of trainable parameters. $\mathbf{b}$ : Weight mixing coefficient $s_{k}[t]$ values. The coefficient distributions loosely resemble periodic switching due to the seeding, but clearly includes additional mixing among weights.

Supplementary Fig. S10: TIDAL-Net performance on SVHN classification under a fixed hidden-parameter budget. a , Classification error when the number of distinct weight banks $L_{\mathrm{W}}$ and the number of temporal steps $L_{\mathrm{T}}$ are varied. Configurations with $L_{\mathrm{W}}>L_{\mathrm{T}}$ are not valid and are shown as gray cells. b , Schematic examples of model structures on the diagonal $L_{\mathrm{W}}=L_{\mathrm{T}}$ . Because the hidden-layer parameter budget is fixed, increasing $L_{\mathrm{W}}$ requires reducing the hidden dimension $H$ of each weight bank. c , Classification error as a function of $L_{\mathrm{T}}$ for fixed $L_{\mathrm{W}}$ . For a fixed hidden-parameter budget, a smaller $L_{\mathrm{W}}$ corresponds to wider hidden layers and generally gives better performance. The best performance is often obtained at $L_{\mathrm{T}}>L_{\mathrm{W}}$ , indicating that temporal repetition can improve performance without increasing the number of trainable hidden-layer parameters. d , Classification error as a function of $L_{\mathrm{W}}$ for fixed $L_{\mathrm{T}}$ . Increasing $L_{\mathrm{W}}$ generally worsens performance because each weight bank becomes narrower under the fixed-budget constraint. Larger $L_{\mathrm{T}}$ values tend to perform better than smaller ones, further supporting the contribution of temporal repetition.

# different layers $L_{\rm{W}}$	1	2	3	4	5	6	7	8
hidden nodes per layer $H$	192	128	104	91	80	73	68	64
# trainable parameters	37057	32897	32761	33489	32401	32413	32845	33281

Appendix

Supplementary Table S2: The number of hidden nodes used in Fig. S10 . Here, the trainable parameters consist of the network weights

W_{\rm{hh}}

and bias terms

b_{\rm{h}}

. This excludes the input-hidden network

W_{\rm{xh}}

, hidden-output network

W_{\rm{hy}}

and residual parameters

\alpha

. Note: Total parameter counts fluctuate because the dimension of hidden nodes needs to be an integer.

Supplementary Fig. S11: —TIDAL-Net performance with Oscillator Network performance. $\mathbf{a}$ : Error Rate $\mathbf{b}$ : Test Loss (Crossentropy) Like TIDAL-Net with ReLU, the error and loss (blue plot) decrease as the number of parameters (orange bars) increase. This means that TIDAL-Net with Oscillator Network has the same advantages that TIDAL-Net with ReLU has.

Supplementary Fig. S12: —TIDAL-Net performance on image classification tasks with MNIST. The number of weights $L_{\mathrm{W}}$ is changed while keeping the number of layers $L_{\mathrm{T}}$ constant. $\mathbf{a},\mathbf{b}$ Error Rate and Parameter Footprint. Blue curves show the mean performance, error bars indicate the standard deviation across trials, and orange bars show the number of trainable hidden-layer parameters, corresponding to the hardware footprint. The gray dashed line is the performance of the stateless RNN limit for $L_{\mathrm{W}}=1$ . While TIDAL-Net can outperform standard RNNs with additional layers like Fig. 5 on the errors, the performance increase is small. Especially, the error bars are too large to distinguish the performance.

	stateless RNN	TIDAL-Net	time-variable DNN
input/output weight	fixed	fixed	fixed
network weight	fixed	periodic switching	dynamical switching

Appendix

Supplementary Table S3: —Comparison among RNNs. TIDAL-Net exists between a stateless RNN and a time-variable DNN.

Supplementary Fig. S13: — Epochs vs TIDAL-Net performance on NLP. Test Loss changes as epoch increases. The loss values generally decrease as training progresses. The loss learning curves flatten out at higher values as

L_{\rm{W}}

increases, which is likely caused by the effect of gradient vanishing/exploding.

Model	SVHN Accuracy	Input	Hidden	Output	Other
Fit-DNN ( $N=100$ , $L=2$ ) [ 48 ]	78.9%	102,500	10,100	1,010	-
Standard RNN	81.2%	196,672	4,160	650	1
TIDAL-Net ( $L_{\rm{W}}=12$ )	86.1%	196,672	49,920	650	1
Linear SVM with HOG [ 69 ]	85.0%	-	-	-	1765
Human [ 69 ]	98.0%

Appendix

Supplementary Table S4: —SVHN classification performance comparison. TIDAL-Net achieves reasonable performance among other previous approaches. Models have different architectures separated into different components. “Input (Hidden, Output)” expresses the number of parameters in the input weight (hidden layers, output weight) as a multilayer perceptron. “Other” expresses the number of parameters coming from other components, such as the strength of residual connections. In addition, there are blanks in Human because we do not know how many parameters are used in the Human brain for image classification.

epoch	$L_{\rm{W}}=1$	$L_{\rm{W}}=4$	$L_{\rm{W}}=8$	$L_{\rm{W}}=12$
$1$	aiuer rol ruitam r-	me sous the the the	tttttttttttttttttttt	are
$100$	and thoopple they so	and the state the st	akiccxeanxtny top th	and the state the st
$200$	and thee the king pe	and the stranger the	akicletn tay tpofray	my lord, and the cou

Appendix

Supplementary Table S5: —Output examples on NLP tasks. TIDAL-Net outputs when using TIDAL-Net as a generative model with the input prompt “to be, or not to be,” from the Fig. 6 setting. As training progresses, more coherent continuations appear. Note, that our models are far smaller than even GPT-2 and thus cannot be expected to produce full sentences consistently.

Architecture	Model	Year	Parameters
Electronic Computers	AlexNet [ 25 ]	2012	60,000,000
	VGG-19 [ 2 ]	2014	144,000,000
	GPT-1 [ 26 ]	2018	117,000,000
	BERT-Large [ 5 ]	2019	340,000,000
	Megatron-LM [ 27 ]	2019	8,300,000,000
	GPT-2 [ 28 ]	2020	1,500,000,000

Appendix

Supplementary Table S6: — The number of trainable parameters shown in Fig. 1 . The numbers of parameters shown here are based on what the original papers claim or our reasonable estimate of the system from information they supply.

Supplementary Fig. S14: —Parameter scaling comparison between CNN and TIDAL-Net. Normalized number of trainable parameters required to reach target error rates of

20\%

18\%

, and

16\%

on image classification. For each model family, the required parameter count is normalized by the parameter count required to reach 20% error. This comparison indicates that TIDAL-Net achieves a competitive accuracy–parameter trade-off while using a temporal weight-reuse mechanism. “NR” denotes “not reached”, indicating that the model did not reach the corresponding target error rate within the tested parameter range.

Time	Component	Main limiting process	Timescale
$T_{\mathrm{upd}}$	MZI phase shifters	Full weight reconfiguration, e.g., thermo-optic tuning	$\mathrm{\mu s}$ – $10\,\mathrm{\mu s}$ or longer
$T_{\mathrm{swt}}$	Optical/electrical switch	Weight-bank selection	sub- $\mathrm{ns}$ – $\mathrm{\mu s}$
$T_{\mathrm{MM}}$	MZI mesh	Optical matrix multiplication through the interferometer mesh	$\mathrm{ps}$ -scale
$T_{\mathrm{state}}$	State-update block	Photodetection, amplification, modulation, nonlinear activation, and optical combining	$\mathrm{ps}$ – $10\,\mathrm{ns}$

Appendix

Supplementary Table S7: Computing-time components for an MZI-based photonic recurrent system. Representative latency components used in the coarse-grained timing model. Here,

T_{\mathrm{upd}}

is the latency of full MZI-weight reconfiguration,

T_{\mathrm{swt}}

is the latency of switching among preconfigured weight banks,

T_{\mathrm{MM}}

is the latency of the MZI-based matrix-multiplication block, and

T_{\mathrm{state}}

is the latency of the subsequent state-update block. The listed timescales are representative values and depend on the device implementation [ 60 , 55 , 56 , 85 , 9 , 61 ] .

Explore similar work

Multi-layer MIMO Relay as Deep Physical Neural Networks: Power Amplifiers as Activation Functions
Jul 20, 2026Meng Hua, Itsik Bergel, Deniz GündüzMultiple-Input Multiple-Output Activation Functions
Department of Electrical and Electronic Engineering, Imperial College London, London SW7 2AZ, U.K. · Faculty of Engineering, Bar-Ilan University, Ramat Gan 5290002, Israel
Fully Analog Resonant Recurrent Neural Network via Metacircuit
Apr 19, 2026Zixin Zhou, Tianxi Jiang, Menglong Yang +3Neuromorphic Computing Analog
Institute of Humanoid Robots, CAS Key Laboratory of Mechanical Behavior and Design of Materials, Department of Precision Machinery and Precision Instrumentation, University of Science and Technology of China, Hefei 230026, People’s Republic of China · State Key Laboratory of Mechanical System and Vibration, Shanghai Jiao Tong University, Shanghai 200240, People’s Republic of China
Physics-Modeled Neural Networks
May 5, 2026Raul Felipe-Sosa, Angel Martin del Rey, Maria Flores CeballosNeural Dynamics Neural Ordinary Differential Equations
aDepartamento de Matemáticas, Escuela Politénica Superior, Universidad Nebrija, 28015 Madrid, Spain · bDepartment of Applied Mathematics, IUFFyM, Universidad de Salamanca, 37008 Salamanca, Spain · cUnidad de Excelencia en Luz y Materia Estructuradas (LUMES), Universidad de Salamanca, 37008 Salamanca, Spain

Jul 20, 2026, cs.LGJ/K move · Enter open · S save

Multi-layer MIMO Relay as Deep Physical Neural Networks: Power Amplifiers as Activation Functions

Meng Hua, Itsik Bergel, Deniz Gündüz

Department of Electrical and Electronic Engineering, Imperial College London, London SW7 2AZ, U.K. · Faculty of Engineering, Bar-Ilan University, Ramat Gan 5290002, Israel

Multiple-Input Multiple-Output Activation Functions Channel State Information

Open paper AskPDF arXiv

Wireless physical neural networks (WPNNs) embed neural computation directly into analog hardware, offering lower energy consumption and latency than conventional digital implementations. In this paper, we propose a deep WPNN in which nonlinear activations are realized by a multi-hop multiple-input multiple-output (MIMO) relay network, in which each relay implements a trainable complex linear gain and bias, followed by the power amplifier's intrinsic nonlinearity acting as an activation function. The cascade of multiple relays therefore realizes an over-the-air fully connected network whose parameters can be trained end-to-end. We develop two transceiver designs for different channel state information (CSI) availability scenarios: a least squares (LS)-based scheme requiring only receiver-side CSI, and a singular-value-decomposition (SVD)-based scheme requiring both transmitter-side and receiver-side CSI. Simulation results show that the proposed architecture enables accurate over-the-air inference for image classification. In particular, the results highlight the advantage of exploiting hardware nonlinearity for enhanced inference capability.

Abstract

Explore similar work

Multi-layer MIMO Relay as Deep Physical Neural Networks: Power Amplifiers as Activation Functions

Fully Analog Resonant Recurrent Neural Network via Metacircuit

Physics-Modeled Neural Networks