Learning Inference Concurrency in DynamicGate MLP Structural and Mathematical Justification

Yongil Choi

cs.LGApr 15, 2026

Learning Inference Concurrency in DynamicGate MLP Structural and Mathematical Justification

Authors: Yongil Choi

Organizations: Sorynorydotcom Co., Ltd./AI Open Research Lab

Abstract

Conventional neural networks strictly separate learning and inference because if parameters are updated during inference, outputs become unstable and even the inference function itself is not well defined [1, 2, 3]. This paper shows that DynamicGate MLP structurally permits learning inference concurrency [4, 5]. The key idea is to separate routing (gating) parameters from representation (prediction) parameters, so that the gate can be adapted online while inference stability is preserved, or weights can be selectively updated only within the inactive subspace [4, 5, 6, 7]. We mathematically formalize sufficient conditions for concurrency and show that even under asynchronous or partial updates, the inference output at each time step can always be interpreted as a forward computation of a valid model snapshot [8, 9, 10]. This suggests that DynamicGate MLP can serve as a practical foundation for online adaptive and on device learning systems [11, 12].

Figures & tables

Figure 1: Conceptual diagram of concurrent learning during inference

Model	Mode	DriftBefore	AdaptAcc	Recovery	CleanDrop	Flip	AR	FLOPs~	Status
Dense	A_none	36.42	-	-	-	-	-	-	SKIP(no trainable params)
Dense	B_theta_only	36.42	-	-	-	-	-	-	SKIP(mode=theta_only)
Dense	C_w_inactive_only	36.42	-	-	-	-	-	-	SKIP(mode=w_inactive_only)
Dense	D_theta_and_w_inactive	36.42	-	-	-	-	-	-	SKIP(mode=theta_and_w_inactive)
DG-Hard	A_none	43.34	-	-	-	-	-	-	SKIP(no trainable params)
DG-Hard	B_theta_only	43.34	-	-	-	-	-	-	SKIP(mode=theta_only)

Table 1: Summary of online adaptation results under drift

Model	CleanAcc	DriftAcc	AdaptAcc	Flip(drift)	AR(drift)	FLOPs~	$\theta$ params
Dense	96.17	28.48	28.56	None	1.000	1.00	0
DG-Hard	94.80	33.97	34.21	0.104	0.297	0.30	65792
DG-Soft	96.32	22.42	74.00	0.953	0.908	0.91	65792
DG-Anneal	95.58	26.67	61.60	0.996	0.913	0.91	65792
MoE-Top1	96.85	36.74	37.48	0.615	0.250	0.25	1028
MoE-Soft	96.90	39.67	44.68	0.429	1.000	1.00	1028

Table 2: Table of performance metrics for each model

Figure 2: (1) Online Adaptation Loss ( $\theta$ -only) A graph showing the loss change during online adaptation steps for each gating/MoE model. The Dense model has no trainable gate/router parameters, so it has no loss curve.

Figure 3: (2) Accuracy under Drift Before vs After $\theta$ -only Online Adaptation: A bar chart comparing how each model’s accuracy changes before and after online adaptation in a drift environment.

Figure 4: (3) Compute Proxy (FLOPs~) under Drift (lower is cheaper) A bar chart showing the computational cost proxy of each model in a drift environment. Corresponds to AR(drift).

Figure 5: (4) Routing Flip Rate under Drift (higher = less stable) A bar chart showing the stability of routing decisions in a drift environment. Higher values mean routing decisions change more frequently.

Model	Mode	Flip	AdaptAcc
DG-Hard	C_w_inactive_only	0.106602	0.7467
DG-Hard	D_theta_and_w_inactive	0.106076	0.7991
DG-Soft	B_theta_only	0.908429	0.7353
DG-Soft	C_w_inactive_only	0.925763	0.7882
DG-Soft	D_theta_and_w_inactive	0.878457	0.8447
DG-Anneal	B_theta_only	0.671266	0.6180

Table 3: Flip and AdaptAcc by model/adaptation mode

Figure 6: Correlation graph between Flip ratio and AdaptAcc

Dynamic Neural Networks

Dynamic neural networks adapt their computation, capacity, structure, or internal state to changing inputs, tasks, and resource budgets.

See all 85 papers

DynamicGate MLP Conditional Computation via Learned Structural Dropout and Input Dependent Gating for Functional Plasticity
Mar 17, 2026Yong Il ChoiDynamic Neural Networks Conditional Computation
Sorynorydotcom Co., Ltd./AI Open Research Lab
Training, learning and inference: unified dynamics of neural systems
Aug 21, 2026Mian WangNeural Network Training Dynamics
Wuhan Polytechnic University, Wuhan 430023, Hubei, China
Align Forward, Adapt Backward: Closing the Discretization Gap in Logic Gate Networks
Mar 14, 2026Youngsung KimNeural Network Optimization Differentiable Logic Gate Networks
Department of Artificial Intelligence and Department of Electrical and Computer Engineering, Inha University, Incheon, Korea.

Mar 17, 2026J/K move · Enter open · S save

DynamicGate MLP Conditional Computation via Learned Structural Dropout and Input Dependent Gating for Functional Plasticity

Yong Il Choi

Sorynorydotcom Co., Ltd./AI Open Research Lab

Dynamic Neural Networks Conditional Computation Efficient Neural Network Inference

Open paper AskPDF arXiv

Dropout is a representative regularization technique that stochastically deactivates hidden units during training to mitigate overfitting. In contrast, standard inference executes the full network with dense computation, so its goal and mechanism differ from conditional computation, where the executed operations depend on the input. This paper organizes DynamicGate-MLP into a single framework that simultaneously satisfies both the regularization view and the conditional-computation view. Instead of a random mask, the proposed model learns gates that decide whether to use each unit (or block), suppressing unnecessary computation while implementing sample-dependent execution that concentrates computation on the parts needed for each input. To this end, we define continuous gate probabilities and, at inference time, generate a discrete execution mask from them to select an execution path. Training controls the compute budget via a penalty on expected gate usage and uses a Straight-Through Estimator (STE) to optimize the discrete mask. We evaluate DynamicGate-MLP on MNIST, CIFAR-10, Tiny-ImageNet, Speech Commands, and PBMC3k, and compare it with various MLP baselines and MoE-style variants. Compute efficiency is compared under a consistent criterion using gate activation ratios and a layerweighted relative MAC metric, rather than wall-clock latency that depends on hardware and backend kernels.

Abstract

Dynamic Neural Networks

Explore similar work

DynamicGate MLP Conditional Computation via Learned Structural Dropout and Input Dependent Gating for Functional Plasticity

Training, learning and inference: unified dynamics of neural systems

Align Forward, Adapt Backward: Closing the Discretization Gap in Logic Gate Networks