DynamicGate MLP Conditional Computation via Learned Structural Dropout and Input Dependent Gating for Functional Plasticity

Yong Il Choi

cs.LGMar 17, 2026

DynamicGate MLP Conditional Computation via Learned Structural Dropout and Input Dependent Gating for Functional Plasticity

Authors: Yong Il Choi

Organizations: Sorynorydotcom Co., Ltd./AI Open Research Lab

Abstract

Dropout is a representative regularization technique that stochastically deactivates hidden units during training to mitigate overfitting. In contrast, standard inference executes the full network with dense computation, so its goal and mechanism differ from conditional computation, where the executed operations depend on the input. This paper organizes DynamicGate-MLP into a single framework that simultaneously satisfies both the regularization view and the conditional-computation view. Instead of a random mask, the proposed model learns gates that decide whether to use each unit (or block), suppressing unnecessary computation while implementing sample-dependent execution that concentrates computation on the parts needed for each input. To this end, we define continuous gate probabilities and, at inference time, generate a discrete execution mask from them to select an execution path. Training controls the compute budget via a penalty on expected gate usage and uses a Straight-Through Estimator (STE) to optimize the discrete mask. We evaluate DynamicGate-MLP on MNIST, CIFAR-10, Tiny-ImageNet, Speech Commands, and PBMC3k, and compare it with various MLP baselines and MoE-style variants. Compute efficiency is compared under a consistent criterion using gate activation ratios and a layerweighted relative MAC metric, rather than wall-clock latency that depends on hardware and backend kernels.

Figures & tables

Panel of Figure 1 — Figure 1: Conceptual comparison: Baseline / Dropout / Pruning / DynamicGate-MLP / RigL / DynamicGate - MLP + RigL

Figure 2: Conceptual diagram of the gating layer

Figure 3: Conceptual illustration of active and silent neurons

Panel of Figure 4 — Figure 4: Conceptual diagram of forward execution with hard gating and STE-based backpropagation

Panel of Figure 5 — Figure 5: Diagram of adding RigL’s dynamic rewiring to the hard-gating (STE) training path. DynamicGate controls input-dependent unit activation $g(x)$ , and RigL periodically rewires a sparse connectivity mask $m(t)$ during training to learn structural sparsity.

Data Set	Domain	Dimensionality	Training Set	Test Set
MNIST	Vision	28×28 grayscale (784)	60,000	10,000
CIFAR-10	Vision	32×32 RGB (3,072)	50,000	10,000
Tiny ImageNet	Vision	200 classes, 64x64 RGB(12,288)	100,000	10,000
Speech	Speech	2624 (=64×41)	86,843	11,305
PBMC3k	Genomics	2000 (HVG)	100,000	10,000

Table 1: Overview of datasets used in this paper

Model	Accuracy(%)	FLOPs Reduction(%)	Params	FLOPs
Baseline MLP	98.07	0.000000	203,530	406,528
Dropout MLP	98.31	0.000000	203,530	406,528
Pruned MLP	98.02	29.622314	203,530	286,105
Dynamic MLP	98.07	21.711912	204,570	318,263

Table 2: Model comparison (Accuracy / Params / FLOPs)

Panel of Figure 6 — Figure 6: Model comparison plots on MNIST

Model	Accuracy (%)	FLOPs(rel)	Notes
Baseline	43.30	1.000
Dropout	41.45	1.000
Pruned	48.90	0.941
Dynamic	43.29	0.843	OpenRate L1/L2: 0.94/0.29

Table 3: Accuracy and relative FLOPs (proxy) comparison

Panel of Figure 7 — Figure 7: Model comparison on CIFAR-10

Model	Accuracy(%)	FLOPs Reduction(%)	Params	FLOPs
B aseline MLP	3.24	0.000	3,197,384	6,393,856
D ropout MLP	2.51	0.000	3,197,384	6,393,856
P runed MLP	4.28	29.519	3,197,384	4,506,418
D ynamicMLP	2.97	80.088	3,197,640	1,273,117

Table 4: Model comparison (Accuracy / Params / FLOPs)

Item	Value
Sample rate	16 kHz
Clip length	1.0 s (fixed length, 16,000 samples)
STFT window / hop	30 ms / 10 ms
Mel bins	40
Feature	log-mel spectrogram (shape: $40\times 98$ )
Augmentation	time shift (100 ms), additive noise (optional)

Table 5: Speech Commands preprocessing parameters (reproducibility)

Model	Val Acc (best)	Test Acc	Gate open $(g_{1}/g_{2})$	RelMAC	MACs (baseline $\rightarrow$ gated)
BaselineMLP	0.8762	0.8656	–	1.000	973,824 $\rightarrow$ 973,824
DynamicGateMLP	0.8739	0.8654	0.924 / 0.820	0.989	973,824 $\rightarrow$ 963,287

Table 6: Model comparison on Speech Commands

	DynamicGateMLP		SwitchMoE_MLP
ep	active_g	corr(h,a)	entropy	cv	MI	cap
21	0.377229	-0.001672	2.044537	0.266152	0.871885	63.291139
22	0.367201	0.002379	2.043974	0.265866	0.899625	63.291139
23	0.359287	-0.006189	2.051539	0.229780	0.848144	63.291139

Table 7: Epoch-wise comparison of DynamicGateMLP vs. SwitchMoE_MLP (MNIST snapshots).

Model	Acc (%)	Macro-F1 (%)	MACs ( $\times 10^{6}$ )	MACs Red. (%)	Time (s)
Baseline	91.83 $\pm$ 1.49	93.29 $\pm$ 1.77	2.505 $\pm$ 0.000	0.00 $\pm$ 0.00	7.20 $\pm$ 0.24
Dropout	92.48 $\pm$ 1.07	94.22 $\pm$ 1.11	2.505 $\pm$ 0.000	0.00 $\pm$ 0.00	7.75 $\pm$ 0.57
Pruned	92.17 $\pm$ 1.46	93.95 $\pm$ 1.36	1.252 $\pm$ 0.000	50.00 $\pm$ 0.00	3.31 $\pm$ 0.38
DynamicGate	92.57 $\pm$ 0.89	94.18 $\pm$ 0.78	0.988 $\pm$ 0.446	60.57 $\pm$ 17.82	8.32 $\pm$ 0.26
RigL-only	93.33 $\pm$ 1.04	94.18 $\pm$ 2.06	0.629 $\pm$ 0.000	74.87 $\pm$ 0.00	9.61 $\pm$ 0.53
DynamicGate+RigL	92.43 $\pm$ 1.34	93.43 $\pm$ 2.41	0.541 $\pm$ 0.033	78.41 $\pm$ 1.33	11.24 $\pm$ 0.30

Table 8: Performance and efficiency averaged over multiple random seeds (mean

\pm

std).

Figure 8: Pareto plot of classification accuracy versus compute reduction on PBMC3K.

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

Dataset	Colab Source Code (URL)
MNIST	https://colab.research.google.com/drive/1PbKL4So4Vqel9VTN-SxFsEcHJGoo7aib?usp=sharing
CIFAR-10	https://colab.research.google.com/drive/1q8aA74ImyfW8QB55RrqImBDSOt2IUtbv?usp=sharing
Tiny ImageNet	https://colab.research.google.com/drive/19V6E8EHPjwdW-zgv1jnyxCgPiRT9_F3M?usp=sharing
SpeechCommands	https://colab.research.google.com/drive/1TZBecTZQxlEu2ME_q3Ta5mM_jgt-fQTZ?usp=sharing
PBMC3k (HVG)	https://colab.research.google.com/drive/1DffTQEO8Ctw0h8UQGL-RajQW6HtBKEjX?usp=sharing

Appendix

Table 9: Colab source code links for each dataset.

Dynamic Neural Networks

Dynamic neural networks adapt their computation, capacity, structure, or internal state to changing inputs, tasks, and resource budgets.

See all 84 papers

Learning Inference Concurrency in DynamicGate MLP Structural and Mathematical Justification
Apr 15, 2026Yongil ChoiDynamic Neural Networks Online Learning
Sorynorydotcom Co., Ltd./AI Open Research Lab
Revisiting Auxiliary Losses for Conditional Depth Routing: An Empirical Study
Apr 19, 2026Qingwei LinEfficient Neural Network Inference Conditional Computation
Independent Researcher
CausalGate: Causal Importance Distillation for Transformer Module Pruning
Jul 21, 2026Kiran Nair, Smriti Regmi, Rodrigue RizkEfficient Transformer Inference LLM Pruning
USD Artificial Intelligence Research Lab, Department of Computer Science University of South Dakota, Vermillion, SD 57069, USA

Apr 15, 2026J/K move · Enter open · S save

Learning Inference Concurrency in DynamicGate MLP Structural and Mathematical Justification

Yongil Choi

Sorynorydotcom Co., Ltd./AI Open Research Lab

Dynamic Neural Networks Online Learning Neural Network Training Dynamics

Open paper AskPDF arXiv

Conventional neural networks strictly separate learning and inference because if parameters are updated during inference, outputs become unstable and even the inference function itself is not well defined [1, 2, 3]. This paper shows that DynamicGate MLP structurally permits learning inference concurrency [4, 5]. The key idea is to separate routing (gating) parameters from representation (prediction) parameters, so that the gate can be adapted online while inference stability is preserved, or weights can be selectively updated only within the inactive subspace [4, 5, 6, 7]. We mathematically formalize sufficient conditions for concurrency and show that even under asynchronous or partial updates, the inference output at each time step can always be interpreted as a forward computation of a valid model snapshot [8, 9, 10]. This suggests that DynamicGate MLP can serve as a practical foundation for online adaptive and on device learning systems [11, 12].

Abstract

Dynamic Neural Networks

Explore similar work

Learning Inference Concurrency in DynamicGate MLP Structural and Mathematical Justification

Revisiting Auxiliary Losses for Conditional Depth Routing: An Empirical Study

CausalGate: Causal Importance Distillation for Transformer Module Pruning