Dynamic Neural Networks

Follow this topic Ask about this topic

Momentum

4 papers in the last four weeks, against 2 the four weeks before. 0.0% of all new papers.

Jul 13Week of Sep 28

Latest papers 56

May 24, 2026q-bio.NC

Growing a Neural Network in Breadth, Depth, and Time

Spatial and temporal resource constraints are critical for both biological and artificial intelligent systems. Here we define differentiable cost terms for breadth, depth, and time within a recurrent convolutional neural network conceived as a finite subset of an infinite lattice. We optimize these costs jointly with task errors via backpropagation. We set different pressures on breadth, depth, and time, which leads to diverse computational graphs emerging organically through training. We find that all three resources can be traded off against each other to achieve a given level of accuracy. Networks grow in all three dimensions with task complexity and spontaneously take more recurrent steps when inputs are occluded. Surprisingly, time used by the model correlates with human reaction times in an object recognition task. Our framework provides a normative account of how resource constraints shape neural architectures, connecting to questions about brain design in neuroscience, and may help illuminate the diversity of neural solutions found in nature.

Eivinas Butkus, Kedar Garzón Gupta, Nikolaus Kriegeskorte

1Columbia University · 2NSF AI Institute for Artificial and Natural Intelligence

Convolutional RNNs Dynamic Neural Networks Neural Network Optimization Recurrent Neural Networks Convolutional Neural Networks Adaptive Computation

May 22, 2026cs.NE

SpikingMoE: SDPrompt-Guided Dynamic Expert Fusion in Spiking Neural Networks

Spiking Neural Networks (SNNs) provide an energy-efficient paradigm for visual recognition. We present SpikingMoE, which integrates a spike-driven Transformer with a Mixture-of-Experts (MoE) framework for dynamic computation. Inspired by the lateral geniculate nucleus (LGN), a spike-driven prompt (SDprompt) enables input-dependent expert routing in a biologically plausible manner. By replacing standard MLPs with spike-compatible expert modules and enforcing binary spike communication, SpikingMoE is designed for neuromorphic hardware. Experiments on CIFAR-10 and CIFAR-100 achieve 94.09% and 74.54% top-1 accuracy, showing that modular expert routing can be incorporated while retaining reasonable performance. To our knowledge, SpikingMoE is the first open-source SNN framework that integrates MoE into a spike-driven Transformer with LGN-inspired routing.

Yukai Yang, Chenxi Qin, Jungang Li +3

Tianjin University, Medical School, Tianjin 300072, China · Southern University of Science and Technology, Department of Computer Science and Engineering, Shenzhen, Guangdong 518055, China · Chinese Academy of Medical Sciences, Institute of Biomedical Engineering, Tianjin 300192, China

Spiking Transformers Dynamic Neural Networks Spiking Neural Networks Neuromorphic Computing Mixture-of-Experts Models

May 20, 2026cs.LG

Winfree Oscillatory Neural Network

Oscillations and synchronization are widely believed to play a fundamental role in representation and computation. However, existing machine learning approaches based on synchronization dynamics have largely been confined to specialized settings such as object discovery, with limited evidence of scalability to standard vision benchmarks or logic reasoning tasks. We propose the Winfree Oscillatory Neural Network (WONN), a dynamical neural architecture based on generalized Winfree dynamics. WONN evolves representations on the torus

(S^1)^d

through structured oscillatory interactions, combining phase-based inductive biases with flexible and hierarchical interaction mechanisms instantiated as either fixed trigonometric mappings or learnable neural networks. We evaluate WONN on image recognition and complex reasoning tasks, including CIFAR, ImageNet, Maze-hard, and Sudoku. Across these domains, WONN achieves competitive or superior performance with strong parameter efficiency. In particular, WONN is, to our knowledge, the first synchronization-based oscillatory architecture to scale competitively to ImageNet-1K. Furthermore, on Maze-hard, WONN achieves 80.1% accuracy using only 1% of the parameters of prior state-of-the-art models. These results suggest that structured oscillatory dynamics provide a scalable and parameter-efficient alternative to conventional neural architectures.

Jiawen Dai, Yue Song

3Shanghai Jiao Tong University · 1Shanghai Qi Zhi Institute · College of AI, Tsinghua University

Dynamic Neural Networks Oscillatory Neural Networks Image Classification

May 19, 2026cs.LG

TIDE: Asymmetric Neural Circuits for Stabilized Temporal Inhibitory-Excitatory Dynamics

Recent Continuous Thought Machine architecture decouples internal computation from external inputs via neural dynamics, but relies on multi-layer perceptrons without stability guarantees. We propose to model neural dynamics using asymmetric Excitatory-Inhibitory (E-I) networks, which can be stabilized via principles from network theory and can be expressed as energy-based systems optimized through a game-theoretic loss. Building on this perspective, we introduce Temporal Inhibitory-Excitatory Dynamic Engine (TIDE), a neuro-inspired architecture that computes internal representations through neural dynamics stabilized by incorporating the Wilson-Cowan dynamics and lateral inhibition. TIDE balances biological realism by, for instance, using Hierarchical Receptive Fields and enforcing Dale's principle to ensure a realistic

80:20

E-I balance ratio with an end-to-end trainable architecture. The aim of this paper is to introduce a new architecture that brings neuro-inspired learning to the forefront. We present proofs of convergence, stability, and complexity bounds, along with empirical ablation studies. Overall, TIDE surpasses CTM with under

50\%

of the training time and improves

\texttt{top-1}

accuracy by an average of

+1.65\%

on ImageNet under various perturbations.

Alexander Kyuroson, Denis Kleyko, Marcus Liwicki

Luleå University of Technology, Luleå, Sweden · Örebro University & RISE Research Institutes of Sweden

Dynamic Neural Networks Neural Processes

May 14, 2026cs.LG

On the Stability of Growth in Structural Plasticity

Standard deep-learning pipelines usually choose the network architecture before training and keep it fixed throughout optimization. In contrast, a model can also be adapted by editing its structure during training, for example by pruning existing hidden-neuron units or growing new ones. Although growth is appealing for adaptive and continual systems, we show that it is not simply the inverse of pruning. Pruning selects among units that have participated in training from the start, whereas growth inserts new units into an already specialized optimization trajectory. We isolate this insertion problem and show that newborn units are often forward-active but backward-starved: they participate in the forward computation, yet receive much weaker gradient signal than incumbent units. This disadvantage is minor in small MLP benchmarks, but becomes clear in harder image-classification settings with a convolutional trunk. In these settings, \textsc{Grow} can achieve high final accuracy during the structural-editing procedure, while \textsc{Prune} is stronger when performance is averaged over the training trajectory or when the final sparse network is retrained from scratch. Interventions targeting optimizer state, insertion, selection, and trainability show that improving the integration of newborn units can improve adaptive performance, but does not automatically produce better final subnetworks. In continual-learning benchmarks stressing plasticity loss, \textsc{Grow} becomes competitive mainly when new units have enough time to integrate. Together, these results suggest that \textsc{Grow} should be evaluated not only as an architecture-search operator, but as a time-sensitive optimization process whose success depends on insertion stability.

Lute Lillo, Nick Cheney

University of Vermont

Dynamic Neural Networks Neural Network Training Dynamics Neural Network Pruning

May 12, 2026cs.CV

Trajectory-Aware Adaptive Inference in Object Detection Models

The increasing integration of sensors in autonomous maritime navigation has led to large-scale multimodal datasets, raising challenges in achieving efficient real-time perception. In such systems, object detection and trajectory perception of nearby vessels are tightly coupled, particularly in dynamic environments such as maritime navigation. However, the efficiency of object detection models during inference remains an often-overlooked aspect. To this end, we build upon an existing object detection framework by incorporating GPS trajectory data into the inference process to enable input-adaptive computation. Specifically, we introduce an early-exit mechanism in a YOLOv8-based detector that incorporates motion cues - such as inter-vessel distances. Frames of vessels that are separated by short distances, converging with high speed, are processed using the full model, while only a subset of the network's architecture is activated otherwise. The difficulty degree (or scene complexity) of a frame or set of frames per second is evaluated by leveraging inter-object distance and the rate at which the distance between them decreases. Experimental results demonstrate that this strategy maintains satisfactory detection performance while significantly reducing inference time and computational cost, thus enabling a flexible trade-off between accuracy and efficiency compared to full-model inference.

Grigorios Papanikolaou, Ioannis Kontopoulos, Giannis Spiliopoulos +2

Department of Electrical and Computer Engineering, National Technical University of Athens, Greece · Department of Product and Systems Design Engineering, University of the Aegean, Syros, Greece

Adaptive Inference Dynamic Neural Networks Object Detection Real-Time Object Detection Early-Exit Neural Networks

May 10, 2026cs.CV

AnyDepth-DETR/-YOLO: Any-depth object detection with a single network

Modern object detectors are static, fixed-depth networks optimized for a single operating point, requiring separate models for different deployment scenarios. We present an any-depth detection framework that enables a single network to span a continuous range of accuracy--efficiency trade-offs by controlling depth at inference time without retraining. Each backbone and neck stage is divided into an essential path, which always executes, and a skippable refinement path; this decomposition preserves the full multi-scale feature hierarchy at every depth configuration, unlike conventional early exiting that discards entire stages. To train such a network, jointly optimizing many sub-networks of varying depth introduces conflicting gradient signals. We address this via self-distillation between only the two extremes, with prediction-level and feature-level alignment losses that enforce stage-wise modularity, ensuring the outputs of each stage remain compatible regardless of the paths taken. Instantiated on RT-DETR and YOLOv12, our full-depth configurations match or surpass their respective SOTA baselines with negligible parameter overhead, while the most efficient configurations achieve up to

1.82\times

speedup at a cost of only 2.0 AP, all from a single set of weights.

Woochul Kang, Hyungseop Lee, Jiho Lee

Incheon Nat’l Univ.

Efficient Neural Network Inference Dynamic Neural Networks Efficient Object Detection Object Detection Real-Time Object Detection

May 8, 2026cs.CV

Amortized-Precision Quantization for Early-Exit Vision Transformers

Vision Transformers (ViTs) achieve strong performance across vision tasks, yet their deployment with low-precision early exiting remains fragile. Existing quantization methods assume static full-depth execution, making them unstable when exit decisions are perturbed by quantization noise, which can amplify errors along dynamic inference paths. In this paper, we introduce Amortized-Precision Quantization (APQ), a utilization-aware formulation that accounts for layer-wise stochastic exposure to quantization noise and reveals depth-precision trade-offs. Building on APQ, we propose Mutual Adaptive Quantization with Early Exiting (MAQEE), a bi-level framework that jointly optimizes exit thresholds and bit-widths under explicit risk control to improve inference stability. MAQEE establishes a superior Pareto frontier in the accuracy-efficiency trade-off, reducing BOPs by up to 95% while maintaining accuracy and outperforming strong baselines by up to 20% across classification, detection, and segmentation tasks.

Rui Fang, Hsi-Wen Chen, Ming-Syan Chen

National Taiwan University

Mixed-Precision Quantization Dynamic Neural Networks Vision Transformer Efficient ViTs Neural Network Quantization Early-Exit Neural Networks

May 7, 2026cs.LG

Perceive, Route and Modulate: Dynamic Pattern Recalibration for Time Series Forecasting

Local temporal patterns in real-world time series continuously shift, rendering globally shared transformations suboptimal. Current deep forecasting models, despite their scale and complexity, rely on fixed weight matrices applied uniformly to all temporal tokens. This creates a static pattern response: models settle into a compromised average, unable to adapt to changing local dynamics. We introduce Dynamic Pattern Recalibration (DPR), a backbone-agnostic mechanism that resolves this via token-level recalibration. Through a lightweight "Perceive-Route-Modulate" pipeline, DPR computes a soft-routing distribution over a learned basis of adaptive response patterns, generating a time-aware modulation vector that recalibrates hidden states via a residual Hadamard product. As a backbone-agnostic adapter, DPR enhances forecasting across diverse architectures with minimal overhead, confirming it addresses a general bottleneck. As a minimalist standalone model, DPRNet achieves competitive performance across 12 benchmarks, validating dynamic recalibration against macroscopic parameter scaling.

Siru Zhong, Zhao Meng, Haohuan Fu +3

The Hong Kong University of Science and Technology (Guangzhou) · Tsinghua University · Squirrel Ai Learning

Dynamic Neural Networks Time Series Forecasting Non-Stationary Time Series Forecasting

May 6, 2026cs.LG

Balancing Stability and Plasticity in Sequentially Trained Early-Exiting Neural Networks

Early-exiting neural networks enable adaptive inference by allowing inputs to exit at intermediate classifiers, reducing computation for easy samples while maintaining high accuracy. In practice, exits can be trained sequentially by incrementally adding them to a shared backbone; however, this sequential training can cause newly introduced exits to interfere with previously learned ones, degrading the performance of earlier classifiers. We address this problem by retaining the knowledge embedded in existing exits while allowing new ones to specialize. We propose two alternative approaches that operate at different levels of the model. The first constrains learning by protecting parameters that are important for previously trained exits, while the second preserves the output distributions of earlier exits as the network adapts. These alternatives directly reflect the stability-plasticity trade-off studied in continual learning. Accordingly, we leverage \textit{Elastic Weight Consolidation} to constrain critical weights and \textit{Learning without Forgetting} to preserve output distributions. Experiments on standard benchmarks show that our approaches consistently improve early-exit performance, achieving higher accuracy over existing sequential training methods and significant performance speedups at low computational budgets.

Alaa Zniber, Ouassim Karrakchou, Mounir Ghogho

⋆TICLab, International University of Rabat, Morocco · ‡ School of Electronic and Electrical Engineering, University of Leeds, UK

Adaptive Inference Dynamic Neural Networks Early-Exit Neural Networks Catastrophic Forgetting

May 5, 2026cs.LG

Physics-Modeled Neural Networks

We introduce \emph{Dynamical Physics-Modeled Neural Networks} (DynPMNNs), a continuous-time deep learning architecture in which each hidden layer is defined as the solution of an ordinary differential equation. Unlike classical feed-forward networks, this approach replaces static activation functions with time-evolving dynamical systems, providing a biologically inspired interpretation of hidden-layer behavior and enabling the integration of physically meaningful models. The framework is rigorously grounded in Reproducing Kernel Banach Spaces (RKBSs), allowing DynPMNNs to be characterized as finite-dimensional solutions of an abstract training problem and revealing structural connections with standard neural networks. We present a concrete implementation based on the FitzHugh--Nagumo model for neuronal activation, where numerical ODE solvers are embedded into the computational graph via Euler-type schemes. Both network weights and dynamical parameters are trained jointly. Through experiments on the California Housing dataset, we compare DynPMNNs with Neural ODEs (NODEs) and Closed-form Continuous-Time Networks (CfCs). Despite using fewer trainable parameters, DynPMNNs achieve competitive performance. These results position DynPMNNs as a principled bridge between dynamical systems and deep learning, with promising directions for further research in expressivity, stability, and physics-based modeling.

Raul Felipe-Sosa, Angel Martin del Rey, Maria Flores Ceballos

aDepartamento de Matemáticas, Escuela Politénica Superior, Universidad Nebrija, 28015 Madrid, Spain · bDepartment of Applied Mathematics, IUFFyM, Universidad de Salamanca, 37008 Salamanca, Spain · cUnidad de Excelencia en Luz y Materia Estructuradas (LUMES), Universidad de Salamanca, 37008 Salamanca, Spain

Dynamic Neural Networks Neural ODEs

May 5, 2026cs.LG

DynaTab: Dynamic Feature Ordering as Neural Rewiring for High-Dimensional Tabular Data

High-dimensional tabular data lacks a natural feature order, limiting the applicability of permutation-sensitive deep learning models. We propose DynaTab, a dynamic feature ordering-enabled architecture inspired by neural rewiring. We introduce a lightweight criterion that predicts when feature permutation will benefit a dataset by quantifying its intrinsic complexity. DynaTab dynamically reorders features via a neural rewiring algorithm and processes them through a compact, dynamic order-aware combination of separate learned positional embedding, importance-based gating, and masked attention layers, compatible with any sequence-sensitive backbone. Trained end-to-end with bespoke dynamic feature ordering (DFO) and dispersion losses, DynaTab achieves statistically significant gains, particularly on high-dimensional datasets, where it is benchmarked against 45 state-of-the-art baselines across 36 different real-world tabular datasets. Our results position DynaTab as a compelling new paradigm for high-dimensional tabular deep learning.

Al Zadid Sultan Bin Habib, Gianfranco Doretto, Donald A. Adjeroh

Lane Department of Computer Science and Electrical Engineering, West Virginia University, Morgantown, WV 26506, USA · Scientific Computing and Imaging Institute & Department of Biomedical Informatics, The University of Utah, Salt Lake City, UT 84112, USA

Dynamic Neural Networks Tabular ML Tabular Prediction

May 4, 2026cs.LG

MPCS: Neuroplastic Continual Learning via Multi-Component Plasticity and Topology-Aware EWC

Continual learning systems face a fundamental tension between plasticity -- acquiring new knowledge -- and stability -- retaining prior knowledge. We introduce MPCS (Multi-Plasticity Continual System), a neuroplastic architecture that integrates eleven complementary mechanisms: task-driven neurogenesis, Fourier-encoded inputs, EWC regularization, meta-replay, mixed consolidation, hybrid gating, synapse pruning/regeneration, Hebbian updates, task similarity routing, adaptive growth control, and continuous neuron importance tracking. We evaluate MPCS on MEP-BENCH, a multi-track benchmark spanning 31 tasks across regression, classification, logic, and mixed domains, using a three-dimensional Pareto criterion over task performance (Perf), representation diversity (RD), and gradient conflict rate (GCR). Across 15 ablation configurations (3 seeds x 4 tracks x 2000 epochs), MPCS achieves a Normalized Efficiency Score of 94.2, placing it on the Pareto frontier among 9 of 14 gate-passing systems. Key findings: (i) Fourier encoding is the single most critical component (removal drops Perf by 30.7 pp and fails the MEP gate on 14% of tasks); (ii) global EWC degrades performance (NES = -4.2); topology-local EWC reduces this penalty (NES 90.5->91.8) but does not eliminate it; removing EWC entirely yields MPCS_EFFICIENT, the highest-Perf system -- establishing a monotone relationship in the high task-similarity regime (s_bar ~= 0.95): global EWC < topology EWC < no EWC; (iii) the Pareto status assessment is predictive: removing the two Pareto-dominated components (EWC + Hebbian) jointly yields MPCS_EFFICIENT, which improves Perf by 0.6 pp at 4.7x lower compute cost (127 vs. 602 min), validating the Pareto frontier as an actionable model-compression guide.

Joern Hentsch

Stability-Plasticity Dilemma Dynamic Neural Networks Continual Learning Elastic Weight Consolidation

May 2, 2026cs.LG

Congestion-Aware Dynamic Axonal Delay for Spiking Neural Networks

Spiking Neural Networks (SNNs) are widely regarded as an energy-efficient paradigm for modeling and processing temporal and event-driven information. Incorporating delays in SNNs has been proven to be an effective mechanism for improving spike alignment in event-driven tasks. However, existing delay learning approaches predominantly assign static delays to individual synapses, resulting in a large number of delay parameters and limited adaptability to input-dependent activity dynamics. To this end, we propose a Congestion-Aware Dynamic Axonal Delay (CADAD) mechanism, which decomposes the delay into a channel-wise static base delay for temporal structuring and a global, activity-conditioned shift that dynamically regulates the state update rate under varying spike intensities. The delay parameters are learned using differentiable linear interpolation and discretized at inference time, preserving the benefits of dynamic delay modulation while incurring only minimal additional cost. Experiments on speech benchmarks, including the Spiking Heidelberg Dataset, Spiking Speech Commands, and Google Speech Commands, demonstrate that introducing congestion-aware delays into synaptic signal transmission effectively improves accuracy on temporal tasks, notably achieving 93.75% accuracy on SHD, 80.69% accuracy on SSC, and 95.58% on GSC-35, while reducing the parameter count by approximately 50% compared to state-of-the-art delay-based methods with the same architecture.

Dewei Bai, Hongxiang Peng, Yunyun Zeng +2

University of Electronic Science and Technology of China

Dynamic Neural Networks Spiking Neural Networks

May 1, 2026cs.LG

LEAP: Layer-wise Exit-Aware Pretraining for Efficient Transformer Inference

Layer-aligned distillation and convergence-based early exit represent two predominant computational efficiency paradigms for transformer inference; yet we establish that they exhibit systematic incompatibility under standard deployment conditions for convergence-based early exit. Distillation objectives that align intermediate student layers to teacher representations suppress the representational convergence that early-exit mechanisms exploit, rendering such mechanisms ineffective on distilled models. We introduce LEAP (Layer-wise Exit-Aware Pretraining), an auxiliary training objective that reconciles this incompatibility. LEAP requires no architectural modifications; it augments standard distillation with a single constraint ensuring intermediate layers approximate final-layer representations. LEAP-MiniLM achieves 1.61

\times

measured wall-clock speedup (batch=1, NVIDIA L4) at

θ

=0.95, with 91.9% of samples exiting by layer 7 and 1.80

\times

theoretical layer reduction, where standard distilled models achieve zero effective speedup. We validate across sentence similarity (STS-B: 0.760

\pm

0.006) and retrieval benchmarks (BEIR), providing operational guidance including latency measurements, decision thresholds, and deployment criteria.

Shashank Kapadia, Deep Naryan Mishra, Sujal Reddy Alugubelli +4

Walmart Inc. · Sam’s Club

Efficient Transformer Inference Dynamic Neural Networks Language Model Distillation Teacher-Student Learning Knowledge Distillation Early-Exit Neural Networks

Apr 29, 2026cs.LG

NORACL: Neurogenesis for Oracle-free Resource-Adaptive Continual Learning

In a continual learning setting, we require a model to be plastic enough to learn a new task and stable enough to not disturb previously learned capabilities. We argue that this dilemma has an architectural root. A finite network has limited representational and plastic resources, yet the required capacity depends on properties of the future task stream that are unknown: how many tasks will be encountered, and how much they overlap in feature space. Regularization-based methods preserve past knowledge within fixed-capacity architectures and therefore implicitly rely on an oracle architecture sized for this unknown future. When tasks are only weakly related, fixed architectures progressively run out of plastic resources; when tasks are few or strongly overlapping, models are often over-provisioned. Inspired by neurogenesis in biology, we propose NORACL to address the stability-plasticity dilemma by tackling the oracle architecture problem through neuronal growth. Starting from a compact network, NORACL grows only when needed by monitoring two complementary signals for representational and plasticity saturation. We evaluate NORACL against oracle-sized static baselines across varying task counts and geometries. Across all settings, NORACL achieves final average accuracies that are better than or on par with oracle-provisioned static baselines while using fewer parameters. Additionally, NORACL yields architectures with interpretable growth, i.e. dissimilar tasks predominantly expand feature-extraction layers, whereas tasks which rely on common features shift growth toward later feature-combination layers. Our analysis further explains why fixed-capacity networks lose plasticity as tasks accumulate, whereas NORACL creates fresh capacity for new tasks through growth. Together, these results show that adaptive neurogenesis pushes the stability-plasticity Pareto frontier of continual learning.

Karthik Charan Raghunathan, Christian Metzner, Laura Kriener +1

Institute of Neuroinformatics University of Zurich & ETH Zurich Zurich, Switzerland

Stability-Plasticity Dilemma Dynamic Neural Networks Continual Learning

Apr 28, 2026cs.LG

SWAN: World-Aware Adaptive Multimodal Networks for Runtime Variations

Multimodal deep neural networks deployed in realistic environments must contend with runtime variations: changes in modality quality, overall input complexity, and available platform resources. Current networks struggle with such fluctuations -- adaptive networks cannot adhere to a strict compute budget, controller-based networks neglect to consider input complexity, and statically provisioned networks fail at all the above. Consequently, they do not extract maximum utility from the expended computational resources. We present SWAN (Sample and World-Aware Multimodal Network), the first adaptive multimodal network that accomplishes all three goals. SWAN employs a quality-aware controller to assign resources among modalities according to a variable user-specified maximum budget. Within this budget, an adaptive gating module further optimizes efficiency by scaling layer utilization according to sample complexity. For further gains, SWAN also employs a token dropping module that masks semantically irrelevant multimodal features before performing detections. We evaluate SWAN in the domain of autonomous driving with complex multi-object 3D detection, reducing FLOPs by up to 49% with minimal degradation.

Jason Wu, Shir-Kang Scott Jin, Yuyang Yuan +4

University of California, Los Angeles · University of California, Riverside · U.S. Army DEVCOM Army Research Laboratory

Efficient Multimodal Inference Dynamic Neural Networks 3D Object Detection

Apr 28, 2026cs.CV

CoRE: Concept-Reasoning Expansion for Continual Brain Lesion Segmentation

Accurate brain lesion segmentation in MRI is vital for effective clinical diagnosis and treatment planning. Due to high annotation costs and strict data privacy regulations, universal models require employing Continual Learning (CL) to adapt to evolving clinical tasks without losing previously acquired knowledge. However, existing CL paradigms often suffer from capacity limits or redundant parameter growth, and even advanced dynamic methods rely mostly on image-perception strategies that struggle to handle the substantial pathological and multimodal heterogeneity inherent in brain imaging. To address this issue, we propose Concept-Reasoning Expansion (CoRE) framework, which establishes a joint decision-making mechanism by integrating visual features with structured concepts. Through the alignment of image tokens with a hierarchical concept library, CoRE simulates clinical reasoning to guide both interpretable expert routing and demand-based model growth. This collaborative process ensures model evolution is grounded in clinical priors, preventing redundant parameter expansion while maximizing knowledge reuse. Extensive evaluations across 12 sequential brain lesion MRI tasks demonstrate that CoRE achieves state-of-the-art performance and provides a high knowledge starting point for efficient future adaptation. Its superior few-shot transferability and clinical interpretability further validate its effectiveness in managing non-stationary clinical data streams. Our code will be released soon.

Qianqian Chen, Anglin Liu, Jingyang Zhang +1

Southeast University, Nanjing, China · The Hong Kong University of Science and Technology (Guangzhou), Guangzhou, China

Dynamic Neural Networks Medical Image Segmentation

Apr 27, 2026cs.LG

A Comparative Analysis on the Performance of Upper Confidence Bound Algorithms in Adaptive Deep Neural Networks

Edge computing environments impose strict constraints on energy consumption and latency, making the deployment of deep neural networks a significant challenge. Therefore, smart and adaptive inference strategies that dynamically balance computational cost or latency with predictive accuracy are critical in edge computing scenarios. In this work, we build on Adaptive Deep Neural Networks (ADNNs) that employ the Multi-Armed Bandit (MAB) framework. Current literature leverages the first version of the Upper Confidence Bound (UCB1) strategy to dynamically select the optimal confidence threshold, enabling efficient early exits without sacrificing accuracy. However, we introduce four additional Upper Confidence Bound strategies in ADNNs, namely UCB-V, UCB-Tuned, UCB-Bayes, and UCB-BwK, and perform, for the first time, a comparative study of these strategies with respect to trade-offs between accuracy, energy consumption, and latency. The proposed UCB strategies are employed on the ResNet and MobileViT neural networks, and are evaluated on the benchmark datasets of CIFAR-10, CIFAR-10.1, and CIFAR-100. Experimental results demonstrate that all strategies achieve sub-linear cumulative regret, with UCB-Bayes converging the fastest, followed by UCB-Tuned and UCB-V. Finally, UCB-V and UCB-Tuned dominate the Pareto Frontiers of accuracy-latency and accuracy-energy trade-offs. The implementation code is available here: https://github.com/gr3gor1/MAB_UCB

Grigorios Papanikolaou, Ioannis Kontopoulos, Konstantinos Tserpes

Department of Electrical and Computer Engineering National Technical University of Athens, Greece

Multi-Armed Bandits Dynamic Neural Networks Energy-Efficient ML Edge Computing UCB Algorithms Adaptive Computation+1

Apr 21, 2026cs.NE

Scalable Memristive-Friendly Reservoir Computing for Time Series Classification

Memristive devices present a promising foundation for next-generation information processing by combining memory and computation within a single physical substrate. This unique characteristic enables efficient, fast, and adaptive computing, particularly well suited for deep learning applications. Among recent developments, the memristive-friendly echo state network (MF-ESN) has emerged as a promising approach that combines memristive-inspired dynamics with the training simplicity of reservoir computing, where only the readout layer is learned. Building on this framework, we propose memristive-friendly parallelized reservoirs (MARS), a simplified yet more effective architecture that enables efficient scalable parallel computation and deeper model composition through novel subtractive skip connections. This design yields two key advantages: substantial training speedups of up to 21x over the inherently lightweight echo state network baseline and significantly improved predictive performance. Moreover, MARS demonstrates what is possible with parallel memristive-friendly reservoir computing: on several long sequence benchmarks our compact gradient-free models substantially outperform strong gradient-based sequence models such as LRU, S5, and Mamba, while reducing full training time from minutes or hours down seconds or even only a few hundred milliseconds. Our work positions parallel memristive-friendly computing as a promising route towards scalable neuromorphic learning systems that combine high predictive capability with radically improved computational efficiency, while providing a clear pathway to energy-efficient, low-latency implementations on emerging memristive and in-memory hardware.

Coşku Can Horuz, Andrea Ceni, Claudio Gallicchio +1

Adaptive AI Research Group University of Lübeck, Germany · Department of Computer Science University of Pisa, Italy

Dynamic Neural Networks Energy-Efficient ML Time Series Classification Reservoir Computing Neuromorphic Computing

Apr 17, 2026cs.LG

When Do Early-Exit Networks Generalize? A PAC-Bayesian Theory of Adaptive Depth

Early-exit neural networks enable adaptive computation by allowing confident predictions to exit at intermediate layers, achieving 2-8

\times

inference speedup. Despite widespread deployment, their generalization properties lack theoretical understanding -- a gap explicitly identified in recent surveys. This paper establishes a unified PAC-Bayesian framework for adaptive-depth networks. (1) Novel Entropy-Based Bounds: We prove the first generalization bounds depending on exit-depth entropy

H(D)

and expected depth

\mathbb{E}[D]

rather than maximum depth

K

, with sample complexity

\mathcal{O}((\mathbb{E}[D] \cdot d + H(D))/ε^2)

. (2) Explicit Constructive Constants: Our analysis yields the leading coefficient

\sqrt{2\ln 2} \approx 1.177

with complete derivation. (3) Provable Early-Exit Advantages: We establish sufficient conditions under which adaptive-depth networks strictly outperform fixed-depth counterparts. (4) Extension to Approximate Label Independence: We relax the label-independence assumption to

ε

-approximate policies, broadening applicability to learned routing. (5) Comprehensive Validation: Experiments across 6 architectures on 7 benchmarks demonstrate tightness ratios of 1.52-3.87

\times

(all

p < 0.001

) versus

>

100

\times

for classical bounds. Bound-guided threshold selection matches validation-tuned performance within 0.1-0.3%.

Dongxin Guo, Jikun Wu, Siu Ming Yiu

The University of Hong Kong, Hong Kong, China · Brain Investing Limited, Hong Kong, China

Neural Network Generalization PAC-Bayesian Generalization Bounds Adaptive Inference Dynamic Neural Networks Early-Exit Neural Networks

Dec 4, 2025cs.CC

Hardware-Algorithm Co-Optimization of Early-Exit Neural Networks for Multi-Core Edge Accelerators

The deployment of Early-Exiting Neural Networks (EENNs) on edge accelerators requires optimizing not only the network architecture but also its hardware deployment. Exit configuration, quantization, and hardware workload mapping interact in non-trivial ways, influencing memory traffic, accelerator utilization, and ultimately the energy-latency trade-off. This work presents a hardware-aware co-design framework for EENNs that jointly optimizes exit configuration, quantization-aware training, and multi-core hardware mapping within a unified NAS process. Leveraging analytical design space exploration, the framework identifies efficient workload mappings for each candidate architecture while providing accurate latency and energy estimates during the search. We further formulate EENN deployment as a constrained multi-objective optimization problem balancing predictive accuracy, energy-latency product, exit overhead, and dynamic inference efficiency. Experimental results on CIFAR-10 demonstrate that the proposed framework achieves over a 50% reduction in energy-latency product compared with static baselines under 8-bit quantization. These results demonstrate that jointly optimizing architecture and deployment is essential for realizing the full efficiency potential of dynamic inference on heterogeneous edge accelerators.

Alaa Zniber, Arne Symons, Ouassim Karrakchou +2

TICLab, International University of Rabat, Morocco · MICAS, KU Leuven, Belgium

Efficient Neural Network Inference Dynamic Neural Networks Edge Inference Hardware-Aware NAS Early-Exit Neural Networks

Oct 9, 2025cs.LG

Deep Neural Networks Inspired by Differential Equations

Deep learning has become a pivotal technology in fields such as computer vision, scientific computing, and dynamical systems, significantly advancing these disciplines. However, neural Networks persistently face challenges related to theoretical understanding, interpretability, and generalization. To address these issues, researchers are increasingly adopting a differential equations perspective to propose a unified theoretical framework and systematic design methodologies for neural networks. In this paper, we provide an extensive review of deep neural network architectures and dynamic modeling methods inspired by differential equations. We specifically examine deep neural network models and deterministic dynamical network constructs based on ordinary differential equations (ODEs), as well as regularization techniques and stochastic dynamical network models informed by stochastic differential equations (SDEs). We present numerical comparisons of these models to illustrate their characteristics and performance. Finally, we explore promising research directions in integrating differential equations with deep learning to offer new insights for developing intelligent computational methods that boast enhanced interpretability and generalization capabilities.

Yongshuai Liu, Lianfang Wang, Kuilin Qin +6

School of Mathematical Sciences, Beijing Normal University, Beijing, China · Department of Mathematics, The Chinese University of Hong Kong, Shatin, Hong Kong

Neural Network Generalization Dynamic Neural Networks Neural ODEs

Jul 2, 2025cs.LG

PERTINENCE: Input-based Opportunistic Neural Network Dynamic Execution

Deep neural networks (DNNs) are widely used for their ability to model complex patterns across domains such as computer vision, speech recognition, and robotics. However, larger models, while often more accurate, are computationally expensive and energy-intensive. Since such a cost is typically needed only for challenging inputs, dynamically selecting lighter models for simpler inputs can improve efficiency with minimal impact on accuracy. We introduce PERTINENCE, a runtime method that selects, from a set of pre-trained models, the lightest model likely to process each input correctly. An ML-based dispatcher performs this selection, and a genetic algorithm explores dispatcher training strategies to identify Pareto-optimal trade-offs between accuracy and computational cost. We evaluate PERTINENCE on CNNs trained on CIFAR-10 and CIFAR-100, ViTs trained on TinyImageNet, and a YOLO-based road occupancy estimation application using real-time intersection camera feeds. Results show that PERTINENCE matches or improves the accuracy of state-of-the-art pre-trained models while reducing operations by up to 36%, with equivalent or lower end-to-end inference time through tunable invocation intervals.

Omkar Shende, Gayathri Ananthanarayanan, Marcello Traiola

Department of Computer Science and Engineering, Indian Institute of Technology Dharwad, Karnataka, India · University of Rennes, CNRS, Inria, IRISA, Rennes, France

Efficient Neural Network Inference Adaptive Inference Dynamic Neural Networks Cost-Aware Inference

May 15, 2025cs.LG

SEAL: Searching Expandable Architectures for Incremental Learning

Incremental learning is a machine learning paradigm where a model learns from a sequential stream of tasks. This setting poses a key challenge: balancing plasticity (learning new tasks) and stability (preserving past knowledge). Neural Architecture Search (NAS), a branch of AutoML, automates the design of the architecture of Deep Neural Networks and has shown success in static settings. However, existing NAS-based approaches to incremental learning often rely on expanding the model at every task, making them impractical in resource-constrained environments. In this work, we introduce SEAL, a NAS-based framework tailored for data-incremental learning, a scenario where disjoint data samples arrive sequentially and are not stored for future access. SEAL adapts the model structure dynamically by expanding it only when necessary, based on a capacity estimation metric. Stability is preserved through cross-distillation training after each expansion step. The NAS component jointly searches for both the architecture and the optimal expansion policy. Experiments across multiple benchmarks demonstrate that SEAL effectively reduces forgetting and enhances accuracy while allocating additional capacity only when required. These results highlight the promise of combining NAS and selective expansion for efficient, adaptive learning in incremental scenarios.

Matteo Gambella, Manuel Roveri

Dipartimento di Elettronica, Informazione e Bioingegneria, Politecnico di Milano, Milan, Italy

Dynamic Neural Networks Continual Learning Incremental Learning Neural Architecture Search

Date pendingcs.LG

PEARL: Structural Privacy-Utility Control in Human-Centric CPS via Personalized Early-Exit Deep Reinforcement Learning

In human-centric Cyber-Physical Systems (CPS), personalized Deep Reinforcement Learning (DRL) agents must share fine-grained control actions with cloud services, exposing sensitive private states to inference attacks by honest-but-curious adversaries. Static privacy models fail to address the dynamic nature of human interactions. This paper introduces PEARL (Personalized Early-exit Adaptive Reinforcement Learning), a novel framework that addresses this challenge through structural privacy control rather than data perturbation. PEARL deploys a dual-path Early-Exit Deep Q-Network (EE-DQN) at the edge, using Mutual Information (MI) between private states and observable actions to train per-branch binary labels: Utility Confidence Labels (UCL), verifying action quality, and Privacy Confidence Labels (PCL), verifying MI leakage remains below a user-defined threshold. At inference, PEARL selects the shallowest exit branch satisfying both UCL and PCL, structurally limiting shared action descriptive power without noise injection. An MI-based feedback loop tracks behavioral drift and triggers retraining when privacy-utility profiles shift, ensuring long-term robustness. Validated on a personalized smart-home HVAC system and a VR smart classroom, PEARL reduces adversarial state-inference accuracy by 25.67% on average with a controlled 10-16% utility cost, establishing a practical, dynamically enforceable privacy-utility tradeoff.

Mojtaba Taherisadr, Salma Elmalaki

University of California, Irvine, USA

RL Control Cyber-Physical Systems Dynamic Neural Networks Privacy-Preserving ML Early-Exit Neural Networks Privacy-Utility Tradeoff