cs.LGDec 14, 2025

PerNodeDrop: A Method Balancing Specialized Subnets and Regularization in Deep Neural Networks

Authors: Gelesh G. Omathil

Organizations: Department of Computer Science, Christ University, Bangalore, India

Abstract

Deep neural networks possess strong representational capacity yet remain vulnerable to overfitting, primarily because neurons tend to co-adapt in ways that, while capturing complex and fine-grained feature interactions, also reinforce spurious and non-generalizable patterns that inflate training performance but reduce reliability on unseen data. Noise-based regularizers such as Dropout and DropConnect address this issue by injecting stochastic perturbations during training, but the noise they apply is typically uniform across a layer or across a batch of samples, which can suppress both harmful and beneficial co-adaptation. This work introduces PerNodeDrop, a lightweight stochastic regularization method. It applies per-sample, per-node perturbations to break the uniformity of the noise injected by existing techniques, thereby allowing each node to experience input-specific variability. Hence, PerNodeDrop preserves useful co-adaptation while applying regularization. This narrows the gap between training and validation performance and improves reliability on unseen data, as evident from the experiments. Although superficially similar to DropConnect, PerNodeDrop operates at the sample level. It drops weights at the sample level, not the batch level. An expected-loss analysis formalizes how its perturbations attenuate excessive co-adaptation while retaining predictive interactions. Empirical evaluations on vision, text, and audio benchmarks indicate improved generalization relative to the standard noise-based regularizer.

Figures & tables

Explore similar work

CardsList
  1. Dropout Neural Network Training Viewed from a Percolation Perspective

    Dec 15, 2025Finley Devlin, Jaron SandersNeural Network Training Dynamics

  2. Adaptive Data Dropout: Towards Self-Regulated Learning in Deep Neural Networks

    Apr 14, 2026Amar Gahir, Varshil Patel, Shreyank N GowdaTraining Data SelectionAdaptive Sampling

  3. Explicit Dropout: Deterministic Regularization for Transformer Architectures

    Apr 22, 2026Vidhi Agrawal, Illia Oleksiienko, Alexandros IosifidisNeural Network Generalization