Structured Neuron Pruning in Deep Neural Networks Using Multi-Armed Bandits

Salem Ameen; Sunil Vadera

cs.LGMay 29, 2026

Structured Neuron Pruning in Deep Neural Networks Using Multi-Armed Bandits

Authors: Salem Ameen, Sunil Vadera

Organizations: School of Science, Engineering and Environment, University of Salford, Salford, United Kingdom

Abstract

Deep neural networks often contain redundant hidden units. Removing individual weights can reduce parameter count, but unstructured sparsity is not always easy to exploit in standard dense implementations. This paper develops a structured pruning framework in which complete neurons are removed using multi-armed bandit (MAB) algorithms. Each candidate neuron is treated as an arm; pulling an arm temporarily masks that neuron, measures the change in loss on a sampled mini-batch, restores the neuron, and updates an estimate of its safe-removal reward. The framework supports stochastic policies, including Epsilon-Greedy, Softmax, UCB1 and Thompson Sampling, and multiplicative-weight policies, including Hedge-style multiplicative weights and EXP3. We evaluate the method on tabular classification, tabular regression and deep neural-network benchmarks covering image, text and reasoning tasks. Statistical comparisons using the Friedman test followed by the Nemenyi post-hoc test show significant differences between methods. On tabular classification tasks, UCB1 obtains the highest mean rank among pruning policies and improves on the unpruned neural network. On regression tasks, UCB1 obtains the highest mean rank and is statistically competitive with, or superior to, several standard regression models according to R^2. On deep-learning tasks, UCB1 and Thompson Sampling obtain the strongest ranks, and several MAB policies significantly outperform the unpruned model, magnitude-based neuron pruning and greedy activation-variation pruning. The results show that MAB-based neuron pruning is an effective and computationally practical approach for structured model reduction.

Explore similar work

May 23, 2026cs.LG

Pruning Deep Neural Networks via the Marchenko--Pastur Distribution

We study a Marchenko--Pastur (MP) random-matrix approach to pruning deep neural networks with very small post-pruning fine-tuning budgets. The main practical contribution is accuracy retention under short calibration and fine-tuning schedules, rather than a long post-pruning reoptimization pipeline. The theory gives deterministic data-path certificates: if the removed component

R

has small propagated logit effect

L_s \| R ψ_1(s) \|_\infty

, pruning decreases an elastic-net objective and preserves samples whose dense margin exceeds twice the perturbation. The zero-budget case gives perfect pruning; a prune--restore extension models weight restoration inside a fixed sparse-execution pattern; and an additive

L_2

-regularized model shows admissible random-like components vanish at the training limit, with persistent spikes stabilizing as the MP bulk collapses. Under iid-Gaussian sufficient conditions, the fitted MP edge

σ_+

gives a high-probability layerwise budget signal. On ImageNet-1k, after only three distillation epochs, ViT-B/16

2{:}4{+}

ToMe reaches

83.41\%

top-1 (

-1.70

pp from dense) at

59.81\%

sparse-execution MAC reduction, with

1.388\times

best-observed A40 native-

2{:}4

backend speedup for the same checkpoint and ToMe graph; a separate no-ToMe A100 endpoint gives

2.705\times

. At structured sparsity, ViT-B/16

6{:}12

reaches

83.74\%

, ViT-L/16

8{:}16

dense+permutation reaches

85.33\%

(

-0.51

pp), and ConvNeXtV2-Base

12{:}16

reaches

86.35\%

(

-0.37

pp). For CNNs, ResNet50

8{:}16

dense+permutation reaches

75.87\%

(

-0.26

pp), and ResNet152d CAST-conv+permutation reaches

81.33\%

(

-1.53

pp) at

{\sim}50\%

MAC accounting with a

1.62\times

A40 im2col

+2{:}4

sparse-GEMM audit.

Leonid Berlyand, Theo Bourdais, Houman Owhadi +1

Resnet-18

Ask PDF

Sep 21, 2026cs.LG

Artificial Structure Function Search: Preserving Artificial Functional Connectivity for Structured Pruning

Structured pruning is a model compression technique that is used to reduce the computational cost of deploying deep neural networks on resource-constrained devices. Popular methods of pruning rely on opaque heuristics or weight-based criteria that give no indication as to the structural dependencies in the network. To address these limitations we present Artificial Structure Function Search (ASF-S): a novel structured pruning framework. ASF-S utilizes Principle Gradient Importance (PGI): a novel prune-candidate selection criteria that is inspired by structure-function relationships in the brain. By ensuring the pruned structure of the model respects topographical organization of the output layer, we define Artificial Functional Connectivity (AFC) for artificial neural networks. AFC provides evidence to demonstrate that accurate smaller networks can be found using careful prune candidate selection criteria. We present results for PGI as a selection criterion and for ASF-S as a pruning framework against recent benchmarks, demonstrating that our method yields model variants with 70% parameter reduction, that can recover baseline accuracy without re-training the pruned layers.

Mindula Illeperuma, Rafael Pina, Charuka Herath +2

Heterogeneous Pruning Scores

Ask PDF

Jun 24, 2026cs.LG

Hierarchical Reinforcement Learning for Neural Network Compression (HiReLC): Pruning and Quantization

We present HiReLC, a hierarchical ensemble-reinforcement learning framework for automated joint quantization and structured pruning of deep neural networks. The framework decomposes the compression search across two levels of abstraction: low-level agents (LLAs) operate independently per block, selecting per-kernel configurations over a multi-discrete action space spanning bitwidth, pruning keep-ratio, quantization type, and granularity, while high-level agents (HLAs) coordinate global budget allocation via ensemble voting guided by Fisher Information-based sensitivity estimates. To mitigate the computational cost of policy evaluation, an iterative active learning loop interleaves surrogate-guided RL optimization with post-compression fine-tuning, using a lightweight MLP surrogate to amortize expensive evaluations and a logit-MSE proxy during cold-start. The surrogate is used for reward shaping rather than as a replacement for final post-compression evaluation. The controller is architecture-agnostic by design, with a modular layer abstraction decoupling the RL environment from the underlying network topology. Experiments across Vision Transformer and CNN benchmarks demonstrate effective parameter-storage compression ratios of 5.99 - 6.72

\times

with a 3.83 % gain in one setting and 0.55 - 5.62 % accuracy drops elsewhere, supporting hierarchical policy decomposition and sensitivity-aware guidance as practical design choices for joint neural network compression.

Kamar Hibatallah Baghdadi, Kawther Guoual Belhamidi, Sara Belhadj +2

Context Compression Hierarchical Reinforcement Learning Deep Q-Networks Pruning Quantization

Ask PDF