Organizations: the School of Mathematics, Southeast University, Nanjing 210096, China · the School of Mathematics, Southeast University, Nanjing 210096, China, and also with the Systems Research Institute of the Polish Academy of Sciences, Warsaw 01-447, Poland · the School of Cyber Science & Engineering, Southeast University, Nanjing 210096, China · the Systems Research Institute of the Polish Academy of Sciences, Warsaw 01-447, Poland; the Institute of Computer Science, AGH University of Krakow, Kraków 30-059, Poland; and also with SAN University, Łódź 90-113, Poland
Efficient deep learning traditionally relies on static heuristics like weight magnitude or activation awareness (e.g., Wanda, RIA). While successful in unstructured settings, we observe a critical limitation when applying these metrics to the structural pruning of deep vision networks. These contemporary metrics suffer from a magnitude bias, failing to preserve critical functional pathways. To overcome this, we propose a decoupled kinetic paradigm inspired by Alternating Gradient Flow (AGF), utilizing an absolute feature-space Taylor expansion to accurately capture the network's structural "kinetic utility". First, we uncover a topological phase transition at extreme sparsity, where AGF successfully preserves baseline functionality and exhibits topological implicit regularization, avoiding the collapse seen in models trained from scratch. Second, transitioning to architectures without strict structural priors, we reveal a phenomenon of Sparsity Bottleneck in Vision Transformers (ViTs). Through a gradient-magnitude decoupling analysis, we discover that dynamic signals suffer from signal compression in converged models, rendering them suboptimal for real-time routing. Finally, driven by these empirical constraints, we design a hybrid routing framework that decouples AGF-guided offline structural search from online execution via zero-cost physical priors. We validate our paradigm on large-scale benchmarks: under a 75% compression stress test on ImageNet-1K, AGF effectively avoids the structural collapse where traditional metrics aggressively fall below random sampling. Furthermore, when systematically deployed for dynamic inference on ImageNet-100, our hybrid approach achieves Pareto-optimal efficiency. It reduces the usage of the heavy expert by approximately 50% (achieving an estimated overall cost of 0.92×) without sacrificing the full-model accuracy.
Figures & tables
Fig. 1 : Overview of the AGF-Guided Efficiency Framework. The pipeline integrates (A) Gradient-based Utility Calibration, (B) Iterative Structural Pruning, and (C) Confidence-based Dynamic Routing. AGF identifies the topological skeleton, while the routing policy handles runtime complexity.
Fig. 2 : Analysis of Metric Stability and Orthogonality (WideResNet on CIFAR-100 at k=32 ). (a) Selection Stability: AGF demonstrates superior batch-to-batch structural consistency compared to the Taylor baseline. (b) Metric Orthogonality ( J≈0 ): A normalized scatter plot reveals a fundamental divergence between static and dynamic metrics. Traditional magnitude metrics ( ℓ1 ) rigidly select high-capacity channels (blue crosses), whereas AGF identifies an entirely distinct, orthogonal subset of dynamic routing hubs with high kinetic potential (red dots).
Strategy
Width ( k )
Mean Acc (%)
Std ( σ )
Best Seed (%)
Baseline (Full)
1024
68.66
N/A
68.66
Moderate Compression ( k=256 , 25% Width)
Random Pruning
256
70.21
0.45
70.66
ℓ1 -Norm Pruning
256
70.32
0.22
70.46
AGF Pruning (Ours)
256
70.75
0.08
70.81
Narrow (Scratch)
256
70.92
0.28
71.17
TABLE I : Detailed Comparison of Pruning vs. Training-from-Scratch. Consistency vs. Peak Performance: While RIA achieves a slight advantage at extreme sparsity ( k=32 ), it exhibits inconsistency at moderate sparsity ( k=128 ), falling behind simple baselines. AGF (Ours) demonstrates the best trade-off: it dominates at k=128 and maintains competitive accuracy with superior stability ( σ=0.12 ) at k=32 , avoiding the high variance of activation-based methods like Wanda ( σ=0.63 ) and RIA ( σ=0.40 ).
Strategy
Calibration Data
Ep 10 (Post-Prune)
Peak Accuracy
Epoch of Peak
vs. Taylor
Taylor (Baseline)
100 Batches
84.26%
84.57%
Ep 20
-
AGF (Dense)
100 Batches
83.84%
84.56%
Ep 17
-0.01%
AGF (Sparse)
10 Batches
83.60%
84.90%
Ep 20
+0.33%
Random
N/A
83.64%
83.61%
-
-0.96%
ℓ1 -Norm
N/A
83.15%
83.53%
-
-1.04%
TABLE II : 20-Epoch Recovery Analysis on ImageNet-100. Comparing the Peak Accuracy of pruning strategies. Surprisingly, AGF with only 10 batches achieves the highest performance ( 84.90% ), outperforming both Taylor and the high-sample AGF variant. This suggests that the stochastic noise in limited-batch gradient estimation acts as a beneficial regularizer for structure selection.
Metric Type
Strategy
Moderate ( k=1536 , 50%)
Extreme ( k=768 , 25%)
Ep 1 Acc
Peak Acc
Ep 1 Acc
Peak Acc
Unpruned Baseline (Full Capacity)
91.50%*
Static/Magnitude (Activation-Aware)
RIA Pruning
-
-
73.61%
82.24%
Wanda Pruning
79.82%
84.96%
66.50%
81.36%
Dynamic/Gradient
AGF Pruning (Ours)
85.00%
86.81%
73.88%
81.89%
*Estimated full fine-tuning performance on CIFAR-100.
TABLE III : Structural Pruning on ViT-Base (MLP Width = 3072). Moderate ( k=1536 ): AGF demonstrates superior recovery and peak performance over magnitude-based methods. Extreme ( k=768 ): All deterministic metrics encounter a severe Sparsity Bottleneck, highlighting the theoretical limitations of single-dimensional proxies and motivating our Hybrid Routing approach.
Regime
Metric
Full
Pruned
Ratio
Signal
ResNet-50 (Kinetic)
ℓ1 Norm
107,142
1,781
60.2 ×
Physical Truth
AGF Utility
0.0048
0.0001
44.7 ×
Aligned
WideResNet (Saturated)
ℓ1 Norm
232,682
1,557
149.4 ×
Physical Truth
AGF Utility
4.89e-4
2.29e-5
21.4 ×
Compressed
TABLE IV : Regime-Dependent Fidelity of Energy Proxies. ResNet-50 (Kinetic): AGF utility ( 44.7× ) remains aligned with the physical capacity gap. WideResNet-18 (Saturated): AGF utility becomes compressed ( 21.4× ) due to vanishing gradients, underestimating the massive physical disparity ( 149.4× ).
Fig. 3 : Difficulty Distribution of Routed Samples. We measure sample difficulty using the prediction entropy of the full-capacity baseline. The lightweight router successfully learns to disentangle the input space without human priors: ”easy” samples (low entropy) are dominantly routed to the Pruned Expert (Green), while ”hard” ambiguous samples (high entropy) are forwarded to the Full Expert (Red). This adaptive decoupling is the core mechanism enabling our system’s high efficiency.
Method
Acc (%)
vs. Random
Route Ratio (Full:Small)
Est. Cost
Static Full
88.74
+4.42
100 : 0
1.00x
Static Pruned
79.80
-4.52
0 : 100
0.85x
Random Policy
84.32
-
50 : 50
0.92x
Ours (Adaptive)
88.78
+4.46
48.5 : 51.5
0.92x
TABLE V : Main Results on ImageNet-100. Our Adaptive Router (Ours) matches the upper bound (Full Model) while reducing the usage of the heavy expert by ≈50% .
Fig. 4 : Accuracy-Efficiency Trade-off on ImageNet-100. Our adaptive method (Red) establishes a superior Pareto frontier compared to static and random baselines, enabling dynamic trade-offs between performance and cost.
Method
Metric Proxy
Acc (%)
vs. Random
vs. Wanda
Reference Upper Bound (100% Capacity)
Unpruned Baseline
Full Capacity
80.35
-
-
Extreme Compression Regime (25% Capacity)
Taylor Pruning
Loss Approx ( ∇W⋅W )
64.42
-0.51
-0.27
ℓ1 -Norm Pruning
Weights ( ∣W∣ )
64.54
-0.39
-0.15
Wanda Pruning
Weights × Act
64.69
-0.24
-
TABLE VI : Extreme Stress Test on ImageNet-1K (ResNet-50). Comparison of pruning strategies under a severe 75% structural compression ( k=64 ) on the bottleneck layers. Reference: The unpruned model achieves 80.35%. The significant absolute drop across all methods reflects an information bottleneck. Notably, magnitude-based heuristics ( ℓ1 , Wanda) and Taylor approximations suffer degradation below the Random baseline. AGF remains competitive with uniform sampling, indicating that the network has reached an intrinsic capacity limit without suffering from negative proxy biases.
Fig. 5 : Qualitative Visualization of Routing Decisions. Top row: ”Easy” samples (e.g., clear objects) routed to the Pruned Expert. Bottom row: ”Hard” samples (e.g., clutter) routed to the Full Expert. The router effectively identifies samples requiring higher capacity for correct classification.
Fig. 6 : Metric Efficiency and Slope Analysis. Main Plot: Both AGF and L1 significantly outperform Random selection. Zoomed Inset: Highlights the critical divergence at high costs. While L1 saturates (Negative Slope), AGF maintains a positive marginal gain .
Appendix figures & tables2 assets
Supplementary material from the paper’s appendix.
Appendix
Threshold ( τ )
AGF (Ours)
ℓ1 -Norm
Random
Regime
Acc (%)
Cost
Acc (%)
Cost
Acc (%)
Cost
0.000
68.54
1.00
68.83
1.00
68.13
1.00
Pruned Only
0.500
69.26
7.28
69.61
7.93
69.25
8.26
Low Cost
0.700
70.94
21.33
71.04
21.35
70.87
21.69
0.800
71.40
28.39
71.47
29.16
71.37
29.30
0.900
72.10
38.08
72.11
39.29
72.08
39.61
Balanced
Appendix
TABLE VII : Full Numerical Sweep of Confidence-Based Routing. We compare the Accuracy (%) and Normalized Cost of three strategies across varying confidence thresholds τ . Bold indicates the best trade-off (Highest Accuracy or Lowest Cost) at critical operating points.
Parameter
Value
Optimizer
SGD (Nesterov)
Momentum
0.9
Weight Decay
5×10−4
Total Epochs
150
Batch Size
32
Initial Learning Rate
1.0×10−3
Appendix
TABLE VIII : Hyperparameters for Teacher Training (CIFAR-100).
Network pruning is used to reduce inference latency and power consumption in large neural networks. However, most methods focus on empirical results at the expense of understanding the pruning process. We introduce Hyperflux, a novel L0 method which models pruning as a continuously evolving system determined by flux, the gradient response to a weight's removal, and pressure, a global regularization driving weights toward pruning. By exploiting this model, Hyperflux's pruning behavior becomes understandable at both microscopic (weight regrowth/pruning) and macroscopic (sparsity convergence, etc.) levels. We also introduce a novel pressure scheduler that reliably targets desired sparsities. Hyperflux achieves competitive results with ResNet-50, VGG-19 and DeiT-T/S on CIFAR-10, CIFAR-100 and ImageNet datasets.
Eugen Barbulescu, Antonio Alexoaie, Lucian Busoniu
Department of Computer Science Technical University of Cluj-Napoca · Department of Automation Technical University of Cluj-Napoca
Deploying deep neural networks (DNNs) on edge devices requires strong compression with minimal accuracy loss. This paper introduces Mix-and-Match Pruning, a globally guided, layer-wise sparsification framework that leverages sensitivity scores and simple architectural rules to generate diverse, high-quality pruning configurations. The framework addresses a key limitation that different layers and architectures respond differently to pruning, making single-strategy approaches suboptimal. Mix-and-Match derives architecture-aware sparsity ranges, e.g., preserving normalization layers while pruning classifiers more aggressively, and systematically samples these ranges to produce ten strategies per sensitivity signal (magnitude, gradient, or their combination). This eliminates repeated pruning runs while offering deployment-ready accuracy-sparsity trade-offs. Experiments on CNNs and Vision Transformers demonstrate Pareto-optimal results, with Mix-and-Match reducing accuracy degradation on Swin-Tiny by 40% relative to standard single-criterion pruning. These findings show that coordinating existing pruning signals enables more reliable and efficient compressed models than introducing new criteria.
Danial Monachan, Samira Nazari, Mahdi Taheri +4
Brandenburg Technical University, Cottbus, Germany · University of Zanjan, Iran · Tallinn University of Technology, Estonia +1
Vision Transformers require significant computational resources and memory bandwidth, severely limiting their deployment on resource-constrained hardware. Most structured pruning methods reduce theoretical cost effectively, yet they typically operate at a single structural granularity and depend on multi-stage pipelines with importance ranking, auxiliary solvers or post-hoc magnitude thresholding, followed by a separate fine-tuning phase to recover accuracy. We propose Hierarchical Auto-Pruning (HiAP), which casts ViT pruning as a single budget-aware learning problem and jointly allocates sparsity across four granularities in one end-to-end phase. HiAP introduces stochastic Gumbel-Sigmoid gates at macro level (attention heads and FFN blocks) and micro level (intra-head dimensions and FFN neurons), and trains them against the task loss together with an analytical MAC cost term. Two penalty weights set how much compute is removed, while the gates gradually harden into a dense, smaller sub-network at convergence. It does not require importance heuristics, ranking metrics, auxiliary solvers or secondary fine-tuning. On ImageNet, HiAP compresses DeiT-Base to 7.4G MACs at 80.88% top-1 80.40% at 7.8G without distillation and DeiT-Small to 3.1G at 79.33%, competitive with substantially more complex pipelines at matched compute. The structurally pruned network can be accelerated natively on stock kernels, and more than 90% of the theoretical MAC reduction is realized as measured throughput at batch 256 on an A100.
Andy Li, Aiden Durrant, Milan Markovic +1
School of Natural and Computing Sciences, University of Aberdeen, UK · School of Computing Sciences, University of East Anglia, UK · Department of Physics and Technology, UiT The Arctic University of Norway, Norway