Alternating Gradient Flow Utility: A Unified Metric for Structural Pruning and Dynamic Routing in Deep Networks

Tianhao Qian; Zhuoxuan Li; Jinde Cao; Xinli Shi; Leszek Rutkowski

cs.CVMar 12, 2026

Alternating Gradient Flow Utility: A Unified Metric for Structural Pruning and Dynamic Routing in Deep Networks

Authors: Tianhao Qian, Zhuoxuan Li, Jinde Cao, Xinli Shi, Leszek Rutkowski

Organizations: the School of Mathematics, Southeast University, Nanjing 210096, China · the School of Mathematics, Southeast University, Nanjing 210096, China, and also with the Systems Research Institute of the Polish Academy of Sciences, Warsaw 01-447, Poland · the School of Cyber Science & Engineering, Southeast University, Nanjing 210096, China · the Systems Research Institute of the Polish Academy of Sciences, Warsaw 01-447, Poland; the Institute of Computer Science, AGH University of Krakow, Kraków 30-059, Poland; and also with SAN University, Łódź 90-113, Poland

Abstract

Efficient deep learning traditionally relies on static heuristics like weight magnitude or activation awareness (e.g., Wanda, RIA). While successful in unstructured settings, we observe a critical limitation when applying these metrics to the structural pruning of deep vision networks. These contemporary metrics suffer from a magnitude bias, failing to preserve critical functional pathways. To overcome this, we propose a decoupled kinetic paradigm inspired by Alternating Gradient Flow (AGF), utilizing an absolute feature-space Taylor expansion to accurately capture the network's structural "kinetic utility". First, we uncover a topological phase transition at extreme sparsity, where AGF successfully preserves baseline functionality and exhibits topological implicit regularization, avoiding the collapse seen in models trained from scratch. Second, transitioning to architectures without strict structural priors, we reveal a phenomenon of Sparsity Bottleneck in Vision Transformers (ViTs). Through a gradient-magnitude decoupling analysis, we discover that dynamic signals suffer from signal compression in converged models, rendering them suboptimal for real-time routing. Finally, driven by these empirical constraints, we design a hybrid routing framework that decouples AGF-guided offline structural search from online execution via zero-cost physical priors. We validate our paradigm on large-scale benchmarks: under a 75% compression stress test on ImageNet-1K, AGF effectively avoids the structural collapse where traditional metrics aggressively fall below random sampling. Furthermore, when systematically deployed for dynamic inference on ImageNet-100, our hybrid approach achieves Pareto-optimal efficiency. It reduces the usage of the heavy expert by approximately 50% (achieving an estimated overall cost of 0.92 $\times$ ) without sacrificing the full-model accuracy.

Figures & tables

Fig. 1 : Overview of the AGF-Guided Efficiency Framework. The pipeline integrates (A) Gradient-based Utility Calibration, (B) Iterative Structural Pruning, and (C) Confidence-based Dynamic Routing. AGF identifies the topological skeleton, while the routing policy handles runtime complexity.

Fig. 2 : Analysis of Metric Stability and Orthogonality (WideResNet on CIFAR-100 at

k=32

). (a) Selection Stability: AGF demonstrates superior batch-to-batch structural consistency compared to the Taylor baseline. (b) Metric Orthogonality (

J\approx 0

): A normalized scatter plot reveals a fundamental divergence between static and dynamic metrics. Traditional magnitude metrics (

\ell_{1}

) rigidly select high-capacity channels (blue crosses), whereas AGF identifies an entirely distinct, orthogonal subset of dynamic routing hubs with high kinetic potential (red dots).

Moderate Compression ( $k=256$ , 25% Width)
Strategy	Width ( $k$ )	Mean Acc (%)	Std ( $\sigma$ )	Best Seed (%)
Baseline (Full)	1024	68.66	N/A	68.66
Random Pruning	256	70.21	0.45	70.66
$\ell_{1}$ -Norm Pruning	256	70.32	0.22	70.46
AGF Pruning (Ours)	256	70.75	0.08	70.81
Narrow (Scratch)	256	70.92	0.28	71.17

TABLE I : Detailed Comparison of Pruning vs. Training-from-Scratch. Consistency vs. Peak Performance: While RIA achieves a slight advantage at extreme sparsity (

k=32

), it exhibits inconsistency at moderate sparsity (

k=128

), falling behind simple baselines. AGF (Ours) demonstrates the best trade-off: it dominates at

k=128

and maintains competitive accuracy with superior stability (

\sigma=0.12

) at

k=32

, avoiding the high variance of activation-based methods like Wanda (

\sigma=0.63

) and RIA (

\sigma=0.40

Strategy	Calibration Data	Ep 10 (Post-Prune)	Peak Accuracy	Epoch of Peak	vs. Taylor
Taylor (Baseline)	100 Batches	84.26%	84.57%	Ep 20	-
AGF (Dense)	100 Batches	83.84%	84.56%	Ep 17	-0.01%
AGF (Sparse)	10 Batches	83.60%	84.90%	Ep 20	+0.33%
Random	N/A	83.64%	83.61%	-	-0.96%
$\ell_{1}$ -Norm	N/A	83.15%	83.53%	-	-1.04%

TABLE II : 20-Epoch Recovery Analysis on ImageNet-100. Comparing the Peak Accuracy of pruning strategies. Surprisingly, AGF with only 10 batches achieves the highest performance (

\mathbf{84.90\%}

), outperforming both Taylor and the high-sample AGF variant. This suggests that the stochastic noise in limited-batch gradient estimation acts as a beneficial regularizer for structure selection.

*Estimated full fine-tuning performance on CIFAR-100.
Metric Type	Strategy	Moderate ( $k=1536$ , 50%)		Extreme ( $k=768$ , 25%)
Metric Type	Strategy	Ep 1 Acc	Peak Acc	Ep 1 Acc	Peak Acc
Unpruned Baseline (Full Capacity)		91.50%*
Static/Magnitude (Activation-Aware)	RIA Pruning	-	-	73.61%	82.24%
Static/Magnitude (Activation-Aware)	Wanda Pruning	79.82%	84.96%	66.50%	81.36%
Dynamic/Gradient	AGF Pruning (Ours)	85.00%	86.81%	73.88%	81.89%

TABLE III : Structural Pruning on ViT-Base (MLP Width = 3072). Moderate (

k=1536

): AGF demonstrates superior recovery and peak performance over magnitude-based methods. Extreme (

k=768

): All deterministic metrics encounter a severe Sparsity Bottleneck, highlighting the theoretical limitations of single-dimensional proxies and motivating our Hybrid Routing approach.

Regime	Metric	Full	Pruned	Ratio	Signal
ResNet-50 (Kinetic)	$\ell_{1}$ Norm	107,142	1,781	60.2 $\times$	Physical Truth
ResNet-50 (Kinetic)	AGF Utility	0.0048	0.0001	44.7 $\times$	Aligned
WideResNet (Saturated)	$\ell_{1}$ Norm	232,682	1,557	149.4 $\times$	Physical Truth
WideResNet (Saturated)	AGF Utility	4.89e-4	2.29e-5	21.4 $\times$	Compressed

TABLE IV : Regime-Dependent Fidelity of Energy Proxies. ResNet-50 (Kinetic): AGF utility (

\mathbf{44.7\times}

) remains aligned with the physical capacity gap. WideResNet-18 (Saturated): AGF utility becomes compressed (

\mathbf{21.4\times}

) due to vanishing gradients, underestimating the massive physical disparity (

\mathbf{149.4\times}

Fig. 3 : Difficulty Distribution of Routed Samples. We measure sample difficulty using the prediction entropy of the full-capacity baseline. The lightweight router successfully learns to disentangle the input space without human priors: ”easy” samples (low entropy) are dominantly routed to the Pruned Expert (Green), while ”hard” ambiguous samples (high entropy) are forwarded to the Full Expert (Red). This adaptive decoupling is the core mechanism enabling our system’s high efficiency.

Method	Acc (%)	vs. Random	Route Ratio (Full:Small)	Est. Cost
Static Full	88.74	+4.42	100 : 0	1.00x
Static Pruned	79.80	-4.52	0 : 100	0.85x
Random Policy	84.32	-	50 : 50	0.92x
Ours (Adaptive)	88.78	+4.46	48.5 : 51.5	0.92x

TABLE V : Main Results on ImageNet-100. Our Adaptive Router (Ours) matches the upper bound (Full Model) while reducing the usage of the heavy expert by

\approx 50\%

Fig. 4 : Accuracy-Efficiency Trade-off on ImageNet-100. Our adaptive method (Red) establishes a superior Pareto frontier compared to static and random baselines, enabling dynamic trade-offs between performance and cost.

Reference Upper Bound (100% Capacity)
Method	Metric Proxy	Acc (%)	vs. Random	vs. Wanda
Unpruned Baseline	Full Capacity	80.35	-	-
Extreme Compression Regime (25% Capacity)
Taylor Pruning	Loss Approx ( $\nabla W\cdot W$ )	64.42	-0.51	-0.27
$\ell_{1}$ -Norm Pruning	Weights ( $\|W\|$ )	64.54	-0.39	-0.15
Wanda Pruning	Weights $\times$ Act	64.69	-0.24	-

TABLE VI : Extreme Stress Test on ImageNet-1K (ResNet-50). Comparison of pruning strategies under a severe 75% structural compression (

k=64

) on the bottleneck layers. Reference: The unpruned model achieves 80.35%. The significant absolute drop across all methods reflects an information bottleneck. Notably, magnitude-based heuristics (

\ell_{1}

, Wanda) and Taylor approximations suffer degradation below the Random baseline. AGF remains competitive with uniform sampling, indicating that the network has reached an intrinsic capacity limit without suffering from negative proxy biases.

Fig. 5 : Qualitative Visualization of Routing Decisions. Top row: ”Easy” samples (e.g., clear objects) routed to the Pruned Expert. Bottom row: ”Hard” samples (e.g., clutter) routed to the Full Expert. The router effectively identifies samples requiring higher capacity for correct classification.

Fig. 6 : Metric Efficiency and Slope Analysis. Main Plot: Both AGF and L1 significantly outperform Random selection. Zoomed Inset: Highlights the critical divergence at high costs. While L1 saturates (Negative Slope), AGF maintains a positive marginal gain .

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Threshold ( $\tau$ )	AGF (Ours)		$\ell_{1}$ -Norm		Random		Regime
Threshold ( $\tau$ )	Acc (%)	Cost	Acc (%)	Cost	Acc (%)	Cost	Regime
0.000	68.54	1.00	68.83	1.00	68.13	1.00	Pruned Only
0.500	69.26	7.28	69.61	7.93	69.25	8.26	Low Cost
0.700	70.94	21.33	71.04	21.35	70.87	21.69
0.800	71.40	28.39	71.47	29.16	71.37	29.30
0.900	72.10	38.08	72.11	39.29	72.08	39.61	Balanced

Appendix

TABLE VII : Full Numerical Sweep of Confidence-Based Routing. We compare the Accuracy (%) and Normalized Cost of three strategies across varying confidence thresholds

\tau

. Bold indicates the best trade-off (Highest Accuracy or Lowest Cost) at critical operating points.

Parameter	Value
Optimizer	SGD (Nesterov)
Momentum	0.9
Weight Decay	$5\times 10^{-4}$
Total Epochs	150
Batch Size	32
Initial Learning Rate	$1.0\times 10^{-3}$

Appendix

TABLE VIII : Hyperparameters for Teacher Training (CIFAR-100).

Neural Network Pruning

Neural network pruning removes weights, neurons, channels, heads, or layers to reduce model size and computation while retaining useful behavior.

See all 289 papers

Hyperflux: Pruning Reveals Importance
Apr 6, 2025Eugen Barbulescu, Antonio Alexoaie, Lucian BusoniuSparse Neural Networks Neural Network Training Dynamics
Department of Computer Science Technical University of Cluj-Napoca · Department of Automation Technical University of Cluj-Napoca
Mix-and-Match Pruning: Globally Guided Layer-Wise Sparsification of DNNs
Mar 17, 2026Danial Monachan, Samira Nazari, Mahdi Taheri +4Neural Network Compression Neural Network Pruning
Brandenburg Technical University, Cottbus, Germany · University of Zanjan, Iran · Tallinn University of Technology, Estonia +1
HiAP: A Multi-Granular Stochastic Auto-Pruning Framework for Vision Transformers
Mar 12, 2026Andy Li, Aiden Durrant, Milan Markovic +1Efficient ViTs Attention Head Pruning
School of Natural and Computing Sciences, University of Aberdeen, UK · School of Computing Sciences, University of East Anglia, UK · Department of Physics and Technology, UiT The Arctic University of Norway, Norway

Apr 6, 2025J/K move · Enter open · S save

Hyperflux: Pruning Reveals Importance

Eugen Barbulescu, Antonio Alexoaie, Lucian Busoniu

Department of Computer Science Technical University of Cluj-Napoca · Department of Automation Technical University of Cluj-Napoca

Sparse Neural Networks Neural Network Training Dynamics Neural Network Pruning

Open paper AskPDF arXiv

Network pruning is used to reduce inference latency and power consumption in large neural networks. However, most methods focus on empirical results at the expense of understanding the pruning process. We introduce Hyperflux, a novel

L_0

method which models pruning as a continuously evolving system determined by flux, the gradient response to a weight's removal, and pressure, a global regularization driving weights toward pruning. By exploiting this model, Hyperflux's pruning behavior becomes understandable at both microscopic (weight regrowth/pruning) and macroscopic (sparsity convergence, etc.) levels. We also introduce a novel pressure scheduler that reliably targets desired sparsities. Hyperflux achieves competitive results with ResNet-50, VGG-19 and DeiT-T/S on CIFAR-10, CIFAR-100 and ImageNet datasets.

Abstract

Neural Network Pruning

Explore similar work

Hyperflux: Pruning Reveals Importance

Mix-and-Match Pruning: Globally Guided Layer-Wise Sparsification of DNNs

HiAP: A Multi-Granular Stochastic Auto-Pruning Framework for Vision Transformers