Unified Response Geometry for Structured Pruning

Kaixiang Shu

cs.CVSep 16, 2026

Unified Response Geometry for Structured Pruning

Authors: Kaixiang Shu

Organizations: Independent Researcher

Abstract

Structured pruning is commonly formulated as ranking individual channels, although channel responses can be complementary or cancel through downstream mixing. Motivated by these response interactions, we formulate pruning as the selection of a subset with large joint response capacity, followed by a separate functional realization step. Our unified response geometry maps each candidate set to $M(D,R)=D^{1/2}RD^{1/2}$ and uses its determinant together with Schur-greedy residuals to select non-redundant coordinates. The same construction yields two information-conditioned instances: an unlabeled instance based on activation covariance, and a task-conditioned instance that combines activation and gradient variance for response scale with gradient correlation for complementarity. To convert the selected subset into an executable network, we fold predictable removed responses into successor weights through ridge compensation and recalibrate batch-normalization statistics, without fine-tuning the network. On ImageNet ResNet-50, the unlabeled instance reaches $65.4\%$ and $53.9\%$ Top-1 accuracy at 30% and 40% deletion, versus $59.8\%$ and $43.1\%$ for strength-only selection; the task-conditioned instance reaches $67.7\%$ and $56.3\%$ under the same protocol. A six-family screen shows architecture-dependent behavior, with positive relative contrasts in several convolutional and expansion-layer settings and clear boundary cases in windowed attention. These results support response geometry as a conditional principle for structured pruning, with its benefit determined jointly by the observed response and the architecture in which that response is realized.

Figures & tables

Figure 1: Overview of Unified Response Geometry for Structured Pruning. Downstream responses can be shared, complementary, or cancelling, so marginal importance does not determine the value of a retained set. The unified matrix combines response scale $D$ and relationship structure $R$ ; determinant-based subset capacity and Schur residuals define the greedy selection. Activation-only and activation–gradient observations instantiate the same objective, after which ridge compensation and BN recovery produce an executable pruned network without backbone or classifier fine-tuning.

Method	30%	40%
L2	47.69 $\pm$ 1.49	31.25 $\pm$ 0.69
Strength-only	59.81 $\pm$ 2.17	43.13 $\pm$ 0.32
Unlabeled instance	65.43 $\pm$ 0.20	53.92 $\pm$ 0.48
Task-conditioned instance	67.66 $\pm$ 2.12	56.25 $\pm$ 2.61

Table 1: Primary BN-recovered Top-1 accuracy (mean

\pm

s.d. over three calibration seeds).

Comparison	Deletion	Mean (pp)	Positive seeds
Unlabeled instance $-$ strength-only	30%	+5.62	3/3
Task-conditioned instance $-$ strength-only	30%	+7.85	3/3
Unlabeled instance $-$ strength-only	40%	+10.79	3/3
Task-conditioned instance $-$ strength-only	40%	+13.12	3/3

Table 2: Pre-specified paired Top-1 differences across calibration seeds.

Figure 2: Calibration-seed contrasts for the two final response-geometry instances. Each point is a paired Top-1 difference from strength-only selection on the same confirmation images. Both instances are positive for all three seeds at both deletion rates.

Method	Deletion	Direct	Compensated	+BN	Change to +BN (pp)
L2	30%	0.6	39.6	49.6	+49.0
L2	40%	0.2	16.2	31.6	+31.3
Strength-only	30%	0.0	52.1	59.6	+59.6
Strength-only	40%	0.2	25.3	44.3	+44.1
Unlabeled instance	30%	0.6	55.0	64.1	+63.5
Unlabeled instance	40%	0.2	27.9	50.2	+50.0

Table 3: Selection versus functional realization in the representative flow diagnostic.

Figure 3: Selection and functional recovery are separate stages in a representative flow diagnostic. Panel a follows direct slicing, ridge compensation and BN recalibration; panels b and c show the corresponding accuracy and teacher-KL increments; panel d reports the final BN endpoint. The same recovery procedure is applied to every selector, so the figure diagnoses the pipeline rather than replacing the multi-seed selection comparison.

Figure 4: Six-family cross-architecture screening. Panel A summarizes the median accuracy trajectory and interquartile spread; panel B reports paired 40% contrasts against regime-matched comparators; panels C–H show each architecture. Negative contrasts remain visible to define the operating boundary.

Model	30%				40%				50%
	Unlabeled	best U	Task-cond.	best L	Unlabeled	best U	Task-cond.	best L	Unlabeled	best U	Task-cond.	best L
ResNet-50	40.9	40.9 (Unlabeled instance)	50.8	50.8 (Task-conditioned instance)	18.8	18.8 (Unlabeled instance)	27.9	27.9 (Task-conditioned instance)	5.4	5.4 (Unlabeled instance)	9.9	9.9 (Task-conditioned instance)
ConvNeXt-B	71.7	72.5 (APoZ)	65.7	72.5 (Taylor-gate)	47.1	47.2 (APoZ)	49.7	56.0 (Fisher-gate)	3.0	3.8 (APoZ)	15.2	24.0 (OBD)
EfficientNet-B3	39.7	41.0 (FPGM)	61.0	61.0 (Task-conditioned instance)	19.4	19.4 (Unlabeled instance)	40.0	40.0 (Task-conditioned instance)	5.2	5.2 (Unlabeled instance)	19.9	19.9 (Task-conditioned instance)
MobileNetV2	15.8	23.5 (L1)	42.6	42.6 (Task-conditioned instance)	4.7	7.4 (LAMP)	16.2	16.2 (Task-conditioned instance)	1.2	1.3 (LAMP)	4.7	4.7 (Task-conditioned instance)
ViT-S/16	52.5	52.5 (Unlabeled instance)	54.0	54.0 (Task-conditioned instance)	31.3	31.3 (Unlabeled instance)	36.3	42.5 (Fisher-gate)	9.8	9.8 (Unlabeled instance)	15.8	21.3 (Fisher-gate)

Table 4: Six-family structural-pruning screening. Entries are Top-1 accuracy (%). “Best U/L” is the best method within the unlabeled or label/gradient comparator family at the same deletion rate; the two final response-geometry instances are shown separately so architecture boundaries remain visible.

Figure 5: Architecture–mechanism alignment. Descriptor–contrast pairs are shown for all six architectures. The strongest directional association is between cancellation residual and the unlabeled contrast at 40%, while low-rank and task-conditioned relationships remain mixed.

Appendix figures & tables13 assets

Supplementary material from the paper’s appendix.

Appendix

Method	Deletion	Top-1 (%)	KL $\downarrow$	Agreement (%)
L2	30%	47.69	2.139	51.35
L2	40%	31.25	3.272	33.06
Strength-only	30%	59.81	1.253	64.58
Strength-only	40%	43.13	2.369	46.13
Unlabeled instance	30%	65.43	0.890	71.50
Unlabeled instance	40%	53.92	1.536	57.80

Appendix

Table 5: Secondary BN metrics averaged over calibration seeds.

Comparison	Deletion	Mean (pp)	Positive seeds
Unlabeled instance $-$ strength-only	30%	+5.62	3/3
Task-conditioned instance $-$ strength-only	30%	+7.85	3/3
Unlabeled instance $-$ strength-only	40%	+10.79	3/3
Task-conditioned instance $-$ strength-only	40%	+13.12	3/3

Appendix

Table 6: Pre-specified paired Top-1 differences across calibration seeds.

Method	ResNet-50	ConvNeXt-B	EfficientNet-B3	MobileNetV2	ViT-S/16	Swin-T
Random	21.7	66.3	31.9	8.9	41.6	60.1
L1	5.6	24.5	36.6	23.5	13.2	63.0
L2	11.9	50.3	40.0	22.7	20.1	65.2
BN- $\gamma$	6.0	24.5	33.2	8.2	13.2	63.0
APoZ	22.9	72.5	39.8	4.7	45.6	63.0
LAMP	11.9	50.3	40.0	22.7	20.1	65.2

Appendix

Table 7: Complete unlabeled comparison at 30% deletion. Top-1 accuracy (%).

Method	ResNet-50	ConvNeXt-B	EfficientNet-B3	MobileNetV2	ViT-S/16	Swin-T
Taylor	37.0	67.4	53.8	32.5	43.5	41.4
GradNorm	44.9	68.3	44.3	18.8	40.7	51.6
Fisher	43.9	67.4	52.4	23.0	44.1	55.6
Taylor-w	41.7	68.5	47.8	24.3	49.9	54.0
Taylor-gate	30.9	72.5	50.0	32.6	47.9	52.0
Fisher-gate	29.9	69.7	51.6	28.7	53.3	58.7

Appendix

Table 8: Complete label/gradient comparator family at 30% deletion. Top-1 accuracy (%).

Method	ResNet-50	ConvNeXt-B	EfficientNet-B3	MobileNetV2	ViT-S/16	Swin-T
Random	4.2	39.2	13.5	1.5	16.1	45.9
L1	0.9	2.0	9.2	3.0	4.4	51.8
L2	3.6	14.0	15.1	7.4	6.7	55.2
BN- $\gamma$	0.4	2.0	11.8	1.7	4.4	51.8
APoZ	5.5	47.2	16.8	1.4	22.1	52.2
LAMP	3.6	14.0	15.1	7.4	6.7	55.2

Appendix

Table 9: Complete unlabeled comparison at 40% deletion. Top-1 accuracy (%).

Method	ResNet-50	ConvNeXt-B	EfficientNet-B3	MobileNetV2	ViT-S/16	Swin-T
Taylor	14.2	46.6	31.2	9.0	24.9	19.4
GradNorm	24.2	51.4	19.0	6.1	18.3	30.0
Fisher	19.6	51.7	21.2	6.5	22.3	38.4
Taylor-w	18.5	52.1	14.9	4.0	30.2	34.8
Taylor-gate	8.2	54.5	23.7	5.8	32.2	31.4
Fisher-gate	7.5	56.0	22.9	5.1	42.5	42.1

Appendix

Table 10: Complete label/gradient comparator family at 40% deletion. Top-1 accuracy (%).

Model	Act. low-rank	Cancellation residual	Act. eff. rank	Grad. low-rank	Grad. eff. rank
ResNet-50	0.083	0.953	0.122	0.020	0.477
ConvNeXt-B	0.126	0.958	0.019	0.030	0.077
EfficientNet-B3	0.135	0.910	0.024	0.018	0.207
MobileNetV2	0.119	0.936	0.035	0.014	0.479
ViT-S/16	0.091	0.969	0.029	0.032	0.074
Swin-T	0.428	0.908	0.002	0.018	0.106

Appendix

Table 11: Architecture-level response descriptors used in the Project1 alignment analysis. Values are parameter-weighted medians across prunable layers.

Descriptor	Unlabeled 30%	Unlabeled 40%	Task-cond. 30%	Task-cond. 40%
activation low-rank	-0.60 (0.243)	-0.49 (0.356)	-0.37 (0.498)	-0.14 (0.803)
cancellation residual	+0.77 (0.104)	+0.89 (0.035)	+0.09 (0.920)	-0.03 (1.000)
activation effective rank	+0.26 (0.659)	+0.14 (0.803)	+0.60 (0.243)	+0.37 (0.498)
gradient low-rank	+0.89 (0.035)	+0.77 (0.104)	-0.37 (0.498)	-0.26 (0.659)
readout alignment	-0.71 (0.137)	-0.77 (0.104)	+0.03 (1.000)	+0.26 (0.659)
gradient effective rank	-0.60 (0.243)	-0.54 (0.298)	+0.37 (0.498)	+0.14 (0.803)

Appendix

Table 12: Spearman correlations between architecture descriptors and matched pruning contrasts. Exact permutation p-values use all 6! permutations; these are descriptive because

n=6

Figure 6: All descriptor–contrast pairs in the six-model mechanism test. Titles report Spearman

\rho

and exact permutation

p

; every pair is shown to avoid selective emphasis.

Figure 7: Complete configured methods at 30% coordinate deletion for the six-model screen.

Figure 8: Complete configured methods at 40% coordinate deletion for the six-model screen.

Figure 9: Unlabeled-instance Top-1 contrast relative to L2 across deletion rates and architectures. The right panel reports the mean contrast and the fraction of architectures with a positive contrast.

Figure 10: Task-conditioned Top-1 contrast relative to Fisher across deletion rates and architectures. The right panel reports the mean contrast and the fraction of architectures with a positive contrast.

Structured Pruning

Structured pruning removes whole channels, heads, layers, or experts to shrink neural networks while preserving accuracy and enabling efficient inference.

See all 66 papers

Oct 11, 2026

WUSPruner: Second-Moment-Optimal Transforms for Structured Pruning

Structured pruning is typically formulated as identifying which model dimensions to remove, often within the model's original representation. However, the effect of coordinate deletion fundamentally depends on the basis in which pruning is performed. This raises a more fundamental question: how should the basis be chosen specifically for coordinate deletion? We introduce WUSPruner, a transform-based framework for structured pruning of large language models. We formulate basis selection as a local linear reconstruction problem that directly minimizes the reconstruction loss induced by pruning. Inspired by the second-moment transformation in WUSH, we consider a WUS transform constructed jointly from activation and weight statistics. We show that under positive-definite second moments, WUS transform minimizes the local reconstruction loss over all invertible changes of basis. In the resulting basis, the activation and weight second moments are simultaneously diagonalized and balanced, causing the deletion error to decompose into independent spectral contributions and yielding a principled criterion for coordinate selection. Guided by this result, we develop a scalable pruning method based on blockwise WUS transformations and compact input projections. Experiments on Llama and Qwen models from 8B to 72B parameters show consistent improvements over structured pruning baselines across language modeling, knowledge, commonsense reasoning, mathematics, and code generation benchmarks, while also delivering practical reductions in inference latency and memory usage.

Shengkun Tang, Dan Alistarh, Zhiqiang Shen

MBZUAI · ISTA

LLM Pruning Structured Pruning LLM Compression

Ask PDF

Sep 21, 2026

Artificial Structure Function Search: Preserving Artificial Functional Connectivity for Structured Pruning

Structured pruning is a model compression technique that is used to reduce the computational cost of deploying deep neural networks on resource-constrained devices. Popular methods of pruning rely on opaque heuristics or weight-based criteria that give no indication as to the structural dependencies in the network. To address these limitations we present Artificial Structure Function Search (ASF-S): a novel structured pruning framework. ASF-S utilizes Principle Gradient Importance (PGI): a novel prune-candidate selection criteria that is inspired by structure-function relationships in the brain. By ensuring the pruned structure of the model respects topographical organization of the output layer, we define Artificial Functional Connectivity (AFC) for artificial neural networks. AFC provides evidence to demonstrate that accurate smaller networks can be found using careful prune candidate selection criteria. We present results for PGI as a selection criterion and for ASF-S as a pruning framework against recent benchmarks, demonstrating that our method yields model variants with 70% parameter reduction, that can recover baseline accuracy without re-training the pruned layers.

Mindula Illeperuma, Rafael Pina, Charuka Herath +2

Intitute for Digital Technologies, Loughborough University London · Center for Neuromorphic Intelligence, Aston University

Model Compression Functional Connectivity Neural Network Pruning

Ask PDF

Sep 14, 2026

Optimal Pruning for Neural Architectures using Fisher Information Distances

A new scheme for parameter pruning is introduced, derived from the differential-geometric distance in model space. Pruning a parameter sets its value to zero, representing a displacement of the model to the hypersurface on which that parameter vanishes. The minimal distance from the unpruned model to this hypersurface is naturally computed via the geodesic distance in the model space as determined by the Fisher information metric. This distance determines the true change in the model, and its performance, under pruning. By analysing progressively more faithful approximations of this geodesic distance a natural hierarchy of optimality for pruning methods is determined. This starts with the traditional magnitude pruning, then develops into new more sophisticated and effective pruning schemes. The method is demonstrated for both fully-connected networks and vision transformers, on MNIST and CIFAR-10, over the complete

0

100\%

pruning range and across five random seeds. It outperforms pruning by parameter magnitude and by the local Fisher information alone in every architecture and dataset combination considered, on both accuracy and the Matthews correlation coefficient. Additionally, analysis of different levels of geodesic approximation produces intermediate pruning schemes that are computationally efficient and maintain near-optimal performance. This geometric picture supplies not only a state-of-the-art pruning methodology for AI models, but also a verified and mathematically-motivated justification for pruning schemes.

David S. Berman, Yen-Yu Fu, Edward Hirst +1

Centre for Theoretical Physics, Queen Mary University of London, Mile End Road, London E1 4NS, UK · Institute of Mathematics, Statistics and Scientific Computing, University of Campinas (Unicamp), 13083-859, Brazil

Neural Network Pruning Neural Network Compression

Ask PDF

Abstract

Structured Pruning

Explore similar work

WUSPruner: Second-Moment-Optimal Transforms for Structured Pruning

Artificial Structure Function Search: Preserving Artificial Functional Connectivity for Structured Pruning

Optimal Pruning for Neural Architectures using Fisher Information Distances