How Does Parameter Pruning Reshape DNN Representations? An Interaction-Driven Exploration

Fangbo Li; Junpeng Zhang; Qihan Ren; Quanshi Zhang

cs.LGSep 6, 2026

How Does Parameter Pruning Reshape DNN Representations? An Interaction-Driven Exploration

Authors: Fangbo Li, Junpeng Zhang, Qihan Ren, Quanshi Zhang

Abstract

This study focuses on the scientific problem of understanding internal factors that govern the diverse performance degradation of deep neural networks (DNNs) when different parameters are pruned. In order to explain why pruning certain parameters leads to significant performance degradation but pruning other parameters does not, we examine how the pruning operation affects the interaction patterns encoded by the DNN. We find that when we progressively increase the pruning ratio, the interaction patterns encoded by DNNs exhibit a distinct three-phase dynamics, i.e., model performance is not largely affected until the pruning operation begins to remove low-order interactions, and low-order interactions exhibit strong generalizability. Moreover, we find that the high sensitivity of DNN performance to the pruning of certain modules is attributed to whether the pruning operation removes generalizable low-order interaction patterns.

Figures & tables

Figure 1: (a) It is proven [ 3 ] that the complex prediction logic can be mathematically decomposed into a small set of AND/OR interactions. (b) Some interactions are removed (shown in light color), while others are preserved (shown in dark color) after pruning. (c) When we progressively increase the pruning ratio, we find that the test accuracy does not decrease noticeably until the pruning operation begins to remove generalizable interactions.

Figure 2: Illustration of removed, preserved, and emerged interactions after pruning.

Figure 3: Changes in the distributions of removed interaction (

{\bf I}^{(\operatorname{m}),+}_{\textnormal{removed}}

and

{\bf I}^{(\operatorname{m}),-}_{\textnormal{removed}}

) and the distributions of preserved interaction (

{\bf I}^{(\operatorname{m}),+}_{\textnormal{preserved}}

and

{\bf I}^{(\operatorname{m}),-}_{\textnormal{preserved}}

) when the pruning ratio increases. These changes can be divided into three phases. The four selected checkpoint pruning ratios correspond to the inflection points of the accuracy curve, and represent the boundaries of the three phases in the progressive parameter pruning process. Please see Appendix F.1 for results on more pruning ratios.

Figure 4: (left) Changes in the ratio of preserved low/high-order interactions

\rho_{\textnormal{low}}

\rho_{\textnormal{high}}

and the classification accuracy when the pruning ratio increases. (right) Changes in the offsetting ratio of removed interactions

\kappa

and the classification accuracy when the pruning ratio increases.

Figure 5: (top) The ratio of preserved low/high-order interactions

\rho_{\textnormal{low}}

\rho_{\textnormal{high}}

after pruning each module. (bottom) The offsetting ratio of removed interactions

\kappa

after pruning. After pruning sensitive modules, low-order interactions are significantly removed, and the removed interactions exhibit poorer offsetting ratio. Please see Appendix G.2 for more results.

Figure 6: The distributions of generalizable interactions (denoted by

{\bf I}^{(\operatorname{m}),+}_{\textnormal{general}}

and

{\bf I}^{(\operatorname{m}),-}_{\textnormal{general}}

) and all interactions (denoted by

{\bf I}^{(\operatorname{m}),+}

and

{\bf I}^{(\operatorname{m}),-}

) over different orders. Low-order interactions exhibit stronger generalizability than high-order ones. Please see Appendix H for results on more DNNs.

Figure 7: (left) The ratio of removed generalizable interactions

\rho_{\textnormal{removed}}

after pruning each module. Pruning insensitive modules removes much less generalizable interactions than pruning sensitive modules. Please see Appendix G for results on more DNNs. (right) The ratio of removed generalizable interactions

\rho_{\textnormal{removed}}

when the pruning ratio progressively increases. The removal of generalizable interactions leads to DNN performance degradation.

Appendix figures & tables42 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 8: Top-5 classification accuracy of pruned VGG-16 and pruned AlexNet on TinyImageNet under different pruning ratio.

Model	$\tau$
LLaMA-7B	0.60
LLaMA-3.1-8B	1.00
DeepSeek-R1-Distill-LLaMA-8B	0.70
Qwen-2.5-7B	0.70
ResNet-50	0.50
ResNet-101	0.45

Appendix

Table 1: Threshold

\tau

used for different DNNs.

Figure 9: The distributions of removed interactions (

{\bf I}^{(\operatorname{m}),+}_{\textnormal{removed}}

and

{\bf I}^{(\operatorname{m}),-}_{\textnormal{removed}}

) and preserved interactions (

{\bf I}^{(\operatorname{m}),+}_{\textnormal{preserved}}

and

{\bf I}^{(\operatorname{m}),-}_{\textnormal{preserved}}

) over different orders encoded by pruned ResNet-56 at different pruning ratios. The distributions are arranged in ascending order of the pruning ratio.

Figure 10: The distributions of removed interactions (

{\bf I}^{(\operatorname{m}),+}_{\textnormal{removed}}

and

{\bf I}^{(\operatorname{m}),-}_{\textnormal{removed}}

) and preserved interactions (

{\bf I}^{(\operatorname{m}),+}_{\textnormal{preserved}}

and

{\bf I}^{(\operatorname{m}),-}_{\textnormal{preserved}}

) over different orders encoded by pruned VGG-19 at different pruning ratios. The distributions are arranged in ascending order of the pruning ratio.

Figure 11: The ratio of preserved low-order/high-order interactions

\rho_{\textnormal{low}}

\rho_{\textnormal{high}}

during progressive parameter pruning on different DNNs.

Figure 12: The offsetting ratio of removed interactions

\kappa

during progressive parameter pruning on different DNNs.

Figure 13: The ratio of removed generalizable interactions

\rho_{\textnormal{removed}}

during progressive parameter pruning on different DNNs.

Perspectives	Phase 1	Phase 2	Phase 3
Testing accuracy	barely affected	significantly decreased	remains low
Low-order interactions	slightly removed	significantly removed	most have been removed
High-order interactions	significantly removed	significantly removed	most have been removed
Offsetting of the removed	extremely high	comparatively low	comparatively low
Generalizability of the removed	slightly increase	significantly increase	most have been removed

Appendix

Table 2: Analysis of three-phase dynamics of interactions from different perspectives.

Figure 14: The distributions of removed interactions (

{\bf I}^{(\operatorname{m}),+}_{\textnormal{removed}}

and

{\bf I}^{(\operatorname{m}),-}_{\textnormal{removed}}

) and preserved interactions (

{\bf I}^{(\operatorname{m}),+}_{\textnormal{preserved}}

and

{\bf I}^{(\operatorname{m}),-}_{\textnormal{preserved}}

) over different orders encoded by LLaMA-7B after pruning each sensitive/insensitive module. The distributions are arranged in ascending order of the increase in test loss after pruning. For clarity, this figure presents the first of two subsets of results.

Figure 15: The distributions of removed interactions (

{\bf I}^{(\operatorname{m}),+}_{\textnormal{removed}}

and

{\bf I}^{(\operatorname{m}),-}_{\textnormal{removed}}

) and preserved interactions (

{\bf I}^{(\operatorname{m}),+}_{\textnormal{preserved}}

and

{\bf I}^{(\operatorname{m}),-}_{\textnormal{preserved}}

Figure 16: The distributions of removed interactions (

{\bf I}^{(\operatorname{m}),+}_{\textnormal{removed}}

and

{\bf I}^{(\operatorname{m}),-}_{\textnormal{removed}}

) and preserved interactions (

{\bf I}^{(\operatorname{m}),+}_{\textnormal{preserved}}

and

{\bf I}^{(\operatorname{m}),-}_{\textnormal{preserved}}

) over different orders encoded by LLaMA-3.1-8B after pruning each sensitive/insensitive module. The distributions are arranged in ascending order of the increase in test loss after pruning. For clarity, this figure presents the first of two subsets of results.

Figure 17: The distributions of removed interactions (

{\bf I}^{(\operatorname{m}),+}_{\textnormal{removed}}

and

{\bf I}^{(\operatorname{m}),-}_{\textnormal{removed}}

) and preserved interactions (

{\bf I}^{(\operatorname{m}),+}_{\textnormal{preserved}}

and

{\bf I}^{(\operatorname{m}),-}_{\textnormal{preserved}}

Figure 18: The distributions of removed interactions (

{\bf I}^{(\operatorname{m}),+}_{\textnormal{removed}}

and

{\bf I}^{(\operatorname{m}),-}_{\textnormal{removed}}

) and preserved interactions (

{\bf I}^{(\operatorname{m}),+}_{\textnormal{preserved}}

and

{\bf I}^{(\operatorname{m}),-}_{\textnormal{preserved}}

) over different orders encoded by DeepSeek-R1-Distill-LLaMA-8B after pruning each sensitive/insensitive module. The distributions are arranged in ascending order of the increase in test loss after pruning. For clarity, this figure presents the first of two subsets of results.

Figure 19: The distributions of removed interactions (

{\bf I}^{(\operatorname{m}),+}_{\textnormal{removed}}

and

{\bf I}^{(\operatorname{m}),-}_{\textnormal{removed}}

) and preserved interactions (

{\bf I}^{(\operatorname{m}),+}_{\textnormal{preserved}}

and

{\bf I}^{(\operatorname{m}),-}_{\textnormal{preserved}}

Figure 20: The distributions of removed interactions (

{\bf I}^{(\operatorname{m}),+}_{\textnormal{removed}}

and

{\bf I}^{(\operatorname{m}),-}_{\textnormal{removed}}

) and preserved interactions (

{\bf I}^{(\operatorname{m}),+}_{\textnormal{preserved}}

and

{\bf I}^{(\operatorname{m}),-}_{\textnormal{preserved}}

) over different orders encoded by Qwen-2.5-7B after pruning each sensitive/insensitive module. The distributions are arranged in ascending order of the increase in test loss after pruning. For clarity, this figure presents the first of two subsets of results.

Figure 21: The distributions of removed interactions (

{\bf I}^{(\operatorname{m}),+}_{\textnormal{removed}}

and

{\bf I}^{(\operatorname{m}),-}_{\textnormal{removed}}

) and preserved interactions (

{\bf I}^{(\operatorname{m}),+}_{\textnormal{preserved}}

and

{\bf I}^{(\operatorname{m}),-}_{\textnormal{preserved}}

Figure 22: The distributions of removed interactions (

{\bf I}^{(\operatorname{m}),+}_{\textnormal{removed}}

and

{\bf I}^{(\operatorname{m}),-}_{\textnormal{removed}}

) and preserved interactions (

{\bf I}^{(\operatorname{m}),+}_{\textnormal{preserved}}

and

{\bf I}^{(\operatorname{m}),-}_{\textnormal{preserved}}

) over different orders encoded by ResNet-50 after pruning each sensitive/insensitive module. The distributions are arranged in ascending order of the increase in test loss after pruning. For clarity, this figure presents the first of two subsets of results.

Figure 23: The distributions of removed interactions (

{\bf I}^{(\operatorname{m}),+}_{\textnormal{removed}}

and

{\bf I}^{(\operatorname{m}),-}_{\textnormal{removed}}

) and preserved interactions (

{\bf I}^{(\operatorname{m}),+}_{\textnormal{preserved}}

and

{\bf I}^{(\operatorname{m}),-}_{\textnormal{preserved}}

Figure 24: The distributions of removed interactions (

{\bf I}^{(\operatorname{m}),+}_{\textnormal{removed}}

and

{\bf I}^{(\operatorname{m}),-}_{\textnormal{removed}}

) and preserved interactions (

{\bf I}^{(\operatorname{m}),+}_{\textnormal{preserved}}

and

{\bf I}^{(\operatorname{m}),-}_{\textnormal{preserved}}

) over different orders encoded by ResNet-101 after pruning each sensitive/insensitive module. The distributions are arranged in ascending order of the increase in test loss after pruning. For clarity, this figure presents the first of two subsets of results.

Figure 25: The distributions of removed interactions (

{\bf I}^{(\operatorname{m}),+}_{\textnormal{removed}}

and

{\bf I}^{(\operatorname{m}),-}_{\textnormal{removed}}

) and preserved interactions (

{\bf I}^{(\operatorname{m}),+}_{\textnormal{preserved}}

and

{\bf I}^{(\operatorname{m}),-}_{\textnormal{preserved}}

Figure 26: The distributions of removed interactions (

{\bf I}^{(\operatorname{m}),+}_{\textnormal{removed}}

and

{\bf I}^{(\operatorname{m}),-}_{\textnormal{removed}}

) and preserved interactions (

{\bf I}^{(\operatorname{m}),+}_{\textnormal{preserved}}

and

{\bf I}^{(\operatorname{m}),-}_{\textnormal{preserved}}

) over different orders encoded by ResNet-152 after pruning each sensitive/insensitive module. The distributions are arranged in ascending order of the increase in test loss after pruning. For clarity, this figure presents the first of two subsets of results.

Figure 27: The distributions of removed interactions (

{\bf I}^{(\operatorname{m}),+}_{\textnormal{removed}}

and

{\bf I}^{(\operatorname{m}),-}_{\textnormal{removed}}

) and preserved interactions (

{\bf I}^{(\operatorname{m}),+}_{\textnormal{preserved}}

and

{\bf I}^{(\operatorname{m}),-}_{\textnormal{preserved}}

Figure 28: The distributions of removed interactions (

{\bf I}^{(\operatorname{m}),+}_{\textnormal{removed}}

and

{\bf I}^{(\operatorname{m}),-}_{\textnormal{removed}}

) and preserved interactions (

{\bf I}^{(\operatorname{m}),+}_{\textnormal{preserved}}

and

{\bf I}^{(\operatorname{m}),-}_{\textnormal{preserved}}

) over different orders encoded by ViT-Small after pruning each sensitive/insensitive module. The distributions are arranged in ascending order of the increase in test loss after pruning. For clarity, this figure presents the first of two subsets of results.

Figure 29: The distributions of removed interactions (

{\bf I}^{(\operatorname{m}),+}_{\textnormal{removed}}

and

{\bf I}^{(\operatorname{m}),-}_{\textnormal{removed}}

) and preserved interactions (

{\bf I}^{(\operatorname{m}),+}_{\textnormal{preserved}}

and

{\bf I}^{(\operatorname{m}),-}_{\textnormal{preserved}}

Figure 30: The distributions of removed interactions (

{\bf I}^{(\operatorname{m}),+}_{\textnormal{removed}}

and

{\bf I}^{(\operatorname{m}),-}_{\textnormal{removed}}

) and preserved interactions (

{\bf I}^{(\operatorname{m}),+}_{\textnormal{preserved}}

and

{\bf I}^{(\operatorname{m}),-}_{\textnormal{preserved}}

) over different orders encoded by ViT-Base after pruning each sensitive/insensitive module. The distributions are arranged in ascending order of the increase in test loss after pruning. For clarity, this figure presents the first of two subsets of results.

Figure 31: The distributions of removed interactions (

{\bf I}^{(\operatorname{m}),+}_{\textnormal{removed}}

and

{\bf I}^{(\operatorname{m}),-}_{\textnormal{removed}}

) and preserved interactions (

{\bf I}^{(\operatorname{m}),+}_{\textnormal{preserved}}

and

{\bf I}^{(\operatorname{m}),-}_{\textnormal{preserved}}

Figure 32: The distributions of removed interactions (

{\bf I}^{(\operatorname{m}),+}_{\textnormal{removed}}

and

{\bf I}^{(\operatorname{m}),-}_{\textnormal{removed}}

) and preserved interactions (

{\bf I}^{(\operatorname{m}),+}_{\textnormal{preserved}}

and

{\bf I}^{(\operatorname{m}),-}_{\textnormal{preserved}}

) over different orders encoded by ViT-Large after pruning each sensitive/insensitive module. The distributions are arranged in ascending order of the increase in test loss after pruning. For clarity, this figure presents the first of two subsets of results.

Figure 33: The distributions of removed interactions (

{\bf I}^{(\operatorname{m}),+}_{\textnormal{removed}}

and

{\bf I}^{(\operatorname{m}),-}_{\textnormal{removed}}

) and preserved interactions (

{\bf I}^{(\operatorname{m}),+}_{\textnormal{preserved}}

and

{\bf I}^{(\operatorname{m}),-}_{\textnormal{preserved}}

Figure 34: The ratio of preserved low-order/high-order interactions

\rho_{\textnormal{low}}

\rho_{\textnormal{high}}

after pruning sensitive/insensitive modules on different DNNs.

Figure 35: The offsetting ratio of removed interactions

\kappa

after pruning sensitive/insensitive modules on different DNNs.

Figure 36: The ratio of removed generalizable interactions

\rho_{\textnormal{removed}}

after pruning sensitive/insensitive modules on different DNNs.

Figure 37: The distributions of interactions and generalizable interactions

{\bf I}^{(\operatorname{m}),+}

{\bf I}^{(\operatorname{m}),-}

{\bf I}^{(\operatorname{m}),+}_{\textnormal{general}}

and

{\bf I}^{(\operatorname{m}),-}_{\textnormal{general}}

encoded by twelve DNNs. Interactions of different orders exhibit different generalizability. Low-order interactions exhibit stronger generalizability than high-order ones.

Figure 38: The distributions of removed interactions

{\bf I}^{(\operatorname{m}),+}_{\textnormal{removed}}

{\bf I}^{(\operatorname{m}),-}_{\textnormal{removed}}

at the end of Phase 1. Interactions of different orders exhibit different generalizability. Removed interactions exhibit strong offsetting effects.

Figure 39: Sorted interaction strength distribution on different input samples. Most interactions exhibit negligible strength.

Figure 40: The logical model representing the inference logic of the DeepSeek-R1-Distill-LLaMA-8B model on the input prompt “New York Department of Health recommends that all people should wear N95, KN95, or KF94 masks in all public.” The predicted next word is “settings.”

Figure 41: The logical model representing the inference logic of the Qwen-2.5-7B model on the input prompt “New York Department of Health recommends that all people should wear N95, KN95, or KF94 masks in all public.” The predicted next word is “settings.”

Figure 42: The logical model representing the inference logic of the DeepSeek-R1-Distill-LLaMA-8B model on the input prompt “On June 11, 2018, OpenAI researchers and engineers published a paper introducing the first generative pre-trained.” The predicted next word is “transformer.”

Figure 43: The logical model representing the inference logic of the Qwen-2.5-7B model on the input prompt “On June 11, 2018, OpenAI researchers and engineers published a paper introducing the first generative pre-trained.” The predicted next word is “transformer.”

Figure 44: The logical model representing the inference logic of the original DeepSeek-R1-Distill-LLaMA-8B model on the input prompt “Earth’s greenhouse effect would be too weak to keep temperature above freezing without.” The predicted next word is “carbon dioxide.”

Figure 45: The logical model representing the inference logic of the pruned DeepSeek-R1-Distill-LLaMA-8B model (under a pruning ratio of 41.01) on the input prompt “Earth’s greenhouse effect would be too weak to keep temperature above freezing without.” The predicted next word is “carbon dioxide.”

Figure 46: The logical model representing the inference logic of the original DeepSeek-R1-Distill-LLaMA-8B model on the input prompt “The government borrows money by selling financial products called.” The predicted next word is “bonds.”

Figure 47: The logical model representing the inference logic of the pruned DeepSeek-R1-Distill-LLaMA-8B model (under a pruning ratio of 41.01) on the input prompt “The government borrows money by selling financial products called.” The predicted next word is “bonds.”

Neural Network Pruning

Neural network pruning removes weights, neurons, channels, heads, or layers to reduce model size and computation while retaining useful behavior.

See all 289 papers

Data Pruning: Redundant, Problematic, and Interdependent Samples
Jun 20, 2026Leon Freese, Marthinus W. TheunissenDataset Pruning Training Data Selection
Faculty of Engineering, North-West University, South Africa · Centre for Artificial Intelligence Research (CAIR), South Africa · National Institute for Theoretical and Computational Sciences (NITheCS), South Africa
Elimination-compensation pruning for fully-connected neural networks
Feb 24, 2026Enrico Ballini, Luca Muscarnera, Alessio Fumagalli +2Neural Network Pruning Neural Network Compression
MOX, Department of Mathematics, Politecnico di Milano, Piazza Leonardo da Vinci 32, 20133 Milano, Italy
Is Oracle Pruning the True Oracle?
Nov 28, 2024Sicheng Feng, Keda Tao, Huan WangNeural Network Pruning
Westlake University, Hangzhou, China · Nankai University, Tianjin, China

Jun 20, 2026J/K move · Enter open · S save

Data Pruning: Redundant, Problematic, and Interdependent Samples

Leon Freese, Marthinus W. Theunissen

Faculty of Engineering, North-West University, South Africa · Centre for Artificial Intelligence Research (CAIR), South Africa · National Institute for Theoretical and Computational Sciences (NITheCS), South Africa

Dataset Pruning Training Data Selection

Open paper AskPDF arXiv

The performance of deep learning models is affected by not only data quantity but also data quality. Data pruning is a process by which practitioners can reduce the size of a dataset by only keeping the most important training data points, thereby achieving similar test set performance. We empirically investigate two popular data pruning methods under noisy and noiseless conditions and show that these methods fail in the presence of significant label noise. We highlight that the success of data pruning is distinctly affected by three factors: redundancy in the dataset, the presence of problematic samples, and interdependence between samples. We perform a detailed investigation on commonly used benchmark classification datasets and neural network architectures. We find that our observations are consistent across data distributions and training protocols.

Abstract

Neural Network Pruning

Explore similar work

Data Pruning: Redundant, Problematic, and Interdependent Samples

Elimination-compensation pruning for fully-connected neural networks

Is Oracle Pruning the True Oracle?