cs.CLMar 25, 2026
SaveDemystifying When Pruning Works via Representation Hierarchies
Organizations: University of Maryland, College Park, USA · Northeastern University, USA
Abstract
Network pruning, which removes less important parameters or architectures, is often expected to improve efficiency while preserving performance. However, this expectation does not consistently hold across language tasks: pruned models can perform well on non-generative tasks but frequently fail in generative settings. To understand this discrepancy, we analyze network pruning from a representation-hierarchy perspective, decomposing the internal computation of language models into three sequential spaces: embedding (hidden representations), logit (pre-softmax outputs), and probability (post-softmax distributions). We find that representations in the embedding and logit spaces are largely robust to pruning-induced perturbations. However, the nonlinear transformation from logits to probabilities amplifies these deviations, which accumulate across time steps and lead to substantial degradation during generation. In contrast, the stability of the categorical-token probability subspace, together with the robustness of the embedding space, supports the effectiveness of pruning for non-generative tasks such as retrieval and multiple-choice selection. Our analysis disentangles the effects of pruning across tasks and provides practical guidance for its application. Code is available at https://github.com/CASE-Lab-UMD/Pruning-on-Representations
Figures & tables
Figure 1: Effect of inter-layer pruning on generative and non-generative tasks . Inter-layer pruning is implemented by removing entire transformer blocks (ShortGPT ( Men et al., 2025 ) ) or attention/MLP layers (Attn/MLP Drop ( He et al., 2026 ) ).
Figure 2: Propagation of pruning-induced perturbations across representation spaces in LLMs. Small embedding perturbations introduced by pruning remain stable in the logit space (i.e., small ), but are amplified by the softmax nonlinearity in the high-dimensional probability space, resulting in large probability shifts and degraded autoregressive generation.
Table 1: Benchmark performance comparison of Mistral models under layer dropping. Results are reported for both non-generative tasks (embedding and multiple-choice benchmarks) and generative tasks. Drop-8A and Drop-8M denote models where 8 attention layers or 8 MLP layers are removed, respectively, while keeping the remaining architecture unchanged.
Figure 3: Impact of intra-layer pruning on non-generative and generative tasks for the default Qwen-2.5-7B-Instruct model, namely HellaSwag ( Zellers et al., 2019 ) and GSM8K ( Cobbe et al., 2021 ) . Results are reported using Wanda ( Sun et al., 2023 ) under unstructured (50%), , and ( Zhou et al., 2021 ) sparsity patterns.
| Question | Natalia sold clips to 48 of her friends in April, and then she sold half as many clips in May. How many clips did Natalia sell altogether in April and May? |
|---|---|
| Method | Output |
| Baseline | Natalia sold 48 clips in April. In May, she sold half as many clips as in April, which amounts to 24 clips. Therefore, the total number of clips sold across April and May is . |
| Drop-4A | Natalia sold 48 clips in April and half as many in May, which is 24 clips. Therefore, the total number of clips sold over April and May is . |
| Drop-8A | Let’s denote the number of clips Alicia, the ILUNNIE M ’<%=>t [K1205359752000_1_61.72510155125556460445874923052e-08T.1000.0 01.10115651362430000.2023-01-02 23:21:07 202277447967445715556447 |
| Drop-4M | Natalia sold 48 clips in April and half that amount in May (24 clips), resulting in a total of clips sold across the two months. |
| Drop-8M | To calculate the total number of clips, we are adding the result of first you and your a year and then the second or your and your a year and your and your and your and your and your and your… |
Table 2: Generated output examples of Qwen-2.5-7B-Instruct under inter-layer pruning (Attention/MLP Drop ( He et al., 2026 ) ). Drop- A and Drop- M denote removing attention layers or MLP layers, respectively. While moderate pruning preserves correct generation, heavier pruning causes severe generation breakdown, including incoherent and repetitive outputs.
Figure 4: Representation similarity across three spaces when each layer is individually dropped for the Qwen-2.5-7B-Instruct model, with layer dropping performed. Mean values are shown as curves and min–max ranges as shaded areas.
Figure 5: Relative orthogonal magnitude in the embedding space ( ) and the logit space ( ) at the 14th attention layer under layer dropping.
Figure 6: Comparison between the ground-truth values and the theoretical estimates across generation steps at the 14th attention layer under layer dropping, measured by (a) angular deviation and (b) KL divergence.
Figure 7: Representation similarities across different spaces between the outputs of the baseline and pruned models (Drop-8A) across generation steps for the default Qwen-2.5-7B-Instruct model. Outliers in the probability space at later decoding steps primarily correspond to predictions involving special tokens.
Figure 8: Comparison between the outputs of the full and pruned models in different subspaces under Drop-8A on multiple-choice prompts: (a) probabilities of top-probability tokens sampled from the full vocabulary, and (b) log-likelihoods restricted to the category-token subspace.
Appendix figures & tables14 assets
Supplementary material from the paper’s appendix.
Appendix
| Multiple-Choice Classification |
|---|
| [MC1] Tom has 15 candies. He eats 4 and gives 3 to his friend. How many candies does Tom have left? Choose the correct answer: A) 6 B) 7 C) 8 D) 9 |
| [MC2] Which planet is known as the Red Planet? Choose the correct answer: A) Venus B) Mars C) Jupiter D) Saturn |
| [MC3] Which of the following animals is a mammal? Choose the correct answer: A) Snake B) Frog C) Dog D) Lizard |
| [MC4] Mark is older than John, and John is older than Alex. Who is the youngest? Choose the correct answer: A) Mark B) John C) Alex D) None of them |
| Mathematical Reasoning |
| [Math1] John has twice as many books as Mary. Together they have 18 books. How many books does John have? |
Table 3: Representative candidate prompts used in our analysis, grouped by task format.
Figure 9: Performance comparison under different intra-layer sparsification strategies with SparseGPT.
Figure 10: Impact of inter-layer pruning on generative (HumanEval) and non-generative (MMLU) tasks .
Figure 11: Effect of temperature on pruning-induced deviation estimation. We compare the ground-truth measurements and theoretical estimates under different temperature settings in terms of (a) angular deviation and (b) KL divergence.
Figure 12: Step-wise perturbation analysis under different compression methods. We compare inter-layer dropping via Attn Drop with 8 layers removed, intra-layer sparsification via Wanda with 4:8 (50%) sparsity, and weight-only quantization using AWQ. Panels (a–c) report cosine similarity between the original and perturbed representations in the embedding, logit, and probability spaces, respectively. Panels (d–f) further characterize the magnitude of logit perturbations and their distributional effects, including the resulting KL divergence in the output probability distribution.
Figure 13: Layer-wise representation similarity between the outputs of the baseline model and its temporarily pruned counterpart . For the pruned model, we temporarily prune a single layer during the forward pass and compute the representation similarity at the same layer, while keeping all other layers identical to the baseline model.
Figure 14: Layer-wise evolution of KL divergence in attention layers across generation steps. For each transformer layer, we compare the ground-truth KL divergence ( blue ) and the theoretical estimates ( orange ). The results show that the proposed estimator closely tracks the true KL behavior across layers, while also revealing that deeper layers generally exhibit larger distributional shifts.
Figure 15: Layer-wise evolution of angular deviation values of the probability space between the inputs and outputs of attention layers. We report the ground-truth measurements ( blue ) and the theoretical estimates ( orange ) across decoding steps. Consistent with the KL analysis, shallow layers remain relatively stable while deeper layers exhibit larger semantic deviations, and our estimator successfully captures these trends.
Figure 16: Layer-wise evolution of angular deviation values on the embedding space between the inputs and outputs of attention layers. We report the ground-truth measurements ( blue ) and the theoretical estimates ( orange ) across decoding steps.
Figure 17: Layer-wise evolution of angular deviation values on the logit space between the inputs and outputs of attention layers. We report the ground-truth measurements ( blue ) and the theoretical estimates ( orange ) across decoding steps.
Figure 18: Relative magnitude ratios of representations in the logit space ( blue ) and the embedding space ( orange ).
Figure 19: Uniform and weighted variances of , where the uniform variance is computed under a uniform distribution ( blue ) and the weighted variance ( orange ) is computed under the vocabulary distribution .
Figure 20: Relative values of the weighted variance of , divided by the relative magnitude ratio of .
Figure 21: Relative values of the weighted variance of , divided by the relative magnitude ratio of .
More in this topic
LLM Pruning
Large language model pruning removes less useful weights or components to reduce memory and inference costs while preserving model capabilities.
See all 178 papers