Unlocking Structured Pruning for Reasoning Language Models via Attention-Preserving MLP Trimming

Woosang Lim; Haneul Yoo; Vincent-Daniel Yun; Sungjoo Yoo

cs.LGOct 13, 2026

Unlocking Structured Pruning for Reasoning Language Models via Attention-Preserving MLP Trimming

Authors: Woosang Lim, Haneul Yoo, Vincent-Daniel Yun, Sungjoo Yoo

Organizations: Seoul National University, Republic of Korea · New York University, United States · University of Southern California, United States

Abstract

Structured pruning reduces LLM inference costs by removing model components to produce smaller dense models. However, existing methods can cause reasoning performance to collapse on long-form reasoning tasks such as AIME. We propose LayerTrim, a training-free structured pruning method that identifies redundant decoder layers and selectively trims their MLP features while preserving attention and decoder depth. LayerTrim is the first training-free structured pruning method designed to preserve long-form reasoning without gradients or recovery training. Across three reasoning language models and five benchmarks, LayerTrim outperforms existing structured pruning methods in most evaluated settings, improving average accuracy by up to 21.4 percentage points over RESP. On Qwen3-14B, LayerTrim achieves 1.24 $\times$ faster prefill and 19.7% lower peak GPU memory than the dense model, while better preserving reasoning ability. It also achieves the shortest average generation length among evaluated pruned models, while retaining the highest average accuracy. These results demonstrate that LayerTrim improves reasoning efficiency through both structural compression and reduced token generation.

Figures & tables

Figure 1: Accuracy versus generated tokens per response on Qwen3-14B with a pruning budget equivalent to 9 of its 40 decoder layers. Both metrics are averaged across five benchmarks, with a maximum generation length of 32K tokens.

Figure 2: Observations motivating LayerTrim on Qwen3-14B . (a) LayerTrim reduces memory usage and inference latency with a pruning budget equivalent to 9 of 40 decoder layers. (b) MLP trimming causes less representation error than attention pruning, module removal, or joint pruning. (c) MLP feature importance is concentrated in a subset of features. (d) Block Influence (BI) ( Men et al., 2025 ) -based layer selection reduces error compared with random or all-layer pruning. Diagnostic settings are given in Appendix D .

Figure 3: Overview of LayerTrim. LayerTrim collects layer-level and MLP feature-level statistics from reasoning traces generated by the dense model. It selects redundant decoder layers and retains MLP features with larger output contributions. The selected MLPs are reduced to smaller dense layers while preserving attention and decoder depth.

Variant	AIME25	GPQA-D	Avg.
Pruning targets
Attention removal	0.0 $\pm$ 0.0	0.7 $\pm$ 0.3	0.4
Attention trimming	0.0 $\pm$ 0.0	1.3 $\pm$ 0.3	0.7
MLP removal	0.0 $\pm$ 0.0	17.5 $\pm$ 1.1	8.8
Joint trimming (25% attn.)	43.3 $\pm$ 3.3	53.2 $\pm$ 4.2	48.3
MLP trimming	62.2 $\pm$ 8.4	55.4 $\pm$ 2.4	58.8

Table 2: Ablations on Qwen3-14B . Scores (%) are mean

\pm

sample std over three seeds. Avg. averages AIME25 and GPQA-Diamond; bold marks the best average in each group. We remove approximately 1.5B parameters in the first group and 2.312B (

N=7

) in the others. See Appendix C for experimental settings.

Panel of Figure 4 — Figure 4: Attention preservation and feature importance on Qwen3-14B . (a) Attention KL divergence across pruning budgets. (b) Overlap with the dense model’s top-8 attended tokens by query–key distance, with approximately 1.5B parameters removed. (c) Summed feature importance scores versus squared MLP output change. Experimental details are provided in Appendix D .

Panel of Figure 5 — Figure 5: Quantitative analysis results comparing LayerTrim to existing pruning methods

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Question: An electron is in the spin state $(3i,4)$ . Find $\langle S_{y}\rangle$ , given $\sigma_{y}=\left(\begin{smallmatrix}0&-i\\ i&0\end{smallmatrix}\right)$ . Options: A. $-12\hbar/25$ ; B. $12\hbar/25$ ; C. $-25\hbar/2$ ; D. $25\hbar/2$ .
Method	Original trace excerpts and qualitative observations
Dense 2,880 tokens Terminated	[…] Let psi = (3i/5, 4/5). Then sigma_y = [[0, -i],[i, 0]]. We compute sigma_y \|psi> = [[0, -i],[i, 0]] * (3i/5, 4/5)^T = (0(3i/5) + -i(4/5), i(3i/5) + 0(4/5)) = (-4i/5, i(3i)/5). But note: i(3i) = 3(i^2) = -3. So second component becomes -3/5. So sigma_y \|psi> = (-4i/5, -3/5). […] Then expectation value of S_y is (hbar/2)(-24/25) = -12 hbar/25. […] Observation: Normalization and the matrix product remain consistent through the answer.
ShortGPT 32,256 tokens Limit hit	[…] The correct answer is letter letter option letter letter letter letter answer letter letter letter letter letter letter letter letter letter letter letter letter letter letter letter letter letter letter letter letter le […] negative polarity positive polarity negative polarity positive polarity negative polarity positive polarity negative polarity positive polarity negative polarity positive polarity […] Observation: Answer-format and polarity loops replace the scientific calculation.
LaCo 32,256 tokens Limit hit	itness_duperge hungerCreatorsgenerated Witness_waycup pred Counter brushingonga cagescreatorBush.IM $\boxed{?}$ 回Pot collectivelyWitnessm […] vaeonusdfa832.readlineLossUGCidalDigest HOLDERacdmumu recModifiedDate falsevale Peiadotent Circuit escapeurf Vitalgist Authority CHANNELembr.qualCas447 multiline Highestline tet RingsanaxIF […] Observation: Fragmented tokens continue to the cap without an interpretable calculation.
GRASP 32,256 tokens Limit hit	[…] I should answer answer. Now the letter letter is answer. It is " answer letter. The letter letter. So letter is " letter. But answer. So letter letter: letter. answer letter. So letter answer letter. Actually letter letter: answer. letter. […] letter letter letter letter. letter letter letter. So letter. So letter. So letter. So letter. So letter. So l […] Observation: Answer-letter language repeats without advancing the derivation.
FLAP 32,256 tokens Limit hit	[…] (-3i)(-4i) + (4)(-3) = 12i^2 - 12 = 12(-1) -12 = -24. […] lambda^2 - (i (-i)) = lambda^2 - (-1) = lambda^2 + 1 = 0, so eigenvalues are $\pm$ i. […] So the answer is -12hbar/25. That is option A. Therefore, answer A is the correct answer. I’ll now provide answer A. Thus answer A: -12hbar/25. Thus answer: Option A. I’ll now answer in the requested format: \boxed{A} is the answer. But wait, but the computed result is -12/25. But the options are in terms of hbar. So the answer is -12hbar/25. I must check the reasoning: […] Observation: The correct option appears, but reconsideration continues to the cap; the saved scorer marks this response correct.

Appendix

Table 8: Qualitative analysis of example reasoning traces on GPQA-Diamond using Phi-4-reasoning-plus at

L_{p}/L_{t}=9/40

and seed

0

. […] denotes omitted text.

Question: Complete has_close_elements(numbers, threshold) : return whether any two input elements differ by less than the threshold. The prompt example ([1.0, 2.0, 3.0], 0.5) requires False .
Method	Original trace excerpts and qualitative observations
Dense 1,348 tokens Terminated	Reasoning: […] So for each i in range(len(numbers)), and for each j in range(i+1, len(numbers)), compare numbers[i] and numbers[j]. Final response: […] def has_close_elements(numbers: List[float], threshold: float) -> bool: for i in range(len(numbers)): for j in range(i + 1, len(numbers)): if abs(numbers[i] - numbers[j]) < threshold: return True return False […] Observation: Distinct-index comparisons are preserved in the final function.
ShortGPT 337 tokens Terminated	Reasoning: […] The function should check if any two numbers in the list are within the given threshold. The function should be written in a Python block with the proper indentation. Final response: […] The function uses the ‘*‘ operator to check if any two numbers in the list are within the given threshold. The function returns ‘True‘ if any two numbers are within the given threshold, and ‘False‘ otherwise. The function is as described, and the code is correct. Observation: The response claims to provide an implementation, but supplies no comparison logic.
LaCo 32,256 tokens Limit hit	[…] imer Spo aggregateTimelineaslasca Jenner官 TC户攀筹progressbar共创的高度GD $\boxed{?}$ 栊cket clam命运随手oidantom迎接系列产品 movement踩春节路上功课NPCasl人性化刹档埠勿官- […] Observation: These mixed-script fragments do not form an interpretable plan or function.
GRASP 32,256 tokens Limit hit	from typing import List, float, 2.0, 3.0, 4.0, 0.0, 2.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0 […] 0.0: 0.0: 0.0: 0.0: 0.0: 0.0: 0.0: 0.0: 0.0: 0.0: 0.0: 0.0: 0.0: 0.0: 0.0: 0.0: 0.0: 0.0: 0.0: Observation: A malformed import leads into numeric repetition, rather than a function body.
FLAP 4,122 tokens Terminated	Reasoning: […] So, I can sort the list first, then check each consecutive pair. If any pair has a difference less than the threshold, then the answer is True. Otherwise, it’s False. Final response: […] def has_close_elements(numbers: List[float], threshold: float) -> bool: sorted_numbers = sorted(numbers) for i in range(len(sorted_numbers) - 1): if sorted_numbers[i+1] - sorted_numbers[i] < threshold: return True return False […] Observation: The intended algorithm survives, but the final if and return statements are incorrectly indented.

Appendix

Table 9: Qualitative analysis of example reasoning traces on HumanEval using Qwen3-14B at

L_{p}/L_{t}=9/40

and seed

0

. […] denotes omitted text.

Question: Let $N$ be the number of positive integer divisors of $17017^{17}$ that leave a remainder of $5$ when divided by $12$ . Find the remainder when $N$ is divided by $1000$ .
Method	Original trace excerpts and qualitative observations
Dense 3,907 tokens Terminated	[…] 7 * 11 = 77, 77 * 13 = 1001, 1001 * 17 = 17017. […] So indeed, the valid cases are: (a even, d odd, b even) and (a odd, d even, b odd). And each gives 9 possibilities for a, 9 for b, 9 for d. So total 9^3 + 9^3 = 2729 = 1458. And then times 18 choices for c yields 145818 = 26244. Then remainder mod 1000 is 26244 mod 1000 = 244. […] Observation: The two parity cases and independent exponent choices produce the correct count.
ShortGPT 32,256 tokens Limit hit	[…] But wait, is that all? Let’s verify product: 7 * 11 = 77, 77 13= 100, 7713= 7710 + 773? Let me compute: 77 * 10=770, 773=231, so product= 101, so no. Let’s compute properly: 711=77; 7713= 7710 +773 = 770 +231 = 1, 770+231= 1,101 indeed. […] And then 1,101 17 equals? 1,10117 = 1,10110 = 11,101? Let me compute: 1,10117 = 1,1017= 7,207, oh sorry, let me compute properly: 1,10117 = 1,10110= 11,101, but then 1,10117 = 1,101(10+7) = 11,101 + 7,207 = 18,308, oh sorry, not exactly. […] divided by prime 17017 divided by prime 17017 divided by prime 17017 divided by prime 17017 divided by prime 17017 divided by Observation: An attempted verification introduces new arithmetic errors and then stalls.
LaCo 32,256 tokens Limit hit	reads acquaintance567eyJremball congressentyAAAIDEOanne chargerbatismars-PackINS sachDDminer drift compensation Painterpressu […] s Tape validenger Observable playEDGE downhill dashojufenpagedpaged acceptedplaetty tun Missspotify yields Delta Lower sweep monitor CommissionPARprotarkin monitored […] Observation: No interpretable factorization or count is visible in these fragments.
GRASP 32,256 tokens Limit hit	[…] 17017 is exactly a product of 17, 19, and 13, but let’s check: 17 times 19 and 13: 17: 19 and then 13? So the factors of 17017 is 13, 17, 19. But check: we see that 17 factor? Actually: 17017 is: product of 17, 19, 13? Let’s check: 17017 is a product of 13, 17, 19. So the digits? Let’s check: 131719? Let’s check: 17*19? Actually, 17 times 19 is 60 maybe. […] yes, yes, yes, yes, yes, yes, yes, yes, yes, yes, yes, yes, yes, yes, yes, yes, yes, yes, yes, yes, yes, yes, Observation: The proposed factors multiply to 4,199; verification collapses into repetition.
FLAP 11,066 tokens Terminated	[…] I want to check one more possibility: Possibly the factorization of 17017 is different. Let’s re-check: 17017 = 17 * 7 * 11 * 13, I should check: 177 = 119, 11911 = 1309, 1309*13 = 17017. So that’s correct. […] For any exponent from 0 to 17, there are 9 odd choices and 10 even choices. For the pattern (odd, even, even): - a odd: 9 choices, b even: 10 choices, c even: 10 choices. Total = 9 $\times$ 10 $\times$ 10 = 900. For the pattern (even, odd, odd): - a even: 10 choices, b odd: 9 choices, c odd: 9 choices. Total = 10 $\times$ 9 $\times$ 9 = 810. […] Thus the total number N is 1710 $\times$ 18 = 30780. […] Observation: Factorization is recovered, but a wrong parity count survives into the answer.

Appendix

Table 10: Qualitative analysis of example reasoning traces on AIME 2026 using Phi-4-reasoning-plus at

L_{p}/L_{t}=7/40

and seed

0

. […] denotes omitted text.

Explore similar work

Structured Thoughts For Improved Reasoning And Context Pruning
Jul 11, 2026Zain Sarwar, Supriyo Chakraborty, Berkcan Kapusuzoglu +5CoT Reasoning Efficient Language Model Reasoning
University of Chicago · Capital One
Prune, Update and Trim: Robust Structured Pruning for Large Language Models
May 18, 2026Diego Coello de Portugal Mecke, Tom Hanika, Lars Schmidt-ThiemeLLM Pruning LLM Inference Acceleration
ISMLL & DARC VWFS · University of Hildesheim · Hildesheim, Germany +1
OS-Pruner: Pruning Chains-of-Thought of Reasoning Models via Optimal Stopping
Jul 13, 2026Mohammed Ehab, Aymane El Gadarri, Vivek F. Farias +2Overthinking in Language Models Sequential Decision Making
MIT · Columbia Business School

Jul 11, 2026J/K move · Enter open · S save

Structured Thoughts For Improved Reasoning And Context Pruning

Zain Sarwar, Supriyo Chakraborty, Berkcan Kapusuzoglu, Chia-Hsuan Lee+4

University of Chicago · Capital One

CoT Reasoning Efficient Language Model Reasoning Structured Reasoning LLM Context Management LLM Reasoning

Open paper AskPDF arXiv

Large language models (LLMs) excel at generating long chains of thought, but long reasoning traces are often verbose and memory-inefficient. In this work, we introduce Structured Thoughts, a framework that organizes reasoning into alternating <try> and <outcome> blocks: <try> captures exploratory scratch work, while <outcome> contains the distilled conclusion of that step. We construct a dataset of structured thoughts by segmenting reasoning traces into <try> blocks and prompting an LLM to summarize each step into its corresponding <outcome>. Fine-tuning pretrained foundation models on this reformatted data produces models that adopt the structured reasoning style, leading to performance gains of up to 8.08% on reasoning benchmarks compared to standard SFT. The explicit structure also enables context pruning: after each <try>/<outcome> pair, the <try> can be pruned, allowing the model to retain conclusions without keeping the full scratch work in the context. A proof-of-concept pruning implementation achieves an average of 85% memory / context savings with an 8.67% performance drop across mathematical tasks.

Abstract

Explore similar work

Structured Thoughts For Improved Reasoning And Context Pruning

Prune, Update and Trim: Robust Structured Pruning for Large Language Models

OS-Pruner: Pruning Chains-of-Thought of Reasoning Models via Optimal Stopping