IrekoGPT: Turning Structured Pruning into Post-Hoc Slimmable LLMs

Pietro Moriello; Pietro Buzzega; Angelo Porrello; Simone Calderara

cs.LGSep 30, 2026

IrekoGPT: Turning Structured Pruning into Post-Hoc Slimmable LLMs

Authors: Pietro Moriello, Pietro Buzzega, Angelo Porrello, Simone Calderara

Organizations: AImageLab, University of Modena and Reggio Emilia, Italy

Abstract

We introduce IrekoGPT, a post-hoc method for converting pretrained LLMs into slimmable models whose width can be adjusted at inference time. Building on SliceGPT, we retain its projection matrices without pruning them, allowing a single model to expose nested subnetworks at different widths. We improve robustness by calibrating each layer across multiple compression ratios, and correct downstream linear layers through gradient-free ridge regression. Across Llama and Qwen models, preliminary results show improvements over naive PCA-based slimming, with the largest gains at high compression. Code is available at https://github.com/aimagelab/IrekoGPT

Figures & tables

	$L_{2,\mathrm{7B}}$	$L_{3.1,\mathrm{8B}}$	$L_{3.2,\mathrm{1B}}$	$Q_{3,\mathrm{1.7B}}$		$L_{2,\mathrm{7B}}$	$L_{3.1,\mathrm{8B}}$	$L_{3.2,\mathrm{1B}}$	$Q_{3,\mathrm{1.7B}}$
Sparsity	Dense				Sparsity	25%
Original	$5.47$	$6.24$	$9.75$	$16.71$	SliceGPT †	$7.50$	$13.73$	$19.13$	$21.53$
PCA	$\mathbf{5.47}$	$6.25$	$9.76$	$16.74$	PCA	$8.62$	$18.69$	$20.32$	$24.43$
ECD	$5.47$	$\mathbf{6.24}$	$\mathbf{9.76}$	$\mathbf{16.72}$	ECD	$7.98$	$15.09$	$19.68$	$\mathbf{22.76}$
IrekoGPT	$5.51$	$6.41$	$9.88$	$17.03$	IrekoGPT	$\mathbf{7.92}$	$\mathbf{14.19}$	$\mathbf{17.77}$	$24.06$
Sparsity	10%				Sparsity	30%

Table 1: WikiText-2 perplexity (lower is better) across models and sparsity levels.

L

denotes Llama and

Q

Qwen; subscripts indicate version and size. SliceGPT † is not slimmable and requires a separate model for each sparsity level. Bold indicates the best result among slimmable variants only.

Figure 1: Layer-wise MSE between compressed and full-width value-projection outputs across increasing sparsity levels. We compare PCA-only, ECD, and IrekoGPT ; lower is better.

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 2: Layer-wise CKA between compressed and full-width value-projection outputs across increasing sparsity levels. All models and all layers shown. We compare PCA-only, ECD, and IrekoGPT ; higher is better.

Figure 3: Layer-wise MSE between compressed and full-width value-projection outputs across increasing sparsity levels. All models and all layers shown. We compare PCA-only, ECD, and IrekoGPT ; lower is better.

Figure 4: Layer-wise difference in CKA between ECD and IrekoGPT w.r.t PCA-only. CKA is still computed between compressed and full-width value-projection outputs.

Figure 5: Layer-wise difference in MSE between ECD and IrekoGPT w.r.t PCA-only. MSE is still computed between compressed and full-width value-projection outputs.

Explore similar work

Prune, Update and Trim: Robust Structured Pruning for Large Language Models
May 18, 2026Diego Coello de Portugal Mecke, Tom Hanika, Lars Schmidt-ThiemeUnstructured Pruning Feed-Forward
ISMLL & DARC VWFS · University of Hildesheim · Hildesheim, Germany +1
DarwinLM: Evolutionary Structured Pruning of Large Language Models
Feb 11, 2025Shengkun Tang, Oliver Sieberling, Eldar Kurtic +2Large Language Model Compression Unstructured Pruning
MBZUAI · ETH Zurich · ISTA +1
Unifying Depth and Width Pruning for LLMs via Binary Knapsack Optimization
Aug 13, 2026Palaash Goel, Ayan Sengupta, Akshay Nambi +1Large Language Model Compression Structured Pruning
Yardi School of Artificial Intelligence, Indian Institute of Technology Delhi · Department of Electrical Engineering, Indian Institute of Technology Delhi · Microsoft Research (MSR) India

May 18, 2026, cs.LGJ/K move · Enter open · S save

Prune, Update and Trim: Robust Structured Pruning for Large Language Models

Diego Coello de Portugal Mecke, Tom Hanika, Lars Schmidt-Thieme

ISMLL & DARC VWFS · University of Hildesheim · Hildesheim, Germany+1

Unstructured Pruning Feed-Forward

Open paper AskPDF arXiv

Large Language Models (LLMs) have experienced significant growth and development in recent years. However, performing inference on LLMs remains costly, especially for long-context inference or in resource-constrained devices. This motivates the development of new post-training pruning (PTP) methods. These methods reduce LLMs' requirements by removing a substantial part of the model's parameters. The discarded weights are selected depending on their impact on the models performance. Current PTP methods prune the models by removing the less informative hidden nodes from the FFN layers, and the least important attention layers. We propose Putri, a PTP method that introduces three changes to the State-of-the-art. First, we update the un-pruned weights of the FFN to compensate for the introduced pruning error. Second, the FFN layers are pruned sequentially, taking into account the updates done to the previous layers. Third, instead of removing full attention layers, we remove individual attention-heads. We extend this method such that it can also address Grouped-Query Attention. In summary, Putri is a structure pruning method which remains simple while showing SOTA performance. Pruning experiments on multiple models with a wide variety of sparsity ranges and on different datasets, validate the generality of Putri. Notably, we demonstrate that, unlike previous methods, Putri can prune LLMs on extreme sparsity ratios. The code is available at: https://github.com/Coello-dev/Putri.

Abstract

Explore similar work

Prune, Update and Trim: Robust Structured Pruning for Large Language Models

DarwinLM: Evolutionary Structured Pruning of Large Language Models

Unifying Depth and Width Pruning for LLMs via Binary Knapsack Optimization