SARATHI: Decoupled Data-Free Probing and Adaptive Reconstruction for Structured LLM Pruning

Sankar Behera; Dhruv Singh; Satyadev Ahlawat; Yamuna Prasad

cs.LGOct 11, 2026

SARATHI: Decoupled Data-Free Probing and Adaptive Reconstruction for Structured LLM Pruning

Authors: Sankar Behera, Dhruv Singh, Satyadev Ahlawat, Yamuna Prasad

Organizations: Department of Computer Science and Engineering, IIT Jammu · Department of Mathematics, IIT Jammu · Department of Electrical Engineering, IIT Jammu

Abstract

Structured pruning reduces the inference cost of large language models (LLMs) by permanently removing neurons and weight rows, yielding dense, hardware-agnostic speedup. Existing post-training methods couple neuron selection and weight reconstruction through a shared calibration dataset, introducing domain bias and out-of-memory failures at high sparsity. We propose SARATHI, a framework that decouples these two stages. A data-free NMF Residual Subspace Probe scores neurons using intrinsic weight geometry, without accessing any calibration data, and carries a formal connection to Robust PCA. Adaptive OBS then reconstructs the pruned weights with a localized calibration batch, operating in $\mathcal{O}(d^2)$ peak memory per layer and eliminating the global memory bottleneck of coupled methods. A Bias Shift Compensation step corrects systematic activation drift in post-LayerNorm architectures that would otherwise cause perplexity collapse at high sparsity. Experiments across six models from two architecture families (gated and non-gated FFN) at 15--40% sparsity show that SARATHI achieves competitive perplexity and matches or exceeds existing structured pruning baselines on zero-shot accuracy.

Figures & tables

Figure 1: The coupling flaw in existing structured pruning vs. SARATHI’s decoupled design. (a) Coupled methods (SoBP, SliceGPT, FLAP) use the same calibration dataset $\mathcal{D}_{\text{calib}}$ for both neuron selection and weight reconstruction, incurring $\mathcal{O}(Ld^{2})$ peak memory and introducing calibration-domain bias into the pruning mask. (b) SARATHI separates these two stages: neuron selection is entirely data-free (NMF Residual Probe, green), while weight reconstruction uses only a small localized batch with $\mathcal{O}(d^{2})$ memory per layer (Adaptive OBS, blue). The result is an architecturally native pruned model with Neural Tangent Kernel (NTK) Cosine Similarity of 1.0000 and no OOM failures, even at 40% sparsity on OPT-6.7B.

Figure 2: Bias Shift Compensation on OPT-2.7B PPL. Without correction (red), PPL collapses to 2,082 at 40% sparsity. SARATHI’s zero-overhead bias update (Eq. 11 , green) reduces PPL by 72%, 88%, and 99.1% at 20%, 30%, and 40%, respectively.

Figure 3: Zero-shot accuracy for LLaMA-3-8B at 30% sparsity. SARATHI-Wanda (blue) and SARATHI-NMF (green) dominate SliceGPT (orange) and Dynamic Slicing (red). Grey dotted polygon = dense baseline. SliceGPT/Dyn. Slicing reported only on 4 tasks ( Dumitru et al., 2024 ) .

LLaMA-3-8B (Dense PPL: 6.14)
Method	15%	20%	30%	40%
SliceGPT ( Ashkboos et al., 2024 )	$\ddagger$	$\ddagger$	13.37	20.69
Dyn. Slicing ( Dumitru et al., 2024 )	$\ddagger$	$\ddagger$	12.96	19.11
SARATHI-Wanda	7.78	8.18	9.04	10.10
SARATHI-NMF	7.28	7.70	8.70	9.92
Mistral-7B (Dense PPL: 5.32)

Table 1: WikiText-2 perplexity (

\downarrow

) across 15–40% sparsity. Bold indicates the best result. OOM = out-of-memory failure.

\dagger

indicates failure due to code errors.

\ddagger

indicates results not reported by the original authors. SliceGPT and Dynamic Slicing results for LLaMA-3-8B and Mistral-7B are from Dumitru et al. (2024) . SliceGPT OPT baselines are from Wei et al. (2024) . All SARATHI models are compressed without retraining.

Method	PIQA	Hella	Wino	ARC-e	ARC-c	Avg (5)
PruneNet ( Sengupta et al., 2025 )	71.11	58.30	61.09	53.20	33.53	55.45
SARATHI-NMF	71.33	45.62	61.48	61.24	31.57	54.25
SARATHI-Wanda	73.45	49.87	66.85	69.70	35.67	59.11

Table 2: Zero-shot accuracy (%) on LLaMA-2-7B at 30% sparsity compared to PruneNet ( Sengupta et al., 2025 ) . Bold indicates the best result. SARATHI results are detailed in Appendix D .

OPT-2.7B
Method	PIQA	Hella	ARC-c	ARC-e	Wino	BoolQ	OBQA	Avg	PPL
Dense	74.81	60.58	31.14	54.42	61.01	63.20	35.40	54.37	12.47
FLAP ( An et al., 2024 )	72.14	54.82	29.27	48.86	57.46	61.28	33.20	51.00	18.32
SliceGPT ( Ashkboos et al., 2024 )	72.85	55.43	30.12	49.75	58.09	59.42	33.60	51.32	16.31
SoBP ( Wei et al., 2024 )	72.69	56.51	30.80	50.80	58.88	56.09	34.40	51.45	13.89
SARATHI-Wanda	72.96	55.94	28.67	53.32	58.17	65.11	33.00	52.45	15.89

Table 3: Zero-shot accuracy (

\uparrow

) at 30% sparsity. Bold indicates the best result. PPL refers to WikiText-2 perplexity. Baselines are sourced from Wei et al. (2024) and Dumitru et al. (2024) .

\dagger

indicates failure due to code errors;

\ddagger

indicates tasks not reported by the original authors. All SARATHI models are evaluated without fine-tuning.

Figure 4: WikiText-2 PPL vs. sparsity for LLaMA-3-8B (left) and Mistral-7B (right). SARATHI-NMF (green, solid) achieves the lowest PPL at every sparsity level for these two architectures. The shaded region between SARATHI-NMF and Dynamic Slicing highlights the growing advantage at higher compression rates. SliceGPT and Dynamic Slicing results are taken from Dumitru et al. (2024) . The dashed horizontal line marks the dense (unpruned) baseline. Numbers annotated at 30% sparsity indicate the absolute PPL gap.

Model	Rank	20%	30%	40%
LLaMA-3-8B	$r=7$	12.38	15.78	20.17
	$r=16$	12.37	15.76	20.01
	$r=32$	12.37	15.72	20.16
	$r=64$	12.39	15.73	20.02
Mistral-7B	$r=7$	10.75	14.37	18.74
	$r=16$	10.86	14.39	19.01

Table 4: PPL sensitivity to NMF rank

r

under Uniform slicing settings. The min-keep floor

\alpha=0.5

is constant. Best per block bolded .

Configuration	20%	30%	40%
Without Bias Comp.	47.06	132.00	2,082
With Bias Comp.	13.25	15.61	18.98
PPL Reduction	72%	88%	99.1%

Table 5: Effect of Bias Shift Compensation on OPT-2.7B WikiText-2 PPL.

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Model	Sparsity	BS=1	BS=16	Speedup	NTK Cos.	NTK Trace	Prune Time
OPT-2.7B	Dense (0%)	46.52	699.30	1.00 $\times$	–	–	–
	20%	51.28	779.17	1.11 $\times$	1.0000	0.9639	16.5 min
	30%	46.55	759.22	1.09 $\times$	1.0000	1.0495	17.7 min
	40%	48.10	740.95	1.06 $\times$	1.0000	1.1277	18.4 min
OPT-6.7B	Dense (0%)	44.12	358.39	1.00 $\times$	–	–	–
	20%	53.05	646.64	1.80 $\times$	1.0000	1.5222	33.7 min

Appendix

Table 6: Hardware performance and structural diagnostics for SARATHI-NMF. Throughput is measured on a single A100 40GB GPU. NTK Cosine Similarity near 1.0000 indicates preserved local gradient dynamics. Mistral-7B exhibits a modest throughput regression at BS=16 due to FFN dimensions shifting away from Tensor-Core-aligned sizes.

Model	Method	Spar.	PIQA	Hella	ARC-c	ARC-e	Wino	BoolQ	OBQA	Avg
OPT-2.7B	SARATHI-W	20%	73.88	59.02	29.69	53.24	60.06	65.57	35.60	53.87
	SARATHI-N	20%	73.99	59.19	29.10	54.00	59.12	63.21	35.00	53.37
	SARATHI-W	40%	71.38	51.83	28.50	49.92	58.72	64.19	32.00	50.93
	SARATHI-N	40%	71.87	51.98	28.41	51.01	58.25	64.04	31.80	51.05
OPT-6.7B	SARATHI-W	20%	76.44	65.28	32.94	59.47	65.04	67.25	38.40	57.83
	SARATHI-N	20%	76.33	65.32	32.68	60.14	65.19	66.64	38.60	57.84

Appendix

Table 7: Extended zero-shot accuracy (

\uparrow

) at 20% and 40% sparsity. SARATHI-W = SARATHI-Wanda; SARATHI-N = SARATHI-NMF. OOM = out-of-memory.

Method	Spar.	PPL	PIQA	Hella	ARC-c	ARC-e	Wino	BoolQ	OBQA	Avg ZS
Dense	0%	5.47	–	–	–	–	–	–	–	–
SARATHI-W	15%	5.83	76.44	54.76	40.44	73.36	67.72	77.00	31.20	60.13
SARATHI-N	15%	6.76	74.97	51.16	37.54	68.39	65.35	66.02	29.00	56.06
SARATHI-W	20%	6.02	75.84	53.37	38.99	72.73	67.80	75.32	31.40	59.35
SARATHI-N	20%	7.26	74.70	49.47	34.98	66.62	62.59	62.78	26.00	53.88
SARATHI-W	30%	6.54	73.45	49.87	35.67	69.70	66.85	67.55	27.20	55.76

Appendix

Table 8: WikiText-2 PPL and average zero-shot accuracy on LLaMA-2-7B. SARATHI-W = SARATHI-Wanda; SARATHI-N = SARATHI-NMF.

Sparsity	Method	PPL	PIQA	Hella	ARC-c	ARC-e	Wino	BoolQ	OBQA	Avg
0%	Dense	4.88	80.52	79.37	49.23	77.48	72.30	80.58	45.20	69.24
20%	SARATHI-U	6.86	76.21	73.09	42.06	67.19	66.13	68.87	40.62	62.02
20%	SARATHI-A	5.42	77.20	72.86	43.34	69.95	71.27	73.55	43.00	64.45
30%	SARATHI-U	7.48	74.32	69.63	40.57	65.18	64.51	65.62	41.21	60.15
30%	SARATHI-A	5.85	74.27	69.30	40.61	67.30	70.40	72.87	43.60	62.62
40%	SARATHI-U	8.55	71.87	64.82	37.02	58.14	63.57	60.31	37.57	56.19

Appendix

Table 9: Per-task zero-shot accuracy (%) and WikiText-2 PPL for LLaMA-2-13B. SARATHI-U = Uniform + OBS; SARATHI-A = Adaptive + OBS (our method). Best results bolded . Dense baseline avg. = 69.24%.

Ranking agreement (probe-independent)
Metric	NMF Probe	SVD Probe
Spearman $\rho$ (neuron ranking)	0.9987	0.9987
Top-25% neuron overlap	98.95%	98.95%
Reconstruction and downstream quality
Rel. reconstruction error	0.6103	0.6078
Coefficient of variation (CV)	0.2805	0.2777

Appendix

Table 10: NMF vs. SVD probe ablation on OPT-6.7B at 30% sparsity (probe-only; Adaptive OBS and Bias Shift Compensation excluded to isolate probe quality). Absolute PPL values are not representative of the full SARATHI pipeline; see Table 1 . Bold = better per metric.

Model	Attn	$d$	$d_{ff}$	KV / Q	Attn / layer	FFN / layer
				heads	(M params)	(M params)
OPT-66B	MHA	9216	36 864	72 / 72	339.74	679.48
LLaMA-2-7B	MHA	4096	11 008	32 / 32	67.11	135.27
Mistral-7B	GQA	4096	14 336	8 / 32	41.94	176.16
LLaMA-3-8B	GQA	4096	14 336	8 / 32	41.94	176.16
LLaMA-2-70B	GQA	8192	28 672	8 / 64	150.99	704.64

Appendix

Table 11: Per-layer parameter counts and Attn : FFN ratio for representative LLMs. Attn / layer counts

\mathbf{W}_{Q},\mathbf{W}_{K},\mathbf{W}_{V},\mathbf{W}_{O}

. FFN / layer counts up, down (and gate for SwiGLU). GQA models use fewer KV heads (column “KV / Q heads”), sharply reducing attention parameters while leaving the FFN unchanged.

LLM Pruning

Large language model pruning removes less useful weights or components to reduce memory and inference costs while preserving model capabilities.

See all 178 papers

LILA: Calibration-Free Structured Pruning of Large Language Models via Latent Spectral Geometry
Sep 10, 2026Sankar Behera, Dhruv Singh, Anshika Agnihotri +3LLM Pruning LLM Compression
CSE, IIT Jammu · Mathematics, IIT Jammu · IT, EC Bikaner +1
Correlation-Aware Structured Pruning for Large Language Models
Aug 23, 2026Sicheng Xu, Hao Shi, Wei Zhang +6LLM Pruning Neural Network Pruning
Department of Mathematical Sciences, Tsinghua University · Theory Lab, Central Research Institute, 2012 Labs, Huawei Tech. Co. Ltd.
Diet Your LLM: Dimension-wise Global Pruning of LLMs via Merging Task-specific Importance Score
Mar 25, 2026Jimyung Hong, Jaehyung KimLLM Pruning Structured Pruning
Yonsei University

Sep 10, 2026J/K move · Enter open · S save

LILA: Calibration-Free Structured Pruning of Large Language Models via Latent Spectral Geometry

Sankar Behera, Dhruv Singh, Anshika Agnihotri, Raj Kumar Choudhary+2

CSE, IIT Jammu · Mathematics, IIT Jammu · IT, EC Bikaner+1

LLM Pruning LLM Compression Structured Pruning Neural Network Pruning

Open paper AskPDF arXiv

Structured pruning of large language models (LLMs) offers hardware-efficient compression, yet existing methods require calibration data, gradient computation, or large auxiliary policy networks at pruning time. LILA (Latent-Informed Layer Analysis) scores neuron importance via the Kolmogorov--Smirnov (KS) distance between empirical singular value distributions of the full and neuron-ablated feed-forward network (FFN) weight matrix, providing a closed-form spectral rule requiring no training, calibration data, or auxiliary network. Without any fine-tuning, LILA surpasses PruneNet (45M-parameter RL policy) by 1.57~~pp in zero-shot accuracy on LLaMA-2-7B at 25% sparsity, and outperforms WikiText-2-calibrated SliceGPT by up to 6.0~~pp across all sparsity levels, while preserving the original architecture. After one epoch of LoRA recovery fine-tuning, LILA achieves highly competitive performance, matching the heavily calibrated SliceGPT baseline to within a 0.48~pp margin across LLaMA-2-7B and Phi-2, despite using zero calibration data. A Neural Tangent Kernel analysis confirms a 22

\times

reduction in functional distortion versus random pruning, providing theoretical grounding for the spectral importance criterion. Finally, extending LILA to dynamically allocate sparsity budgets via KS-scores yields state-of-the-art generative preservation at moderate compression, while uncovering fundamental single-layer architectural bottlenecks at higher compression regimes.

Abstract

LLM Pruning

Explore similar work

LILA: Calibration-Free Structured Pruning of Large Language Models via Latent Spectral Geometry

Correlation-Aware Structured Pruning for Large Language Models

Diet Your LLM: Dimension-wise Global Pruning of LLMs via Merging Task-specific Importance Score