Calibration Language Shapes Block-Influence Pruning in Multilingual Language Models

Vedant Pandhare; Piyush Dhoka

cs.CLOct 15, 2026

Calibration Language Shapes Block-Influence Pruning in Multilingual Language Models

Authors: Vedant Pandhare, Piyush Dhoka

Organizations: Independent Researcher

Abstract

Block Influence (BI) ranks Transformer blocks for depth pruning from their input-output cosine change, but the ranking depends on the calibration distribution. We study whether English calibration transfers to Indian languages in Qwen2.5-1.5B-Instruct, Llama-3.2-1B, and Gemma-2-2B. English, Hindi, Marathi, and Tamil use disjoint Wikipedia articles for calibration and evaluation; every selector receives the same 150-chunk budget, and Tamil is excluded from multilingual aggregation to test cross-script transfer. Across all three models, late-block BI is 20.7-55.5% higher for the three Indian languages than for English, and every predeclared late-minus-middle contrast has a 95% bootstrap interval above zero. This mismatch changes the low-BI blocks selected for removal despite high global rank correlations. Averaging BI over equal English, Hindi, and Marathi budgets mitigates the resulting perplexity damage for Qwen at six removed blocks and for Gemma at all tested depths, including held-out Tamil. The mitigation is not universal: on Llama, balancing improves Hindi and Marathi at two removed blocks but severely harms Tamil, while larger pruning budgets select the same blocks and collapse perplexity regardless of calibration. Rank-mean and minimax-rank aggregators do not dominate the raw mean. These results support a diagnostic claim rather than a universal pruning rule: English-only BI systematically misses late-layer activity for Indian-language text, but multilingual calibration must still be validated per architecture and pruning budget.

Figures & tables

Model	Parameters	Blocks	$k=2$	$k=4$	$k=6$
Qwen2.5-1.5B-Instruct	1.54B	28	7.1%	14.3%	21.4%
Llama-3.2-1B	1.24B	16	12.5%	25.0%	37.5%
Gemma-2-2B	2.61B	26	7.7%	15.4%	23.1%

Table 1: Models and pruning fractions. Fractions count decoder blocks, not parameters.

Figure 1: Language-conditioned Block Influence. The gray span is the predeclared late region; the final output-adjacent block is excluded from regional contrasts. Curves share a coarse shape but separate near the low-BI late blocks that can enter the pruning set.

Model	Language	Late shift (%)	Late–middle	95% CI
Qwen2.5-1.5B-Instruct	HI	51.0	0.391	[0.378, 0.405]
	MR	55.5	0.392	[0.378, 0.406]
	TA	39.6	0.363	[0.350, 0.376]
Llama-3.2-1B	HI	20.7	0.221	[0.210, 0.233]
	MR	31.8	0.308	[0.297, 0.319]
	TA	25.3	0.428	[0.417, 0.439]

Table 2: Target-language BI relative to English in the predeclared late region. The late–middle column is Eq. ( 6 ); intervals bootstrap independent calibration articles.

Figure 2: Late-minus-middle log BI ratio to English, with 95% bootstrap intervals. All nine contrasts are above zero.

Model	$k$	HI	MR	TA
Qwen2.5-1.5B-Instruct	2	1.00 (=)	1.00 (=)	1.00 (=)
	4	1.004 [1.000, 1.012]	1.003 [1.000, 1.009]	1.003 [1.000, 1.008]
	6	1.449 [1.434, 1.465]	1.504 [1.484, 1.522]	1.372 [1.361, 1.384]
Llama-3.2-1B	2	1.640 [1.553, 1.730]	3.098 [2.768, 3.487]	0.057 [0.053, 0.061]
	4	1.00 (=)	1.00 (=)	1.00 (=)
	6	1.00 (=)	1.00 (=)	1.00 (=)

Table 3: Perplexity ratio

R_{\ell,k}=\mathrm{PPL}(S_{k}^{\textsc{en}})/\mathrm{PPL}(S_{k}^{\mathrm{mean}})

with hierarchical 95% intervals. The marker = denotes identical layer sets for all five seeds. Tamil is held out from balanced selector construction.

Figure 3: English-only versus raw-mean balanced calibration. Values above one favor balanced selection; values below one favor English-only selection. The log scale exposes the Llama–Tamil reversal.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Model	Repository identifier	Revision
Qwen	Qwen/Qwen2.5-1.5B-Instruct	989aa7980e4c...
Llama	meta-llama/Llama-3.2-1B	4e20de362430...
Gemma	google/gemma-2-2b	c5ebcd40d208...

Appendix

Table 4: Model identifiers and immutable repository revisions used for the reported runs.

Model	Language	PPL	BPC	Tokens/word
Qwen2.5-1.5B	EN	10.207	0.790	1.533
	HI	3.827	1.776	4.996
	MR	5.256	2.284	6.730
	TA	3.620	1.920	9.553
Llama-3.2-1B	EN	10.158	0.755	1.459
	HI	6.093	1.357	2.854

Appendix

Table 5: Intact-model perplexity (PPL), bits per character (BPC), and mean tokenizer tokens per whitespace-delimited word. Values are descriptive; PPL is not compared across tokenizers.

Model	Selector	Statistic	$k=2$	$k=4$	SD	$k=6$
Qwen2.5-1.5B-Instruct	EN-only	Mean	6.68	20.72	0.00	88.89
		Worst	8.45	25.61	0.00	107.5
	Raw mean	Mean	6.68	20.36	0.81	30.84
		Worst	8.45	25.27	0.76	38.03
	Rank mean	Mean	6.68	18.90	0.00	30.84
		Worst	8.45	23.91	0.00	38.03

Appendix

Table 6: Target-language relative PPL degradation (%) for English-only and balanced BI selectors.

Model	Selector	$k=2$	$k=4$	$k=6$
Qwen	EN	15,16	14–17	12,14–17,26
Qwen	Raw mean	15,16	14–17	12–17
Llama	EN	12,13	11–14	7,10–14
Llama	Raw mean	12,14	11–14	7,10–14
Gemma	EN	12,23	10–12,23	9–12,22,23
Gemma	Raw mean	10,12	9–12	8–12,23

Appendix

Table 7: Full-pool BI pruning sets for the two primary selectors. Indices are zero-based.

	Qwen			Llama			Gemma
$k$	EN	LB	Random	EN	LB	Random	EN	LB	Random
2	6.7	6.7	72.8	1461.7	2694.4	429.7	50.8	16.8	198.6
4	20.7	20.4	1122.1	$9.06{\times}10^{5}$	$9.06{\times}10^{5}$	$3.13{\times}10^{5}$	78.5	34.2	$7.48{\times}10^{4}$
6	88.9	30.8	$1.81{\times}10^{7}$	$8.76{\times}10^{5}$	$8.76{\times}10^{5}$	$1.01{\times}10^{7}$	242.3	132.5	$2.11{\times}10^{8}$

Appendix

Table 8: Mean relative PPL degradation (%) over Hindi, Marathi, and Tamil. Random is averaged over ten block sets; EN and raw mean are averaged over five calibration seeds.

LLM Pruning

Large language model pruning removes less useful weights or components to reduce memory and inference costs while preserving model capabilities.

See all 178 papers

Apr 27, 2026

Rethinking Layer Redundancy: Calibration Matters More Than Search in LLM Depth Pruning

Depth pruning improves the inference efficiency of large language models by removing Transformer blocks. Prior work typically treats layer redundancy as an inherent structural property of pretrained networks, emphasizing importance criteria and search algorithms to identify removable layers. In this study, we empirically investigate depth pruning from a functional perspective. Evaluating representative LLM families across diverse calibration configurations and multiple search algorithms, we show that different configurations produce different pruning patterns. Furthermore, under a fixed calibration configuration, complex search algorithms yield marginal performance improvements over simple one-shot methods, converging to similar pruned subsets. Overall, our results suggest that the calibration configuration plays a substantially larger role than the choice of search algorithm in shaping pruning patterns and calibration perplexity, while contributing comparably to variance in downstream reasoning accuracy. This indicates that future pruning efforts may benefit from prioritizing the calibration configuration over search complexity.

Minkyu Kim, Vincent-Daniel Yun, Youngrae Kim +3

Neural Superintelligence Lab, MODULABS, Republic of Korea · University of Southern California, United States · Boston University, United States +2

LLM Pruning LLM Inference Acceleration Neural Network Pruning

Ask PDF

Jun 2, 2026

Rethinking Calibration for High-Sparsity LLM Pruning: From Single Sources to Capability Coverage

Post-training LLM pruning relies on a small calibration set, usually chosen heuristically and judged by WikiText perplexity or by mean accuracy on general-domain commonsense benchmarks. This single-axis view hides capability-specific effects: at 60% sparsity, strategies separated by

2.85

points in averaged commonsense accuracy diverge by

51.9

points in code retention, and across 15 sources no single one preserves all capabilities. We treat calibration as a capability-coverage problem. Balanced, a uniform four-way mixture under a fixed budget, tests the principle on real corpora: it outperforms every evaluated single source, and its margin over C4 grows from

+4.6

points at 30% sparsity to

+18.8

at 60%. The advantage holds across model scale and a second model family, and transfers to held-out benchmarks disjoint from calibration construction. When aligned corpora are unavailable, IGSP approximates that mixture from the base model and the evaluation taxonomy alone, outperforming prior self-generation methods by

7.23

12.79

points on the LLaMA main suite. On Qwen2.5-7B, IGSP exceeds Balanced on both the main suite and held-out benchmarks. Capability coverage is a first-order design variable for high-sparsity pruning.

Hu Xu, Zhaolong Xing, Congcong Liu +5

Shanghai Jiao Tong University

LLM Pruning Model Calibration

Ask PDF

Sep 24, 2026

Speech Block Influence: Component-Specific Layer Scoring for Pruning Speech LLMs

Speech LLMs are costly to deploy in resource-constrained settings. Layer pruning can cut this cost, but existing scoring metrics transfer poorly to speech LLMs: they assume a decoder-only architecture with homogeneous token sequences, whereas speech LLMs add encoder and adapter components and process multimodal sequences. We propose Speech Block Influence (SBI), the first layer-importance scoring framework designed for speech LLM pruning that consists of two component-specific scores: SBI-Enc measures the effect of encoder-layer removal at the adapter's output to better reflect downstream impact; SBI-Dec measures layer-wise input-output similarity over text-token positions only to avoid audio-token dominance. Across three speech LLMs, SBI improves pruning robustness, with stronger encoder performance at higher pruning rates and more reliable decoder layer selection by scoring text tokens rather than the audio-dominated full sequence. We further find that text-only calibration yields decoder rankings highly correlated with those from speech-text calibration, suggesting a cheaper alternative to measure decoder layer importance.

Siyu Yao, Du Q. Huynh, Lian Xu +1

The University of Western Australia

LLM Pruning Speech Language Models

Ask PDF

Abstract

LLM Pruning

Explore similar work

Rethinking Layer Redundancy: Calibration Matters More Than Search in LLM Depth Pruning

Rethinking Calibration for High-Sparsity LLM Pruning: From Single Sources to Capability Coverage

Speech Block Influence: Component-Specific Layer Scoring for Pruning Speech LLMs