Capability Scaling-Down Laws for LLM Compression

Xueqi Cheng; Liang Wu; Kelly Wan; Liangjie Hong; Yushun Dong

cs.LGOct 1, 2026

Capability Scaling-Down Laws for LLM Compression

Authors: Xueqi Cheng, Liang Wu, Kelly Wan, Liangjie Hong, Yushun Dong

Organizations: Florida State University · Nokia Applied Research

Abstract

LLM compression reduces inference costs and memory requirements, but selecting a method and configuration remains largely empirical because comparable resource reductions can produce different capability losses. We systematically investigate capability scaling-down laws for LLM compression across pruning, quantization, and distillation. Our framework measures capability loss in mathematics, code generation, and question answering, and relates these measurements to model size, training stage, compression settings, data availability, and training exposure. We develop simple predictive relations and evaluate their accuracy, measurement efficiency, and generalization to unseen configurations and model states. Sharing the density response across pruning levels halves the configuration measurements needed to fit a pruning predictor: on new Pythia states, on pre-registered OLMo-2 test states and under Wanda pruning, the compact relation matches a regression fitted with all measurements on math and code to within 0.020 nats per token, with coefficients refitted for each setting. Controlled distillation experiments show that the cost of heavy data reuse recurs across question-answering distributions, while the net benefit depends on the evaluation distribution. We further evaluate the decision value of these predictions by comparing numerical selection with configuration medians and fixed method priorities. Independent evaluations across two model families show that selection captures most of the available cross-method benefit for question answering within the tested candidate sets, where a fixed method priority attains the same regret, with smaller opportunities for mathematics and code. These results clarify the predictive scope of capability scaling-down laws and their use in compression method selection. Our code is publicly available at: https://github.com/LabRAI/scaling_down_law.

Figures & tables

Figure 1: Capability loss change from each model’s dense loss across twelve public models, under pruning to

d=0.7

(filled markers, left in each model’s slot) and per-channel int4 (open markers, right), on a symmetric logarithmic axis that is linear within

\pm 0.1

nats; grey bars span each model’s three responses under each method. The ordering of capabilities changes across series, and at the same setting the magnitudes differ by two orders of magnitude across all twelve models.

Panel of Figure 2 — Figure 2: Compression responses on the controlled Pythia panel, the structure the predictors of Section 4 build on: thin lines are the development states and bold lines their median, on a symmetric logarithmic axis that is linear within $\pm 0.1$ nats. (a) Loss change against retained density on the nine development states: mathematics and code rise along one shape with a state-dependent scale, while question answering follows no shared curve. (b) Loss change against bit-width under per-channel round-to-nearest. (c) Mathematics loss change against group size at 3, 4 and 5 bits on the six grouped-quantization development states.

Panel of Figure 3 — Figure 3: Distillation transfer responses under one protocol: loss change in nats per native token from each student’s initial state against the reuse ratio $E=T/D_{U}$ , at the matched budget nearest 200k supervised tokens, each panel on its own scale; dotted 270M, dashed 1B, solid 4B, markers the two pool seeds. Panel (d) rescores the same trajectories at that budget on three question-answering distributions: a fresh 2Wiki sample disjoint from the probe, MuSiQue and TriviaQA.

		Pythia, magnitude		OLMo-2, magnitude		Pythia, Wanda
Form	Grid	Math	Code	Math	Code	Math	Code
Compact power form	Half	0.073	0.104	0.017	0.020	0.023	0.026
Quadratic strength form	Half	0.115	0.142	0.026	0.045	0.026	0.048
Strength only	Half	0.098	0.166	0.018	0.029	0.031	0.046
Median density curve	Half	0.145	0.212	0.026	0.039	0.029	0.046
Per-density regression	Half	0.108	0.136	0.025	0.035	0.024	0.032

Table 1: Measurement efficiency of the compact pruning form in three evaluations, mean absolute error in nats per token on four test states in no fit. The half grid keeps every development state at half the densities, 18 of 36 configurations per capability on Pythia and 12 of 24 on OLMo-2; the full-grid per-density regression is the pre-specified comparator. On OLMo-2 and under Wanda every form was pre-specified before the test; on the Pythia magnitude panel the forms other than the compact form are retrospective refits.

Panel of Figure 4 — Figure 4: Measurement efficiency of the compact pruning form on mathematics and code, in nats per native token: error on new Pythia states against development measurements per capability; error per state in the pre-specified Pythia confirmation (states 1 to 4: 160M at step 80k, 410M at 112k, 1.4B and 1B at 48k), the power form at half budget and the others at full budget; and, on four OLMo-2 test states, the form fitted on 12 configurations (circles) against the per-density regression fitted on 24 (squares).

Pruning, $n=84$
		Error			Gain over baseline
Test	Delivered relation	Math	Code	QA	Math	Code	QA
Three new checkpoints	Median density curve †	0.277	0.214	0.221	$-$ 0.047	$+$ 0.008	$+$ 0.000
Quantization, $n=54$
Group sizes 32 and 512	Interpolation; median for QA †	0.065	0.116	0.458	$+$ 0.268	$+$ 0.562	$+$ 0.000
A new 1.4 billion stage	Development median †	0.088	0.153	0.141	$+$ 0.262	$+$ 0.410	$+$ 0.000

Table 2: Prediction at equal development budget. Test: configurations or states unseen by the relation. Delivered relation: predictor recommended on all evidence; a dagger marks selection after seeing test results. Error: test mean absolute error in nats per token, per capability (QA: question answering). Gain: strongest development baseline error minus delivered error; positive favours the delivered relation. Baselines are chosen inside the development folds and scored on the same cells.

n

, on each method line: development configuration measurements per capability. Appendix Table 8 names baselines and pre-specified candidates; Fig. 11 compares each row with its baseline.

Panel of Figure 5 — Figure 5: Independent validation of the final rule on four references that supplied it no outcome (R1–R4, Appendix Table 20 ): opportunity over the best quantization candidate and regret of the final rule and of quantization only, means in nats per native token over the sixteen paired budgets.

Appendix figures & tables41 assets

Supplementary material from the paper’s appendix.

Appendix

Cohort	Models and states	Interventions	Role in the study
Heterogeneous panel	Twelve public checkpoints from 5 series and 4 families: Qwen3 (0.6B, 1.7B, 4B), Gemma-3 (270M to 27B), Gemma-4 (31B), Muse (30B), OLMo-3 (7B, 32B); two further Qwen3 checkpoints (8B, 14B) measured prospectively at four densities.	Global magnitude pruning at ten densities in $[0.3,\allowbreak 0.9]$ ; per-channel round-to-nearest quantization at 8, 6, 5, 4 and 3 bits.	Establishes the phenomena and the limits of cross-family transfer; screens the measurement noise.
Controlled Pythia panel	Development states: 160M, 410M and 1.4B at steps 16k, 64k and 143k (nine states). Held-out states: 160M, 410M and 1.4B at step 96k; 1B at steps 32k, 96k and 112k; 6.9B at steps 32k and 112k.	Pruning at four development densities (0.9 to 0.6) with tests at 0.75, 0.65 and 0.55; quantization at 8, 6, 4 and 3 bits with a test at 5 bits.	Development set for pruning and quantization; tests along the source axis and the configuration axis.
Distillation, controlled	The nine Pythia development states as students; one fixed teacher pool of 600 traces per domain; Low-rank adaptation adapters under one recipe.	A fixed distillation recipe.	Tests source transfer of the signed response $\delta_{c}$ .
Distillation, pools	Gemma-3-1B student; pools of 75 and 600 traces per domain for fitting and 225 for testing; three pool seeds and two training seeds.	Processed-token budget $T$ and reuse ratio $E=T/D_{U}$ .	Tests transfer to an unseen trace pool.
Distillation, heterogeneous	Gemma-3 students from 270M to 12B, OLMo-3-7B and Qwen3 0.6B to 4B; two teachers accessed through an application programming interface; full traces compared with answer-only traces.	Varied distillation recipes.	Describes measured responses.

Appendix

Table 3: Cohorts of models and states, compression interventions, and study roles. Rows distinguish development cohorts, pre-specified prediction tests, and descriptive post-hoc measurements; each cohort is scored with the same 64 probes per capability. Densities and reuse counts are dimensionless, bit widths are in bits, and budgets are in tokens. The 2.8B Pythia checkpoints are excluded because their available revisions loaded identical weights, one state under three names.

Panel of Figure 6 — Figure 6: Learning-rate pilot for the 1B (dashed) and 4B (solid) students at one pool: final-checkpoint loss change per capability against the configured peak learning rate; positive values indicate a loss above that of the initial student at the same pool and budget.

Panel of Figure 7 — Figure 7: Measurement support for capability specificity. (a)–(c) Out-of-fold MAE of predicting the secondary-benchmark response of a capability from the primary-benchmark response of the same capability (diamond) or of another capability, against zero and train-mean controls, on the six-model, five-density pruning panel; protocols 1–3 hold out models, densities, or both, and 4–6 repeat them at $d\geq 0.75$ . (d)–(f) Paired same-over-cross gain with 95% model-bootstrap intervals over models.

Model	Loss change at density 0.7			Math at density 0.5	Threshold density	QA least; most affected	Lowest QA change (density)
	Math	Code	QA
Qwen3-0.6 billion	+0.49	+0.56	-0.26	+4.72	0.60	3/3; 0/3	-0.26 (0.70)
Qwen3-1.7 billion	+0.24	+0.19	-0.51	+3.60	0.60	6/6; 0/6	-0.90 (0.62)
Qwen3-4 billion	-0.00	-0.09	-0.79	+1.31	0.50	5/5; 0/5	-1.96 (0.55)
Qwen3-8 billion	+0.03	+0.10	-0.41	+1.03	0.50	4/4; 0/4	-1.39 (0.50)
Qwen3-14 billion	+0.06	+0.03	-0.45	+0.67	0.40	4/5; 0/5	-0.80 (0.60)

Appendix

Table 4: The table compares the pruning loss change across models and capabilities. Loss changes are in nats per native token; density is dimensionless. Ranking cells count least-affected and most-affected outcomes out of the set of densities measured for each model.

Model	Series	Loss change at 4 bits			Loss change at 3 bits			All measured bit widths
		Math	Code	QA	Math	Code	QA
Qwen3-0.6 billion	Qwen3	+0.56	+0.73	+0.21	+8.83	+10.83	+6.61	8,6,5,4,3
Qwen3-1.7 billion	Qwen3	+0.40	+0.16	-0.21	+9.53	+9.46	+8.10	8,6,5,4,3
Qwen3-4 billion	Qwen3	+0.13	+0.19	+0.39	+10.73	+11.85	+5.99	8,6,5,4,3
Qwen3-8 billion	Qwen3	+0.28	+0.48	-0.80	+12.86	+13.55	+9.45	8,6,5,4,3
Qwen3-14 billion	Qwen3	+0.09	+0.31	-0.23	+9.30	+10.79	+6.45	8,6,4,3

Appendix

Table 5: The table compares the quantization loss change across models and capabilities. Loss changes are in nats per native token, and bit widths are in bits. The last column lists all measured widths.

		Math		Code		QA
Model	Density	Magnitude	Wanda	Magnitude	Wanda	Magnitude	Wanda
Qwen3, 4B	0.9	+0.01	+0.01	+0.02	+0.01	-0.00	-0.00
	0.8	+0.00	+0.02	-0.01	+0.00	-0.29	+0.04
	0.7	-0.00	+0.05	-0.09	+0.04	-0.79	+0.08
	0.6	+0.15	+0.10	-0.00	+0.13	-1.76	-0.01
	0.5	+1.31	+0.19	+0.90	+0.24	-1.48	-0.41

Appendix

Table 6: Loss change in nats per native token from the dense model under two pruning criteria at the same retained density, on the V6 probes: global magnitude pruning of every weight matrix (the panel’s criterion) and Wanda, which scores each weight by its magnitude times the norm of its input activation over 128 C4 documents and prunes each output row of the block linear layers, leaving embeddings dense. A slash marks a density the magnitude panel did not measure.

Capability	Exponent	Intercept	Source size	Initial loss	Pretraining tokens
Math	3.75	+3.275	+0.319	+3.850	+0.250
Code	3.05	+4.499	+0.273	+5.357	+0.347
QA	3.05	-7.409	-0.481	+5.486	+0.106

Appendix

Table 7: The table gives the pre-specified pruning exponent and coefficients for each capability. Coefficients act on standardized inputs; the resulting loss change is in nats per native token. The note gives the prediction formula and the constants used to standardize each input.

Prediction task	Pre-specified candidate	Candidate error (nats)	Delivered relation	Strongest development baseline
Three unseen checkpoints pruned to densities 0.575, 0.675 and 0.85	Five-parameter power form	0.243 / 0.236 / 0.678	Source-free median density curve Retrospective	Per-density regression; median for QA
Pythia 410 million and 1.4 billion at unseen group sizes 32 and 512, bit widths 3 to 5	Source regression, the median and no change	0.215 / 0.557 / 0.457	Interpolate math and code; use the median for question answering Retrospective	Bilinear regression; no change; median
An unseen 1.4 billion stage at bit widths 3 to 5, group sizes 32 to 512	Source regression, the median and no change	0.303 / 0.153 / 0.222	Development median Retrospective	Bilinear regression; no change; median
Gemma 270 million distilled on six new pools at 50 to 200 thousand tokens	Reuse forms, with a budget and pool form for question answering	0.074 / 0.019 / 0.515	The same predictor	Budget regression; reuse regression for Code and QA
Gemma 1 billion distilled on six new pools at 50 to 200 thousand tokens	Reuse forms, with a budget and pool form for question answering	0.057 / 0.049 / 0.463	The same predictor	Loss regression; reuse regression; size regression

Appendix

Table 8: Pre-specified candidates for Table 2 , their errors on the same cells, delivered relations, and strongest development baselines. Errors are mean absolute errors in nats per token; errors and baseline names follow math, code and question answering order unless qualified. The distillation rows are the 270 million and 1 billion students.

Compression method	Capability	Separate capability response	Shared response with scale	Shared response with offset	Shared response only	Gain over scaled shared response [95% confidence interval]
Pruning	Math	0.2279	0.2328	0.2249	0.1677	0.0048 [-0.0495, 0.0591]
	Code	0.1736	0.1776	0.1750	0.1098	0.0040 [-0.0181, 0.0228]
	QA	0.1818	0.3500	0.4626	0.6165	0.1683 [0.1054, 0.2434]
	Capability mean	0.1944	0.2535	0.2875	0.2980	0.0590 [0.0206, 0.1025]
Grouped quantization	Math	0.3234	0.3343	0.3380	0.3439	0.0109 [0.0024, 0.0160]
	Code	0.3838	0.3744	0.4485	0.4312	-0.0094 [-0.0234, 0.0226]

Appendix

Table 9: The table compares separate capability responses with shared responses on held-out confirmation panels. Mean absolute errors and gains are in nats per native token. Positive gains favor separate responses; brackets give paired confidence intervals for the error differences.

Panel of Figure 8 — Figure 8: Measured responses and pre-specified relations on native configuration axes. (a) Pythia-1.4B at step 112k: measured densities, the delivered power form (solid, Eq. 3 ) and the pre-specified median development curves (dashed); shading marks the refit range $d\in[0.55,0.9]$ . (b)–(d) Math responses to group size at 3, 4 and 5 bits on Pythia-410M at 143k (solid) and 1.4B at 16k (dotted): pre-specified piecewise interpolation, per-configuration medians (dashed), and predictions at the unmeasured $g=32,512$ (hollow) with their later measurements (filled). (e)–(f) Gemma-3 270M and 1B trajectories at $U=200$ : measured responses at the recorded reuse ratio, the pre-specified selected forms (capability colours) and the strongest pre-specified baseline (grey), all on the native configuration axes.

Law	What is shared	What varies	Held-out error, shared against capability-specific
Pruning power	Power family	Source coefficients and exponent	0.494 against 0.494
Grouped quantization	Separable family	Source coefficients and exponents	2.962 against 2.398
Distillation exposure	$\log(1+E)$ family	Source coefficients; inactive $D_{0}$	0.542 against 0.275
Pruning sharing	One Exponent	Source coefficients	0.491 against 0.494
Quantization sharing	two-dimensional term vectors	Capability offsets	3.999 against 2.398
Pruning Math	Power family	Source amplitudes, Exponent	Exponent minimum, median, maximum: 3.00, 4.60, 6.00

Appendix

Table 10: The table compares a shared response family with capability-specific responses, and reports the spread of exponents fitted separately per source. Errors are mean absolute errors in nats per native token, pooled equally over capabilities. Distillation carries no pretraining-token input, so that term is inactive.

Panel of Figure 9 — Figure 9: The four-corner second difference per student and readout on its own scale: the measured value with its pre-specified interval, the additive prediction of zero and the pre-specified interaction prediction. Both predictions lie inside the QA intervals, so the design did not separate the two structures; the math and code readouts were specified in advance as underpowered before the trajectories were trained.

Figure 10: Corner residuals for the two students and the pre-specified additive corner contrasts on the fresh 2Wiki, MuSiQue and TriviaQA distributions, whose pre-specified additive prediction is zero; the legend of Fig. 9 applies to every panel, with the same colours and markers.

Panel of Figure 11 — Figure 11: Generalization of the pre-specified relations by kind of novelty: each row compares the error of the relation, a hollow diamond, with the development-selected baseline, a hollow circle, on the same cells, one sub-row per capability; the segment is coloured where the relation is lower.

Student	Capability	Selected	Baseline	Relation error	Baseline error	Improvement [95% confidence interval]
Gemma-3-270 million	Math	Reuse response	Budget only	0.0743	0.0654	-0.0088 [-0.0098, -0.0081]
	Code	Reuse response	Reuse only	0.0193	0.0230	0.0038 [0.0025, 0.0050]
	QA	Joint budget-pool response	Reuse only	0.5150	0.6097	0.0947 [0.0774, 0.1093]
Gemma-3-1 billion	Math	Reuse response	Response surface, initial loss	0.0569	0.0335	-0.0234 [-0.0235, -0.0233]
	Code	Reuse response	Reuse only	0.0485	0.0532	0.0047 [0.0042, 0.0050]
	QA	Joint budget-pool response	Response surface, student size	0.4634	0.4502	-0.0132 [-0.0198, -0.0098]

Appendix

Table 11: The table compares distillation relations with baselines selected before confirmation. Mean absolute errors and paired improvements are in nats per native token. Improvement is baseline error minus relation error; brackets give paired confidence intervals. Pool sizes count traces per domain; budgets count supervised tokens.

Panel of Figure 12 — Figure 12: Measurement efficiency for question answering, the counterpart of Fig. 4 . The median density curve is the better predictor at every development budget, and on the four confirmation states the three predictors are within 0.006 nats of each other, 0.177 for the per-density regression, 0.179 for the power form and 0.183 for the median curve, so nothing here displaces the median curve as the delivered rule for this capability; on OLMo-2 the reduced-grid form and the full-grid regression are within 0.002 nats on the probes and 0.009 apart on the new items, both far below the zero-change baseline of 0.13 and 0.16.

Predictor (development measurements per capability)	Math	Code	QA
Power form (18)	0.073	0.104	0.179
Power form (36)	0.062	0.109	0.223
Per-density regression (36)	0.060	0.084	0.177
Median density curve (36)	0.112	0.170	0.183
Median density curve (18)	0.145	0.212	0.186

Appendix

Table 12: Pre-specified measurement-efficiency confirmation on four Pythia states outside every fit, 160M at step 80000, 410M at step 112000, 1.4B at step 48000 and 1B at step 48000, with 24 cells per capability: mean absolute error in nats per token of each pre-specified predictor. The reduced-budget power form used the densities 0.9 and 0.7 of each development state, 18 measurements per capability.

Measurements per capability	Predictor	Math	Code	QA
9	Power form	0.25	0.30	0.61
9	Per-density regression	/	/	/
9	Median curve	0.33	0.47	0.58
18	Power form	0.20	0.17	0.52
18	Per-density regression	0.24 (6/20 fits)	0.28 (6/20 fits)	0.76 (6/20 fits)
18	Median curve	0.21	0.21	0.33

Appendix

Table 13: Subsampling study on the nine-state development panel: median mean absolute error in nats on the new-state cells over twenty fixed replicates, when the nine-state development panel retains 9, 18 or all 36 density measurements per capability in total. Where fewer than twenty replicates could be fitted, the count is given; a slash means no replicate could be fitted.

		Probes			New items
Predictor	Development grid	Math	Code	QA	Math	Code	QA
Compact power form	Reduced (12)	0.017	0.020	0.062	0.012	0.029	0.045
Quadratic strength form	Reduced (12)	0.026	0.045	0.061	0.030	0.056	0.041
Strength only	Reduced (12)	0.018	0.029	0.076	0.019	0.037	0.050
Median density curve	Reduced (12)	0.026	0.039	0.074	0.030	0.051	0.048
Per-density regression	Reduced (12)	0.025	0.035	0.062	0.030	0.046	0.043

Appendix

Table 14: Second-family confirmation on OLMo-2: mean absolute error in nats per token over 4 test states (1B and 7B intermediate checkpoints used in no fit) at three densities fixed before the freeze, 0.85, 0.75 and 0.65, on the 64 development probes per capability and on 64 new items per capability. Every predictor is fitted once on the six development states, on the reduced grid of two densities per state or on the full grid of four, with the same labels, the same tuning grids and the same leave-one-state-out ridge rule, and pre-specified before any test state was pruned. The reduced grid costs 59 percent of the full grid’s GPU seconds and 60 percent of its evaluation tokens, dense anchors and model loading included.

		Probes			New items
Predictor	Development grid	Math	Code	QA	Math	Code	QA
Compact power form	Reduced (18)	0.023	0.026	0.058	0.025	0.030	0.048
Quadratic strength form	Reduced (18)	0.026	0.048	0.038	0.028	0.052	0.042
Strength only	Reduced (18)	0.031	0.046	0.039	0.033	0.050	0.039
Median density curve	Reduced (18)	0.029	0.046	0.033	0.029	0.050	0.049
Per-density regression	Reduced (18)	0.024	0.032	0.060	0.026	0.034	0.049

Appendix

Table 15: Measurement-efficiency confirmation under Wanda pruning on Pythia: mean absolute error in nats per token over 4 test states (the four A11 states, in no fit) at the six A11 densities, three of them unseen in development, on the 64 development probes per capability and on 64 new items per capability. Every predictor is fitted once on the nine development states, on the reduced grid of two densities per state or on the full grid of four, with the same labels, the same tuning grids and the same leave-one-state-out ridge rule, and pre-specified before any test state was pruned. The reduced grid costs 70 percent of the full grid’s GPU seconds, the Wanda calibration pass counted once per state, and 60 percent of its evaluation tokens, dense anchors and model loading included in both grids, as in the two earlier confirmations.

		18 measurements			36 measurements
Form	Parameters	Math	Code	QA	Math	Code	QA
Compact power form	5	0.073	0.104	0.179	0.062	0.109	0.223
Linear strength (fixed exponent one)	4	0.151	0.183	0.227	0.250	0.310	0.359
Quadratic strength, same inputs	5	0.115	0.142	0.206	0.187	0.238	0.317
Strength only, no source inputs	2	0.098	0.166	0.243	0.084	0.153	0.239
Median density curve	one per density	0.145	0.212	0.186	0.112	0.170	0.183

Appendix

Table 16: Same-complexity baselines for the measurement-efficiency confirmation. Every form is fitted on the same development rows as the compact power form, at 18 and at 36 measurements per capability, with the same inputs and estimator, and scored on the same 24 confirmation cells per capability; entries are mean absolute errors in nats per token. The pruning-law form is the literature relation

L=L_{0}P_{0}d^{\alpha}

( Sengupta et al., 2026 ) fitted on capability loss, with one exponent and one scale per capability and no calibration on the target. The last two rows calibrate one number on each target’s mildest pruned measurement and are scored on the remaining five densities, twenty cells per capability, so they are not comparable with the rows above. All forms other than the compact power form, the median curve and the per-density regression at 36 measurements were fitted after the confirmation, on its stored development rows.

Predictor	Target measurement used	Math	Code	QA	Pooled
Pruning-law form	none	0.167	0.151	0.117	0.145
Pruning-law form	one accuracy at density 0.9	0.139	0.128	0.090	0.119
Loss route: median curve and link	none	0.264	0.253	0.274	0.264
Loss route: median curve and link	one loss at density 0.9	0.323	0.313	0.301	0.312
Link on the measured loss	the target’s own loss	0.159	0.180	0.102	0.147

Appendix

Table 17: Prediction of task accuracy after magnitude pruning, the endpoint of the pruning law, on Gemma 3 1B and 4B and Qwen3 1.7B and 4B at densities 0.8, 0.7 and 0.6, leave-one-model-out: mean absolute error in accuracy points over 36 cells per column, 64 probes per capability. The pruning-law form fits one exponent per capability on the other three models; the loss route predicts the loss with the final rule’s development median curve and maps it to accuracy with a logistic link fitted on the other three models; the last row applies that link to the measured loss. Calibration uses one measurement of the target at density 0.9. Paired over cells, the loss route minus the law is +0.119 [+0.042, +0.200] without calibration and +0.193 [+0.104, +0.285] with it.

Readout	Student	Budget (k tokens)	Outcome	Measured interval (k independent tokens)	Recommended pool (k independent tokens)
					Boundary relation	Fixed reuse	Per-student interpolation
Math	1 billion	50	non-monotone	unresolved	32.2, unresolved	9.0, unresolved	158.4, unresolved
Math	1 billion	100	Crossed in range	$[9.0,\allowbreak\,32.5]$	32.2, inside	32.2, inside	32.2, inside
Math	1 billion	200	Crossed in range	$[9.0,\allowbreak\,32.5]$	32.2, inside	32.2, inside	32.2, inside
Math	4 billion	50	Upper bound	$\leq 9.1$	38.1, $\geq 1.43$	9.0, inside	158.4, $\geq 2.86$
Math	4 billion	100	Crossed in range	$[8.9,\allowbreak\,38.7]$	38.1, inside	38.1, inside	none

Appendix

Table 18: Measured data requirements at the primary loss tolerance

\tau=0.25

nats are shown for both Gemma 3 students, identified by parameter count. Budgets count supervised training tokens, including reuse; k means thousands. The largest pool varies the training seed; smaller pools vary the pool draw. Intervals are observed brackets, not confidence intervals. Each boundary-rule recommendation gives a pool size followed by inside or its natural-log distance to the interval. Inside includes bracket endpoints and does not verify success. For a one-sided bound, distance is only a lower bound. Unresolved means no measured interval; none means the rule abstains.

Panel of Figure 13 — Figure 13: Measured data requirements of distillation at a tolerance of 0.25 nats: for each student and budget, the range of independent tokens between the largest failing and the smallest passing pool; an upward marker at the largest pool means the constraint is not yet met, a downward marker at the smallest pool that it is met everywhere, and a cross that the seeds disagree. The largest pool varies the training seed, and the smaller pools vary which traces are drawn into the pool.

Panel of Figure 14 — Figure 14: Falsification of the logarithmic reuse term. The pool-change effect divided by its logarithmic bracket against the supervised budget, per capability, for the (a) 1B and (b) 4B students; the heavy line is the per-budget median over pool pairs and the band their range. A logarithmic reuse term would predict a flat line in budget.

Panel of Figure 15 — Figure 15: Determinants of the distillation response and the displacement account of the compression loss change. (a) The reuse exponent $p$ across training folds per capability with the full-development value and its conditional interval; $p=0$ is the logarithmic term and $p=1$ linear. (b) Median relative error of the second-order displacement account as a function of loss-change magnitude under pruning, grouped quantization and per-channel quantization.

Family	Configuration: median relative error
Pruning	$d=0.9$ : 0.9%; 0.85: 4.3%; 0.8: 5.3%; 0.75: 11.1%; 0.7: 16.0%; 0.65: 28.2%; 0.6: 44.4%
Grouped quantization	5 bits: 1.6–2.7% at each group size; 4 bits: 3.3% to 9.2% by group size; 3 bits: 35.9%
Per-channel quantization	8 bits: 2.5%; 6 bits: 3.0%; 5 bits: 4.2%; 4 bits: 23.8%; 3 bits: 53.9%

Appendix

Table 19: Median relative error of the coefficient-free second-order account, by family and configuration. Rows identify compression families; the second column pairs each setting with its median relative error in percent, from a post-hoc analysis of the measured damage.

Role	Model and stage	Matrix parameters	Weights seen in earlier work	What is new in this round
Reference 1	Pythia 410M, training step 120000	302M	No	No earlier use; first compression
Student for reference 1	Pythia 160M, training step 120000	85M	No	No earlier use; first training
Reference 2	Pythia 1.4B, training step 120000	1.21B	No	No earlier use; first compression
Student for reference 2	Pythia 410M, training step 120000	302M	No	No earlier use; first training
Reference 3	Gemma 3, 1B	698M	Yes	New candidates, familiar weights
Student for reference 3	Gemma 3, 270M	100M	Yes	Retrained from familiar weights

Appendix

Table 20: Identity of the eight roles in the selection validation. All are pretrained base releases, none instruction tuned, and the eight roles draw on six distinct snapshots because two serve twice. The two Pythia states appear nowhere in the earlier records; the Gemma releases were measured in earlier development, so for them this round is a new set of candidates on familiar weights. Every candidate in the round is a fresh run, and no measurement on any of these weights entered the fitted selection rule. Parameter counts cover the text-decoder matrices. Exact revisions and their identifiers, the evidence for each entry, and the times at which the predictions were fixed are recorded in the repository audit.

Reference	Objective	Opportunity	Rule regret	Quantization-only regret
Pythia 410M, step 120000	Math	0.000	0.000	0.000
	Code	0.000	0.000	0.000
	QA	0.117	0.027	0.592
	Largest increase	0.001	0.000	0.001
Pythia 1.4B, step 120000	Math	0.000	0.000	0.000
	Code	0.000	0.000	0.000

Appendix

Table 21: Independent cross-method selection validation. None of the four references supplied an intervention outcome to the rule, and each carries twelve candidates at seventeen storage budgets, including a distillation candidate trained in this round. Opportunity is the loss of the best feasible quantization candidate minus that of the best feasible candidate across all methods; policy regret is the loss of the candidate that policy selected minus that same all-method optimum. All values are means in nats per native token of that family over the sixteen budgets at which both policies find a feasible candidate, and are never pooled across families; no quantization setting fits the smallest budget. All four references meet the pre-specified criterion, which compares the two policies per reference after averaging over the four objectives. Read by objective, the rule is ahead or level throughout except code on Gemma 3, 1B, where it is 0.0001 nats behind.

Reference	Objective	Pristine opportunity	Distillation opportunity change	Other methods	Pre-specified opportunity
Pythia 410M	Math	0.000000	0.000000	0.000000	0.000000
	Code	0.000000	0.000000	0.000227	0.000227
	QA	0.005770	0.083639	0.027125	0.116534
	Largest increase	0.000000	0.000000	0.000939	0.000939
Pythia 1.4B	Math	0.000276	-0.000276	0.000000	0.000000
	Code	0.007068	-0.007068	0.000037	0.000037

Appendix

Table 22: This analysis is retrospective. The pristine student

S_{0}

was not in the pre-specified candidate set, and the rule’s selections remain unchanged.

Reference	Objective	Opportunity	Priority	Median	Relations	Rule	Rule gain
Pythia 410M, step 120000	Math	0.000	0.000	0.000	0.000	0.000	0.000
	Code	0.000	0.000	0.000	0.000	0.000	0.000
	QA	0.117	0.027	0.027	0.059	0.027	0.089
	Largest increase	0.001	0.000	0.000	0.000	0.000	0.001
Pythia 1.4B, step 120000	Math	0.000	0.000	0.000	0.045	0.000	0.000
	Code	0.000	0.000	0.000	0.005	0.000	0.000

Appendix

Table 23: Retrospective policy ablation on the independent selection validation. Every policy chooses from the same twelve candidates at the same seventeen budgets; entries are mean regret against the all-method optimum in nats per native token over the sixteen budgets at which both the rule and quantization find a candidate, with the opportunity of Table 21 repeated. The development-priority policy ranks methods by their oracle win rate on the development panel and takes the largest feasible storage within the method, with no numeric prediction; the median policy predicts every candidate from development medians and the student from the development constant; the relations policy uses the source-conditioned relations of Section 4 where their inputs exist and the selected distillation forms; the final rule is the policy fixed before the round, and its quantization-only counterpart is in Table 21 . The rule gain is the loss of the best quantization candidate minus that of the rule’s choice, the gain the rule realises even against a perfect quantization selection. The three added policies were defined after the round from development data alone.

Reference	Model (policies)	Loss	Exact match, 32 tokens	Exact match, 96 tokens	Token F1, 96 tokens	Truncated (percent)	Loss difference from the quantization-only choice
Pythia 410M, step 120000	pruned to density 0.8 (oracle)	5.138	0.109	0.117	0.140	32	-0.66 [-0.75, -0.57]
	grouped 4 bits, group 32 (quantization oracle)	5.388	0.094	0.102	0.122	38	-0.41 [-0.51, -0.30]
	per-channel 4 bits (rule, quantization only, oracle, quantization oracle)	5.797	0.044	0.049	0.069	42	0.00 (reference)
	per-channel 6 bits (quantization oracle)	5.224	0.174	0.143	0.183	23	-0.57 [-0.69, -0.46]
	distilled student (rule, oracle)	5.246	0.029	0.089	0.126	73	-0.55 [-0.68, -0.42]
	initial student	5.219	0.070	0.078	0.110	51	-0.58 [-0.70, -0.46]

Appendix

Table 24: Fresh-item check of the models the policies chose for question answering, on 384 unused 2WikiMultihopQA items: completion loss in nats per native token; exact match under the 32-token readout and under the 96-token readout with the gold-blind extraction rule, with that readout’s token F1 and the percentage of generations reaching the cap without a stop marker; and the difference of token-weighted mean losses from the quantization-only choice, the statistic of the loss column, with its 95% bootstrap interval over items, negative favouring the row. Parentheses name the policies that chose the model.

Candidate	Nominal storage ratio	Bytes as a fraction of the dense reference
Dense reference	1.000	1.000
Pruning, density 0.9	0.900	bitmap 0.963; compressed sparse row 1.802
Pruning, density 0.8	0.800	bitmap 0.863; compressed sparse row 1.602
Pruning, density 0.7	0.700	bitmap 0.762; compressed sparse row 1.402
Pruning, density 0.6	0.600	bitmap 0.662; compressed sparse row 1.202
Per-channel 8 bits	0.500	0.501

Appendix

Table 25: Storage of the selection candidates in bytes, as a fraction of the dense reference stored in 16-bit weights, beside the nominal storage ratio the selection uses. Pruning is priced in a bitmap format, 16-bit kept weights plus one bit per position, and in compressed sparse row format, 16-bit kept weights with 16-bit column indices and 32-bit row pointers; quantization packs its bits exactly and adds one 16-bit scale per output row or per group; the distilled student is stored in 16-bit weights. Byte ratios agree across the four references to three decimals except for the student, whose four values follow the order Pythia 410M, Pythia 1.4B, Gemma 3 1B, Gemma 3 4B, the four references of the validation.

Panel of Figure 16 — Figure 16: Capability-specific selection maps on held-out Pythia states under the earlier laws, retrospectively and leave-one-state-out. For each state and nominal storage budget the colour is the method with the lowest predicted loss among feasible measured configurations; grey cross-hatching marks cells with no feasible configuration, diagonal hatching the no-clear-winner heuristic, and a hollow circle with cross-hatching a cell in which the measured oracle chooses another method.

Capability	Policy	Regret	Method agreement (percent)
Math	Final selection rule	0.0000	98.5
Math	Source-conditioned	0.0149	98.5
Math	Quantization only	0.0068	94.1
Math	Cheapest feasible	1.7716	73.5
Code	Final selection rule	0.0043	95.6
Code	Source-conditioned	0.1124	100.0

Appendix

Table 26: The table compares pre-specified selection policies on the independent confirmation panel. Mean regret is in nats per native token, and method agreement is the percentage of evaluated cells in which the policy’s method matches the oracle’s.

Compression method	Capability	State status	Response predictor
Pruning	Mathematics	Seen source state; new density	Pruning power form
Pruning	Code	Seen source state; new density	Pruning power form
Pruning	Question answering	Seen source state; new density	Pruning development median curve
Pruning	Mathematics	New size or new stage	Pruning development median curve
Pruning	Code	New size or new stage	Pruning development median curve
Pruning	Question answering	New size or new stage	Pruning development median curve

Appendix

Table 27: The table gives the selection predictor for each compression method, capability and source-state status. A prediction adds the listed response, in nats per native token, to a dense anchor: the source loss for pruning and quantization, the initial student’s loss for distillation. Each method is fitted on its own development panel, listed below. The rule was pre-specified before confirmation.

Compression method	Capability	State status	Response predictor
Grouped round-to-nearest quantization	Mathematics	Seen state	Piecewise source interpolation
Grouped round-to-nearest quantization	Code	Seen state	Piecewise source interpolation
Grouped round-to-nearest quantization	Question answering	Seen state	Per-configuration development median
Grouped round-to-nearest quantization	Mathematics	New state	Per-configuration development median
Grouped round-to-nearest quantization	Code	New state	Per-configuration development median
Grouped round-to-nearest quantization	Question answering	New state	Per-configuration development median

Appendix

Table 28: The table continues the pre-specified selection rules. Absolute losses are in nats per native token.

Figure 17: Mean regret per objective on the four earlier Pythia states, in nats per native token on a logarithmic scale, for the final rule, the earlier source-conditioned laws, quantization-only and the cheapest feasible configuration; the saved pooled means of the confirmation round are plotted without recomputation.

Panel of Figure 18 — Figure 18: Full selection maps on the four fresh states under the final rule and under the earlier source-conditioned laws, in row pairs, against nominal storage budget; colour: predicted method; hollow circle with cross-hatching: the oracle differs; diagonal hatching: no clear winner. The last panel uses the largest loss increase across capabilities as the objective.

Readout	Original	With end marker	Difference [95%]
Pythia 1.4B reference, student Pythia 410M
Loss, nats per token	4.860	4.853	-0.007 [-0.060, +0.047]
Exact match, 32 new tokens	0.065	0.083	+0.018 [-0.010, +0.049]
Exact match, 96 new tokens	0.076	0.076	+0.000 [-0.034, +0.031]
Token F1, 96 new tokens	0.117	0.106	-0.011 [-0.045, +0.023]
Reached the 96-token cap	0.628	0.005	-0.622 [-0.672, -0.573]

Appendix

Table 29: End-marker control on the 384 fresh question-answering items for the selection-round students of the Pythia 1.4B and Gemma 3 1B references: the original student against a student retrained on the same 1,800 teacher answers, recipe, seed and schedule with the tokenizer’s end-of-sequence token appended to every target and supervised, 1,781 extra supervised tokens per epoch. Differences are control minus original, with 95 percent intervals from a paired item bootstrap; the loss is token-weighted as in Eq. 1.

Budget	Information available	Cost to disclose
Metadata only (K0)	$N_{0}$ , public $D_{0}$ , architecture, configuration	$D_{0}$ may be undisclosed
Plus dense anchor (K0)	Plus the dense losses $L_{0,c}$ , measured once	One forward pass per probe
One target calibration (K1)	Plus one pre-specified target measurement	Its configuration and cost
Oracle diagnostic	Rescaling on measured target outcomes	None; measures form misfit

Appendix

Table 30: Information budgets: K0 uses no compressed-target calibration, K1 one calibration measurement, and the oracle row is a post-hoc diagnostic.

LLM Compression

Large language model compression reduces the memory, compute, and context costs of LLMs while aiming to preserve their capabilities.

See all 273 papers

Jun 23, 2026

Scaling Laws for Task-Specific LLM Distillation

Large Language Models (LLMs) achieve strong performance across a growing range of domains, yet their scale poses deployment challenges in applications where latency and cost constraints are critical. This paper derives empirical scaling laws for domain-specific LLM compression, quantifying how in-domain and general knowledge performance scale with dataset size, compression ratio, supervision format, and iterative pruning schedule. Using quantitative finance as our application domain, we compare logit-based and LoRA-based distillation under iterative structural pruning, introducing a blended chain-of-thought supervision loss that stabilizes KL-divergence distillation over reasoning traces. In-domain task quality degrades predictably under compression while general-knowledge benchmarks collapse well before the same point; supervision format is the key driver of this tradeoff, with chain-of-thought supervision actively recovering general knowledge that pruning erases. We release the headline dataset FinHeadlineMix, scaling law results, and practical recommendations to provide a reusable framework for domain-specific compression decisions.

Lavinia Ghita, Dhruv Desai, Ioana Boier

NVIDIA

LLM Pruning LLM Compression Language Model Scaling Laws Language Model Distillation

Ask PDF

Mar 17, 2026

Frequency Matters: Fast Model-Agnostic Data Curation for Pruning and Quantization

Post-training model compression is essential for enhancing the portability of Large Language Models (LLMs) while preserving their performance. While several compression approaches have been proposed, less emphasis has been placed on selecting the most suitable set of data (the so-called calibration data) for finding the compressed model configuration. The choice of calibration data is a critical step in preserving model capabilities both intra- and inter-tasks. In this work, we address the challenge of identifying high-performance calibration sets for both pruning and quantization by analyzing intrinsic data properties rather than model-specific signals. We introduce ZipCal, a model-agnostic data curation strategy that maximizes lexical diversity based on Zipfian power laws. Experiments demonstrate that our method outperforms standard uniform random sampling across various pruning benchmarks. Notably, it also performs on par, in terms of downstream performance, with a state-of-the-art method that relies on model perplexity. The latter becomes prohibitively expensive for large-scale models and datasets, while ZipCal is on average

\sim

240

\times

faster due to its tractable linear complexity. We make the code and the experiments available at https://github.com/FrancescoMonaco/ZipCal.

Francesco Pio Monaco, Elia Cunegatti, Flavio Vella +1

University of Trento

LLM Compression LLM Pruning LLM Quantization Post-Training Quantization

Ask PDF

Mar 17, 2026

Capability-Guided Compression: Toward Interpretability-Aware Budget Allocation for Large Language Models

Large language model compression has made substantial progress through pruning, quantization, and low-rank decomposition, yet a fundamental limitation persists across all existing methods: compression budgets are allocated without any representation of what individual model components functionally encode. We term this the capability-blind compression problem and argue it is a root cause of two well-documented failures -- the insensitivity of perplexity-based evaluation to reasoning capability loss, and the abrupt phase transitions in model performance recently characterized by Ma et al. (2026). We propose Capability-Guided Compression (CGC), a framework that addresses this by using Sparse Autoencoder (SAE)-derived capability density maps to allocate differential compression budgets across transformer components. Capability density is a formally defined scalar measure combining the feature breadth, activation entropy, and cross-input consistency of a component's SAE feature activation distribution. We prove theoretically that components with higher capability density exhibit lower structural redundancy and reach their individual phase transition points at lower compression ratios, providing the first pre-compression mechanism for component-level phase transition prediction. Experiments on GPT-2 Medium confirm that capability density is statistically independent of Wanda importance scores (Spearman rho = -0.054, n = 384 heads), establishing it as a genuinely novel compression signal orthogonal to all existing importance metrics. We report a negative result on PPL-based compression comparison and provide a principled diagnosis identifying GPT-2 Medium as an insufficient test bed for the full CGC hypothesis. The theoretical framework, density formalism, and orthogonality finding constitute a foundation for capability-aware compression research.

Rishaank Gupta

Independent Researcher

Phase Transitions in Neural Networks Model Compression LLM Compression

Ask PDF

Abstract

LLM Compression

Explore similar work

Scaling Laws for Task-Specific LLM Distillation

Frequency Matters: Fast Model-Agnostic Data Curation for Pruning and Quantization

Capability-Guided Compression: Toward Interpretability-Aware Budget Allocation for Large Language Models