Compression Technique Interactions in Small Language Models: A Factorial Ablation Study
Organizations: Amazon Web Services, Seattle, WA, USA
Abstract
Deploying small language models on resource-constrained hardware requires compression, yet practitioners lack systematic guidance on how quantization, pruning, and post-compression recovery interact when combined. We present a factorial ablation study evaluating 72 compression configurations across four SLM architectures (2.7B-7B parameters), extended with a 48-configuration ordering analysis, benchmark accuracy validation on 30 compressed models, and robustness checks including multi-seed validation and full-corpus perplexity evaluation. INT8 quantization is effectively lossless (<0.5% accuracy degradation), while NF4 quantization combined with L1 magnitude pruning causes model collapse in all 16 tested combinations. LoRA-based fine-tuning reduces degraded perplexity by up to 97% after aggressive pruning; an ablation against standard fine-tuning without a teacher model shows the recovery is driven by LoRA adaptation itself rather than the distillation signal. Compression ordering shows no significant effect among viable configurations (ANOVA p = 0.96) but determines viability for NF4+pruning combinations. INT8 quantization alone provides 40-45% memory reduction with no quality loss. On-device CPU deployment via llama.cpp confirms these findings transfer off the GPU: 4-bit quantization runs 46-54% faster than 8-bit for a perplexity difference under 0.5 points, while un-structured pruning yields no CPU latency or memory benefit, consistent with its dense storage format.
Figures & tables
| Model | Provider | Params | License |
|---|---|---|---|
| Phi-2 | Microsoft | 2.7B | MIT |
| Qwen3-4B | Alibaba | 4.0B | Apache-2.0 |
| Nemotron-Mini-4B | NVIDIA | 4.0B | NVIDIA OML |
| Qwen2.5-7B-Inst. | Alibaba | 7.0B | Apache-2.0 |
| Model | PPL | MMLU | HSwag | ARC | BoolQ | Mem |
|---|---|---|---|---|---|---|
| Phi-2 | 20.7 | .545 | .559 | .530 | .834 | 5.2 |
| Qwen3-4B | 26.7 | .684 | .523 | .508 | .851 | 7.5 |
| Nemotron | 18.5 | .569 | .559 | .506 | .826 | 7.8 |
| Qwen-7B | 11.8 | .718 | .621 | .527 | .864 | 14.2 |
| Model | Prune | No LoRA | With LoRA | Reduction |
|---|---|---|---|---|
| Qwen-7B | 50% | 920.6 | 24.3 | 97.4% |
| Qwen3-4B | 50% | 148.2 | 43.2 | 70.9% |
| Nemotron | 30% | 50.1 | 22.0 | 56.0% |
| Phi-2 | 50% | 55.8 | 31.7 | 43.2% |
| Model | Prune | Pruned | Self-Distill | LoRA-Only |
|---|---|---|---|---|
| Phi-2 | 50% | 59.73 | 34.74 | 36.11 |
| Qwen3-4B | 50% | 146.19 | 46.45 | 39.71 |
| Qwen-7B | 50% | 1089.95 | 28.80 | 27.48 |
| Nemotron | 30% | 58.01 | 23.15 | 24.95 |
| Ordering | INT4 | INT8 |
|---|---|---|
| 20.78 | 19.71 | |
| 20.67 | 19.73 | |
| – | 21.45 | |
| – | 20.75 |
| Tier | Models that fit | Example | RSS (MB) | tok/s | PPL |
|---|---|---|---|---|---|
| 2 GB | None | – | – | – | – |
| 4 GB | Phi-2 | Phi-2 q4 | 2848 | 17.5 | 11.01 |
| 8 GB | + Qwen3-4B, Nemotron | Nemotron q8 | 4356 | 9.8 | 9.25 |
| 12 GB | All four | Qwen2.5-7B q4 | 7545 | 7.3 | 6.91 |
| Target | Config | Cost / benefit |
|---|---|---|
| GPU server ( 16 GB) | INT8 only | 0.5% acc. loss; 40–45% memory |
| GPU, sparse runtime | INT8 + 30% prune + recovery | 1% acc. loss; validate per arch. |
| Mobile (4–6 GB) | INT4, no pruning | 1–2.6% acc. loss |
| CPU, latency-bound | 4-bit (q4) | 46–54% faster; 0.5 PPL vs. q8 |
| Avoid | NF4 + pruning, quantize-first | collapse (16/16 configs) |
Appendix figures & tables6 assets
Supplementary material from the paper’s appendix.
Appendix
| Model | Quant | Prune | Recov | PPL | MMLU | HSwag | ARC | BoolQ | Avg Acc | Acc |
| Phi-2 (2.7B) | ||||||||||
| none | none | no | 20.67 | 0.545 | 0.559 | 0.530 | 0.834 | 0.617 | – | |
| none | none | yes | 20.91 | 0.543 | 0.556 | 0.534 | 0.836 | 0.617 | 0.0% | |
| INT8 | none | no | 21.25 | 0.543 | 0.555 | 0.531 | 0.829 | 0.615 | 0.3% | |
| INT8 | none | yes | 20.99 | 0.542 | 0.554 | 0.522 | 0.837 | 0.614 | 0.5% | |
| INT8 | 30% | yes | 21.14 | 0.531 | 0.551 | 0.522 | 0.834 | 0.610 | ||
| Model | Config | PPL | |
|---|---|---|---|
| Phi-2 | Baseline | 22.28 | 0.00 |
| Phi-2 | INT8+30%+R | 22.86 | 0.07 |
| Phi-2 | INT4 | 23.62 | 0.00 |
| Phi-2 | FP16+50%+R | 34.79 | 0.28 |
| Qwen3-4B | Baseline | 26.89 | 0.00 |
| Qwen3-4B | INT8+30%+R | 27.35 | 0.18 |
| Model | q4 tok/s | q8 tok/s | Speedup | q4 PPL | q8 PPL |
|---|---|---|---|---|---|
| Phi-2 | 17.5 | 11.5 | +52% | 11.01 | 10.74 |
| Qwen3-4B | 12.5 | 8.1 | +54% | 14.57 | 14.15 |
| Nemotron-4B | 14.4 | 9.8 | +46% | 9.50 | 9.25 |
| Qwen2.5-7B | 7.3 | 4.8 | +52% | 6.91 | 7.31 |
| Config | RSS (MB) | tok/s | PPL | Min fit | |
|---|---|---|---|---|---|
| Phi-2 (2.7B) | |||||
| phi2-baseline-q4 | 2848 | 17.50 | 0.09 | 11.01 | 4 GB |
| phi2-baseline-q8 | 2957 | 11.53 | 0.10 | 10.74 | 4 GB |
| phi2-int4-q4 | 2848 | 17.93 | 0.07 | 11.64 | 4 GB |
| phi2-int4-d-q4 | 2848 | 17.94 | 0.13 | 11.65 | 4 GB |
| phi2-int8-q8 | 2957 | 11.48 | 0.09 | 10.85 | 4 GB |
Small Language Models
Small language models are compact neural language systems designed to deliver useful text and reasoning capabilities with limited compute, memory, and energy.
See all 296 papers