When Bigger Becomes Brittle: Why Compression Hurts More as Language Models Scale
Organizations: Indraprastha Institute of Information Technology, Delhi · TurboML · Independent Researcher · Microsoft ·
Abstract
Quantization and pruning reduce the cost of deploying language models, but their effects are usually reported as benchmark scores. Such scores record only the loss that remains at the output. Inside the model, compression damages some components more than others, and the rest of the network can partly compensate. The observed loss therefore reflects both the damage and this compensation, which benchmark scores cannot separate. How the two combine, and whether their balance changes with model size, remains unclear. We study this compensation for in-context retrieval, a capability carried by a small, identifiable set of attention heads. To separate the two, the expected loss from ablating these heads is decomposed exactly into the part restored by the rest of the network, attributed to the components responsible, and the part that reaches the output. This decomposition is then compared across model sizes. For damage of comparable size, smaller models restore most of the loss, whereas larger models retain most of it, because the compensating responses offset less of it as models grow. Compressing the heads produces the same pattern. Compression that is nearly harmless in small models can therefore be costly in large ones.
Figures & tables
| Output harm | ||||
|---|---|---|---|---|
| Model | Deficit | Natural | Backup held clean | Both held clean |
| 410M | 0.797 | 0.078 | 0.728 | |
| 1B | 0.795 | 0.815 | ||
| 1.4B | 0.756 | 0.354 | 0.464 | |
| 2.8B | 0.888 | 0.476 | 1.316 | 1.770 |
| 6.9B | 0.829 | 0.561 | 0.675 | 0.905 |
Appendix figures & tables17 assets
Supplementary material from the paper’s appendix.
Appendix
| Model | Acc. | Logit diff | ||
|---|---|---|---|---|
| 410M | 0.99 | 4.31 | 98 | 2 |
| 1B | 1.00 | 5.05 | 27 | 2 |
| 1.4B | 1.00 | 4.21 | 105 | 2 |
| 2.8B | 1.00 | 4.30 | 283 | 2 |
| 6.9B | 1.00 | 3.95 | 266 | 2 |
| 12B | 1.00 | 4.32 | 389 | 2 |
LLM Compression
Large language model compression reduces the memory, compute, and context costs of LLMs while aiming to preserve their capabilities.
See all 301 papers