cs.AINov 5, 2026

When Bigger Becomes Brittle: Why Compression Hurts More as Language Models Scale

Authors: Gautam Gupta, Dikshant Kukreja, Unique Kumar, Avni Mittal, Deepanshu Dabas, Kshitij Sah, Erik Cambria, Rajiv Ratn Shah, 

Organizations: Indraprastha Institute of Information Technology, Delhi · TurboML · Independent Researcher · Microsoft · 

Abstract

Quantization and pruning reduce the cost of deploying language models, but their effects are usually reported as benchmark scores. Such scores record only the loss that remains at the output. Inside the model, compression damages some components more than others, and the rest of the network can partly compensate. The observed loss therefore reflects both the damage and this compensation, which benchmark scores cannot separate. How the two combine, and whether their balance changes with model size, remains unclear. We study this compensation for in-context retrieval, a capability carried by a small, identifiable set of attention heads. To separate the two, the expected loss from ablating these heads is decomposed exactly into the part restored by the rest of the network, attributed to the components responsible, and the part that reaches the output. This decomposition is then compared across model sizes. For damage of comparable size, smaller models restore most of the loss, whereas larger models retain most of it, because the compensating responses offset less of it as models grow. Compressing the heads produces the same pattern. Compression that is nearly harmless in small models can therefore be costly in large ones.

Figures & tables

Appendix figures & tables17 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. Compression Technique Interactions in Small Language Models: A Factorial Ablation Study

    Oct 30, 2026Bhushit Nema, Siddharth Krishnamurthy, Anjali KadiyalaSmall Language ModelsLLM Quantization

  2. ACE-Compression: An Ability-Centric Evaluation of Compressed Large Language Models

    Oct 20, 2026Jiahui Ning, Liang Chen, Yuchen Liang +2LLM CompressionLLM Evaluation

  3. Language model capabilities fail in a shared, predictable order under compression and perturbation

    Nov 8, 2026Zhixiang Wang, Ulas BagciLanguage Model RobustnessLanguage Model Performance Prediction