cs.LGNov 4, 2026

The Behavioral Rate of Fine-Tuning: Compression as a Measure of Useful Information

Authors: Joey David, Paul Caillon, Alexandre Allauzen

Organizations: Miles Team, LAMSADE, Université Paris Dauphine-PSL, Paris, France · ESPCI PSL, Paris, France

Abstract

Low-Rank Adaptation (LoRA) adapts a frozen language model through a compact learned update, making it a natural setting for asking how much information fine-tuning needs to write. Yet adapter rank, parameter count, or numerical reconstruction error do not directly tell us how many bits are required to preserve the acquired behavior. We therefore define an operational behavioral rate R⋆R^\star as the smallest serialized size of a compressed adapter that, once reloaded into the frozen base model, retains a fixed fraction of its held-out fine-tuning gain. Across LoRA fine-tuning experiments, this rate increases with distinct training content at fixed optimization budgets, but is not determined by intuitive metrics like source description length, fine-tuning gain, or weight-space reconstruction error. Most importantly, the same datasets exhibit substantially different, and even reordered, rates across frozen base models, showing that fine-tuning compressibility is a property of the dataset--receiver pair rather than of the dataset alone. However, we show that correction spectra measured on the data are able to rank these rates within a receiver. Finally, we study a striking noise-reducing effect of compression: adapters trained on deliberately corrupted datasets fall far below the frozen model, yet after compression score above it (23.7% to 82.9% on GSM8K, against 77.6% for frozen Qwen2.5-7B), consistently across tested families and sizes. The damage is unevenly concentrated in the adapter's leading singular directions, which compression appears to filter. Away from its training task, the compressed corrupted adapter also generalizes better than a clean fine-tune: on harder math it retains 97% of base accuracy, versus 44% for the clean adapter.

Figures & tables

Appendix figures & tables14 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. How Much Can a LoRA Adapter Memorize? Measuring Adapter Capacity in Bits

    Jul 23, 2026Kaizhen Tan, Heqing Du, Yang FengMemorization in Language ModelsPrivacy Leakage in Language Models

  2. LoRA on a Budget: Function-Aware Post-Hoc Compression for Adapter Fleets

    Oct 26, 2026Shenghui Li, Thiemo voigtLow-Rank CompressionAdaptive Rank Allocation

  3. Beyond LoRA: Is Sparsity-Induced Adaptation Better?

    Jun 11, 2026Elijah Cadenhead, Cristian McGee, Xin Li +2Low-Rank AdaptationParameter-Efficient Fine-Tuning