The Behavioral Rate of Fine-Tuning: Compression as a Measure of Useful Information
Organizations: Miles Team, LAMSADE, Université Paris Dauphine-PSL, Paris, France · ESPCI PSL, Paris, France
Abstract
Low-Rank Adaptation (LoRA) adapts a frozen language model through a compact learned update, making it a natural setting for asking how much information fine-tuning needs to write. Yet adapter rank, parameter count, or numerical reconstruction error do not directly tell us how many bits are required to preserve the acquired behavior. We therefore define an operational behavioral rate as the smallest serialized size of a compressed adapter that, once reloaded into the frozen base model, retains a fixed fraction of its held-out fine-tuning gain. Across LoRA fine-tuning experiments, this rate increases with distinct training content at fixed optimization budgets, but is not determined by intuitive metrics like source description length, fine-tuning gain, or weight-space reconstruction error. Most importantly, the same datasets exhibit substantially different, and even reordered, rates across frozen base models, showing that fine-tuning compressibility is a property of the dataset--receiver pair rather than of the dataset alone. However, we show that correction spectra measured on the data are able to rank these rates within a receiver. Finally, we study a striking noise-reducing effect of compression: adapters trained on deliberately corrupted datasets fall far below the frozen model, yet after compression score above it (23.7% to 82.9% on GSM8K, against 77.6% for frozen Qwen2.5-7B), consistently across tested families and sizes. The damage is unevenly concentrated in the adapter's leading singular directions, which compression appears to filter. Away from its training task, the compressed corrupted adapter also generalizes better than a clean fine-tune: on harder math it retains 97% of base accuracy, versus 44% for the clean adapter.
Figures & tables
Appendix figures & tables14 assets
Supplementary material from the paper’s appendix.
Appendix
| corruption | configs | |
|---|---|---|
| rationale permuted | each worked explanation/CoT moved to a different problem; question and answer line kept | 19 |
| response mismatched | the whole response dealt to a different item, for tasks without an answer line | 10 |
| rationale steps shuffled | steps of the explanation reordered; steps, arithmetic and answer kept | 2 |
| arithmetic corrupted | half the computed results moved by up to a third, and the answer follows the wrong chain | 2 |
| labels scrambled / flipped | each multiple-choice letter, or short text label, moved to another of its choices | 2 + 2 |
| codec | bits / value | gain kept (%) | relative RMSE |
|---|---|---|---|
| midtread, 2 levels (exact zero) | 0.63 | 61.1 | 0.881 |
| adaptive, may drop rows, target 1 | 0.98 | 2.6 | 0.537 |
| uniform zero-free, 1 bit | 1.02 | 85.7 | 0.563 |
| adaptive, keeps rows, target 1.1 | 1.03 | 85.9 | 0.561 |
| LoRAQuant [email protected] | 1.59 | 95.4 | – |
| uniform zero-free, 2 bits | 1.97 | 98.6 | 0.314 |
| receiver | measure | locked | corpora | RMSE | base RMSE | |
|---|---|---|---|---|---|---|
| Llama-3.1-8B | Fisher log-volume | yes | 5 | 0.357 | 0.327 | |
| Qwen3-8B | Fisher effective rank | yes | 4 | 0.408 | 0.306 | |
| Qwen2.5-7B, new panel | coherent energy / entropy | yes | 5 | 0.10 | 0.296 | 0.241 |
| Llama-3.1-8B | coherent energy / entropy | no | 5 | 0.70 | 0.132 | 0.327 |
| Qwen3-8B | coherent energy / entropy | no | 4 | 0.80 | 0.163 | 0.306 |
| Qwen3-8B | coherent energy | no | 4 | 1.00 | 0.206 | 0.306 |
| probe | tier | base | clean | compr. | clean/base | compr./base |
|---|---|---|---|---|---|---|
| gsm8k | 0 | 0.756 | 0.810 | 0.814 | 1.13 | 1.13 |
| asdiv | 1 | 0.885 | 0.925 | 0.902 | 1.07 | 1.03 |
| gsm symbolic | 1 | 0.762 | 0.830 | 0.790 | 1.10 | 1.09 |
| svamp | 1 | 0.790 | 0.847 | 0.890 | 1.08 | 1.15 |
| aqua rat † | 2 | 0.402 | 0.059 | 0.394 | 0.02 | 1.00 |
| math500 † | 2 | 0.585 | 0.495 | 0.590 | 0.89 | 1.07 |
| candidate | result |
|---|---|
| distinct rows | ranks four conditions correctly within a corpus, but reverses across corpora |
| behavioral change (bits saved) | slope 1.90 within a 0.16-wide window; 0.12 on answer rewrites and approximately flat under optimizer-budget changes |
| optimizer depth | increasing the number of updates by on the same rows changes bits saved by 0.04 while decreases |
| base surprisal on the corpus | with |
| weight reconstruction error | approximately constant to within 1% across all five corpora at a fixed rate |
| local curvature ( , ) | the second-order term adds no ranking power, and the local expansion breaks down below one bit per value |
| condition | GSM8K | GSM-Symbolic |
|---|---|---|
| rank 1 @ 1 bit | 83.24 | 77.60 |
| rank 1 @ 16 bits | 42.81 | 33.80 |
| full rank @ 1 bit | 65.05 | 55.00 |
| full rank @ 16 bits (damaged) | 23.81 | 15.53 |
| random rank-1 projection @ 16 bits | 80.09 | 72.13 |
| full update scaled to the rank-1 norm | 23.93 | 13.80 |
| aligned | permuted | |||||||
|---|---|---|---|---|---|---|---|---|
| model | raw | 2 bits | 1.5 bits | 1 bit | raw | 2 bits | 1.5 bits | 1 bit |
| Mistral-7B | 68.9 | 67.3 | 64.2 | 59.3 | 15.7 | 15.3 | 14.3 | 24.2 |
| Qwen2.5-7B | 83.2 | 83.4 | 82.9 | 82.4 | 23.7 | 23.2 | 27.4 | 65.8 |
| Llama-3.1-8B | 73.2 | 72.2 | 70.0 | 65.7 | 19.8 | 19.4 | 17.1 | 34.7 |
| size | base | damaged | clean | rank 1 @ 1 bit | overshoot |
|---|---|---|---|---|---|
| 0.5B | 23.0 | 9.4 | 45.2 | 25.0 | +2.0 |
| 1.5B | 51.6 | 13.6 | 67.6 | 59.0 | +7.4 |
| 3B | 60.4 | 17.4 | 74.0 | 71.0 | +10.6 |
| 7B | 75.6 | 22.8 | 81.0 | 80.8 | +5.2 |
| 14B, XBRL tagging | 24.6 | 53.0 | – | 73.8 | +49.2 |
| 32B, XBRL tagging | 32.4 | 51.8 | – | 74.4 | +42.0 |
Fine-Tuning
Fine-tuning adapts pretrained models to new tasks, domains, and behaviors by updating some or all of their parameters.
See all 258 papers