SteerDistill: Quantizing LLM Agents via Closed-Loop Teacher Steering
Organizations: Institute for AI Industry Research (AIR), Tsinghua University · University of Science and Technology of China
Abstract
Low-bit quantization reduces the inference cost of LLM agents but can compromise task success. Examining failed rollouts, we find that quantized agents overlook task requirements or environmental conditions, producing environment transitions that diverge from successful full-precision references. Correcting these behavioral divergences remains difficult: offline data may miss the student's failure contexts, while its own rollouts may provide few correct responses in those contexts. We introduce SteerDistill, a quantization-aware distillation method in which a full-precision teacher guides the student to revise failed responses and iteratively refines its guidance using feedback from student execution. This closed-loop teacher steering produces locally verified behavioral corrections. We internalize these corrections by training the student without guidance to match the steered student's predictions at corrected turns and the full-precision teacher's at other turns, enabling unassisted inference. Across five agentic benchmarks and three models, SteerDistill exceeds the strongest quantized baseline by 1.67--2.18 points in average score under W4A16 and stays within one point of BF16. On -Bench, it outperforms QeRL by 1.84 percentage points in success rate using 67% fewer training GPU hours and produces shorter successful trajectories than the quantized baselines.
Figures & tables
| Model | Precision | Method | BFCL V4 Overall | -Bench Pass@1 | WebShop SR | ALFWorld SR | Search QA EM | Avg. |
| Qwen3 8B | BF16 | – | 46.89 | 32.64 | 68.58 | 79.92 | 39.65 | 53.54 |
| W4A16 | GPTQ | 40.73 | 26.85 | 59.24 | 71.32 | 33.64 | 46.36 | |
| EfficientQAT | 41.77 | 28.00 | 61.34 | 73.18 | 35.08 | 47.87 | ||
| BitDistiller | 42.46 | 28.31 | 62.07 | 73.91 | 35.69 | 48.49 | ||
| On-Policy QAD | 44.43 | 28.77 | 64.72 | 76.53 | 37.47 | 50.38 | ||
| QeRL | 44.91 | 29.84 | 65.29 | 77.04 | 37.94 | 51.00 |
| Configuration | Overall | Non-Live | Live | Multi- Turn | Web Search | Memory | Irrel. |
| BF16 | 46.89 | 87.14 | 81.76 | 48.53 | 18.57 | 19.32 | 78.63 |
| On-Policy QAD | 44.43 | 85.72 | 80.19 | 45.14 | 16.86 | 17.16 | 74.88 |
| Full SteerDistill (Ours) | 46.84 | 88.29 | 81.71 | 48.47 | 18.44 | 18.79 | 78.56 |
| Q1: Intervention localization (default: Behavioral divergence) | |||||||
| Random eligible turn | 43.52 | 85.20 | 79.72 | 44.00 | 16.10 | 16.30 | 73.48 |
| First action mismatch | 45.01 | 86.20 | 80.55 | 45.95 | 17.20 | 17.45 | 76.20 |
Appendix figures & tables16 assets
Supplementary material from the paper’s appendix.
Appendix
| Symbol | Meaning |
| Policies and conditioning | |
| Frozen full-precision teacher. | |
| , | Student master weights and trainable quantized policy. |
| , , | Training cycle, frozen student weights, and corresponding policy. |
| , , | Trajectory, turn, and response-token indices. |
| , | Student trajectory and its fixed successful teacher reference. |
| Method | Epochs | Learning Rate | Batch Size (trajectories) |
| EfficientQAT | 16 | ||
| BitDistiller | 2 | 16 | |
| On-Policy QAD | 2 | 16 | |
| QeRL | 2 | 16 | |
| Ours | 2 | 16 |
| Model | Precision | Method | BFCL V4 Overall | -Bench Pass@1 |
| Qwen3 8B | BF16 | – | 40.40 | 30.00 |
| W4A16 | BitDistiller | 35.34 | 24.29 | |
| On-Policy QAD | 36.16 | 24.58 | ||
| QeRL | 38.85 | 28.92 | ||
| Ours | 39.67 | 29.05 |
| Method | -Bench | WebShop | Search QA (EM) | ||||||
| Pass@4 | Score | NQ † | TriviaQA ⋆ | PopQA ⋆ | HotpotQA † | 2Wiki ⋆ | MuSiQue ⋆ | Bamboogle ⋆ | |
| BF16 | 54.20 | 78.69 | 43.36 | 62.42 | 41.96 | 36.58 | 35.12 | 15.92 | 42.18 |
| GPTQ | 45.50 | 70.86 | 37.62 | 56.04 | 36.05 | 30.44 | 28.39 | 12.23 | 34.71 |
| EfficientQAT | 47.10 | 72.73 | 39.08 | 57.69 | 37.48 | 32.02 | 30.17 | 12.88 | 36.25 |
| BitDistiller | 47.55 | 73.41 | 39.71 | 58.31 | 38.11 | 32.71 | 30.86 | 13.21 | 36.92 |
| On-Policy QAD | 48.20 | 75.61 | 41.18 | 60.22 | 39.69 | 34.47 | 32.86 | 14.46 | 39.44 |
| Method | Input Tokens | Cached Input | Output Tokens | Interaction Steps | Tool Calls |
| BF16 | 185K | 148K | 3.9K | 11.4 | 4.2 |
| GPTQ | 270K | 223K | 5.2K | 13.8 | 5.25 |
| EfficientQAT | 243K | 199K | 4.7K | 13.1 | 4.92 |
| BitDistiller | 235K | 192K | 4.5K | 12.9 | 4.78 |
| On-Policy QAD | 214K | 173K | 4.2K | 12.3 | 4.55 |
| QeRL | 206K | 166K | 4.1K | 12.1 | 4.43 |
LLM Quantization
Large language model quantization reduces numerical precision to lower memory use and accelerate training or inference.
See all 443 papers