BQ-LoRA: Binary-Quantized Low-Rank Adapters as Implicit Regularizers for Parameter-Efficient Fine-Tuning
Organizations: Independent Researcher · KAIST, DeepAuto.ai
Abstract
Parameter-efficient fine-tuning of quantized large language models is dominated by QLoRA, which combines 4-bit weight quantization with FP16 low-rank adapters. We ask whether the FP16 adapter precision is necessary. Replacing it with a single binary bit per adapter parameter, combined with a learned per-layer scale and a Scaled Straight-Through Estimator, reduces adapter storage by 13-16x. We introduce two variants -- NormScale, which binarizes under a norm-preserving scale with a clipped STE, and LearnThresh, which makes the binarization threshold learnable -- unified under a framework we call BQ-LoRA. Our main theoretical contribution is a tight characterization of the BQ-LoRA hypothesis class: it has bounded entrywise norm, a typical-set spectral norm bound that is times sharper than the worst case, a full-rank, near-isotropic update, and consequently a Rademacher complexity that yields an explicit generalization bound. FP16 LoRA has no comparable a priori norm or isotropy control. On Qwen2.5 (1.5B-72B), BQ-LoRA exceeds QLoRA's benchmark averages at every scale by 0.8-3.5 points; on LLaMA-2 (7B/70B) it remains competitive at the same storage reduction but does not uniformly lead the strongest FP16-adapter baselines (LoftQ at 7B, QLoRA at 70B). On Qwen2.5-7B the binary adapter preserves the 4-bit base WikiText-2 PPL within 0.06 (9.86 vs. 9.80) while QLoRA worsens it by 0.59. The reduction applies to inference-time adapter storage and serving; training memory matches QLoRA and step time rises by under 4%. We translate the storage saving into a deployment study, BQ-SGMV, by extending Punica's SGMV kernel with a binary inner-product primitive: the same A100-80GB holds 15.6x more adapters in HBM, and our research prototype eliminates cache misses under a skewed synthetic workload.
Figures & tables
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
| Checkpoint (# matrices) | mean ratio | worst ratio | within of random | frac |
|---|---|---|---|---|
| Qwen2.5-7B, NormScale (224) | 1.000 | 1.034 | 94% | 0.500 |
| Qwen2.5-7B, LearnThresh (224) | 1.000 | 1.035 | 95% | 0.501 |
| LLaMA-2-7B, NormScale (256) | 1.000 | 1.014 | 95% | 0.500 |
| LLaMA-2-7B, LearnThresh (256) | 1.001 | 1.048 | 89% | 0.501 |
| Family | Method | PR (of 16) | Top-1 share | mean | |
|---|---|---|---|---|---|
| Qwen2.5-7B | QLoRA | 8.07 | 0.526 | 0.077 | – |
| NormScale | 15.92 | 0.079 | 0.162 | 0.0052 | |
| LLaMA-2-7B | QLoRA | 8.36 | 0.505 | 0.064 | – |
| NormScale | 15.97 | 0.072 | 0.215 | 0.0086 |
Low-Rank Adaptation
Low-rank adaptation (LoRA) fine-tunes pretrained models with compact weight updates, reducing trainable parameters and deployment costs.
See all 585 papers