cs.LGNov 17, 2026

BQ-LoRA: Binary-Quantized Low-Rank Adapters as Implicit Regularizers for Parameter-Efficient Fine-Tuning

Authors: Juyoung Park, Heejun Lee

Organizations: Independent Researcher · KAIST, DeepAuto.ai

Abstract

Parameter-efficient fine-tuning of quantized large language models is dominated by QLoRA, which combines 4-bit weight quantization with FP16 low-rank adapters. We ask whether the FP16 adapter precision is necessary. Replacing it with a single binary bit per adapter parameter, combined with a learned per-layer scale ss and a Scaled Straight-Through Estimator, reduces adapter storage by 13-16x. We introduce two variants -- NormScale, which binarizes under a norm-preserving scale with a clipped STE, and LearnThresh, which makes the binarization threshold learnable -- unified under a framework we call BQ-LoRA. Our main theoretical contribution is a tight characterization of the BQ-LoRA hypothesis class: it has bounded entrywise norm, a typical-set spectral norm bound that is Θ(r)Θ(r) times sharper than the worst case, a full-rank, near-isotropic update, and consequently a Rademacher complexity that yields an explicit O~(smax⁡αdout/(rn))\tilde{O}(s_{\max}α\sqrt{d_{out}/(rn)}) generalization bound. FP16 LoRA has no comparable a priori norm or isotropy control. On Qwen2.5 (1.5B-72B), BQ-LoRA exceeds QLoRA's benchmark averages at every scale by 0.8-3.5 points; on LLaMA-2 (7B/70B) it remains competitive at the same storage reduction but does not uniformly lead the strongest FP16-adapter baselines (LoftQ at 7B, QLoRA at 70B). On Qwen2.5-7B the binary adapter preserves the 4-bit base WikiText-2 PPL within 0.06 (9.86 vs. 9.80) while QLoRA worsens it by 0.59. The reduction applies to inference-time adapter storage and serving; training memory matches QLoRA and step time rises by under 4%. We translate the storage saving into a deployment study, BQ-SGMV, by extending Punica's SGMV kernel with a binary inner-product primitive: the same A100-80GB holds 15.6x more adapters in HBM, and our research prototype eliminates cache misses under a skewed synthetic workload.

Figures & tables

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. Signs Beat Floats: Low-Rank Double-Binary Adaptation for On-Device Fine-Tuning

    May 22, 2026Yoshihiko Fujisawa, Yuma Ichikawa, Yudai Fujimoto +2LLM QuantizationMemory-Efficient Fine-Tuning

  2. ProjQ: Project-and-Quantize for Adapter-Aware LLM Compression

    May 30, 2026Wenya Yu, Chao Zhang, Li Wang +2LLM QuantizationPost-Training Quantization