BiLoCo: Binary Low-Rank Corrections for Post-Training Quantization

David Jin; Beshr IslamBouli; Tarushii Goel; Han Guo; Yoon Kim

cs.LGOct 23, 2026

BiLoCo: Binary Low-Rank Corrections for Post-Training Quantization

Authors: David Jin, Beshr IslamBouli, Tarushii Goel, Han Guo, Yoon Kim

Organizations: Massachusetts Institute of Technology

Abstract

Post-training quantization (PTQ) to FP4 has emerged as a key technique for reducing large language model inference cost, especially on Blackwell GPUs, which offer native FP4 tensor core support. Direct PTQ to FP4, however, still leaves a noticeable accuracy gap. Recent work on SVDQuant mitigates the errors introduced through FP4 quantization for diffusion models by adding high-precision low-rank corrections obtained via SVD. However, this high-precision low-rank correction requires many bits per rank, and it runs on the same tensor cores as the main FP4 GEMM. We introduce BiLoCo, a low-rank correction method whose left and right factors are stored as binary ${\{\pm 1\}}$ sign vectors rather than as BF16 numbers. The signed format uses $16{\times}$ less storage per rank-one component, and since the correction now consists of additions/subtractions, this can be run on CUDA cores concurrently with the FP4 tensor-core GEMM. On 4B--72B LLMs, BiLoCo matches or improves memory-matched SVDQuant for PTQ. We implement BiLoCo on B300 as a fused single kernel, where the FP4 GEMM runs on tensor cores, the bit-packed correction on the CUDA cores of the same launch, and the two are combined in the GEMM epilogue. With batch size $1$ , the fused kernel is at most $11\%$ slower than the FP4-only GEMM for every rank up to $512$ . At rank $128$ , it is at most $20\%$ slower up to batch size $8$ .

Figures & tables

Figure 1: BiLoCo inference for performing

y=xR+xVU^{\top}.

A native-FP4 residual GEMM on tensor cores runs alongside a signed add/subtract correction on CUDA cores.

Figure 2: Fraction

\gamma=\|\widehat{W}\|_{F}/\|W\|_{F}

of the Frobenius norm in the low-rank part versus the storage budget

b

, on

63

Qwen3-4B projection matrices (

7

projection types in

9

decoder layers). At each budget, BiLoCo uses binary rank

r=b

and SVDQuant uses BF16 rank

k=b/16

, so both use the same storage. Lines are medians and bands are interquartile ranges.

model	$\sigma_{1}/\sigma_{32}$ med.	$\gamma$ , SVD $k{=}32$	$\gamma$ , sign $r{=}512$	ratio	Pearson
Qwen3-4B	$1.93$	$29.3\%$	$52.9\%$	$1.92\times$	$-0.65$
Qwen3-14B	$2.00$	$26.3\%$	$47.3\%$	$1.77\times$	$-0.65$
Llama-3-8B	$2.24$	$27.9\%$	$50.6\%$	$1.76\times$	$-0.81$

Table 1: Fraction

\gamma

at equal storage across three model families. Signed rank

512

and BF16 SVD rank

32

use the same vector-storage budget.

Model	Dataset	BF16	NVFP4	SVDQuant $k{=}32$	BiLoCo $r{=}128$	BiLoCo $r{=}512$
Qwen3-4B	Wiki2	$14.22$	$15.13$	$15.07$	$15.09{\scriptstyle\,\pm 0.58}$	$\mathbf{14.94}{\scriptstyle\,\pm 0.37}$
Qwen3-4B	C4	$15.25$	$15.88$	$15.85$	$15.88{\scriptstyle\,\pm 0.23}$	$\mathbf{15.81}{\scriptstyle\,\pm 0.17}$
Llama-3-8B	Wiki2	$6.27$	$7.00$	$6.92$	$6.93{\scriptstyle\,\pm 0.01}$	$\mathbf{6.84}{\scriptstyle\,\pm 0.01}$
Llama-3-8B	C4	$6.69$	$7.68$	$7.59$	$7.61{\scriptstyle\,\pm 0.01}$	$\mathbf{7.48}{\scriptstyle\,\pm 0.01}$
Qwen3-14B	Wiki2	$8.89$	$9.36$	$9.19$	$9.21{\scriptstyle\,\pm 0.03}$	$\mathbf{9.17}{\scriptstyle\,\pm 0.04}$
Qwen3-14B	C4	$11.19$	$11.55$	$11.46$	$11.49{\scriptstyle\,\pm 0.03}$	$\mathbf{11.44}{\scriptstyle\,\pm 0.02}$

Table 2: Perplexity of BiLoCo vs. baseline methods. BiLoCo with

b{=}512

is memory-matched to SVDQuant writh

k=32

. BiLoCo entries report mean

\,\pm\,

std over sign initializations. Bold marks the best corrected FP4 method, with ties bolded.

	KL mean ( $\downarrow$ )		KL p99 ( $\downarrow$ )
Model	BiLoCo $r{=}512$	SVDQuant $k{=}32$	BiLoCo $r{=}512$	SVDQuant $k{=}32$
Qwen3-4B	$\mathbf{0.0740}$	$0.0835$	$\mathbf{0.1421}$	$0.1649$
Qwen3-14B	$\mathbf{0.0480}$	$0.0517$	$\mathbf{0.1303}$	$0.1671$
Llama-3-8B	$\mathbf{0.0756}$	$0.0824$	$\mathbf{0.3252}$	$0.3409$
Qwen3-32B	$\mathbf{0.0589}$	$0.0605$	$\mathbf{0.3175}$	$0.3363$
Qwen2.5-72B	$0.0515$	$\mathbf{0.0497}$	$\mathbf{0.1937}$	$0.2097$

Table 3: KL divergence to the BF16 model at

b{=}512

on FineWeb document. Lower is better.

Method	Qwen3-4B	Qwen3-14B	Llama-3-8B	Qwen3-32B	Qwen2.5-72B
BF16 (reference)	$0.686$	$0.772$	$0.803$	$0.803$	$0.857$
SVDQuant $k{=}32$ ( $b{=}512$ )	$0.669$	$0.759$	$\mathbf{0.780}$	$0.796$	$0.849$
BiLoCo $r{=}128$ ( $b{=}128$ )	$0.667$	$\mathbf{0.763}$	$0.777$	$0.795$	$0.846$
BiLoCo $r{=}512$ ( $b{=}512$ )	$\mathbf{0.670}$	$\mathbf{0.763}$	$0.777$	$\mathbf{0.799}$	$\mathbf{0.856}$

Table 4: Mean accuracy on WinoGrande (5-shot) and HellaSwag (10-shot). Higher is better.

	Rank $r$
Batch $M$	$16$	$32$	$64$	$128$	$256$	$512$
$1$	$6.17$ $1.06\times$	$6.18$ $1.06\times$	$6.23$ $1.06\times$	$6.33$ $1.08\times$	$6.39$ $1.09\times$	$6.49$ $1.11\times$
$2$	$6.36$ $1.08\times$	$6.36$ $1.09\times$	$6.36$ $1.09\times$	$6.32$ $1.08\times$	$6.36$ $1.09\times$	$6.71$ $1.15\times$
$4$	$6.36$ $1.09\times$	$6.36$ $1.09\times$	$6.46$ $1.10\times$	$6.47$ $1.11\times$	$6.90$ $1.18\times$	$8.45$ $1.44\times$
$8$	$6.42$ $1.10\times$	$6.42$ $1.10\times$	$6.82$ $1.17\times$	$6.74$ $1.15\times$	$8.02$ $1.37\times$	$10.10$ $1.74\times$
$16$	$6.55$ $1.11\times$	$6.52$ $1.11\times$	$7.48$ $1.28\times$	$9.08$ $1.55\times$	$14.70$ $2.51\times$	$17.81$ $3.04\times$

Table 5: Decode latency of the BiLoCo kernel on B300 (

K{=}N{=}5120

). Each entry gives the time per launch in

\mu

s and, below it, the ratio to the padded FP4-only kernel measured in the same run (

5.8

–

5.9\,\mu

s). Gray entries are more than

20\%

slower than the FP4-only kernel.

Shape ( $K{\times}N$ )	Role	padded	unpadded	$r{=}128$	$r{=}256$	$r{=}512$
$5120{\times}5120$	attn q/o	$5.85$	$7.54$	$6.33$ $1.08\times$	$6.39$ $1.09\times$	$6.49$ $1.11\times$
$5120{\times}1024$	attn k/v	$5.72$	$5.62$	$6.10$ $1.09\times$	$6.55$ $1.17\times$	$7.00$ $1.25\times$
$17408{\times}5120$	MLP down	$12.42$	$22.65$	$12.93$ $1.04\times$	$12.97$ $1.04\times$	$13.18$ $1.06\times$

Table 6: BiLoCo kernel across Qwen3-14B projection shapes (

M{=}1

, B300). Each entry gives the time per launch in

\mu

s and, below it, the ratio to the fastest FP4-only GEMM of the same shape measured in the same run.

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Model	Projection	SVDQuant $k{=}32$	BiLoCo $r{=}128$	BiLoCo $r{=}256$
Qwen3-4B	q_proj	$28.1\%$	$35.4\%$	$\mathbf{44.6\%}$
	k_proj	$36.9\%$	$43.5\%$	$\mathbf{55.8\%}$
	v_proj	$24.7\%$	$38.3\%$	$\mathbf{51.8\%}$
	o_proj	$21.5\%$	$30.8\%$	$\mathbf{41.7\%}$
	gate_proj	$18.5\%$	$26.5\%$	$\mathbf{34.9\%}$
	up_proj	$15.0\%$	$23.4\%$	$\mathbf{32.3\%}$

Appendix

Table 7: Frobenius-norm capture

\mathrm{capA}(r)=\|\widehat{W}_{r}\|_{F}/\|W\|_{F}

per projection type, as the median over

9

evenly spaced decoder layers per model. SVDQuant uses rank

k{=}32

(

b{=}512

), and BiLoCo uses

r{=}128

(

b{=}128

, a quarter of that storage) and

r{=}256

(

b{=}256

, half). Bold marks BiLoCo

r{=}256

, which is the highest in every row. Higher is better.

Figure 3: Frobenius-relative capture rate over

128

peel iterations on Qwen3-4B projection matrices.

construction	WikiText-2	C4
random signs $+$ joint coefficient refit	$7.000{\scriptstyle\,\pm 0.001}$	$7.684{\scriptstyle\,\pm 0.008}$
greedy learned signs	$6.842{\scriptstyle\,\pm 0.007}$	$7.475{\scriptstyle\,\pm 0.018}$
greedy learned signs $+$ global refit (Algorithm 1 )	$6.843{\scriptstyle\,\pm 0.017}$	$7.480{\scriptstyle\,\pm 0.011}$
$+$ pm-altmin, $10$ sweeps	$6.838{\scriptstyle\,\pm 0.013}$	$7.459{\scriptstyle\,\pm 0.014}$

Appendix

Table 8: Solver ablation on Llama-3-8B at

r{=}512

: perplexity (mean

\,\pm\,

std over three sign seeds) for four ways of constructing the low-rank component. Without a low-rank component (NVFP4 only), perplexity is

6.998

on WikiText-2 and

7.680

on C4.

method	solver	other	layer total	PTQ total
SVDQuant, BF16 rank $32$ (SVD)	$170.4$	$2.5$	$174.3$	$222$
greedy BiLoCo, $r{=}512$	$86.3$	$0.7$	$90.3$	$116$
greedy $+$ global refit, $r{=}512$	$91.3$	$0.2$	$92.5$	$118$
batched pm-altmin, $r{=}512$	$213.9$	$0.7$	$215.9$	$263$
full pm-altmin, $r{=}512$	$1070.2$	$0.2$	$1071.3$	$1097$

Appendix

Table 9: PTQ time on Llama-3-8B in seconds on one node. “Other” is the NVFP4 quantization of the residual and bookkeeping. The PTQ total also includes collecting the calibration activations.

	WikiText-2	C4
BF16	$6.266$	$6.690$
NVFP4 only	$6.998$	$7.680$
SVDQuant, BF16 rank $32$	$6.919$	$7.592$
BiLoCo $r{=}32$	$6.983$	$7.656$
BiLoCo $r{=}128$	$6.926$	$7.605$
BiLoCo $r{=}256$	$6.892$	$7.549$

Appendix

Table 10: BiLoCo perplexity versus rank on Llama-3-8B (C4-train calibration, pm-altmin)

backbone / correction	WikiText-2	C4
no preprocessing, NVFP4 only	$7.056$	$7.751$
presmoothing, NVFP4 only	$6.998$	$7.680$
QuaRot, NVFP4 only	$7.206$	$7.981$
QuaRot $+$ SVDQuant	$7.130$	$7.904$
QuaRot $+$ BiLoCo	$\mathbf{6.999}$	$\mathbf{7.731}$
SpinQuant, NVFP4 only	$7.240$	$8.035$

Appendix

Table 11: Perplexity of Llama-3-8B with full-width rotated backbones (seed

42

, lower is better). SVDQuant uses BF16 rank

32

, and BiLoCo uses

r{=}512

with the pm-altmin solver (Appendix A.2 ).

Model	Method	$b$	ARC-e	ARC-c	PIQA	WinoG.	OBQA	mean
Qwen3-4B	SVDQuant $k{=}32$	512	0.775	0.467	0.738	0.639	0.296	0.583
	BiLoCo $r{=}128$	128	0.789	0.485	0.727	0.618	0.298	0.583
	BiLoCo $r{=}512$	512	0.782	0.486	0.724	0.652	0.286	0.586
Qwen3-14B	SVDQuant $k{=}32$	512	0.830	0.557	0.780	0.717	0.344	0.646
	BiLoCo $r{=}128$	128	0.830	0.560	0.795	0.688	0.344	0.643
	BiLoCo $r{=}512$	512	0.828	0.583	0.795	0.715	0.348	0.654

Appendix

Table 12: Standard PTQ downstream task breakdown. Higher is better.

			KL mean ( $\downarrow$ )		KL p99 ( $\downarrow$ )
Model	Method	$b$	BiLoCo	SVDQ	BiLoCo	SVDQ
Qwen3-4B	$r{=}128$ vs. $k{=}8$	128	0.0914	0.0901	0.1849	0.1965
Qwen3-4B	$r{=}512$ vs. $k{=}32$	512	0.0740	0.0835	0.1421	0.1649
Qwen3-14B	$r{=}128$ vs. $k{=}8$	128	0.0548	0.0541	0.2026	0.1618
Qwen3-14B	$r{=}512$ vs. $k{=}32$	512	0.0480	0.0517	0.1303	0.1671
Llama-3-8B	$r{=}128$ vs. $k{=}8$	128	0.0856	0.0881	0.3611	0.4789

Appendix

Table 13: KL divergence to the BF16 model at

b{=}128

and

b{=}512

on held-out FineWeb samples. Lower is better.

LLM Quantization

Large language model quantization reduces numerical precision to lower memory use and accelerate training or inference.

See all 380 papers

Mar 26, 2026

GlowQ: Group-Shared LOw-Rank Approximation for Quantized LLMs

Quantization techniques such as BitsAndBytes, AWQ, and GPTQ are widely used as a standard method in deploying large language models but often degrades accuracy when using low-bit representations, e.g., 4 bits. Low-rank correction methods (e.g., LQER, QERA, ASER) has been proposed to mitigate this issue, however, they restore all layers and insert error-correction modules into every decoder block, which increases latency and memory overhead. To address this limitation, we propose GlowQ, a group-shared low-rank approximation for quantized LLMs that caches a single shared right factor per input-sharing group and restores only the groups or layers that yield the highest accuracy benefit. GlowQ computes the high-precision projection once per input-sharing group and reuses it across its modules, reducing parameter and memory overhead, and retaining the expressivity of layer-specific corrections. We also propose a selective variant, GlowQ-S, that applies the cached shared module only where it provides the largest benefit. Compared with strong baselines, our approach reduces TTFB by (5.6%) and increases throughput by (9.6%) on average, while reducing perplexity on WikiText-2 by (0.17%) and increasing downstream accuracy by 0.42 percentage points. The selective model GlowQ-S further reduces latency, cutting TTFB by (23.4%) and increasing throughput by (37.4%), while maintaining accuracy within 0.2 percentage points on average. Code is available at https://github.com/ahnselim/GlowQ.

Selim An, Il hong Suh, Yeseong Kim

Department of Artificial Intelligence, DGIST, Korea · COGA robotics, Korea · Department of Electrical Engineering, POSTECH, Korea

LLM Quantization LLM Inference Acceleration Low-Bit Quantization

Ask PDF

Apr 20, 2026

LoRaQ: Optimized Low Rank Approximation for 4-bit Quantization

Post-training quantization (PTQ) is essential for deploying large diffusion transformers on resource-constrained hardware, but aggressive 4-bit quantization significantly degrades generative performance. Low-rank approximation methods have emerged as a promising solution by appending auxiliary linear branches to restore performance. However, current state-of-the-art approaches assume these branches must retain high precision (W16A16) and rely on heavy, data-dependent calibration for initialization. We challenge both limitations with LoRaQ (Low-Rank Approximated Quantization), a simple, data-free calibration approach that optimizes quantization error compensation. By overcoming the need for high-precision branches, LoRaQ enables the first fully sub-16 bit pipeline, allowing the low-rank branch itself to be quantized. We demonstrate that, at equal memory overhead, LoRaQ outperforms the state-of-the-art methods in their native implementations on Pixart-

Σ

and SANA. We also analyze mixed-precision configurations, showing that setups such as W8A8, W6A6, and W4A8 for the low-rank branch, alongside a W4 main layer, yield superior results while maintaining a fully quantized architecture compatible with modern mixed-precision hardware.

Yann Bouquet, Alireza Khodamoradi, Sophie Yáng Shen +2

Research and Advancement Development Team, Advanced Micro Devices, Inc, USA · Computer Vision Laboratory, Ecole Polytechnique F´ed´erale de Lausanne, Lausanne, Switzerland.

Mixed-Precision Quantization Diffusion Transformer Diffusion Model Quantization 4-Bit Quantization Low-Rank Approximation Post-Training Quantization+1

Ask PDF

May 31, 2026

GPTQ-intrinsic LoRA: A Near-optimal Algorithm for Low-precision Quantization with Low-rank Adaptation

Post-training quantization is widely used for compressing large neural networks, but aggressive low-bit quantization can significantly degrade model quality. A common remedy is to augment the quantized weights with a low-rank correction, leading to approximations of the form

W\approx Q+LR

. In this paper, we study this low-precision plus low-rank representation through the layer-wise reconstruction objective

\|XW-X(Q+LR)\|_F^2

, where

X

is a calibration matrix. We establish, to our knowledge, the first information-theoretic lower bounds for this problem under finite-alphabet and bounded low-rank compensation constraints. We then propose GPTQ-intrinsic LoRA, a training-free algorithm that incorporates the low-rank correction directly into a GPTQ-style quantization pass by appropriately augmenting the calibration Hessian. For the choice

L=V_r

, where

V_r

contains the top right singular vectors of

X

, we prove layer-wise reconstruction error bounds in which the usual GPTQ dependence on

\|X\|_F^2

is replaced by the rank-

r

residual

\|X-X_r\|_F^2

, up to regularization terms. Under natural structural assumptions, these bounds match the information-theoretic lower bounds in their dominant scaling, up to constants and mild factors. We also introduce Bid-Up, a fixed-grid quantization refinement step that can be alternated with optimal low-rank compensation with guaranteed non-increasing layer-wise reconstruction error. Experiments on Qwen3 language models and DeiT vision transformers show that GPTQ-intrinsic LoRA improves over GPTQ and GPTQ followed by low-rank compensation, with additional gains from refinement loops.

Shihao Zhang, Rayan Saab

Department of Mathematics, University of California San Diego

LLM Quantization Low-Rank Adaptation Low-Rank Compression Post-Training Quantization Low-Bit Quantization Neural Network Quantization

Ask PDF

Abstract

LLM Quantization

Explore similar work

GlowQ: Group-Shared LOw-Rank Approximation for Quantized LLMs

LoRaQ: Optimized Low Rank Approximation for 4-bit Quantization

GPTQ-intrinsic LoRA: A Near-optimal Algorithm for Low-precision Quantization with Low-rank Adaptation