BiLoCo: Binary Low-Rank Corrections for Post-Training Quantization

David Jin; Beshr IslamBouli; Tarushii Goel; Han Guo; Yoon Kim

cs.LGOct 23, 2026

BiLoCo: Binary Low-Rank Corrections for Post-Training Quantization

Authors: David Jin, Beshr IslamBouli, Tarushii Goel, Han Guo, Yoon Kim

Organizations: Massachusetts Institute of Technology

Abstract

Post-training quantization (PTQ) to FP4 has emerged as a key technique for reducing large language model inference cost, especially on Blackwell GPUs, which offer native FP4 tensor core support. Direct PTQ to FP4, however, still leaves a noticeable accuracy gap. Recent work on SVDQuant mitigates the errors introduced through FP4 quantization for diffusion models by adding high-precision low-rank corrections obtained via SVD. However, this high-precision low-rank correction requires many bits per rank, and it runs on the same tensor cores as the main FP4 GEMM. We introduce BiLoCo, a low-rank correction method whose left and right factors are stored as binary ${\{\pm 1\}}$ sign vectors rather than as BF16 numbers. The signed format uses $16{\times}$ less storage per rank-one component, and since the correction now consists of additions/subtractions, this can be run on CUDA cores concurrently with the FP4 tensor-core GEMM. On 4B--72B LLMs, BiLoCo matches or improves memory-matched SVDQuant for PTQ. We implement BiLoCo on B300 as a fused single kernel, where the FP4 GEMM runs on tensor cores, the bit-packed correction on the CUDA cores of the same launch, and the two are combined in the GEMM epilogue. With batch size $1$ , the fused kernel is at most $11\%$ slower than the FP4-only GEMM for every rank up to $512$ . At rank $128$ , it is at most $20\%$ slower up to batch size $8$ .

Figures & tables

Figure 1: BiLoCo inference for performing

y=xR+xVU^{\top}.

A native-FP4 residual GEMM on tensor cores runs alongside a signed add/subtract correction on CUDA cores.

Figure 2: Fraction

\gamma=\|\widehat{W}\|_{F}/\|W\|_{F}

of the Frobenius norm in the low-rank part versus the storage budget

b

, on

63

Qwen3-4B projection matrices (

7

projection types in

9

decoder layers). At each budget, BiLoCo uses binary rank

r=b

and SVDQuant uses BF16 rank

k=b/16

, so both use the same storage. Lines are medians and bands are interquartile ranges.

model	$\sigma_{1}/\sigma_{32}$ med.	$\gamma$ , SVD $k{=}32$	$\gamma$ , sign $r{=}512$	ratio	Pearson
Qwen3-4B	$1.93$	$29.3\%$	$52.9\%$	$1.92\times$	$-0.65$
Qwen3-14B	$2.00$	$26.3\%$	$47.3\%$	$1.77\times$	$-0.65$
Llama-3-8B	$2.24$	$27.9\%$	$50.6\%$	$1.76\times$	$-0.81$

Table 1: Fraction

\gamma

at equal storage across three model families. Signed rank

512

and BF16 SVD rank

32

use the same vector-storage budget.

Model	Dataset	BF16	NVFP4	SVDQuant $k{=}32$	BiLoCo $r{=}128$	BiLoCo $r{=}512$
Qwen3-4B	Wiki2	$14.22$	$15.13$	$15.07$	$15.09{\scriptstyle\,\pm 0.58}$	$\mathbf{14.94}{\scriptstyle\,\pm 0.37}$
Qwen3-4B	C4	$15.25$	$15.88$	$15.85$	$15.88{\scriptstyle\,\pm 0.23}$	$\mathbf{15.81}{\scriptstyle\,\pm 0.17}$
Llama-3-8B	Wiki2	$6.27$	$7.00$	$6.92$	$6.93{\scriptstyle\,\pm 0.01}$	$\mathbf{6.84}{\scriptstyle\,\pm 0.01}$
Llama-3-8B	C4	$6.69$	$7.68$	$7.59$	$7.61{\scriptstyle\,\pm 0.01}$	$\mathbf{7.48}{\scriptstyle\,\pm 0.01}$
Qwen3-14B	Wiki2	$8.89$	$9.36$	$9.19$	$9.21{\scriptstyle\,\pm 0.03}$	$\mathbf{9.17}{\scriptstyle\,\pm 0.04}$
Qwen3-14B	C4	$11.19$	$11.55$	$11.46$	$11.49{\scriptstyle\,\pm 0.03}$	$\mathbf{11.44}{\scriptstyle\,\pm 0.02}$

Table 2: Perplexity of BiLoCo vs. baseline methods. BiLoCo with

b{=}512

is memory-matched to SVDQuant writh

k=32

. BiLoCo entries report mean

\,\pm\,

std over sign initializations. Bold marks the best corrected FP4 method, with ties bolded.

	KL mean ( $\downarrow$ )		KL p99 ( $\downarrow$ )
Model	BiLoCo $r{=}512$	SVDQuant $k{=}32$	BiLoCo $r{=}512$	SVDQuant $k{=}32$
Qwen3-4B	$\mathbf{0.0740}$	$0.0835$	$\mathbf{0.1421}$	$0.1649$
Qwen3-14B	$\mathbf{0.0480}$	$0.0517$	$\mathbf{0.1303}$	$0.1671$
Llama-3-8B	$\mathbf{0.0756}$	$0.0824$	$\mathbf{0.3252}$	$0.3409$
Qwen3-32B	$\mathbf{0.0589}$	$0.0605$	$\mathbf{0.3175}$	$0.3363$
Qwen2.5-72B	$0.0515$	$\mathbf{0.0497}$	$\mathbf{0.1937}$	$0.2097$

Table 3: KL divergence to the BF16 model at

b{=}512

on FineWeb document. Lower is better.

Method	Qwen3-4B	Qwen3-14B	Llama-3-8B	Qwen3-32B	Qwen2.5-72B
BF16 (reference)	$0.686$	$0.772$	$0.803$	$0.803$	$0.857$
SVDQuant $k{=}32$ ( $b{=}512$ )	$0.669$	$0.759$	$\mathbf{0.780}$	$0.796$	$0.849$
BiLoCo $r{=}128$ ( $b{=}128$ )	$0.667$	$\mathbf{0.763}$	$0.777$	$0.795$	$0.846$
BiLoCo $r{=}512$ ( $b{=}512$ )	$\mathbf{0.670}$	$\mathbf{0.763}$	$0.777$	$\mathbf{0.799}$	$\mathbf{0.856}$

Table 4: Mean accuracy on WinoGrande (5-shot) and HellaSwag (10-shot). Higher is better.

	Rank $r$
Batch $M$	$16$	$32$	$64$	$128$	$256$	$512$
$1$	$6.17$ $1.06\times$	$6.18$ $1.06\times$	$6.23$ $1.06\times$	$6.33$ $1.08\times$	$6.39$ $1.09\times$	$6.49$ $1.11\times$
$2$	$6.36$ $1.08\times$	$6.36$ $1.09\times$	$6.36$ $1.09\times$	$6.32$ $1.08\times$	$6.36$ $1.09\times$	$6.71$ $1.15\times$
$4$	$6.36$ $1.09\times$	$6.36$ $1.09\times$	$6.46$ $1.10\times$	$6.47$ $1.11\times$	$6.90$ $1.18\times$	$8.45$ $1.44\times$
$8$	$6.42$ $1.10\times$	$6.42$ $1.10\times$	$6.82$ $1.17\times$	$6.74$ $1.15\times$	$8.02$ $1.37\times$	$10.10$ $1.74\times$
$16$	$6.55$ $1.11\times$	$6.52$ $1.11\times$	$7.48$ $1.28\times$	$9.08$ $1.55\times$	$14.70$ $2.51\times$	$17.81$ $3.04\times$

Table 5: Decode latency of the BiLoCo kernel on B300 (

K{=}N{=}5120

). Each entry gives the time per launch in

\mu

s and, below it, the ratio to the padded FP4-only kernel measured in the same run (

5.8

–

5.9\,\mu

s). Gray entries are more than

20\%

slower than the FP4-only kernel.

Shape ( $K{\times}N$ )	Role	padded	unpadded	$r{=}128$	$r{=}256$	$r{=}512$
$5120{\times}5120$	attn q/o	$5.85$	$7.54$	$6.33$ $1.08\times$	$6.39$ $1.09\times$	$6.49$ $1.11\times$
$5120{\times}1024$	attn k/v	$5.72$	$5.62$	$6.10$ $1.09\times$	$6.55$ $1.17\times$	$7.00$ $1.25\times$
$17408{\times}5120$	MLP down	$12.42$	$22.65$	$12.93$ $1.04\times$	$12.97$ $1.04\times$	$13.18$ $1.06\times$

Table 6: BiLoCo kernel across Qwen3-14B projection shapes (

M{=}1

, B300). Each entry gives the time per launch in

\mu

s and, below it, the ratio to the fastest FP4-only GEMM of the same shape measured in the same run.

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Model	Projection	SVDQuant $k{=}32$	BiLoCo $r{=}128$	BiLoCo $r{=}256$
Qwen3-4B	q_proj	$28.1\%$	$35.4\%$	$\mathbf{44.6\%}$
	k_proj	$36.9\%$	$43.5\%$	$\mathbf{55.8\%}$
	v_proj	$24.7\%$	$38.3\%$	$\mathbf{51.8\%}$
	o_proj	$21.5\%$	$30.8\%$	$\mathbf{41.7\%}$
	gate_proj	$18.5\%$	$26.5\%$	$\mathbf{34.9\%}$
	up_proj	$15.0\%$	$23.4\%$	$\mathbf{32.3\%}$

Appendix

Table 7: Frobenius-norm capture

\mathrm{capA}(r)=\|\widehat{W}_{r}\|_{F}/\|W\|_{F}

per projection type, as the median over

9

evenly spaced decoder layers per model. SVDQuant uses rank

k{=}32

(

b{=}512

), and BiLoCo uses

r{=}128

(

b{=}128

, a quarter of that storage) and

r{=}256

(

b{=}256

, half). Bold marks BiLoCo

r{=}256

, which is the highest in every row. Higher is better.

Figure 3: Frobenius-relative capture rate over

128

peel iterations on Qwen3-4B projection matrices.

construction	WikiText-2	C4
random signs $+$ joint coefficient refit	$7.000{\scriptstyle\,\pm 0.001}$	$7.684{\scriptstyle\,\pm 0.008}$
greedy learned signs	$6.842{\scriptstyle\,\pm 0.007}$	$7.475{\scriptstyle\,\pm 0.018}$
greedy learned signs $+$ global refit (Algorithm 1 )	$6.843{\scriptstyle\,\pm 0.017}$	$7.480{\scriptstyle\,\pm 0.011}$
$+$ pm-altmin, $10$ sweeps	$6.838{\scriptstyle\,\pm 0.013}$	$7.459{\scriptstyle\,\pm 0.014}$

Appendix

Table 8: Solver ablation on Llama-3-8B at

r{=}512

: perplexity (mean

\,\pm\,

std over three sign seeds) for four ways of constructing the low-rank component. Without a low-rank component (NVFP4 only), perplexity is

6.998

on WikiText-2 and

7.680

on C4.

method	solver	other	layer total	PTQ total
SVDQuant, BF16 rank $32$ (SVD)	$170.4$	$2.5$	$174.3$	$222$
greedy BiLoCo, $r{=}512$	$86.3$	$0.7$	$90.3$	$116$
greedy $+$ global refit, $r{=}512$	$91.3$	$0.2$	$92.5$	$118$
batched pm-altmin, $r{=}512$	$213.9$	$0.7$	$215.9$	$263$
full pm-altmin, $r{=}512$	$1070.2$	$0.2$	$1071.3$	$1097$

Appendix

Table 9: PTQ time on Llama-3-8B in seconds on one node. “Other” is the NVFP4 quantization of the residual and bookkeeping. The PTQ total also includes collecting the calibration activations.

	WikiText-2	C4
BF16	$6.266$	$6.690$
NVFP4 only	$6.998$	$7.680$
SVDQuant, BF16 rank $32$	$6.919$	$7.592$
BiLoCo $r{=}32$	$6.983$	$7.656$
BiLoCo $r{=}128$	$6.926$	$7.605$
BiLoCo $r{=}256$	$6.892$	$7.549$

Appendix

Table 10: BiLoCo perplexity versus rank on Llama-3-8B (C4-train calibration, pm-altmin)

backbone / correction	WikiText-2	C4
no preprocessing, NVFP4 only	$7.056$	$7.751$
presmoothing, NVFP4 only	$6.998$	$7.680$
QuaRot, NVFP4 only	$7.206$	$7.981$
QuaRot $+$ SVDQuant	$7.130$	$7.904$
QuaRot $+$ BiLoCo	$\mathbf{6.999}$	$\mathbf{7.731}$
SpinQuant, NVFP4 only	$7.240$	$8.035$

Appendix

Table 11: Perplexity of Llama-3-8B with full-width rotated backbones (seed

42

, lower is better). SVDQuant uses BF16 rank

32

, and BiLoCo uses

r{=}512

with the pm-altmin solver (Appendix A.2 ).

Model	Method	$b$	ARC-e	ARC-c	PIQA	WinoG.	OBQA	mean
Qwen3-4B	SVDQuant $k{=}32$	512	0.775	0.467	0.738	0.639	0.296	0.583
	BiLoCo $r{=}128$	128	0.789	0.485	0.727	0.618	0.298	0.583
	BiLoCo $r{=}512$	512	0.782	0.486	0.724	0.652	0.286	0.586
Qwen3-14B	SVDQuant $k{=}32$	512	0.830	0.557	0.780	0.717	0.344	0.646
	BiLoCo $r{=}128$	128	0.830	0.560	0.795	0.688	0.344	0.643
	BiLoCo $r{=}512$	512	0.828	0.583	0.795	0.715	0.348	0.654

Appendix

Table 12: Standard PTQ downstream task breakdown. Higher is better.

			KL mean ( $\downarrow$ )		KL p99 ( $\downarrow$ )
Model	Method	$b$	BiLoCo	SVDQ	BiLoCo	SVDQ
Qwen3-4B	$r{=}128$ vs. $k{=}8$	128	0.0914	0.0901	0.1849	0.1965
Qwen3-4B	$r{=}512$ vs. $k{=}32$	512	0.0740	0.0835	0.1421	0.1649
Qwen3-14B	$r{=}128$ vs. $k{=}8$	128	0.0548	0.0541	0.2026	0.1618
Qwen3-14B	$r{=}512$ vs. $k{=}32$	512	0.0480	0.0517	0.1303	0.1671
Llama-3-8B	$r{=}128$ vs. $k{=}8$	128	0.0856	0.0881	0.3611	0.4789

Appendix

Table 13: KL divergence to the BF16 model at

b{=}128

and

b{=}512

on held-out FineWeb samples. Lower is better.

LLM Quantization

Large language model quantization reduces numerical precision to lower memory use and accelerate training or inference.

See all 386 papers

GlowQ: Group-Shared LOw-Rank Approximation for Quantized LLMs
Mar 26, 2026Selim An, Il hong Suh, Yeseong KimLLM Quantization LLM Inference Acceleration
Department of Artificial Intelligence, DGIST, Korea · COGA robotics, Korea · Department of Electrical Engineering, POSTECH, Korea
LoRaQ: Optimized Low Rank Approximation for 4-bit Quantization
Apr 20, 2026Yann Bouquet, Alireza Khodamoradi, Sophie Yáng Shen +2Mixed-Precision Quantization Diffusion Transformer
Research and Advancement Development Team, Advanced Micro Devices, Inc, USA · Computer Vision Laboratory, Ecole Polytechnique F´ed´erale de Lausanne, Lausanne, Switzerland.
GPTQ-intrinsic LoRA: A Near-optimal Algorithm for Low-precision Quantization with Low-rank Adaptation
May 31, 2026Shihao Zhang, Rayan SaabLLM Quantization Low-Rank Adaptation
Department of Mathematics, University of California San Diego

Mar 26, 2026J/K move · Enter open · S save

GlowQ: Group-Shared LOw-Rank Approximation for Quantized LLMs

Selim An, Il hong Suh, Yeseong Kim

Department of Artificial Intelligence, DGIST, Korea · COGA robotics, Korea · Department of Electrical Engineering, POSTECH, Korea

LLM Quantization LLM Inference Acceleration Low-Bit Quantization

Open paper AskPDF arXiv

Quantization techniques such as BitsAndBytes, AWQ, and GPTQ are widely used as a standard method in deploying large language models but often degrades accuracy when using low-bit representations, e.g., 4 bits. Low-rank correction methods (e.g., LQER, QERA, ASER) has been proposed to mitigate this issue, however, they restore all layers and insert error-correction modules into every decoder block, which increases latency and memory overhead. To address this limitation, we propose GlowQ, a group-shared low-rank approximation for quantized LLMs that caches a single shared right factor per input-sharing group and restores only the groups or layers that yield the highest accuracy benefit. GlowQ computes the high-precision projection once per input-sharing group and reuses it across its modules, reducing parameter and memory overhead, and retaining the expressivity of layer-specific corrections. We also propose a selective variant, GlowQ-S, that applies the cached shared module only where it provides the largest benefit. Compared with strong baselines, our approach reduces TTFB by (5.6%) and increases throughput by (9.6%) on average, while reducing perplexity on WikiText-2 by (0.17%) and increasing downstream accuracy by 0.42 percentage points. The selective model GlowQ-S further reduces latency, cutting TTFB by (23.4%) and increasing throughput by (37.4%), while maintaining accuracy within 0.2 percentage points on average. Code is available at https://github.com/ahnselim/GlowQ.

Abstract

LLM Quantization

Explore similar work

GlowQ: Group-Shared LOw-Rank Approximation for Quantized LLMs

LoRaQ: Optimized Low Rank Approximation for 4-bit Quantization

GPTQ-intrinsic LoRA: A Near-optimal Algorithm for Low-precision Quantization with Low-rank Adaptation