FluxBin: Flexible LUT-based Ultra-low-bit LLM Inference by Algorithm-Kernel Synergy

Qingyao Yang; Runming Yang; He Xiao; Wendong Xu; Junyu Chen; Haobo Liu; Chenchen Ding; Ruihan Hu

cs.LGAug 16, 2026

FluxBin: Flexible LUT-based Ultra-low-bit LLM Inference by Algorithm-Kernel Synergy

Authors: Qingyao Yang, Runming Yang, He Xiao, Wendong Xu, Junyu Chen, Haobo Liu, Chenchen Ding, Ruihan Hu, +2 more

Organizations: The University of Hong Kong · Harbin Institute of Technology

Abstract

While binary quantization theoretically promises extreme compression and acceleration for Large Language Models (LLMs), existing research often overlooks the necessity of specialized hardware kernels, thus failing to unleash the full acceleration potential due to persistent reliance on expensive floating-point arithmetic or runtime dequantization overheads. To bridge this gap, we propose FluxBin (Flexible LUT-based Ultra-low-bit eXecution with Binary bases), an algorithm-kernel co-design that synergizes post-training quantization with a highly optimized CUDA kernel. Algorithmically, we introduce Decoupled Row-Column Binary Decomposition to enhance representational capacity while maintaining hardware efficiency, complemented by a Hessian-guided saliency-aware hybrid bases that preserve critical information. At the kernel level, we implement a Lookup Table Building Approach with Scale Fusion to reduce floating-point arithmetic, featuring a Virtual Columnar Mapping that transforms irregular, sparse, and salient matrices into dense execution. Extensive evaluations demonstrate FluxBin achieves up to $5.92\times$ speedup and $10.19\times$ energy savings across diverse model architectures, delivering comparable accuracy to heavily fine-tuned methods. This effectively enables the deployment of 70B-scale models on one single A100 GPU with a $4\times$ memory reduction. Code is available at https://github.com/nicyyyy/FluxBin.

Figures & tables

Figure 1: Illustration of LUT-based Multiplication.

Figure 2: The framework of FluxBin Binary Bases Quantization.

Panel of Figure 3 — Figure 3: FluxBin kernel architecture and VCM strategy.

Figure 4: (a) E2E throughput scaling. (b) Average power. (c) Decode-only throughput.

LLaMA-2	Method	FT/Dist ∗	W bit †	Speed (tok/s) $\uparrow$	Wiki $\downarrow$	PIQA $\uparrow$	ARC-e $\uparrow$	ARC-c $\uparrow$	BoolQ $\uparrow$	HS ‡ $\uparrow$	WG ‡ $\uparrow$	Avg. $\uparrow$
7B	FP16		16	42.94	5.47	78.13	75.46	43.00	79.36	57.08	69.38	67.07
	GPTQ	✗	3.125	–	8.42	73.50	63.43	30.29	67.00	48.57	64.96	57.96
	OmniQuant	✓	3	83.40	6.62	74.70	64.26	35.92	66.29	51.73	63.45	59.39
	SLiM-LLM	✓	2	73.70	16.38	63.43	46.42	23.63	61.71	33.22	55.64	47.34
	AQLM	✓	2.02	98.50	6.59	72.69	65.11	32.85	69.24	49.34	62.59	58.64
	QuIP#	✓	2.02	131.97	8.22	75.10	64.60	34.60	74.31	42.94	64.90	59.41

Table 1: E2E speed and zero-shot accuracy on LLaMA-2. ∗ FT/Dist used. † Avg bit incl. scales (Sec. C.2 ). ‡ HS=HellaSwag, WG=Winogrande. Best / 2nd / 3rd .

Figure 5: (a) Latency breakdown of LUT-based kernel. (b) Speed-accuracy trade-off on LLaMA-2.

Config.	w/o	Wiki PPL $\downarrow$	PIQA $\uparrow$
Baseline		8.20	75.50
Column Scaling	✗	8.96	71.65
Saliency Search	✗	26.02	66.92
Hybrid Base	✗	13.71	69.79
Config.	w/o	Speed (tok/s) $\uparrow$	Max Mem. (MB) $\downarrow$
Baseline		250.87	3475.21

Table 2: Ablation study on LLaMA-2-7B (baseline: FluxBin-2b-s16-g128). Removing a component is denoted by ✗.

Appendix figures & tables13 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 8

	Baselines			FluxBin
Model Name	FP16	AQLM 2bit	QuIP# 2bit	2b-g128	2b-s8-g128	2b-s16-g128	4b-g128
LLaMA-2-7B	13330.61	5247.35	3461.87	2895.24	3474.30	3475.21	4684.38
LLaMA-2-13B	25681.85	4058.02	5785.77	4558.59	6172.64	6173.32	8474.27
LLaMA-2-70B	OOM	20415.70	19360.84	20251.61	26393.52	26393.13	38633.95
Qwen3-8B	15779.09	–	–	4557.90	5185.48	5187.33	6444.57
Qwen3-14B	28322.55	–	–	7043.43	8215.49	8221.31	10486.19

Appendix

Table 3: Max GPU Memory (MB) Allocated in Inference with Input Length 512 and Output Length 512.

Figure 6: Throughput Comparison on LLaMA-3.1 Across All Batch Sizes

Figure 7: Throughput Comparison on Qwen3 Across All Batch Sizes

LLaMA-3.1	Method	#W bit	Speed (tok/s) $\uparrow$	ARC-e $\uparrow$	ARC-c $\uparrow$	HS ∗ $\uparrow$	WG ∗ $\uparrow$	MMLU † $\uparrow$	Avg. $\uparrow$
8B-Instruct	FP16	16	41.51	82.15	52.05	60.77	74.35	69.40	67.74
	FlexRound	2.125	-	24.75	24.57	43.78	55.16	24.27	34.51
	AQLM	2	124.50	46.25	30.89	61.40	58.25	42.29	47.82
	GPTQ	3.125	–	72.14	37.97	53.73	68.75	46.90	55.90
	FluxBin-2b-s8-g128	2.63	217.02	69.23	35.41	45.16	68.19	43.90	52.38
	FluxBin-2b-s16-g128	2.75	217.03	69.91	36.77	44.49	68.43	45.15	52.95

Appendix

Table 4: Comparison of quantization methods on LLaMA-3.1 models. Higher accuracy and speed are better (

\uparrow

). ∗ HS = HellaSwag, WG = Winogrande. † MMLU: Evaluated using 5-shot in-context learning. We highlight the best , second , and third results (excluding FP16).

Qwen3	Method	#W bit	Speed (tok/s) $\uparrow$	Wiki $\downarrow$	PIQA $\uparrow$	ARC-e $\uparrow$	ARC-c $\uparrow$	BoolQ $\uparrow$	HS ∗ $\uparrow$	WG ∗ $\uparrow$	Avg. $\uparrow$
8B	FP16	16	28.60	9.72	76.39	83.29	55.38	86.57	57.16	68.03	71.14
	SLiM-LLM	2	-	30.61	60.11	43.81	23.97	62.99	34.83	55.80	46.92
	GPTQ	3.125	-	11.18	74.16	77.90	48.63	84.07	53.42	65.43	67.27
	FluxBin-2b-s8-g128	2.63	121.85	13.46	74.37	74.41	42.58	83.33	47.90	66.30	64.82
	FluxBin-2b-s16-g128	2.75	121.97	12.44	74.21	74.71	43.77	82.75	49.09	66.77	65.22
14B	FP16	16	26.54	8.64	80.36	84.13	58.62	89.42	60.94	72.69	74.36

Appendix

Table 5: Comparison of quantization methods on Qwen3 models. Higher accuracy and speed are better (

\uparrow

). ∗ HS = HellaSwag, WG = Winogrande. We highlight the best , second , and third results (excluding FP16).

Config.	W bit †	Wiki $\downarrow$	PIQA $\uparrow$	ARC-e $\uparrow$	ARC-c $\uparrow$	BoolQ $\uparrow$	HS ‡ $\uparrow$	WG ‡ $\uparrow$	Avg. $\uparrow$
2b-g64	2.51	10.27	73.50	63.43	30.29	67.00	48.57	64.96	57.96
2b-g128	2.25	13.71	69.79	56.23	25.00	65.41	38.64	56.69	51.96
2b-g256	2.13	12.79	64.91	48.91	22.35	62.08	34.52	50.43	47.20
2b-s8-g64	3.25	7.95	73.01	67.59	32.51	73.70	47.05	66.06	59.99
2b-s8-g128	2.63	8.76	72.52	65.45	30.97	69.88	45.64	64.09	58.09
2b-s8-g256	2.31	8.91	72.47	64.98	30.46	71.25	44.97	64.88	58.17

Appendix

Table 6: Ablation study across different group sizes (g) in fixed configurations of LLaMA-2-7B. Lower Wiki PPL is better, while higher accuracy is better for downstream tasks. † The average bit includes weight and scale factors. The calculation for FluxBin is derived in Sec. C.2 . ‡ HS = HellaSwag, WG = Winogrande.

nsample	Time (min) $\downarrow$	Wiki $\downarrow$	PIQA $\uparrow$	ARC-e $\uparrow$	ARC-c $\uparrow$	BoolQ $\uparrow$	HS ‡ $\uparrow$	WG ‡ $\uparrow$	Avg. $\uparrow$
64	13.22	8.92	72.31	64.60	30.55	70.06	45.61	64.01	57.86
128	17.02	8.85	72.74	65.49	30.97	69.88	45.38	63.54	58.00
256	25.53	8.63	74.10	64.44	33.28	71.13	45.51	64.17	58.77
512	41.78	8.54	73.34	65.78	32.00	71.13	45.89	63.30	58.57
1024	74.58	8.76	72.63	65.40	30.80	69.82	45.67	64.09	58.07

Appendix

Table 7: Ablation study of the number of calibration samples (nsample) on evaluation time and performance on LLaMA-2-7B, with FluxBin configuration 2b-s8-g128. Lower PTQ time and Wiki PPL are better (

\downarrow

), while higher accuracy is better (

\uparrow

). ‡ HS = HellaSwag, WG = Winogrande.

Model	Quantization	Avg Power (W) $\downarrow$	Total Energy (J) $\downarrow$	Avg GPU Util. (%)	Avg Mem. Util. (%)
LLaMA-2-7B	FP16	168.66	3467.69	67.01	18.09
	FluxBin-2b-g128	106.75	462.98	99.83	5.55
	FluxBin-2b-s8-g128	110.80	425.05	99.81	6.27
	FluxBin-2b-s16-g128	107.40	434.34	99.87	6.27
	FluxBin-4b-g128	109.40	401.16	99.85	7.65
LLaMA-2-13B	FP16	225.49	5908.03	86.83	33.16

Appendix

Table 8: Power, energy, and resource utilization comparison on LLaMA-2 models. Lower power and energy are better (

\downarrow

); GPU and memory utilization are reported as diagnostics of the compute/bandwidth balance rather than as quality targets.

$\mu$	BS	Shared Mem. (KB)	Throughput $\uparrow$	Power (W) $\downarrow$	Energy (J) $\downarrow$
4	1	3	213.97	107.13	397.15
	4	12	656.50	146.97	802.74
	8	24	952.86	172.31	1301.12
8	1	17	213.96	105.41	451.18
	4	68	655.58	140.04	792.82
	8	136	953.04	169.48	1220.45

Appendix

Table 9: Impact of sub-vectors size

\mu

on block-level shared memory usage and end-to-end generation performance of FluxBin-2b-g128 LLaMA-2-7B. Higher throughput is better (

\uparrow

), while lower power and energy are better (

\downarrow

				FluxBin-2b-g128		FluxBin-4b-g128
M	K	$M_{\text{tile}}$	$K_{\text{tile}}$	Latency (µs) $\downarrow$	TFLOPS $\uparrow$	Latency (µs) $\downarrow$	TFLOPS $\uparrow$
4096	4096	32	32	$113.3$	$0.30$	$109.6$	$0.31$
		32	64	$143.0$	$0.23$	$108.4$	$0.31$
		32	128	$228.9$	$0.15$	$108.4$	$0.31$
		64	32	$58.0$	$0.58$	$72.9$	$0.46$
		64	64	$67.6$	$0.50$	$66.9$	$0.50$

Appendix

Table 10: Effect of tile sizes on FluxBin kernel performance.

M	K	$K_{\text{tile}}$	$M_{\text{tile}}$	$M_{s\_tile}$	Lat (µs) $\downarrow$	TFLOPS $\uparrow$	Lat (µs) $\downarrow$	TFLOPS $\uparrow$
					FluxBin-2b-s8-g128		FluxBin-2b-s16-g128
4096	4096	32	32	16	$121.0$	$0.29$	$124.4$	$0.28$
				32	$119.5$	$0.29$	$123.2$	$0.28$
				64	$118.3$	$0.29$	$121.9$	$0.28$
			64	16	$67.3$	$0.51$	$69.0$	$0.50$
				32	$65.7$	$0.53$	$67.9$	$0.51$

Appendix

Table 11: Effect of tile sizes on FluxBin kernel performance on hybrid bases.

Weight Size ( $M\times N$ )	Baselines		FluxBin
Weight Size ( $M\times N$ )	AQLM 2bit	QuIP# 2bit	2b-g128	2b-s8-g128	2b-s16-g128	4b-g128
4096 $\times$ 4096	79	112	41	47	49	56
5120 $\times$ 5120	111	174	63	73	74	87
4096 $\times$ 11008	169	272	102	109	111	158
5120 $\times$ 13824	277	411	167	166	169	252

Appendix

Table 12: Per-operator GEMV kernel latency (

\mu

s) across weight matrix sizes (

M\times N

), measured for the 2-bit quantized baselines and FluxBin. Lower is better (

\downarrow

); the best result in each row is in bold . Each kernel is measured in isolation with its official implementation.

Explore similar work

May 1, 2026

BWLA: Breaking the Barrier of W1AX Post-Training Quantization for LLMs

Large language models (LLMs) have driven major progress in NLP, yet their substantial memory and compute demands still hinder practical deployment. Binarization can compress weights to 1 bit, fundamentally lowering compute and bandwidth cost. However, existing methods cannot address activation heavy tails and thus must keep activations in high precision, preventing true end-to-end acceleration. To overcome this limitation, we propose BWLA (Binarized Weights and Low-bit Activations), the first post-training quantization framework that preserves high accuracy while achieving 1-bit weight quantization together with low-bit activations (e.g., 6 bits). The Orthogonal-Kronecker Transformation (OKT) learns an orthogonal mapping via EM minimization, converting unimodal weights into symmetric bimodal forms while suppressing activation tails and incoherence. The Proximal SVD Projection (PSP) then performs lightweight low-rank refinement through proximal SVD projection, further enhancing quantizability with minimal overhead. On Qwen3-32B, BWLA reaches a Wikitext2 perplexity of 11.92 under 6-bit activations (vs. 38 from SOTA), improves five zero-shot tasks by more than 70%, and delivers 3.26 times inference speedup, demonstrating strong potential for real-world LLM compression and acceleration.

Zhixiong Zhao, Zukang Xu, Dawei Yang

1Houmo AI

LLM Quantization LLM Inference Acceleration Activation Quantization Post-Training Quantization Low-Bit Quantization

Ask PDF

Sep 5, 2026

All for 1-Bit: Towards Genuine 1-Bit Post-Training Quantization for LLMs

Large language models (LLMs) have achieved remarkable progress, yet their massive storage and memory-bandwidth demands still hinder efficient deployment. Weight binarization is a promising solution, but existing binarization-based post-training quantization (PTQ) methods usually far exceed the nominal 1-bit storage target due to hidden overhead. To address this gap, we propose All for 1-Bit (AF1), a genuine 1-bit PTQ framework for LLMs. AF1 comprises two complementary components: (1) Null-space-Aware Binary Factorization (NABF) for improving binary reconstruction through Hessian-aware surrogate reparameterization, null-space-aware binary factorization, and scale-only global reconstruction; and (2) Hierarchical Shapley Allocation (HiSA) for assigning structural capacity using hierarchical Shapley sensitivity. Together, they preserve model accuracy under a strict 1.0-BPW budget in the PTQ setting. Experiments on LLaMA, Qwen, and Gemma families show that AF1 consistently outperforms existing binarization-based PTQ methods in perplexity and zero-shot accuracy. Compared with BF16, AF1 achieves an average 2.5 times inference speedup and over 90% memory reduction across evaluated models, providing a practical path toward deployable genuine 1-bit compression for LLMs. The code for reproducibility is available at https://github.com/Kishon-zzx/AF1.

Zhixiong Zhao, Zukang Xu, Guangyu Sun +2

Houmo AI · School of Integrated Circuits, Peking University

LLM Quantization Post-Training Quantization Neural Network Quantization LLM Inference Acceleration

Ask PDF

Apr 21, 2026

LBLLM: Lightweight Binarization of Large Language Models via Three-Stage Distillation

Deploying large language models (LLMs) in resource-constrained environments is hindered by heavy computational and memory requirements. We present LBLLM, a lightweight binarization framework that achieves effective W(1+1)A4 quantization through a novel three-stage quantization strategy. The framework proceeds as follows: (1) initialize a high-quality quantized model via PTQ; (2) quantize binarized weights, group-wise bitmaps, and quantization parameters through layer-wise distillation while keeping activations in full precision; and (3) training learnable activation quantization factors to dynamically quantize activations to 4 bits. This decoupled design mitigates interference between weight and activation quantization, yielding greater training stability and better inference accuracy. LBLLM, trained only using 0.016B tokens with a single GPU, surpasses existing state-of-the-art binarization methods on W2A4 quantization settings across tasks of language modeling, commonsense QA, and language understanding. These results demonstrate that extreme low-bit quantization of LLMs can be both practical and highly effective without introducing any extra high-precision channels or rotational matrices commonly used in recent PTQ-based works, offering a promising path toward efficient LLM deployment in resource-limited situations.

Siqing Song, Chuang Wang, Yong Lang +2

MAIS, Institute of Automation, Chinese Academy of Sciences, China · School of Artificial Intelligence, University of Chinese Academy of Sciences, China · Central Media Technology Institute, Huawei

LLM Quantization Activation Quantization Post-Training Quantization Low-Bit Quantization

Ask PDF