FluxBin: Flexible LUT-based Ultra-low-bit LLM Inference by Algorithm-Kernel Synergy

Qingyao Yang; Runming Yang; He Xiao; Wendong Xu; Junyu Chen; Haobo Liu; Chenchen Ding; Ruihan Hu

cs.LGAug 16, 2026

FluxBin: Flexible LUT-based Ultra-low-bit LLM Inference by Algorithm-Kernel Synergy

Authors: Qingyao Yang, Runming Yang, He Xiao, Wendong Xu, Junyu Chen, Haobo Liu, Chenchen Ding, Ruihan Hu, +2 more

Organizations: The University of Hong Kong · Harbin Institute of Technology

Abstract

While binary quantization theoretically promises extreme compression and acceleration for Large Language Models (LLMs), existing research often overlooks the necessity of specialized hardware kernels, thus failing to unleash the full acceleration potential due to persistent reliance on expensive floating-point arithmetic or runtime dequantization overheads. To bridge this gap, we propose FluxBin (Flexible LUT-based Ultra-low-bit eXecution with Binary bases), an algorithm-kernel co-design that synergizes post-training quantization with a highly optimized CUDA kernel. Algorithmically, we introduce Decoupled Row-Column Binary Decomposition to enhance representational capacity while maintaining hardware efficiency, complemented by a Hessian-guided saliency-aware hybrid bases that preserve critical information. At the kernel level, we implement a Lookup Table Building Approach with Scale Fusion to reduce floating-point arithmetic, featuring a Virtual Columnar Mapping that transforms irregular, sparse, and salient matrices into dense execution. Extensive evaluations demonstrate FluxBin achieves up to $5.92\times$ speedup and $10.19\times$ energy savings across diverse model architectures, delivering comparable accuracy to heavily fine-tuned methods. This effectively enables the deployment of 70B-scale models on one single A100 GPU with a $4\times$ memory reduction. Code is available at https://github.com/nicyyyy/FluxBin.

Figures & tables

Figure 1: Illustration of LUT-based Multiplication.

Figure 2: The framework of FluxBin Binary Bases Quantization.

Panel of Figure 3 — Figure 3: FluxBin kernel architecture and VCM strategy.

Figure 4: (a) E2E throughput scaling. (b) Average power. (c) Decode-only throughput.

LLaMA-2	Method	FT/Dist ∗	W bit †	Speed (tok/s) $\uparrow$	Wiki $\downarrow$	PIQA $\uparrow$	ARC-e $\uparrow$	ARC-c $\uparrow$	BoolQ $\uparrow$	HS ‡ $\uparrow$	WG ‡ $\uparrow$	Avg. $\uparrow$
7B	FP16		16	42.94	5.47	78.13	75.46	43.00	79.36	57.08	69.38	67.07
	GPTQ	✗	3.125	–	8.42	73.50	63.43	30.29	67.00	48.57	64.96	57.96
	OmniQuant	✓	3	83.40	6.62	74.70	64.26	35.92	66.29	51.73	63.45	59.39
	SLiM-LLM	✓	2	73.70	16.38	63.43	46.42	23.63	61.71	33.22	55.64	47.34
	AQLM	✓	2.02	98.50	6.59	72.69	65.11	32.85	69.24	49.34	62.59	58.64
	QuIP#	✓	2.02	131.97	8.22	75.10	64.60	34.60	74.31	42.94	64.90	59.41

Table 1: E2E speed and zero-shot accuracy on LLaMA-2. ∗ FT/Dist used. † Avg bit incl. scales (Sec. C.2 ). ‡ HS=HellaSwag, WG=Winogrande. Best / 2nd / 3rd .

Figure 5: (a) Latency breakdown of LUT-based kernel. (b) Speed-accuracy trade-off on LLaMA-2.

Config.	w/o	Wiki PPL $\downarrow$	PIQA $\uparrow$
Baseline		8.20	75.50
Column Scaling	✗	8.96	71.65
Saliency Search	✗	26.02	66.92
Hybrid Base	✗	13.71	69.79
Config.	w/o	Speed (tok/s) $\uparrow$	Max Mem. (MB) $\downarrow$
Baseline		250.87	3475.21

Table 2: Ablation study on LLaMA-2-7B (baseline: FluxBin-2b-s16-g128). Removing a component is denoted by ✗.

Appendix figures & tables13 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 8

	Baselines			FluxBin
Model Name	FP16	AQLM 2bit	QuIP# 2bit	2b-g128	2b-s8-g128	2b-s16-g128	4b-g128
LLaMA-2-7B	13330.61	5247.35	3461.87	2895.24	3474.30	3475.21	4684.38
LLaMA-2-13B	25681.85	4058.02	5785.77	4558.59	6172.64	6173.32	8474.27
LLaMA-2-70B	OOM	20415.70	19360.84	20251.61	26393.52	26393.13	38633.95
Qwen3-8B	15779.09	–	–	4557.90	5185.48	5187.33	6444.57
Qwen3-14B	28322.55	–	–	7043.43	8215.49	8221.31	10486.19

Appendix

Table 3: Max GPU Memory (MB) Allocated in Inference with Input Length 512 and Output Length 512.

Figure 6: Throughput Comparison on LLaMA-3.1 Across All Batch Sizes

Figure 7: Throughput Comparison on Qwen3 Across All Batch Sizes

LLaMA-3.1	Method	#W bit	Speed (tok/s) $\uparrow$	ARC-e $\uparrow$	ARC-c $\uparrow$	HS ∗ $\uparrow$	WG ∗ $\uparrow$	MMLU † $\uparrow$	Avg. $\uparrow$
8B-Instruct	FP16	16	41.51	82.15	52.05	60.77	74.35	69.40	67.74
	FlexRound	2.125	-	24.75	24.57	43.78	55.16	24.27	34.51
	AQLM	2	124.50	46.25	30.89	61.40	58.25	42.29	47.82
	GPTQ	3.125	–	72.14	37.97	53.73	68.75	46.90	55.90
	FluxBin-2b-s8-g128	2.63	217.02	69.23	35.41	45.16	68.19	43.90	52.38
	FluxBin-2b-s16-g128	2.75	217.03	69.91	36.77	44.49	68.43	45.15	52.95

Appendix

Table 4: Comparison of quantization methods on LLaMA-3.1 models. Higher accuracy and speed are better (

\uparrow

). ∗ HS = HellaSwag, WG = Winogrande. † MMLU: Evaluated using 5-shot in-context learning. We highlight the best , second , and third results (excluding FP16).

Qwen3	Method	#W bit	Speed (tok/s) $\uparrow$	Wiki $\downarrow$	PIQA $\uparrow$	ARC-e $\uparrow$	ARC-c $\uparrow$	BoolQ $\uparrow$	HS ∗ $\uparrow$	WG ∗ $\uparrow$	Avg. $\uparrow$
8B	FP16	16	28.60	9.72	76.39	83.29	55.38	86.57	57.16	68.03	71.14
	SLiM-LLM	2	-	30.61	60.11	43.81	23.97	62.99	34.83	55.80	46.92
	GPTQ	3.125	-	11.18	74.16	77.90	48.63	84.07	53.42	65.43	67.27
	FluxBin-2b-s8-g128	2.63	121.85	13.46	74.37	74.41	42.58	83.33	47.90	66.30	64.82
	FluxBin-2b-s16-g128	2.75	121.97	12.44	74.21	74.71	43.77	82.75	49.09	66.77	65.22
14B	FP16	16	26.54	8.64	80.36	84.13	58.62	89.42	60.94	72.69	74.36

Appendix

Table 5: Comparison of quantization methods on Qwen3 models. Higher accuracy and speed are better (

\uparrow

). ∗ HS = HellaSwag, WG = Winogrande. We highlight the best , second , and third results (excluding FP16).

Config.	W bit †	Wiki $\downarrow$	PIQA $\uparrow$	ARC-e $\uparrow$	ARC-c $\uparrow$	BoolQ $\uparrow$	HS ‡ $\uparrow$	WG ‡ $\uparrow$	Avg. $\uparrow$
2b-g64	2.51	10.27	73.50	63.43	30.29	67.00	48.57	64.96	57.96
2b-g128	2.25	13.71	69.79	56.23	25.00	65.41	38.64	56.69	51.96
2b-g256	2.13	12.79	64.91	48.91	22.35	62.08	34.52	50.43	47.20
2b-s8-g64	3.25	7.95	73.01	67.59	32.51	73.70	47.05	66.06	59.99
2b-s8-g128	2.63	8.76	72.52	65.45	30.97	69.88	45.64	64.09	58.09
2b-s8-g256	2.31	8.91	72.47	64.98	30.46	71.25	44.97	64.88	58.17

Appendix

Table 6: Ablation study across different group sizes (g) in fixed configurations of LLaMA-2-7B. Lower Wiki PPL is better, while higher accuracy is better for downstream tasks. † The average bit includes weight and scale factors. The calculation for FluxBin is derived in Sec. C.2 . ‡ HS = HellaSwag, WG = Winogrande.

nsample	Time (min) $\downarrow$	Wiki $\downarrow$	PIQA $\uparrow$	ARC-e $\uparrow$	ARC-c $\uparrow$	BoolQ $\uparrow$	HS ‡ $\uparrow$	WG ‡ $\uparrow$	Avg. $\uparrow$
64	13.22	8.92	72.31	64.60	30.55	70.06	45.61	64.01	57.86
128	17.02	8.85	72.74	65.49	30.97	69.88	45.38	63.54	58.00
256	25.53	8.63	74.10	64.44	33.28	71.13	45.51	64.17	58.77
512	41.78	8.54	73.34	65.78	32.00	71.13	45.89	63.30	58.57
1024	74.58	8.76	72.63	65.40	30.80	69.82	45.67	64.09	58.07

Appendix

Table 7: Ablation study of the number of calibration samples (nsample) on evaluation time and performance on LLaMA-2-7B, with FluxBin configuration 2b-s8-g128. Lower PTQ time and Wiki PPL are better (

\downarrow

), while higher accuracy is better (

\uparrow

). ‡ HS = HellaSwag, WG = Winogrande.

Model	Quantization	Avg Power (W) $\downarrow$	Total Energy (J) $\downarrow$	Avg GPU Util. (%)	Avg Mem. Util. (%)
LLaMA-2-7B	FP16	168.66	3467.69	67.01	18.09
	FluxBin-2b-g128	106.75	462.98	99.83	5.55
	FluxBin-2b-s8-g128	110.80	425.05	99.81	6.27
	FluxBin-2b-s16-g128	107.40	434.34	99.87	6.27
	FluxBin-4b-g128	109.40	401.16	99.85	7.65
LLaMA-2-13B	FP16	225.49	5908.03	86.83	33.16

Appendix

Table 8: Power, energy, and resource utilization comparison on LLaMA-2 models. Lower power and energy are better (

\downarrow

); GPU and memory utilization are reported as diagnostics of the compute/bandwidth balance rather than as quality targets.

$\mu$	BS	Shared Mem. (KB)	Throughput $\uparrow$	Power (W) $\downarrow$	Energy (J) $\downarrow$
4	1	3	213.97	107.13	397.15
	4	12	656.50	146.97	802.74
	8	24	952.86	172.31	1301.12
8	1	17	213.96	105.41	451.18
	4	68	655.58	140.04	792.82
	8	136	953.04	169.48	1220.45

Appendix

Table 9: Impact of sub-vectors size

\mu

on block-level shared memory usage and end-to-end generation performance of FluxBin-2b-g128 LLaMA-2-7B. Higher throughput is better (

\uparrow

), while lower power and energy are better (

\downarrow

				FluxBin-2b-g128		FluxBin-4b-g128
M	K	$M_{\text{tile}}$	$K_{\text{tile}}$	Latency (µs) $\downarrow$	TFLOPS $\uparrow$	Latency (µs) $\downarrow$	TFLOPS $\uparrow$
4096	4096	32	32	$113.3$	$0.30$	$109.6$	$0.31$
		32	64	$143.0$	$0.23$	$108.4$	$0.31$
		32	128	$228.9$	$0.15$	$108.4$	$0.31$
		64	32	$58.0$	$0.58$	$72.9$	$0.46$
		64	64	$67.6$	$0.50$	$66.9$	$0.50$

Appendix

Table 10: Effect of tile sizes on FluxBin kernel performance.

M	K	$K_{\text{tile}}$	$M_{\text{tile}}$	$M_{s\_tile}$	Lat (µs) $\downarrow$	TFLOPS $\uparrow$	Lat (µs) $\downarrow$	TFLOPS $\uparrow$
					FluxBin-2b-s8-g128		FluxBin-2b-s16-g128
4096	4096	32	32	16	$121.0$	$0.29$	$124.4$	$0.28$
				32	$119.5$	$0.29$	$123.2$	$0.28$
				64	$118.3$	$0.29$	$121.9$	$0.28$
			64	16	$67.3$	$0.51$	$69.0$	$0.50$
				32	$65.7$	$0.53$	$67.9$	$0.51$

Appendix

Table 11: Effect of tile sizes on FluxBin kernel performance on hybrid bases.

Weight Size ( $M\times N$ )	Baselines		FluxBin
Weight Size ( $M\times N$ )	AQLM 2bit	QuIP# 2bit	2b-g128	2b-s8-g128	2b-s16-g128	4b-g128
4096 $\times$ 4096	79	112	41	47	49	56
5120 $\times$ 5120	111	174	63	73	74	87
4096 $\times$ 11008	169	272	102	109	111	158
5120 $\times$ 13824	277	411	167	166	169	252

Appendix

Table 12: Per-operator GEMV kernel latency (

\mu

s) across weight matrix sizes (

M\times N

), measured for the 2-bit quantized baselines and FluxBin. Lower is better (

\downarrow

); the best result in each row is in bold . Each kernel is measured in isolation with its official implementation.

Explore similar work

BWLA: Breaking the Barrier of W1AX Post-Training Quantization for LLMs
May 1, 2026Zhixiong Zhao, Zukang Xu, Dawei YangLLM Quantization LLM Inference Acceleration
1Houmo AI
All for 1-Bit: Towards Genuine 1-Bit Post-Training Quantization for LLMs
Sep 5, 2026Zhixiong Zhao, Zukang Xu, Guangyu Sun +2LLM Quantization Post-Training Quantization
Houmo AI · School of Integrated Circuits, Peking University
LBLLM: Lightweight Binarization of Large Language Models via Three-Stage Distillation
Apr 21, 2026Siqing Song, Chuang Wang, Yong Lang +2LLM Quantization Activation Quantization
MAIS, Institute of Automation, Chinese Academy of Sciences, China · School of Artificial Intelligence, University of Chinese Academy of Sciences, China · Central Media Technology Institute, Huawei

May 1, 2026J/K move · Enter open · S save

BWLA: Breaking the Barrier of W1AX Post-Training Quantization for LLMs

Zhixiong Zhao, Zukang Xu, Dawei Yang

1Houmo AI

LLM Quantization LLM Inference Acceleration Activation Quantization Post-Training Quantization Low-Bit Quantization

Open paper AskPDF arXiv

Large language models (LLMs) have driven major progress in NLP, yet their substantial memory and compute demands still hinder practical deployment. Binarization can compress weights to 1 bit, fundamentally lowering compute and bandwidth cost. However, existing methods cannot address activation heavy tails and thus must keep activations in high precision, preventing true end-to-end acceleration. To overcome this limitation, we propose BWLA (Binarized Weights and Low-bit Activations), the first post-training quantization framework that preserves high accuracy while achieving 1-bit weight quantization together with low-bit activations (e.g., 6 bits). The Orthogonal-Kronecker Transformation (OKT) learns an orthogonal mapping via EM minimization, converting unimodal weights into symmetric bimodal forms while suppressing activation tails and incoherence. The Proximal SVD Projection (PSP) then performs lightweight low-rank refinement through proximal SVD projection, further enhancing quantizability with minimal overhead. On Qwen3-32B, BWLA reaches a Wikitext2 perplexity of 11.92 under 6-bit activations (vs. 38 from SOTA), improves five zero-shot tasks by more than 70%, and delivers 3.26 times inference speedup, demonstrating strong potential for real-world LLM compression and acceleration.

Abstract

Explore similar work

BWLA: Breaking the Barrier of W1AX Post-Training Quantization for LLMs

All for 1-Bit: Towards Genuine 1-Bit Post-Training Quantization for LLMs

LBLLM: Lightweight Binarization of Large Language Models via Three-Stage Distillation