LoopQuant: Accurate and Efficient Post-Training Quantization for Looped Transformers

Zhixiong Zhao; Zukang Xu; Xing Hu; Lifeng Liu; Guangyu Sun; Dawei Yang

cs.LGOct 12, 2026

LoopQuant: Accurate and Efficient Post-Training Quantization for Looped Transformers

Authors: Zhixiong Zhao, Zukang Xu, Xing Hu, Lifeng Liu, Guangyu Sun, Dawei Yang

Organizations: School of Integrated Circuits, Peking University · Houmo AI

Abstract

Looped Language Models (LoopLMs) improve parameter efficiency by recursively reusing shared Transformer modules, yet their repeated computation over evolving recurrent states poses new challenges for low-bit deployment. Existing post-training quantization (PTQ) methods, largely designed for feed-forward Transformers, fail to account for cross-loop activation dynamics, heterogeneous loop sensitivity, and recursive error propagation, leading to substantial performance degradation at low precision. To address these challenges, we propose LoopQuant, a loop-aware PTQ framework for accurate and efficient low-bit quantization of LoopLMs. LoopQuant is built on three core components: 1) Cross-Loop Distribution Canonicalization (CLDC) learns a shared quantization-friendly representation across recurrent states while preserving parameter sharing; 2) Interaction-Aware Loop-wise Precision Allocation (ILPA) allocates activation precision under a fixed bit budget by modeling task-level loop sensitivity and cross-loop interactions; and 3) Loop-Conditioned Task Calibration (LCTC) further aligns quantized inference with the low-bit recursive trajectory through lightweight task-space calibration. Extensive experiments across LoopLM architectures demonstrate that LoopQuant consistently outperforms existing low-bit PTQ methods. On Ouro-2.6B, LoopQuant under W4A4 surpasses all W4A8 baselines in performance, while reducing model memory by 65.3% with only 3.4% additional storage overhead over naive INT4 and achieving a 1.7 times decoding speedup over BF16. These results demonstrate a strong balance among accuracy, compression, and inference efficiency, providing a practical and deployment-friendly solution for low-bit LoopLM inference. The code is available at [https://github.com/Kishon-zzx/LoopQuant].

Figures & tables

Figure 1: Accuracy retention of PTQ methods on four LoopLMs under different W/A precisions. LoopQuant consistently preserves higher accuracy, with W4A4 surpassing baselines at W4A8.

Figure 2: Overview of LoopQuant. LoopQuant consists of three stages: CLDC canonicalizes cross-loop activation distributions, ILPA allocates activation precision under a fixed bit budget, and LCTC aligns quantized models with recursive task behavior.

Method	W/A	Ouro-1.4B			Ouro-2.6B			Nanbeige4.2-3B			Huginn-0125
Method	W/A	Avg. $\uparrow$	WT2 $\downarrow$	C4 $\downarrow$	Avg. $\uparrow$	WT2 $\downarrow$	C4 $\downarrow$	Avg. $\uparrow$	WT2 $\downarrow$	C4 $\downarrow$	Avg. $\uparrow$	WT2 $\downarrow$	C4 $\downarrow$
BF16	16/16	66.85	8.60	12.61	71.12	7.61	11.42	58.68	24.17	31.45	49.89	12.10	16.28
SmoothQuant	4/8	60.65	11.25	15.38	67.12	9.48	13.46	56.10	28.63	32.38	46.95	13.41	17.65
QuaRot	4/8	62.53	10.78	15.21	68.30	9.09	13.02	53.83	34.09	41.54	47.73	13.38	17.63
SpinQuant	4/8	62.69	10.79	15.06	67.00	9.11	13.03	54.47	31.10	35.86	47.33	13.34	17.62
FlatQuant	4/8	61.37	10.65	14.84	67.31	9.26	13.18	55.14	25.99	32.93	47.31	13.45	17.71

Table 1: Comparison of LoopQuant with representative low-bit PTQ methods on LoopLMs. Bold and underline denote the best and second-best quantized results.

Method	W/A	Ouro-1.4B-thinking			Ouro-2.6B-thinking
Method	W/A	MMLU	GSM8K	HumanEval	MMLU	GSM8K	HumanEval
BF16	16/16	67.71	69.57	71.34	76.20	73.31	82.32
SmoothQuant	4/8	59.42	34.38	63.24	68.97	60.56	74.22
QuaRot	4/8	57.24	34.23	58.98	71.50	65.94	79.71
SpinQuant	4/8	56.68	53.25	60.80	71.26	63.67	75.44
FlatQuant	4/8	60.50	59.09	61.02	70.21	63.59	77.27

Table 2: Evaluation on reasoning-oriented LoopLMs.

W/A	CLDC			ILPA	LCTC	Storage Overhead	Wiki2	C4
W/A	$\mathbf{D}_{m}^{\star}$	$\mathbf{R}_{m}^{\star}$	$\mathbf{\Pi}_{m}^{\star}$	ILPA	LCTC	Storage Overhead	PPL ( $\downarrow$ )	PPL ( $\downarrow$ )
W4A6	✗	✗	✗	✗	✗	0	47.39	53.65
	✓	✗	✗			+1.42 MB (0.04%)	36.91	41.52
	✓	✓	✗			+46.84 MB (1.22%)	28.41	36.31
	✓	✓	✓			+52.51 MB (1.37%)	27.85	35.10
	✓			✓	✗	+52.51 MB (1.37%)	27.05	34.17

Table 3: Impact of different components in LoopQuant.

Model	Allocation	Loop-wise Bits	Wiki2	Avg.
Model	Strategy	( $t=1\rightarrow 6$ )	PPL ( $\downarrow$ )	Acc. $\uparrow$
Huginn-0125 ( $T=6$ )	Uniform	[6,6,6,6,6,6]	13.82	46.92
	Random	3 $\times$ A4 + 3 $\times$ A8	15.21	44.83
	Local Sensitivity	[4,8,4,8,8,4]	13.05	48.01
	Marginal Sensitivity	[4,8,8,4,4,8]	13.67	47.54
	ILPA	[4,4,4,8,8,8]	12.47	49.09

Table 4: Comparison of loop-wise bit allocation strategies.

Figure 3: Efficiency analysis. Left : Memory consumption and C4 perplexity of BF16, INT4, and LoopQuant across three LoopLMs, where gray regions denote the embedding and LM head. Right : Decoding throughput (Tokens/Sec) and average zero-shot accuracy under the same settings.

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

Model	Method	W/A	AE $\uparrow$	AC $\uparrow$	HS $\uparrow$	LO $\uparrow$	LS $\uparrow$	PQ $\uparrow$	WG $\uparrow$	Avg. 7 $\uparrow$
Ouro-1.4B	BF16	16/16	76.81	52.13	71.59	65.17	57.75	78.51	65.98	66.85
	SmoothQuant	4/8	69.42	46.21	66.33	54.69	48.67	75.31	63.90	60.65
	QuaRot	4/8	74.85	50.65	67.55	56.51	51.39	74.28	62.48	62.53
	SpinQuant	4/8	73.59	48.00	68.02	58.99	52.18	75.48	62.56	62.69
	FlatQuant	4/8	73.72	47.49	66.68	55.25	48.57	75.04	62.87	61.37
	DartQuant	4/8	71.27	46.21	65.44	56.02	47.74	75.20	61.77	60.52

Appendix

Table 5: Detailed zero-shot accuracy on ARC-Easy (AE), ARC-Challenge (AC), HellaSwag (HS), LAMBADA-openai (LO), LAMBADA-standard (LS), PIQA (PQ), and WinoGrande (WG). Bold and underline denote the best and second-best quantized results within each comparison group.

Model	W/A	Method	HS $\uparrow$	WG $\uparrow$	LB $\uparrow$	AC $\uparrow$	WT2 $\downarrow$
Ouro-1.4B	W4A8	LoopQ	69.54	64.56	62.55	51.45	13.29
	W4A6	LoopQuant	68.75	64.67	60.22	51.69	9.48
	W4A4	LoopQ	65.82	59.51	56.59	48.81	15.81
	W4A4	LoopQuant	68.94	63.37	59.36	51.42	9.86
Ouro-2.6B	W4A8	LoopQ	75.55	67.80	68.37	58.19	11.38
	W4A6	LoopQuant	76.25	68.26	68.04	57.45	8.04

Appendix

Table 6: Comparison with LoopQ under different low-bit quantization settings. LoopQ results are directly taken from its original paper due to the lack of public implementation. We only compare overlapping models and evaluation metrics.

Figure 4: Sensitivity of LoopQuant to calibration-data configurations. (a) WikiText2 perplexity with different numbers of calibration samples. (b) WikiText2 perplexity under different random sampling seeds. All other quantization hyperparameters are kept unchanged.

Model	Calibration Data	WikiText2 $\downarrow$	C4 $\downarrow$	Avg. Acc. $\uparrow$
Ouro-1.4B	WikiText2	9.86	14.00	64.04
Ouro-1.4B	C4	10.57	11.86	63.79
Ouro-2.6B	WikiText2	8.27	12.24	68.89
Ouro-2.6B	C4	9.41	10.15	68.07

Appendix

Table 7: Ablation on the choice of calibration corpus under W4A4. We compare WikiText2 and C4 calibration data on Ouro-1.4B and Ouro-2.6B. Bold and underline denote the best and second-best results within each model, respectively.

Method	Allocation	Wiki2 $\downarrow$	Avg. Acc. $\uparrow$
QuaRot	Uniform A6	14.02	46.28
QuaRot + ILPA	[4,4,4,8,8,8]	13.59	47.35
SpinQuant	Uniform A6	14.21	45.97
SpinQuant + ILPA	[4,4,4,8,8,8]	13.63	47.04
LoopQuant	ILPA + CLDC + LCTC	12.47	49.09

Appendix

Table 8: Effectiveness of ILPA with different quantization backbones on Huginn-0125 (

T=6

) under the same average activation precision budget of 6 bits.

W/A	LCTC			Storage Overhead	WikiText2 $\downarrow$	C4 $\downarrow$
W/A	$\mathbf{c}_{m,t}$	$\mathbf{g}_{m,t}$	$\bm{\beta}_{m,t}$	Storage Overhead	WikiText2 $\downarrow$	C4 $\downarrow$
W4A6	$\times$	$\times$	$\times$	0	27.05	34.17
	$\checkmark$	$\times$	$\times$	+0.49 MB (0.03%)	26.48	33.38
	$\checkmark$	$\checkmark$	$\times$	+0.97 MB (0.05%)	26.09	32.74
	$\checkmark$	$\checkmark$	$\checkmark$	+1.46 MB (0.08%)	25.84	32.19

Appendix

Table 9: Ablation of the three calibration parameters in LCTC under W4A6. We progressively introduce clipping, gain, and bias while keeping CLDC and ILPA fixed. The storage overhead is measured relative to the W4 base model.

Panel of Figure 5 — Figure 5: Layer-wise relative $\ell_{2}$ representation error on GSM8K under W4A4 quantization for Ouro-1.4B-thinking (top) and Ouro-2.6B-thinking (bottom). The error is measured against the corresponding BF16 hidden representation at each decoder layer. Dashed vertical lines indicate the boundaries between successive recurrent passes in the unrolled reasoning computation.

Panel of Figure 6 — Figure 6: Channel-wise input activation distributions of the $q$ projection (top) and $k$ projection (bottom) in the first recurrent layer of Ouro-1.4B. Within each image, the left and right columns show the raw input and the corresponding representation after CLDC, respectively, while the four rows correspond to different recurrent loops.

Panel of Figure 7 — Figure 7: Channel-wise input activation distributions of the $v$ projection (top) and $o$ projection (bottom) in the first recurrent layer of Ouro-1.4B, before and after CLDC. The four rows correspond to different recurrent loops. CLDC substantially suppresses channel-wise outliers and produces more balanced activation ranges across hidden dimensions.

Panel of Figure 8 — Figure 8: Channel-wise input activation distributions of the MLP gate projection (top) and up projection (bottom) in the first recurrent layer of Ouro-1.4B. Raw activations exhibit sparse but large channel-wise outliers across recurrent loops, whereas CLDC produces substantially more compact and homogeneous percentile envelopes.

Figure 9: Channel-wise input activation distributions of the MLP down projection in the first recurrent layer of Ouro-1.4B. The left and right columns show the raw input and the representation after CLDC, respectively, across four recurrent loops. The raw input contains highly concentrated channel-wise outliers, including extreme values above 20 in the first loop, while most percentile statistics remain close to zero. CLDC strongly suppresses these extreme channels and transforms the activation space into a substantially more compact and balanced distribution.

Post-Training Quantization

Post-Training Quantization compresses trained neural networks into low-bit weights, activations, or caches to reduce memory use and inference cost.

See all 366 papers

LoopQ: Quantization for Looped Language Models
May 8, 2026Rui Fang, Hsi-Wen Chen, Chia-Ming Cheng +1Recurrent Transformers Language Modeling
National Taiwan University
Loopy: Low-Bit Quantization Framework for Looped Language Models
Oct 4, 2026Zeyu LI, Yipu ZHANG, Jintao Chen +2LLM Quantization Post-Training Quantization
The Hong Kong University of Science and Technology · Duke Kunshan University
ResidualQuant: KV Cache Quantization for Looped Transformers with 2-Bit Residuals
Oct 7, 2026Heejun Kim, Junyoung Lee, SangLyul Cho +3KV-Cache Quantization Efficient Transformer Inference
KAIST · Yonsei University · Seoul National University

May 8, 2026J/K move · Enter open · S save

LoopQ: Quantization for Looped Language Models

Rui Fang, Hsi-Wen Chen, Chia-Ming Cheng, Ming-Syan Chen

National Taiwan University

Recurrent Transformers Language Modeling Activation Quantization Post-Training Quantization Low-Bit Quantization Recursive Language Models

Open paper AskPDF arXiv

Looped language models (LoopLMs) increase effective computational depth by repeatedly applying shared Transformer blocks without increasing the parameter count. However, this reuse makes them particularly sensitive to post-training quantization (PTQ). We systematically study PTQ in LoopLMs and identify three sources of degradation: distribution shift across roles, state reuse across loop transitions, and recursive error accumulation. We propose LoopQ, a loop-aware PTQ framework that preserves a shared quantized backbone while introducing lightweight adaptations. LoopQ combines loop-aware activation scaling and selective transformations to accommodate changing hidden-state distributions, cross-loop state alignment to stabilize transitions, and trajectory-aware calibration to limit accumulated error. Under W4A4 quantization, LoopQ achieves a 67.8% average relative improvement in five-task mean accuracy across four backbones and a conservative estimate of 75.2% for the average relative perplexity reduction across eight backbone-dataset pairs, compared with the strongest static PTQ baseline for each comparison.

Abstract

Post-Training Quantization

Explore similar work

LoopQ: Quantization for Looped Language Models

Loopy: Low-Bit Quantization Framework for Looped Language Models

ResidualQuant: KV Cache Quantization for Looped Transformers with 2-Bit Residuals