Loopy: Low-Bit Quantization Framework for Looped Language Models

Zeyu LI; Yipu ZHANG; Jintao Chen; Xin LI; Wei ZHANG

cs.LGOct 4, 2026

Loopy: Low-Bit Quantization Framework for Looped Language Models

Authors: Zeyu LI, Yipu ZHANG, Jintao Chen, Xin LI, Wei ZHANG

Organizations: The Hong Kong University of Science and Technology · Duke Kunshan University

Abstract

Looped language models provide a parameter-efficient way to scale iterative test-time computation by repeatedly executing a shared recurrent core. Post-training quantization (PTQ) can reduce the memory footprint and inference cost of looped language models, but errors introduced by a quantized shared core affect subsequent cores. Among PTQ methods, channel scaling and orthogonal rotations preserve the floating-point computation while producing representations with different quantization quality. We find that quantization configuration candidate rankings can change with recurrent depth, motivating configuration selection at the target deployment depth. However, evaluating every candidate over the full calibration set at this depth is costly. We therefore propose Loopy, a PTQ framework that formulates shared-core quantization through a recurrent-depth-aware objective, selecting shared low-bit representations by their final prediction loss at the target deployment depth. Channel scaling and orthogonal rotations parameterize the candidate representations. To approximately solve this selection problem efficiently, Loopy progressively allocates calibration windows to promising candidates while preserving complete target-depth execution, using only forward evaluations. Across eight settings, Loopy achieves the state-of-the-art results among different baselines. On Ouro-1.4B under W4A4, Loopy reduces LAMBADA perplexity by 36.5% relative to SpinQuant. Our code is available at https://github.com/Shameless0817/Loopy-review.git.

Figures & tables

Figure 2: Loopy method overview. Top: channel scaling and rotation define quantized representations shared across recurrent steps. Bottom: progressive selection allocates more calibration windows to surviving candidates while preserving the target depth $T$ .

Panel of Figure 3 — Figure 3: Rotation improves activation-side quantization on Ouro-1.4B. (a) Attention and feed-forward network (FFN) inputs of physical block 22 at recurrent step 4, before and after rotation. Appendix F provides all 24 blocks. (b) Activation normalized mean-squared error (NMSE) distributions without and with rotation, on a logarithmic scale.

Figure 4: Accuracy across recurrent depths. WinoGrande accuracy (%) for 20 configurations of Ouro-1.4B W4A4. Columns identify scaling power $p$ and rotation seed $s$ and are ordered by decreasing accuracy at $T=4$ . Rows show recurrent depth. A shared color scale preserves absolute differences across depths. Appendix H shows the corresponding within-depth rankings.

Figure 5: Operator-wise quantization transforms . Channel scaling and rotation in attention and FFN, with weight-side transforms merged before quantization and input transforms applied online.

	Ouro-2.6B			Ouro-1.4B			Huginn-0125			Recurrent-Llama-3.2-1B
Method	WT2	LMB	Avg.	WT2	LMB	Avg.	WT2	LMB	Avg.	WT2	LMB	Avg.
FP ref.	10.51	4.23	69.65	12.03	5.29	64.66	13.21	7.30	50.42	20.72	17.23	38.63
W4A8
AWQ	11.46	5.22	66.45	13.64	7.68	60.51	14.14	8.85	47.37	28.16	50.70	35.10
SmoothQuant	19.44	15.20	56.15	53.27	134.60	37.78	—	—	25.82	243.77	—	26.40
QuaRot	11.76	5.30	66.76	14.25	8.29	59.09	—	—	25.61	26.03	31.15	36.60

Table 1: W4A8 and W4A4 results on looped language models. WT2 and LMB denote WikiText-2 and LAMBADA perplexity (

\downarrow

); Avg. is mean task accuracy (%,

\uparrow

). Detailed accuracy results are shown in Tables 5 and 6 . Bold indicates the best low-bit result per model and precision.

	WT2 PPL $\downarrow$		Avg. Acc. (%) $\uparrow$
Model	LoopQ †	Loopy	LoopQ †	Loopy
Ouro-1.4B	15.81	14.73	57.08	57.75
Ouro-2.6B	15.89	12.35	58.13	64.37

Table 2: W4A4 comparison with LoopQ.

\dagger

: published results ( Fang et al., 2026 ) ; Avg. Acc. averages the five tasks in Table 1 .

Figure 6: Selection quality–calibration cost trade-off. WikiText-2 perplexity versus calibration time for (a) Ouro-1.4B W4A4 ( $T=4$ ) and (b) Huginn-0125 W4A8 ( $T=32$ ). Black stars denote exhaustive-search references. Lower perplexity is better.

W/A	KV setting	PPL $\downarrow$	$\Delta$ PPL (%)	Avg. Acc. (%) $\uparrow$
W4A4	BF16 KV	11.4174	—	54.5266
	BF16 KV+ $H$	11.4139	$-0.03$	54.4818
	KV8	11.4109	$-0.06$	54.1423
	KV4	12.1660	$+6.56$	51.9853
	KV4+ $H$	11.8062	$+3.41$	53.3883
W4A8	BF16 KV	9.3425	—	61.7154

Table 3: KV quantization results.

\Delta

PPL is relative to BF16 KV within each W/A setting; Avg. Acc. averages four tasks.

H

denotes the Q/K Hadamard transform.

	W4A4		W4A8
Method	B1/128	B4/2048	B1/128	B4/2048
QuaRot	$9.08^{*}$	$35.14^{*}$	8.80	34.75
SpinQuant	$9.16^{*}$	$35.85^{*}$	9.14	35.78
Loopy (online)	10.01	40.93	10.22	41.17
Loopy (offline) †	11.21	45.45	11.37	45.83

Table 4: Decoding throughput on NVIDIA H20 (tokens/s). B/input denotes batch size and input length. Protocol details and additional configurations appear in Appendix D .

Appendix figures & tables15 assets

Supplementary material from the paper’s appendix.

Appendix

Model	Method	WT2 PPL	LMB PPL	HS	WG	LMB Acc.	ARC-C	MMLU	Avg.
Ouro- 2.6B	FP ref.	10.51	4.23	76.45	69.53	70.08	58.36	73.83	69.65
	AWQ	11.4603	5.2171	74.71	67.80	64.22	55.89	69.63	66.45
	SmoothQuant †	19.4442	15.2011	68.29	61.96	45.37	50.26	54.85	56.15
	QuaRot †	11.7643	5.3048	74.13	68.43	64.70	56.83	69.69	66.76
	SpinQuant †	12.0654	5.5639	74.28	66.93	63.13	57.76	68.93	66.21
	FlatQuant †	14.5179	8.1443	71.83	66.69	56.12	54.10	63.35	62.42

Appendix

Table 5: Complete W4A8 results. HS: HellaSwag; WG: WinoGrande; LMB: LAMBADA.

\dagger

: adapted baselines.

Model	Method	WT2 PPL	LMB PPL	HS	WG	LMB Acc.	ARC-C	MMLU	Avg.
Ouro- 2.6B	FP ref.	10.51	4.23	76.45	69.53	70.08	58.36	73.83	69.65
	AWQ	340.35	$1.35\!\times\!10^{4}$	30.5	49.7	1.9	24.5	23.4	26.00
	SmoothQuant	$1.01\!\times\!10^{4}$	$1.25\!\times\!10^{6}$	26.3	48.4	0.0	24.7	23.2	24.52
	QuaRot	15.27	9.73	69.1	60.9	52.1	49.2	53.4	56.94
	SpinQuant	16.62	8.18	70.4	63.1	56.2	53.5	61.2	60.88
	FlatQuant	$1.57\!\times\!10^{4}$	$3.77\!\times\!10^{6}$	26.2	47.9	0.0	24.9	23.8	24.56

Appendix

Table 6: Complete W4A4 results. Avg. is the arithmetic mean of the five displayed task accuracies (%). Large perplexities are shown in scientific notation.

W/A	KV setting	ARC-C	WinoGrande	LAMBADA	HellaSwag
W4A4	BF16 KV	44.20	62.51	45.97	65.43
	BF16 KV+ $H$	44.97	60.85	46.59	65.51
	KV8	45.65	58.96	46.48	65.48
	KV4	40.78	58.64	44.34	64.17
	KV4+ $H$	40.96	61.88	45.68	65.04
W4A8	BF16 KV	51.02	64.56	61.48	69.80

Appendix

Table 7: Per-task accuracy (%) for KV quantization.

H

denotes the Q/K Hadamard transform.

Model	BF16	W4A4	W3A4	W3A3
Ouro-1.4B	12.03	14.73	23.77	1653.88
Ouro-2.6B	10.51	12.35	17.65	361.85

Appendix

Table 8: WikiText-2 perplexity under lower-bit quantization. All quantized results use Loopy. Lower is better.

Figure 7: Calibration depth at fixed deployment depth. Token-normalized WikiText-2 perplexity with 2048-token windows for (a) Ouro-2.6B W4A8, (b) Ouro-1.4B W4A4, and (c) Huginn-0125 W4A8. Orange markers indicate matched calibration and deployment depths. Each configuration has one run; no error bars are shown. Vertical scales differ across panels.

	W4A4		W4A8
Method	B1/128	B4/2048	B1/128	B4/2048
QuaRot	$9.08^{*}$	$35.14^{*}$	8.80	34.75
SpinQuant	$9.16^{*}$	$35.85^{*}$	9.14	35.78
Loopy (online)	10.01	40.93	10.22	41.17
Loopy (offline)	11.21	45.45	11.37	45.83
+ down rotation	11.17	45.56	11.37	45.60

Appendix

Table 9: Complete H20 decoding throughput (tokens/s,

\uparrow

). The last two rows add online rotations to Loopy (offline).

Figure 8: Activation distributions in blocks 1–4 of Ouro-1.4B at recurrent step 4. Each row shows attention input, rotated attention input, FFN input, and rotated FFN input, from left to right.

Figure 9: Activation distributions in blocks 5–8 of Ouro-1.4B at recurrent step 4. Each row shows attention input, rotated attention input, FFN input, and rotated FFN input, from left to right.

Figure 10: Activation distributions in blocks 9–12 of Ouro-1.4B at recurrent step 4. Each row shows attention input, rotated attention input, FFN input, and rotated FFN input, from left to right.

Figure 11: Activation distributions in blocks 13–16 of Ouro-1.4B at recurrent step 4. Each row shows attention input, rotated attention input, FFN input, and rotated FFN input, from left to right.

Figure 12: Activation distributions in blocks 17–20 of Ouro-1.4B at recurrent step 4. Each row shows attention input, rotated attention input, FFN input, and rotated FFN input, from left to right.

Figure 13: Activation distributions in blocks 21–24 of Ouro-1.4B at recurrent step 4. Each row shows attention input, rotated attention input, FFN input, and rotated FFN input, from left to right.

Figure 14: Activation distributions across recurrent steps. Input activations to the query projection in physical block 12 of Ouro-1.4B for the same input window. Panels labeled Step 0–3 correspond to the first through fourth recurrent calls. The overall activation structure remains similar, while local peaks and maximum magnitudes vary across calls despite shared weights.

Panel of Figure 15 — Figure 15: Outlier-channel stability at the block-12 query-projection input. Left to right, stepwise Top-1%/5% channel overlap, maximum absolute activation for fixed channels (logarithmic color scale), and token exceedance frequency. The frequency threshold is 1.84375; the four-step common Max Top-1% fraction is 81.0%.

Figure 17: Within-depth configuration rankings. WinoGrande accuracy ranks for Ouro-1.4B W4A4, using the same candidate pool and column order as Figure 4 . Darker cells indicate better ranks (1 is best); stars mark the displayed top-ranked configuration at each depth.

Explore similar work

LoopQ: Quantization for Recursive Transformers
May 8, 2026Rui Fang, Hsi-Wen Chen, Ming-Syan ChenPost-Training Quantization Transformer Architectures
National Taiwan University
Closing the Loop: Practical Training Recipes for Looped Language Models
Sep 30, 2026Andrei Marchenko, Viacheslav Bezrukov, Oleg Kashurin +5Recurrent Model Loop
RND NLP, DAIMLD, Russian Federation
Recurrent Residual Quantization: A Progressive Multi-Precision Representation for LLMs
Aug 4, 2026Yu Luo, Bo Dong, Wenhua Cheng +1Large Language Model Quantization Nvidia
Intel

May 8, 2026, cs.LGJ/K move · Enter open · S save

LoopQ: Quantization for Recursive Transformers

Rui Fang, Hsi-Wen Chen, Ming-Syan Chen

National Taiwan University

Post-Training Quantization Transformer Architectures

Open paper AskPDF arXiv

Looped language models (LoopLMs) improve parameter efficiency by recursively reusing Transformer blocks, enabling deeper computation under a fixed model size. However, this reuse makes LoopLMs more fragile under post-training quantization (PTQ). We present the first systematic study of quantization in LoopLMs and identify three challenges: distribution shift across roles, state reuse across loop transitions, and recursive error accumulation. To address these challenges, we propose LoopQ, a loop-aware PTQ framework that preserves a shared quantized backbone while introducing lightweight adaptations. LoopQ combines activation scaling, selective transformation, cross-loop state alignment, and trajectory-aware optimization to reduce distributional mismatch within loops and error accumulation across loops. Experiments across seven benchmarks show that, under W4A4 quantization, LoopQ improves average downstream accuracy by 68.8% and reduces average perplexity by 87.7% compared with the strongest static PTQ baseline.

Abstract

Explore similar work

LoopQ: Quantization for Recursive Transformers

Closing the Loop: Practical Training Recipes for Looped Language Models

Recurrent Residual Quantization: A Progressive Multi-Precision Representation for LLMs