Looped Language Models (LoopLMs) improve parameter efficiency by recursively reusing shared Transformer modules, yet their repeated computation over evolving recurrent states poses new challenges for low-bit deployment. Existing post-training quantization (PTQ) methods, largely designed for feed-forward Transformers, fail to account for cross-loop activation dynamics, heterogeneous loop sensitivity, and recursive error propagation, leading to substantial performance degradation at low precision. To address these challenges, we propose LoopQuant, a loop-aware PTQ framework for accurate and efficient low-bit quantization of LoopLMs. LoopQuant is built on three core components: 1) Cross-Loop Distribution Canonicalization (CLDC) learns a shared quantization-friendly representation across recurrent states while preserving parameter sharing; 2) Interaction-Aware Loop-wise Precision Allocation (ILPA) allocates activation precision under a fixed bit budget by modeling task-level loop sensitivity and cross-loop interactions; and 3) Loop-Conditioned Task Calibration (LCTC) further aligns quantized inference with the low-bit recursive trajectory through lightweight task-space calibration. Extensive experiments across LoopLM architectures demonstrate that LoopQuant consistently outperforms existing low-bit PTQ methods. On Ouro-2.6B, LoopQuant under W4A4 surpasses all W4A8 baselines in performance, while reducing model memory by 65.3% with only 3.4% additional storage overhead over naive INT4 and achieving a 1.7 times decoding speedup over BF16. These results demonstrate a strong balance among accuracy, compression, and inference efficiency, providing a practical and deployment-friendly solution for low-bit LoopLM inference. The code is available at [https://github.com/Kishon-zzx/LoopQuant].
Figures & tables
Figure 1: Accuracy retention of PTQ methods on four LoopLMs under different W/A precisions. LoopQuant consistently preserves higher accuracy, with W4A4 surpassing baselines at W4A8.
Figure 2: Overview of LoopQuant. LoopQuant consists of three stages: CLDC canonicalizes cross-loop activation distributions, ILPA allocates activation precision under a fixed bit budget, and LCTC aligns quantized models with recursive task behavior.
Method
W/A
Ouro-1.4B
Ouro-2.6B
Nanbeige4.2-3B
Huginn-0125
Avg. ↑
WT2 ↓
C4 ↓
Avg. ↑
WT2 ↓
C4 ↓
Avg. ↑
WT2 ↓
C4 ↓
Avg. ↑
WT2 ↓
C4 ↓
BF16
16/16
66.85
8.60
12.61
71.12
7.61
11.42
58.68
24.17
31.45
49.89
12.10
16.28
SmoothQuant
4/8
60.65
11.25
15.38
67.12
9.48
13.46
56.10
28.63
32.38
46.95
13.41
17.65
QuaRot
4/8
62.53
10.78
15.21
68.30
9.09
13.02
53.83
34.09
41.54
47.73
13.38
17.63
SpinQuant
4/8
62.69
10.79
15.06
67.00
9.11
13.03
54.47
31.10
35.86
47.33
13.34
17.62
FlatQuant
4/8
61.37
10.65
14.84
67.31
9.26
13.18
55.14
25.99
32.93
47.31
13.45
17.71
Table 1: Comparison of LoopQuant with representative low-bit PTQ methods on LoopLMs. Bold and underline denote the best and second-best quantized results.
Method
W/A
Ouro-1.4B-thinking
Ouro-2.6B-thinking
MMLU
GSM8K
HumanEval
MMLU
GSM8K
HumanEval
BF16
16/16
67.71
69.57
71.34
76.20
73.31
82.32
SmoothQuant
4/8
59.42
34.38
63.24
68.97
60.56
74.22
QuaRot
4/8
57.24
34.23
58.98
71.50
65.94
79.71
SpinQuant
4/8
56.68
53.25
60.80
71.26
63.67
75.44
FlatQuant
4/8
60.50
59.09
61.02
70.21
63.59
77.27
Table 2: Evaluation on reasoning-oriented LoopLMs.
W/A
CLDC
ILPA
LCTC
Storage Overhead
Wiki2
C4
Dm⋆
Rm⋆
Πm⋆
PPL ( ↓ )
PPL ( ↓ )
W4A6
✗
✗
✗
✗
✗
0
47.39
53.65
✓
✗
✗
+1.42 MB (0.04%)
36.91
41.52
✓
✓
✗
+46.84 MB (1.22%)
28.41
36.31
✓
✓
✓
+52.51 MB (1.37%)
27.85
35.10
✓
✓
✗
+52.51 MB (1.37%)
27.05
34.17
Table 3: Impact of different components in LoopQuant.
Model
Allocation
Loop-wise Bits
Wiki2
Avg.
Strategy
( t=1→6 )
PPL ( ↓ )
Acc. ↑
Huginn-0125 ( T=6 )
Uniform
[6,6,6,6,6,6]
13.82
46.92
Random
3 × A4 + 3 × A8
15.21
44.83
Local Sensitivity
[4,8,4,8,8,4]
13.05
48.01
Marginal Sensitivity
[4,8,8,4,4,8]
13.67
47.54
ILPA
[4,4,4,8,8,8]
12.47
49.09
Table 4: Comparison of loop-wise bit allocation strategies.
Figure 3: Efficiency analysis. Left : Memory consumption and C4 perplexity of BF16, INT4, and LoopQuant across three LoopLMs, where gray regions denote the embedding and LM head. Right : Decoding throughput (Tokens/Sec) and average zero-shot accuracy under the same settings.
Appendix figures & tables11 assets
Supplementary material from the paper’s appendix.
Appendix
Model
Method
W/A
AE ↑
AC ↑
HS ↑
LO ↑
LS ↑
PQ ↑
WG ↑
Avg. 7 ↑
Ouro-1.4B
BF16
16/16
76.81
52.13
71.59
65.17
57.75
78.51
65.98
66.85
SmoothQuant
4/8
69.42
46.21
66.33
54.69
48.67
75.31
63.90
60.65
QuaRot
4/8
74.85
50.65
67.55
56.51
51.39
74.28
62.48
62.53
SpinQuant
4/8
73.59
48.00
68.02
58.99
52.18
75.48
62.56
62.69
FlatQuant
4/8
73.72
47.49
66.68
55.25
48.57
75.04
62.87
61.37
DartQuant
4/8
71.27
46.21
65.44
56.02
47.74
75.20
61.77
60.52
Appendix
Table 5: Detailed zero-shot accuracy on ARC-Easy (AE), ARC-Challenge (AC), HellaSwag (HS), LAMBADA-openai (LO), LAMBADA-standard (LS), PIQA (PQ), and WinoGrande (WG). Bold and underline denote the best and second-best quantized results within each comparison group.
Model
W/A
Method
HS ↑
WG ↑
LB ↑
AC ↑
WT2 ↓
Ouro-1.4B
W4A8
LoopQ
69.54
64.56
62.55
51.45
13.29
W4A6
LoopQuant
68.75
64.67
60.22
51.69
9.48
W4A4
LoopQ
65.82
59.51
56.59
48.81
15.81
W4A4
LoopQuant
68.94
63.37
59.36
51.42
9.86
Ouro-2.6B
W4A8
LoopQ
75.55
67.80
68.37
58.19
11.38
W4A6
LoopQuant
76.25
68.26
68.04
57.45
8.04
Appendix
Table 6: Comparison with LoopQ under different low-bit quantization settings. LoopQ results are directly taken from its original paper due to the lack of public implementation. We only compare overlapping models and evaluation metrics.
Figure 4: Sensitivity of LoopQuant to calibration-data configurations. (a) WikiText2 perplexity with different numbers of calibration samples. (b) WikiText2 perplexity under different random sampling seeds. All other quantization hyperparameters are kept unchanged.
Model
Calibration Data
WikiText2 ↓
C4 ↓
Avg. Acc. ↑
Ouro-1.4B
WikiText2
9.86
14.00
64.04
C4
10.57
11.86
63.79
Ouro-2.6B
WikiText2
8.27
12.24
68.89
C4
9.41
10.15
68.07
Appendix
Table 7: Ablation on the choice of calibration corpus under W4A4. We compare WikiText2 and C4 calibration data on Ouro-1.4B and Ouro-2.6B. Bold and underline denote the best and second-best results within each model, respectively.
Method
Allocation
Wiki2 ↓
Avg. Acc. ↑
QuaRot
Uniform A6
14.02
46.28
QuaRot + ILPA
[4,4,4,8,8,8]
13.59
47.35
SpinQuant
Uniform A6
14.21
45.97
SpinQuant + ILPA
[4,4,4,8,8,8]
13.63
47.04
LoopQuant
ILPA + CLDC + LCTC
12.47
49.09
Appendix
Table 8: Effectiveness of ILPA with different quantization backbones on Huginn-0125 ( T=6 ) under the same average activation precision budget of 6 bits.
W/A
LCTC
Storage Overhead
WikiText2 ↓
C4 ↓
cm,t
gm,t
βm,t
W4A6
×
×
×
0
27.05
34.17
✓
×
×
+0.49 MB (0.03%)
26.48
33.38
✓
✓
×
+0.97 MB (0.05%)
26.09
32.74
✓
✓
✓
+1.46 MB (0.08%)
25.84
32.19
Appendix
Table 9: Ablation of the three calibration parameters in LCTC under W4A6. We progressively introduce clipping, gain, and bias while keeping CLDC and ILPA fixed. The storage overhead is measured relative to the W4 base model.
Figure 5: Layer-wise relative ℓ2 representation error on GSM8K under W4A4 quantization for Ouro-1.4B-thinking (top) and Ouro-2.6B-thinking (bottom). The error is measured against the corresponding BF16 hidden representation at each decoder layer. Dashed vertical lines indicate the boundaries between successive recurrent passes in the unrolled reasoning computation.
Figure 6: Channel-wise input activation distributions of the q projection (top) and k projection (bottom) in the first recurrent layer of Ouro-1.4B. Within each image, the left and right columns show the raw input and the corresponding representation after CLDC, respectively, while the four rows correspond to different recurrent loops.
Figure 7: Channel-wise input activation distributions of the v projection (top) and o projection (bottom) in the first recurrent layer of Ouro-1.4B, before and after CLDC. The four rows correspond to different recurrent loops. CLDC substantially suppresses channel-wise outliers and produces more balanced activation ranges across hidden dimensions.
Figure 8: Channel-wise input activation distributions of the MLP gate projection (top) and up projection (bottom) in the first recurrent layer of Ouro-1.4B. Raw activations exhibit sparse but large channel-wise outliers across recurrent loops, whereas CLDC produces substantially more compact and homogeneous percentile envelopes.
Figure 9: Channel-wise input activation distributions of the MLP down projection in the first recurrent layer of Ouro-1.4B. The left and right columns show the raw input and the representation after CLDC, respectively, across four recurrent loops. The raw input contains highly concentrated channel-wise outliers, including extreme values above 20 in the first loop, while most percentile statistics remain close to zero. CLDC strongly suppresses these extreme channels and transforms the activation space into a substantially more compact and balanced distribution.
Looped language models (LoopLMs) increase effective computational depth by repeatedly applying shared Transformer blocks without increasing the parameter count. However, this reuse makes them particularly sensitive to post-training quantization (PTQ). We systematically study PTQ in LoopLMs and identify three sources of degradation: distribution shift across roles, state reuse across loop transitions, and recursive error accumulation. We propose LoopQ, a loop-aware PTQ framework that preserves a shared quantized backbone while introducing lightweight adaptations. LoopQ combines loop-aware activation scaling and selective transformations to accommodate changing hidden-state distributions, cross-loop state alignment to stabilize transitions, and trajectory-aware calibration to limit accumulated error. Under W4A4 quantization, LoopQ achieves a 67.8% average relative improvement in five-task mean accuracy across four backbones and a conservative estimate of 75.2% for the average relative perplexity reduction across eight backbone-dataset pairs, compared with the strongest static PTQ baseline for each comparison.
Looped language models provide a parameter-efficient way to scale iterative test-time computation by repeatedly executing a shared recurrent core. Post-training quantization (PTQ) can reduce the memory footprint and inference cost of looped language models, but errors introduced by a quantized shared core affect subsequent cores. Among PTQ methods, channel scaling and orthogonal rotations preserve the floating-point computation while producing representations with different quantization quality. We find that quantization configuration candidate rankings can change with recurrent depth, motivating configuration selection at the target deployment depth. However, evaluating every candidate over the full calibration set at this depth is costly. We therefore propose Loopy, a PTQ framework that formulates shared-core quantization through a recurrent-depth-aware objective, selecting shared low-bit representations by their final prediction loss at the target deployment depth. Channel scaling and orthogonal rotations parameterize the candidate representations. To approximately solve this selection problem efficiently, Loopy progressively allocates calibration windows to promising candidates while preserving complete target-depth execution, using only forward evaluations. Across eight settings, Loopy achieves the state-of-the-art results among different baselines. On Ouro-1.4B under W4A4, Loopy reduces LAMBADA perplexity by 36.5% relative to SpinQuant. Our code is available at https://github.com/Shameless0817/Loopy-review.git.
Zeyu LI, Yipu ZHANG, Jintao Chen +2
The Hong Kong University of Science and Technology · Duke Kunshan University
Looped Transformers improve parameter efficiency by repeatedly applying shared Transformer blocks over multiple recurrent loops, increasing computational depth without increasing the parameter count. However, KV cache memory still scales with the number of loops, becoming a key memory bottleneck that limits batch size and inference throughput. KV cache quantization can alleviate this bottleneck, but existing methods often suffer substantial accuracy degradation at aggressive low-precision regimes. We observe that looped Transformers offer a unique opportunity: KV states across loops are highly similar. Based on this observation, we propose ResidualQuant, which uses the final-loop KV states as a reference and represents the remaining loops with low-precision residuals. Our method further combines least-square scaling and rotations applied to the residuals, as well as loop-wise mixed precision, to enable accurate quantization down to INT2 while retaining efficient reconstruction. Across multiple looped Transformer models and mathematical reasoning and code generation benchmarks, ResidualQuant consistently improves the accuracy-memory tradeoff over state-of-the-art rotation-based KV quantization. In particular, our method retains accuracy close to BF16 under mixed-precision settings while reducing theoretical KV storage by 80.7%, achieving up to 13.0% higher accuracy than the rotation-based baseline at the same memory budget. On an RTX 5090, the reduced KV memory traffic improves fixed-batch decode throughput by up to 2.73x, while the smaller memory footprint enables up to 2x larger batches, improving peak throughput by up to 4.15x.
Heejun Kim, Junyoung Lee, SangLyul Cho +3
KAIST · Yonsei University · Seoul National University