Quantizing Looped Transformers: Feedback Exposure and Calibration Blindness

Nux Li

cs.LGSep 25, 2026

Quantizing Looped Transformers: Feedback Exposure and Calibration Blindness

Authors: Nux Li

Organizations: Meta

Abstract

Looped transformers reuse weights across recurrence steps, making low-bit quantization especially attractive. We identify two distinct failure modes of standard post-training quantization. On Huginn-3.5B, per-channel INT4 fails primarily at the non-residual loop-entry adapter, while quantizing the residual core is much less damaging. We call this feedback exposure: a quantized layer perturbs the recurrent state without an identity path, and the resulting error is fed back at later steps. Controlled experiments on linear filters and Mamba state-space models show that feedback exposure also occurs outside transformers. Grouped INT4 reveals a separate failure, calibration blindness: our one-step GPTQ baseline builds its Hessian from step-0 activations, leaving input directions used later in the recurrence nearly unweighted. Across nine checkpoints from seven looped architectures, one-step GPTQ is worse than round-to-nearest (RTN) on the primary task metric for five checkpoints. Accumulating the GPTQ Hessian across recurrence steps outperforms both one-step GPTQ and RTN on all nine checkpoints and recovers bf16-level accuracy on Huginn. These results separate two questions for PTQ on looped models: where quantization error enters the recurrence, and which states calibration sees.

Figures & tables

Figure 1: Per-channel INT4 feedback exposure. (a) Huginn’s non-residual adapter-only agreement falls to 10.5%, while quantizing all 16 residual-core projections together retains 87.9%. Two bf16 runs with independently sampled initial states reach 98.9% agreement. (b) On 15 fixed prompts, COCONUT agreement rises from 44.8% after the quantized base forward to 61.5% after 8 latent recurrences.

Layer	Dimension	1-step rank	1-step %	32-step rank	Gain
Adapter	10,560	43	0.41%	335	7.8 $\times$
Core Wqkv (avg)	5,280	$\sim$ 1,050	20%	$\sim$ 1,450	1.3 $\times$
mlp.proj (block 3)	17,920	65	0.36%	89	1.4 $\times$

Table 1: Per-layer thresholded Hessian rank at

10^{-3}\lambda_{\max}

on 50 GSM8K prompts for Huginn-3.5B. The adapter’s step-0 Hessian covers only 43 of 10,560 input dimensions. Accumulating 32 recurrence steps raises this to 335.

Figure 2: Maximum accumulated-to-step-0 Hessian rank ratio across shared layers versus the RTN-to-full-precision gap closed by trajectory calibration. Marker shape shows whether one-step GPTQ outperforms RTN. The hollow COCONUT point is an in-sample top-1 diagnostic (

n{=}30

); all 9 filled points have rank gain

\geq 2\times

Figure 3: Huginn-3.5B calibration horizon (grouped INT4,

g{=}128

). (a) Adapter Hessian rank. (b) Exact match on GSM8K test IDs 0–99, held out from calibration, with Wilson 95% intervals and matched bf16/RTN references. Panels use separate runs.

Full GSM8K exact-match accuracy $\uparrow$
Checkpoint	$\bm{N_{\mathrm{cal}}}$	Base	RTN	1-step	$N$ -step	Gap closed
Huginn 3.5B a	32	34.9	27.9	26.9	35.0	102%
R-Llama 1.4B	32	50.2	39.5	40.3	45.3	54%
Full WikiText-2 perplexity $\downarrow$
Ouro 1.4B	4	11.51	14.38	14.67	12.97	49%
Ouro 2.6B b	4	10.10	11.94	13.61	11.62	17%

Table 2: Simulated grouped-INT4 recovery (

g{=}128

N_{\mathrm{cal}}

is the calibration horizon; gap closed is

(N\text{-step}-RTN)/(Base-RTN)

, computed from unrounded results. Base is bf16 except Parcae (f32). The best quantized result in each row is bold.

Appendix figures & tables26 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 4 — Figure 4: Huginn recurrence loop. The adapter projects $\mathrm{cat}(s_{t},e)$ back to the model width with no residual path. Adapter perturbations therefore enter the recurrence without an identity bypass, while core residual paths provide an identity channel that can partially buffer small perturbations.

Arm	Quantized layers	Top-1 (N=32)	Top-1 endpoints ( $N{=}1\rightarrow 32$ )
0	None (bf16 control)	98.9%	68.8% $\to$ 98.9%
1	Adapter only	10.5%	24.8% $\to$ 10.5%
2	Core only (16 projections)	87.9%	68.4% $\to$ 87.9%
3	All (adapter + core)	10.6%	25.0% $\to$ 10.6%
4	Gaussian noise (norm-matched)	25.0%	49.3% $\to$ 25.0%

Appendix

Table 3: Controlled injection for Huginn-3.5B (50 prompts; data from Figure 1 ). Arms 1–3 use deterministic per-channel INT4 FakeQuant on the listed layers, with all other layers bf16. Arm 4 adds one fixed Gaussian weight-noise draw with the same Frobenius norm as the adapter’s per-channel INT4 weight error. Two bf16 runs with independently sampled initial states reach 98.9% agreement.

Arm	Quantized layers	Top-1 (N=32)
0	None (bf16 control)	98.7%
1	Adapter only	49.1%
2	Core blocks only (6 layers)	74.8%
3	All (adapter + core)	34.6%

Appendix

Table 4: Controlled injection for R-Llama-1.4B (50 prompts, per-channel INT4 FakeQuant). The adapter/core split replicates on a second architecture: adapter-only INT4 drops agreement to 49.1%, while core-only preserves 74.8%. The gap is narrower than Huginn (25.7 pp vs. 77.4 pp). Two bf16 runs with independently sampled initial states reach 98.7% agreement.

Configuration	Top-1 (N=32)	GSM8K strict/flex
bf16	$\sim$ 100%	34% / 41%
Per-channel INT4, no rotation	8.5%	0% / 0%
Per-channel RTN + rotation	82.6%	13% / 15%
Per-channel $N$ -step + rotation	92.7%	24% / 25%

Appendix

Table 5: Huginn-3.5B per-channel INT4 diagnostic. Top-1 agreement uses 20 prompts disjoint from calibration. GSM8K uses the first 100 test examples. For trajectory GPTQ, those 100 GSM8K inputs are also used without labels for Hessian calibration, so the corresponding task score is diagnostic.

Rank and top-1 diagnostic
$N_{\mathrm{cal}}$	Hessian rank	Rank (% of dim.)	Top-1 (32-step)
1	43	0.41%	88.2%
4	234	2.22%	90.9%
8	282	2.67%	92.5%
32	335	3.17%	92.5%
RTN baseline			88.0%

Appendix

Table 6: Huginn-3.5B calibration-horizon diagnostics (deterministic grouped-INT4 FakeQuant,

g{=}128

, no rotation). The upper block reports Hessian rank and top-1 agreement. The lower block is a separate held-out GSM8K evaluation on test examples 0–99, held fixed across rows. For comparison, the separate step-31-only Hessian has rank 361 (3.42%), while the 32-step accumulated Hessian has rank 335. Each is evaluated against its own relative threshold (Section 5 ).

Model	$\bm{N_{\mathrm{cal}}}$	WikiText-2 PPL	LAMBADA acc
Ouro 1.4B	1	14.67	—
	2	13.17	—
	3	13.05	—
	4	12.97	—
Parcae 1.3B	1	27.35	0.341
	2	99.59 †	0.326

Appendix

Table 7: Calibration horizons across architectures (grouped-INT4 FakeQuant,

g{=}128

). Each row reports one calibration horizon. Ouro-1.4B obtains 88% of the

N{=}1

-to-

N{=}4

PPL improvement by

N_{\mathrm{cal}}{=}2

. Parcae-1.3B improves sharply by

N_{\mathrm{cal}}{=}4

; at

N_{\mathrm{cal}}{=}2

, one layer has only 3 eigenvalues above the rank threshold and PPL is worse than RTN.

Model	Config	ARC-C (%)	HellaSwag (%)	WikiText-2 PPL
Huginn 3.5B	bf16	37.4%	66.6%	14.02
	$N$ -step + rotation	37.0%	65.3%	14.54
R-Llama 1.4B	bf16	37.1%	46.1%	30.49
	$N$ -step + rotation	36.3%	45.2%	32.64

Appendix

Table 8: Auxiliary Huginn/R-Llama benchmarks for

g{=}128

with trajectory calibration plus adapter rotation. ARC-C and HellaSwag are accuracy (higher is better); WikiText-2 is perplexity (lower is better). The main-table rows (Table 2 ) isolate trajectory calibration without rotation. For comparison, Section 5 reports 14.89 PPL for the non-rotated Huginn trajectory-calibration run.

Checkpoint	Metric	RTN	1-step	$N$ -step
Ouro-2.6B	GSM8K strict/flex ( $n{=}1{,}319$ )	78.8 / 81.5	72.6 / 74.6	79.8 / 82.8
Parcae-1.3B	LAMBADA accuracy	0.3117	0.3410	0.4157

Appendix

Table 9: Supporting metrics for Ouro-2.6B and Parcae-1.3B. Table 2 uses WikiText-2 perplexity as the primary metric for these checkpoints, and the GSM8K and LAMBADA results preserve the same method ordering.

Configuration	Strict (%)	Flexible (%)
bf16	34.87 (460/1,319)	42.23 (557/1,319)
RTN	8.11 (107/1,319)	8.64 (114/1,319)
1-step GPTQ	5.84 (77/1,319)	6.44 (85/1,319)
Equal-row trajectory GPTQ	17.44 (230/1,319)	17.82 (235/1,319)
Full-trajectory GPTQ	20.70 (273/1,319)	23.28 (307/1,319)

Appendix

Table 10: Full-test Huginn GSM8K at grouped INT3 (

g{=}128

;

n{=}1{,}319

). Equal-row calibration uses the one-step activation-row budget distributed across all 32 recurrence steps.

Checkpoint	Base	RTN	1-step	$N$ -step
Ouro 1.4B	0.6591	0.7193	0.7238	0.6912
Ouro 2.6B	0.6240	0.6689	0.7014	0.6582
LoopFormer 278M	0.9334	0.9600	1.0327	0.9516
Parcae 140M	0.9847	1.0326	1.0053	0.9899
Parcae 1.3B	0.8032	1.0514	0.8895	0.8311
PonderPythia 2.8B	0.6646	0.6837	0.6817	0.6757

Appendix

Table 11: Direct bits per byte on WikiText-2 for the seven perplexity checkpoints in Table 2 ; lower is better. Base is bf16 except for Parcae (f32). The best quantized result in each row is bold. All Nanbeige conditions retain the checkpoint’s native 131,072-token context configuration.

Configuration	Strict (%)	Flex (%)
RTN (no rotation, no GPTQ)	25%	28%
1-step GPTQ (rotated variant)	25%	28%
Adapter-bf16 + RTN (no rotation)	32%	35%
Adapter $N$ -step + rotation (core 1-step)	23%	29%
$N$ -step all layers (no rotation)	30%	37%
$N$ -step all layers (rotated variant)	34%	39%

Appendix

Table 12: Component ablation on Huginn-3.5B (

g{=}128

n{=}100

). Scores are diagnostic because GPTQ uses these unlabeled inputs for Hessian calibration. Bold marks the feedback-protection and best-quantized results.

GPTQ
Configuration	WikiText-2 PPL
RTN $g{=}128$	14.98
Step 0 (1-step)	15.36
Step 31	15.07
32-step	14.89
AWQ

Appendix

Table 13: Calibration-step sensitivity on Huginn-3.5B WikiText-2 (

g{=}128

, deterministic FakeQuant). Bold marks the best result within each method family.

Config	Top-1 (N=32)	GSM8K ( $n{=}100$ ) strict/flex	GSM8K ( $n{=}1{,}319$ ) strict/flex
Random orthogonal	91.0%	23% / 24%	26.9% / 27.5%
Hadamard	90.7%	29% / 33%	29.4% / 34.0%
bf16	$\sim$ 100%	34% / 41%	34.9% / 42.2%

Appendix

Table 14: Hadamard vs. random orthogonal at per-channel INT4 with

N

-step GPTQ (both at block size 512). Top-1 agreement uses 20 prompts disjoint from calibration. The GSM8K task evaluations include the unlabeled inputs used for Hessian calibration and are reported as diagnostics.

Perturbation	Adapter	Core	Gap (Core $-$ Adapter)
Trained rank-4 update	13.4%	77.8%	64.4 pp
Random rank-4 update	71.8%	95.8%	24.0 pp
Gaussian noise	89.3%	96.5%	7.2 pp

Appendix

Table 15: Perturbation-type generalization under the perturbation scales defined above. Top-1 agreement with bf16 at

N{=}32

. The trained rank-4 update is evaluated on 60 prompts; the random rank-4 update on 60 prompts for each of 3 seeds; Gaussian noise on 10 prompts for each of 3 seeds.

Projection	Ablation sens.	Random-direction rank	Hessian-trace rank
transformer.adapter	0.118 ( $2.3\times$ next)	17/17 (last)	3/17
core_block.3.mlp.fc	0.051	4/17	4/17
core_block.3.mlp.proj	0.044	14/17	1/17
core_block.1.mlp.proj	0.041	15/17	2/17
core_block.2.mlp.fc	0.040	5/17	5/17

Appendix

Table 16: Top five projections by ablation sensitivity from rankings across 17 shared projections under single-projection grouped INT4 ablation (

g{=}128

; 10 prompts, averaged across 32 steps). Ablation sensitivity is the fraction of per-token top-1 predictions that disagree with bf16 (0 = no effect, 1 = all predictions change). The adapter is a

2.3\times

outlier by ablation but ranks last by random-direction gain and third by Hessian trace.

Group size	$N{=}1$	$N{=}8$	$N{=}32$
$g{=}32$	68.3%	90.6%	88.4%
$g{=}64$	68.8%	90.5%	87.5%
$g{=}128$	68.7%	87.7%	86.5%
$g{=}256$	69.4%	86.6%	83.9%
Per-channel	18.8%	7.8%	7.6%

Appendix

Table 17: Group-size sweep (simulated INT4, full model, 30 prompts). Grouped settings retain high final-step agreement, while per-channel quantization collapses to 7.6%. Bold marks the collapsed per-channel endpoint.

Architecture	Bottleneck layer	Step-0 rank	Max rank gain	1-step vs. RTN	Gap closed
Ouro 1.4B	layer-0 down_proj	0.02%	6.2 $\times$ $\dagger$	worse	49%
Ouro 2.6B	layer-0 down_proj	0.02%	5.4 $\times$ $\dagger$	worse	17%
Nanbeige4.2-3B-Base	layer-1 down_proj	0.02%	3.67 $\times$	worse	58%
R-Llama 1.4B	adapter	0.02%	3.0 $\times$	better	54%
Parcae 1.3B	core.7 mlp.proj	0.03%	6.9 $\times$	better	96%
Huginn 3.5B	core.3 mlp.proj	0.36%	7.8 $\times$	worse	102%

Appendix

Table 18: Cross-architecture Hessian-rank survey at

10^{-3}\lambda_{\max}

. The bottleneck has the lowest step-0 rank ratio among GPTQ-quantized shared layers. Max rank gain is the largest accumulated-to-step-0 ratio across shared layers. Bold marks checkpoints where one-step GPTQ is worse than RTN.

1-step GPTQ worse than RTN
Checkpoint	Screened layer	Step 0	Step 1
Ouro 1.4B	layer-0 down_proj	$1/5{,}632$	$974/5{,}632$
Ouro 2.6B	layer-0 down_proj	$1/5{,}632$	$1{,}010/5{,}632$
Huginn 3.5B	adapter	$43/10{,}560$	$259/10{,}560$
LoopFormer 278M	c_fc block 0	$31/2{,}048$	$154/2{,}048$
1-step GPTQ better than RTN

Appendix

Table 19: Step-1 Hessian ranks for the 8 checkpoints with retained per-step traces. For

H_{t}=X_{t}^{\top}X_{t}

, each count is the number of eigenvalues above

10^{-3}\lambda_{\max}(H_{t})

; denominators are input dimensions. Rows use the rank-survey bottleneck. Huginn instead uses its adapter, where its maximum rank gain occurs.

Filter	Bits	$\|z\|_{\max}$ (orig)	$\|z\|_{\max}$ (A-quant)	$\|z\|_{\max}$ (BCD-quant)	A-quant stable?
Butterworth 4	4	0.915	2.593	0.915	No
	6	0.915	1.287	0.915	No
	8	0.915	1.134	0.915	No
	10	0.915	1.011	0.915	No
	12	0.915	0.934	0.915	Yes
	16	0.915	0.916	0.915	Yes

Appendix

Table 20: IIR stability under quantization.

|z|_{\max}

is the largest pole magnitude; rows with A-quant stable? = No are unstable (

|z|_{\max}>1

). Bold rows mark each filter’s stability boundary. BCD quantization leaves poles unchanged, while A-quantization destabilizes low-bit regimes.

$\bm{\alpha}$	Divergence at last $t$	vs. $\alpha{=}0$
0.00	0.127	baseline
0.10	0.130	$+2\%$
0.15	0.120	$-6\%$
0.25	0.111	$-13\%$
0.50	0.101	$-20\%$

Appendix

Table 21: Identity injection in Mamba SSM. Adding

\alpha\cdot h_{t-1}

to the recurrence reduces A_log sensitivity monotonically from

\alpha\geq 0.15

, consistent with the Huginn identity-bypass result. Intermediate values interpolate smoothly between the shown rows.

Figure 5: Eigenvalue spectrum of the row-normalized calibration Hessian $\widetilde{H}=2X^{\top}X/m$ . (a) Adapter ( $d{=}10{,}560$ ): the spectrum concentrates in 2–3 dominant eigenvalues at step 0, with a sharp drop exceeding 2 orders of magnitude by index 5. Increasing prompts from 50 to 1,000 (dashed) deflates near-threshold eigenvalues below the $10^{-3}\lambda_{\max}$ cutoff, while $\lambda_{\max}$ is stable ( $1.00\times$ ratio). Accumulating 32 steps (dash-dot) lifts the bulk spectrum, populating 335 directions above threshold. (b) Core Wqkv block 0 ( $d{=}5{,}280$ ): the spectrum decays gradually with no sharp cutoff, and 2,111 eigenvalues exceed the threshold at 50 prompts, $49\times$ more than the adapter. The core layer has a much broader step-0 spectrum than the adapter, while 32-step accumulation substantially broadens the adapter spectrum.

Adapter ( $d=10{,}560$ )
Condition	$\bm{10^{-1}}$	$\bm{10^{-2}}$	$\bm{10^{-3}}$	$\bm{10^{-4}}$	$\bm{10^{-5}}$	$\bm{10^{-6}}$
50p, 1-step	1	2	43	2,120	2,919	2,919
1000p, 1-step	1	2	20	3,047	7,462	9,621
50p, 32-step	1	16	335	1,795	4,796	6,213
Core Wqkv@0 ( $d=5{,}280$ )
50p, 1-step	12	323	2,111	2,870	2,870	2,919

Appendix

Table 22: Hessian rank across thresholds

\tau

relative to

\lambda_{\max}

, with effective rank (erank) and stable rank. Core Wqkv from block 0 provides the core-layer comparison. The focal

\tau{=}10^{-3}

column is bold.

Model	$N$	RTN	$N$ -step	Skip-0	$\bm{\Delta}$
Ouro 2.6B	4	11.94	11.62	11.10	$-$ 0.52
Parcae 1.3B	8	51.30	20.84	20.58	$-$ 0.26
Huginn 3.5B	32	14.98	14.89	14.92	$+$ 0.03
R-Llama 1.4B	32	34.02	33.46	33.65	$+$ 0.19
LoopFormer 278M	8	35.10	34.08	34.33	$+$ 0.25
Ouro 1.4B	4	14.38	12.97	13.78	$+$ 0.81

Appendix

Table 23: Step-0 exclusion ablation. WikiText-2 PPL (

\downarrow

) for grouped-INT4

g{=}128

\Delta

is skip-0 minus

N

-step; negative means skip-0 is better.

Model	Config	Strict (%)	Wilson 95% CI
Huginn 3.5B	bf16	34.9	[32.4, 37.5]
Huginn 3.5B	RTN	27.9	[25.5, 30.4]
Huginn 3.5B	1-step GPTQ	26.9	[24.6, 29.4]
Huginn 3.5B	$N$ -step GPTQ	35.0	[32.5, 37.6]
R-Llama 1.4B	bf16	50.2	[47.5, 52.9]
R-Llama 1.4B	RTN	39.5	[36.9, 42.2]

Appendix

Table 24: Wilson 95% confidence intervals for the main GSM8K rows evaluated on all 1,319 test examples, computed from the exact underlying counts.

Rows	Calibration data	Hessian collection	Solver
Huginn/R-Llama	First 50 GSM8K training questions	1 or 32 recurrence steps	GPTQ, 1% damping for core and 10% for adapter
Ouro	WikiText-2 train, 128 $\times$ 2048 tokens	1 or 4 recurrence steps	Column-wise GPTQ, 0.01 damping
LoopFormer	WikiText-2 train, 128 $\times$ 1024 tokens	1 or 8 loop iterations	Column-wise GPTQ, 0.01 damping
Parcae	WikiText-2 train, 128 $\times$ 1024 tokens	1 or 8 core iterations	Column-wise GPTQ, 0.01 damping
PonderPythia	WikiText-2 train, 128 $\times$ 2048 tokens	First pass or all 4 shared-stack passes	Column-wise GPTQ, 0.01 damping
Nanbeige	WikiText-2 train, 128 $\times$ 2048 tokens	First pass or both shared-stack passes	Column-wise GPTQ, 0.01 damping

Appendix

Table 25: Calibration and solver details for the recovery rows. All rows use deterministic FakeQuant with

g{=}128

, no act-order, and unchanged inference graphs within each row.

Asset / source	License
Checkpoints
Huginn-3.5B ( 0f0fa0ba ); R-Llama-1.4B ( a5f6f126 ); Ouro-1.4B ( 574fa66c ); Ouro-2.6B ( 1ed04250 )	Apache-2.0 (pinned checkpoint cards).
PonderPythia-2.8B ( 5bd111c1 ); Nanbeige4.2-3B-Base ( 4a38e817 )	Apache-2.0 (pinned checkpoint cards).
Mamba-130M ; Mamba-2.8B	Checkpoint terms not stated; code Apache-2.0.
LoopFormer-278M ( e7c87640 ); Parcae-140M ( 236739ce ); Parcae-1.3B ( 517f8e6a )	Checkpoint terms not stated; code MIT.
COCONUT-GPT2-124M	Checkpoint terms not stated; code MIT.

Appendix

Table 26: Third-party asset licenses. No third-party checkpoints, datasets, or tooling are redistributed with this submission.

LLM Quantization

Large language model quantization reduces numerical precision to lower memory use and accelerate training or inference.

See all 381 papers

LoopQ: Quantization for Looped Language Models
May 8, 2026Rui Fang, Hsi-Wen Chen, Chia-Ming Cheng +1Recurrent Transformers Language Modeling
National Taiwan University
LoopQuant: Accurate and Efficient Post-Training Quantization for Looped Transformers
Oct 12, 2026Zhixiong Zhao, Zukang Xu, Xing Hu +3Post-Training Quantization LLM Quantization
School of Integrated Circuits, Peking University · Houmo AI
From Attention Sensitivity to Layer Role: Revisiting Mixed-Precision Quantization of Transformers
Sep 28, 2026Nafiseh HosseinpourFardi, Negar Alihadi, Mahmoudreza Babaei +2Mixed-Precision Quantization LLM Quantization
Sharif University of Technology, Tehran, Iran · GISMA University of Applied Sciences, Potsdam, Germany · BRAINS, Brandenburg Research Center for Applied Intelligent Systems, Potsdam, Germany +3

May 8, 2026J/K move · Enter open · S save

LoopQ: Quantization for Looped Language Models

Rui Fang, Hsi-Wen Chen, Chia-Ming Cheng, Ming-Syan Chen

National Taiwan University

Recurrent Transformers Language Modeling Activation Quantization Post-Training Quantization Low-Bit Quantization Recursive Language Models

Open paper AskPDF arXiv

Looped language models (LoopLMs) increase effective computational depth by repeatedly applying shared Transformer blocks without increasing the parameter count. However, this reuse makes them particularly sensitive to post-training quantization (PTQ). We systematically study PTQ in LoopLMs and identify three sources of degradation: distribution shift across roles, state reuse across loop transitions, and recursive error accumulation. We propose LoopQ, a loop-aware PTQ framework that preserves a shared quantized backbone while introducing lightweight adaptations. LoopQ combines loop-aware activation scaling and selective transformations to accommodate changing hidden-state distributions, cross-loop state alignment to stabilize transitions, and trajectory-aware calibration to limit accumulated error. Under W4A4 quantization, LoopQ achieves a 67.8% average relative improvement in five-task mean accuracy across four backbones and a conservative estimate of 75.2% for the average relative perplexity reduction across eight backbone-dataset pairs, compared with the strongest static PTQ baseline for each comparison.

Abstract

LLM Quantization

Explore similar work

LoopQ: Quantization for Looped Language Models

LoopQuant: Accurate and Efficient Post-Training Quantization for Looped Transformers

From Attention Sensitivity to Layer Role: Revisiting Mixed-Precision Quantization of Transformers