TTQ: Activation-Aware Test-Time Quantization to Accelerate LLM Inference On The Fly

Toshiaki Koike-Akino; Jing Liu; Ye Wang

cs.LGMar 11, 2026

TTQ: Activation-Aware Test-Time Quantization to Accelerate LLM Inference On The Fly

Authors: Toshiaki Koike-Akino, Jing Liu, Ye Wang

Organizations: Mitsubishi Electric Research Laboratories (MERL), 201 Broadway, Cambridge, MA 02139, USA

Abstract

To tackle the huge computational demand of large foundation models, activation-aware compression techniques without retraining have been introduced. However, since these methods highly rely on calibration data, domain shift issues may arise for unseen downstream tasks. We propose a test-time quantization (TTQ) framework which compresses large models on the fly at inference time to resolve this issue. With an efficient online calibration, instant activation-aware quantization can adapt every prompt regardless of the downstream tasks, yet achieving inference speedup. Several experiments demonstrate that TTQ can improve the quantization performance over state-of-the-art baselines.

Figures & tables

Figure 1: (a) Offline static quantization (e.g., AWQ/GPTQ) requires calibration data, incurs domain shift risk, and cannot be recalibrated after deployment. (b) Our TTQ is online dynamic quantization, with zero offline calibration, and capable of on-device self-calibration at inference time.

	TTQ		AWQ (C4 Calib)
Calib Tokens $T$	0 (r=0)	0 (r=16)	$2^{11}$	$2^{12}$	$2^{13}$	$2^{14}$	$2^{15}$	$2^{16}$	$2^{17}$
WT2 Perplexity ( $\downarrow$ )	24.93	24.17	25.73	25.56	25.62	25.55	25.42	25.07	25.02

Table 1: Calibration length impact at 3-bit quantization with

g=32

for OPT-350M model.

Panel of Table 2 — Table 2: Groupsize impact on WT2 perplexity at 3-bit quantization for Qwen3-1.7B model.

OPT-125M (WT2: 27.7, PTB: 39.0, C4: 26.6, Avg: 31.1 )
Quantization $q$	2 bits	3 bits	4 bits	5 bits
RTN	5058.5	56.3	33.5	31.8
AWQ (WT2 Calib)	381.7	37.4	32.3	31.4
AWQ (PTB Calib)	375.3	37.3	32.2	31.3
AWQ (C4 Calib)	451.7	37.7	32.6	31.3
TTQ ( $r=0$ )	257.4	36.6	31.9	31.2

Table 3: Perplexity (

\downarrow

) of OPT/Qwen3/Gemma3 models with different quantization methods. It shows macro average across WT2/PTB/C4 datasets. Groupsize is

g=32

for all cases. Calibration token length is

T=2^{17}

for AWQ. Bold and underline denote the best and second best, respectively. Asterisk “*” indicates reaching competitive performance to the original un-compressed LLM.

Appendix figures & tables16 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 2 — Figure 2: Histogram of top-5 hyperparameter selections $(\alpha,\lambda,p)$ for different OPT models with quantization bits of $q\in\{2,3,4,5\}$ .

Panel of Table 4 — Table 4: Runtime Speed (k tokens/sec) of Query Projection Module for Qwen3 Models with 4-bit AWQ and TTQ, on NVIDIA A40 GPU.

Panel of Table 5 — Table 5: Runtime Speed (k tokens/sec) of Query Projection Module for Qwen3 Models with 4-bit AWQ and TTQ, on NVIDIA A100 GPU.

Panel of Table 6 — Table 6: Runtime Speed (k tokens/sec) of Query Projection Module for Qwen3 Models with 4-bit AWQ and TTQ, on NVIDIA L40 GPU.

Panel of Table 7 — Table 7: Runtime Speed (k tokens/sec) of Query Projection Module for Qwen3 Models with 4-bit AWQ and TTQ, on NVIDIA RTX3090 GPU.

Panel of Table 8 — Table 8: Runtime Speed (k tokens/sec) of Query Projection Module for Qwen3 Models with 4-bit AWQ and TTQ, on NVIDIA RTX4090 GPU.

OPT-125M (WT2: 27.7, PTB: 39.0, C4: 26.6, Avg: 31.1 ±6.8 )
Quantization $q$	2 bits	3 bits	4 bits	5 bits
RTN	5058.5 ±833.7	56.3 ±10.5	33.5 ±7.0	31.8 ±7.0
AWQ (WT2 Calib)	381.7 ±159.8	37.4 ±7.4	32.3 ±6.5	31.4 ±6.1
AWQ (PTB Calib)	375.3 ±84.9	37.3 ±6.9	32.2 ±6.3	31.3 ±6.0
AWQ (C4 Calib)	451.7 ±181.7	37.7 ±7.5	32.6 ±6.6	31.3 ±6.0
TTQ ( $r=0$ )	257.4 ±65.3	36.6 ±8.2	31.9 ±7.0	31.2 ±6.8

Appendix

Table 9: Perplexity (

\downarrow

) of OPT models with different quantization methods. It shows macro average and standard deviation across WT2/PTB/C4 datasets. Groupsize is

g=32

for all cases. Calibration token length is

T=2^{17}

for AWQ. Bold and underline denote the best and second best, respectively. Asterisk “*” indicates reaching competitive performance to the original un-compressed LLM.

Qwen3-0.6B (WT2: 21.0, PTB: 43.8, C4: 30.3, Avg: 31.7 ±11.5 )
Quantization $q$	2 bits	3 bits	4 bits	5 bits
RTN	8.2e6 ±4.4e6	127.3 ±54.1	38.2 ±14.7	33.5 ±12.4
AWQ (WT2 Calib)	9739.1 ±2765.7	49.4 ±20.2	33.5 ±11.7	32.4 ±11.8
AWQ (PTB Calib)	17344.3 ±1962.7	47.9 ±17.3	34.1 ±12.5	32.0 ±11.5
AWQ (C4 Calib)	5388.1 ±1862.6	48.3 ±17.3	33.0 ±11.6	32.1 ±11.8
TTQ ( $r=0$ )	2827.8 ±883.8	44.7 ±16.9	33.0 ±12.1	31.9 ±11.3

Appendix

Table 10: Perplexity (

\downarrow

) of Qwen3 models with different quantization methods. It shows macro average and standard deviation across WT2/PTB/C4 datasets. Groupsize is

g=32

for all cases. Calibration token length is

T=2^{17}

for AWQ. Bold and underline denote the best and second best, respectively. Asterisk “*” indicates reaching competitive performance to the original un-compressed LLM.

Gemma3-270M (WT2: 70.1, PTB: 698.7, C4: 68.1, Avg: 279.0 ±353.5 )
Quantization $q$	2 bits	3 bits	4 bits	5 bits
RTN	2.6e11 ±1.2e11	1795.0 ±2207.5	391.9 ±505.5	315.0 ±410.9
AWQ (WT2 Calib)	89188.4 ±58307.2	517.4 ±690.0	279.6 ±348.6	306.0 ±405.2
AWQ (PTB Calib)	1.9e5 ±2.0e9	537.7 ±711.8	310.0 ±408.2	293.4 ±386.4
AWQ (C4 Calib)	1.3e5 ±1.6e5	598.1 ±832.9	326.4 ±420.5	*276.8 ±358.0
TTQ ( $r=0$ )	30199.0 ±23705.9	387.7 ±489.0	* 277.0 ±357.1	* 262.5 ±335.2

Appendix

Table 11: Perplexity (

\downarrow

) of Gemma3 models with different quantization methods. It shows macro average and standard deviation across WT2/PTB/C4 datasets. Groupsize is

g=32

for all cases. Calibration token length is

T=2^{17}

for AWQ. Bold and underline denote the best and second best, respectively. Asterisk “*” indicates reaching competitive performance to the original un-compressed LLM.

Qwen3-VL-2B: Acc 80.35%
Quantization $q$	2 bits	3 bits	4 bits	5 bits
RTN	0.00%	9.23%	73.27%	80.22%
AWQ (COCO-Cap Calib)	1.41%	75.67%	79.04%	79.89%
AWQ (OK-VQA Calib)	1.37%	74.68%	79.47%	79.80%
AWQ (ChartQA Calib)	0.67%	75.52%	79.29%	79.80%
AWQ (TextVQA Calib)	0.82%	72.15%	78.85%	80.21%

Appendix

Table 12: Accuracy (

\uparrow

) of Qwen3-VL models with different quantization methods on TextVQA benchmark. Groupsize is g = 32 for all cases. Bold and underline denote the best and second best, respectively. Asterisk “*” indicates reaching competitive performance to the original un-compressed VLM.

Benthmark	Libero Spatial	Libero Object	Libero Goal	Libero 10	Avg
BF16	97.5%	100.0%	97.0%	96.5%	97.75%
RTN	57.0%	65.0%	27.5%	2.0%	37.88%
AWQ (Spatial Calib)	90.5%	100.0 %	85.0%	82.0%	89.34%
AWQ (Object Calib)	91.5%	98.5%	92.0%	78.0%	90.00%
AWQ (Goal Calib)	92.5%	100.0 %	93.5 %	84.5%	92.63%
AWQ (10 Calib)	94.0 %	99.5 %	92.5 %	76.5%	90.63%

Appendix

Table 13: Success rate (

\uparrow

) of

\pi_{0.5}

VLA model with different quantization methods on LIBERO robot manipulation benchmark. We use

q=2

bits and

g=64

groupsize. Bold and underline denote the best and second best, respectively.

Model	layers	heads	hidden size	head dim	MLP dim	Huggingface ID
125M	12	12	768	64	3072	facebook/opt-125m
350M	24	16	1024	64	4096	facebook/opt-350m
1.3B	24	32	2048	64	8192	facebook/opt-1.3b
2.7B	32	32	2560	80	10240	facebook/opt-2.7b
6.7B	32	32	4096	128	16384	facebook/opt-6.7b
13B	40	40	5120	128	20480	facebook/opt-13b

Appendix

Table 14: OPT model parameters Zhang et al. (2022)

Model	layers	heads	KV heads	hidden size	head dim	MLP dim	Huggingface ID
0.6B	28	16	8	1024	128	3072	Qwen/Qwen3-0.6B
1.7B	28	16	8	2048	128	6144	Qwen/Qwen3-1.7B
4B	36	32	8	2560	128	9728	Qwen/Qwen3-4B
8B	36	32	8	4096	128	12288	Qwen/Qwen3-8B
14B	40	40	8	5120	128	17408	Qwen/Qwen3-14B
32B	64	64	8	5120	128	25600	Qwen/Qwen3-32B

Appendix

Table 15: Architecture parameters of Qwen3 dense models Yang et al. (2025)

Model	layers	heads	KV heads	hidden size	head dim	MLP dim	Huggingface ID
270M	18	4	1	640	256	2048	google/gemma-3-270m-it
1B	26	4	1	1152	256	6912	google/gemma-3-1b-it
4B	34	8	4	2560	256	10240	google/gemma-3-4b-it
12B	48	16	8	3840	256	15360	google/gemma-3-12b-it
27B	62	32	16	5376	256	21504	google/gemma-3-27b-it

Appendix

Table 16: Gemma3 instruction-tuned text transformer parameters Gemma Team et al. (2025)

Model	layers	heads	KV heads	hidden size	head dim	MLP dim	Hugginface ID
2B	24	16	8	2048	128	11008	Qwen/Qwen3-VL-2B-Instruct
4B	32	32	8	4096	128	22016	Qwen/Qwen3-VL-4B-Instruct
8B	36	32	8	4096	128	22016	Qwen/Qwen3-VL-8B-Instruct
32B	64	40	8	5120	128	27392	Qwen/Qwen3-VL-32B-Instruct

Appendix

Table 17: Qwen3-VL transformer parameters Bai et al. (2025)

Module	layers	heads	KV heads	hidden size	head dim	MLP dim
LLM: Gemma-2B	18	8	1	2048	256	16384
Vision Encoder: SigLIP ViT-L	24	16	16	1024	64	4096
Diffusion Policy: Gemma-300M	18	8	1	1024	128	4096

Appendix

Table 18:

\pi_{0.5}

VLA transformer parameters Intelligence et al. (2025) : Huggingface ID lerobot/pi05_libero_finetuned

Explore similar work

InfoQuant: Shaping Activation Distributions for Low-Bit LLM Quantization
May 25, 2026Ke Li, Dong An, Xiaoling Zang +6LLM Quantization Activation Quantization
School of Software Technology, Zhejiang University · Ant Group · College of Computer Science and Technology, Zhejiang University of Technology +3
You Had One Job: Per-Task Quantization Using LLMs' Hidden Representations
Nov 9, 2025Amit LeVi, Raz Lapid, Rom Himelstein +3Mixed-Precision Quantization LLM Quantization
1Technion—Israel Institute of Technology · 2Intuit · 3Ben-Gurion University of the Negev +1
JustQuant: You Don't Need Smoothing, SVD, or Rotation for 4-Bit Activation Quantization
Sep 27, 2026Kaicheng Yang, Kaisen Yang, Chunyu Liu +8Diffusion Model Quantization Quantization-Aware Training
Shanghai Jiao Tong University · Tsinghua University · The Hong Kong Polytechnic University

May 25, 2026J/K move · Enter open · S save

InfoQuant: Shaping Activation Distributions for Low-Bit LLM Quantization

Ke Li, Dong An, Xiaoling Zang, Can Ye+5

School of Software Technology, Zhejiang University · Ant Group · College of Computer Science and Technology, Zhejiang University of Technology+3

LLM Quantization Activation Quantization Post-Training Quantization Low-Bit Quantization Neural Network Quantization

Open paper AskPDF arXiv

Low-bit activation quantization remains a major bottleneck in efficient large language model (LLM) deployment. The difficulty is not only that activations contain outliers, but that their distributions are often poorly matched to a low-bit uniform quantizer. Existing post-training quantization (PTQ) methods suppress peaks, balance channels, or minimize reconstruction error, yet they rarely specify what activation distribution is actually easy to discretize. As a result, activations may appear numerically smoother while still incurring large quantization error because the quantization range remains wide or most values collapse into a few levels near the mean. We recast activation transformation as quantizer-facing distribution design and analyze quantization error from an information-theoretic perspective. Our analysis shows that quantization-friendly activations should jointly have a smaller numerical range and sufficient dispersion within that range. Guided by this analysis, we propose InfoQuant, a train-free method that employs Peak Suppression Orthogonal Transformation (PSOT) to shape activations into more quantization-friendly distributions. We further introduce adaptive outlier-token selection to improve the robustness of PSOT during optimization. Across multiple LLM families, InfoQuant consistently outperforms prior PTQ and end-to-end training baselines. Under W4A4KV4, it preserves 97% of floating-point accuracy on average and reduces the LLaMA-2 13B performance gap by 42% over the previous state of the art. Code is available at https://github.com/LLIKKE/InfoQuant

Abstract

Explore similar work

InfoQuant: Shaping Activation Distributions for Low-Bit LLM Quantization

You Had One Job: Per-Task Quantization Using LLMs' Hidden Representations

JustQuant: You Don't Need Smoothing, SVD, or Rotation for 4-Bit Activation Quantization