JustQuant: You Don't Need Smoothing, SVD, or Rotation for 4-Bit Activation Quantization

Kaicheng Yang; Kaisen Yang; Chunyu Liu; Xianglong Yan; Haotong Qin; Junyi Wu; Tianao Zhang; Xun Zhang

cs.AISep 27, 2026

JustQuant: You Don't Need Smoothing, SVD, or Rotation for 4-Bit Activation Quantization

Authors: Kaicheng Yang, Kaisen Yang, Chunyu Liu, Xianglong Yan, Haotong Qin, Junyi Wu, Tianao Zhang, Xun Zhang, +3 more

Organizations: Shanghai Jiao Tong University · Tsinghua University · The Hong Kong Polytechnic University

Abstract

Recent generative models have become increasingly powerful, but their inference cost continues to grow. Model quantization offers a promising way to compress these models and accelerate inference. However, at 4 bits, activation quantization is substantially more challenging than weight quantization. Recent post-training quantization (PTQ) and quantization-aware training (QAT) methods have made progress in 4-bit activation quantization by introducing smoothing, SVD branches, rotations, mixed precision, or advanced formats such as NVFP4. These additional operators and data types impose demanding requirements on inference engines and hardware, limiting the broad adoption of low-precision models. Can quantization be achieved using only plain low-bit operators? To answer this question, we propose JustQuant, a simple yet effective framework that moves the complexity of low-bit quantization from deployment-time operators into the training process. We first revisit model quantization from the perspective of knowledge distillation and show that a key reason existing PTQ and QAT methods fail is that they typically exploit supervision at only a single level. We then introduce Theseus QAD, a quantization-aware distillation method that progressively applies multi-level supervision, analogous to the gradual replacement process in the Ship of Theseus. Extensive experiments on DiT and diffusion large language models show two distinct regimes. For smaller models, Theseus QAD can serve as a lightweight warm-up stage that substantially improves subsequent QAT with plain operators, while naive QAD may collapse in the same setting. For larger models, Theseus QAD provides a stronger distillation training path than ordinary QAD. Across both regimes, JustQuant improves low-bit quantization quality while avoiding the complex operators required by many existing PTQ methods.

Figures & tables

Panel of Figure 1 — Figure 1: FLUX-family 12B experiments. JustQuant pushes naive W4A4 quality close to BF16 with only 18 hours on 4 H200 GPUs, while preserving a clean inference path.

Figure 2: The false prosperity of low-bit activation quantization. Across eight recent conferences, $\leq$ 4-bit activation papers increasingly rely on extra operators. This deployment-side complexity can obscure the practical speedups promised by low-bit activations.

	Small model		Large model
Method	Quality	Deploy.	Quality	Deploy.
PTQ	✓	✗	✓	✗
QAT	✓	✗	✗	✓
QAD	✗	✗	✓	✓
JustQuant	✓	✓	✓	✓

Table 1: Different regimes favor different quantization recipes.

Panel of Figure 3 — Figure 3: Naive W1.58A4 ImageNet samples and deployment summary. With a 24-hour GPT5.6-sol kernel-optimization budget, the clean JustQuant inference path realizes latency gains more easily than operator-heavy alternatives, even on a 0.6B DiT where small matrices and non-GEMM time make compute-bound speedup difficult.

Figure 4: Theseus QAD. Theseus QAD expands distillation from local blocks to the full model while allocating training steps geometrically across path lengths.

Figure 5: Why activation quantization is hard to QAT.

Figure 6: Effect of Theseus QAD on W1.58A4 DiT. (a) Intermediate activations after Theseus QAD warm-up and QAT resemble those of the FP model, whereas direct W1.58A4 QAT yields more disordered activations. (b) Theseus QAD substantially improves generation quality and diversity.

1:	$\phi\leftarrow\theta,\quad(w_{0},\ldots,w_{K})\leftarrow\textsc{BinaryMergeWidths}(L)$
2:	$(N_{0},\ldots,N_{K})\leftarrow\textsc{Allocate}(T_{1},\gamma)$ (see text)
3:	for $k=0,\ldots,K$ and $u=1,\ldots,N_{k}$ do
4:	$(x,t)\sim\mathcal{D}$ , $h\leftarrow x,\quad\mathcal{L}_{\mathrm{S1}}\leftarrow 0$
5:	for $g=0,w_{k},2w_{k},\ldots,L-w_{k}$ do
6:	$a\leftarrow h,\quad h\leftarrow B_{g+w_{k}-1}\circ\cdots\circ B_{g}(a)$
7:	$s\leftarrow\widetilde{B}_{g+w_{k}-1}\circ\cdots\circ\widetilde{B}_{g}(a)$

Figure 7: Theseus QAD warm-up and Stage-1 allocation. The left panel shows the online Stage-1 loop: during one FP forward, each reached segment is immediately matched by its quantized counterpart, and the student is updated once after the accumulated loss. The right panel reports the same sweep, allocating 10k Stage-1 steps across replacement widths

(1,2,4,8,16,28)

, followed by 10k-step official QAT.

	Small model	Large model
DiT	DiT-XL/2, 0.6B ( Peebles and Xie, 2023 )	FLUX.1, 12B ( Black Forest Labs, 2024 )
DLLM	ELF-B, 105M ( Hu et al., 2026 )	LLaDA, 8B ( Nie et al., 2025 )

Table 2: Experimental summary.

Setting	Method	W/A	Type/Step	FID $\downarrow$	sFID $\downarrow$	IS $\uparrow$	Additional Operator
10K samples 50 steps	FP ( Peebles and Xie, 2023 )	32/32	-/7000k	6.78	20.56	243.70	–
	SVDQuant ( Li et al., 2025 )	4/4	PTQ/0k	81.28	67.27	29.17	Smooth, SVD
	ConvRot ( Huang et al., 2025a )	4/4	PTQ/0k	18.69	32.06	139.54	Rotation
	QuaRot ( Ashkboos et al., 2024 )	4/4	PTQ/0k	53.31	56.74	53.12	Rotation
	VETA-DiT ( Xu et al., 2025 )	4/4	PTQ/14.4k	9.87	25.52	202.55	Smooth, Rotation
	TreeQ ( Yang et al., 2025 )	4/4	PTQ/36.9k	6.92	20.86	219.66	MP, SVD, Rotation

Table 3: DiT-XL/2 on ImageNet

256\times 256

. Further evaluation details are provided in Appendix B .

W/A	Arm	Qwen3-8B-Base		GPT-2-Large		Add. Op.
W/A	Arm	gen-PPL $\downarrow$ (4seed)	$\Delta$ PPL vs FP	gen-PPL $\downarrow$ (4seed)	$\Delta$ PPL vs FP	Add. Op.
4/4	FP	$14.729\pm 0.152$	–	$19.029\pm 0.305$	–	–
	JustQuant	$14.854\pm 0.344$	$+0.125$	$19.180\pm 0.417$	$+0.152$	None
	RobuQ	$16.751\pm 0.384$	$+2.022$	$22.089\pm 0.529$	$+3.061$	Hadamard, SVD
	Direct QAT	$19.625\pm 0.327$	$+4.896$	$25.011\pm 0.465$	$+5.983$	None
1.58/4	FP	$14.823\pm 0.208$	–	$19.254\pm 0.253$	–	–

Table 5: ELF-B OpenWebText generation. W4A4 and W1.58A4 results are reported with generation perplexity from two judge models, following the evaluation protocol of ELF ( Hu et al., 2026 ) .

Method	LR	Steps	FID $\downarrow$	sFID $\downarrow$	IS $\uparrow$
QAT	1e-4	20k	24.60	28.39	90.01
QAT	1e-6	20k	27.52	35.68	84.70
QAT	2e-6	20k	25.77	36.19	89.09
QAT	5e-6	20k	25.64	34.68	85.60
QAT	1e-5	20k	24.94	33.47	88.06
QAT	2e-5	20k	18.27	29.78	114.82

Table 7: Ablation Study.

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 8 — Figure 8: DiT-XL/2 W1.58A4 qualitative comparisons. The two panels show ImageNet $256\times 256$ samples from the full-precision model and the compared low-bit methods under the W1.58A4 setting.

Figure 9: FLUX.1-schnell W4A4 INT4 qualitative comparison. Each row shows one prompt, and the columns compare FP, ConvRot, SVDQuant, and JustQuant under the W4A4 INT4 setting.

	DiT-XL/2, ImageNet $256^{2}$
Setting	W1.58A32	W4A4	W1.58A4
Stage 1 (Theseus QAD)
Dataset	ImageNet, 128k VAE latents	ImageNet, 128k VAE latents	ImageNet, 128k VAE latents
Learning rate	$1\times 10^{-4}$	$1\times 10^{-4}$	$1\times 10^{-4}$
Batch size	256	128	256
Training steps	10k	10k	10k

Appendix

Table 8: DiT-XL/2 experimental details.

Explore similar work

May 25, 2026cs.LG

InfoQuant: Shaping Activation Distributions for Low-Bit LLM Quantization

Low-bit activation quantization remains a major bottleneck in efficient large language model (LLM) deployment. The difficulty is not only that activations contain outliers, but that their distributions are often poorly matched to a low-bit uniform quantizer. Existing post-training quantization (PTQ) methods suppress peaks, balance channels, or minimize reconstruction error, yet they rarely specify what activation distribution is actually easy to discretize. As a result, activations may appear numerically smoother while still incurring large quantization error because the quantization range remains wide or most values collapse into a few levels near the mean. We recast activation transformation as quantizer-facing distribution design and analyze quantization error from an information-theoretic perspective. Our analysis shows that quantization-friendly activations should jointly have a smaller numerical range and sufficient dispersion within that range. Guided by this analysis, we propose InfoQuant, a train-free method that employs Peak Suppression Orthogonal Transformation (PSOT) to shape activations into more quantization-friendly distributions. We further introduce adaptive outlier-token selection to improve the robustness of PSOT during optimization. Across multiple LLM families, InfoQuant consistently outperforms prior PTQ and end-to-end training baselines. Under W4A4KV4, it preserves 97% of floating-point accuracy on average and reduces the LLaMA-2 13B performance gap by 42% over the previous state of the art. Code is available at https://github.com/LLIKKE/InfoQuant

Ke Li, Dong An, Xiaoling Zang +6

School of Software Technology, Zhejiang University · Ant Group · College of Computer Science and Technology, Zhejiang University of Technology +3

Large Language Model Quantization Distributions Outliers

Ask PDF

Jun 10, 2026cs.LG

DynamicPTQ: Mitigating Activation Quantization Collapse via Residual-Stream Dynamics

Post-training quantization (PTQ) is essential for efficient large language model inference, but reliably quantizing activations remains challenging when weights, activations, and KV caches are all quantized to 4-bit precision. A key difficulty lies in massive activations, whose extreme values dominate the activation range and amplify quantization errors. State-of-the-art methods mainly mitigate massive activations through transformation-based smoothing, such as orthogonal rotations and affine scaling, but overlook the cross-layer dynamics of the residual stream. In this paper, we show that massive activations emerge and disappear in a phase-wise pattern across network depth, triggering large residual changes. These changes cause newly injected layer-wise updates to dominate the 4-bit quantization scale and weaken historical residual information. To characterize this behavior, we introduce Jump Ratio and Historical Feature SNR. This suggests that static transformation-based smoothing cannot fully resolve dynamic quantization instability caused by cross-layer residual changes. Based on this analysis, we propose DynamicPTQ, a Dynamic Post-Training Quantization policy for phase-aware mixed-precision activation quantization. DynamicPTQ identifies quantization-sensitive layers from residual-stream dynamics and assigns 8-bit activation precision only to these layers, while keeping weights, KV caches, and other activations in 4-bit precision. It can be directly integrated with strong PTQ baselines such as QuaRot, SpinQuant, and FlatQuant. Experiments on LLaMA-2 and LLaMA-3 show that DynamicPTQ consistently improves perplexity and zero-shot QA performance under W4A4KV4 quantization, while achieving 1.05 to 1.07 times throughput improvement with modest memory overhead. These results demonstrate a practical path toward robust low-bit LLM inference.

Zimo Zhao, Maolin Wang, Bowen Yu +3

City University of Hong Kong · Hong Kong, China · Zhejiang University of Technology +1

Post-Training Quantization Mixed-Precision Quantization LLM Inference Optimization Residual Stream

Ask PDF

May 9, 2026cs.LG

AAAC: Activation-Aware Adaptive Codebooks for 4-bit LLM Weight Quantization

Post-training weight-only quantization to 4 bits is widely used to reduce the memory and compute costs of large language model inference. Existing PTQ methods, such as AWQ and GPTQ, improve how weights are mapped onto a fixed 4-bit grid through scaling, clipping, or error compensation. To further improve accuracy, methods such as OmniQuant and QuIP# uses gradient-assisted algorithms at the cost of hours of quantization time. In this work, we propose AAAC (Activation-Aware Adaptive Codebooks), a lightweight method for 4-bit LLM weight quantization. AAAC replaces the fixed scalar codebook used in standard quantization with two small learned scalar codebooks (64 bytes) per layer. Each group of weights selects the codebook that minimizes activation-weighted reconstruction error, encoding the choice in the unused sign bit of the group's positive scale and adding zero storage overhead. AAAC completes in 3--30 minutes on a single GPU, and adds no memory beyond the model itself. We evaluate against AWQ, GPTQ, IF4, GPTVQ, OmniQuant, SqueezeLLM, and QuIP# across model families. AAAC outperforms baselines at orders-of-magnitude less quantization time.

Beshr IslamBouli, David Jin

Massachusetts Institute of Technology, Cambridge, MA, USA.

Large Language Model Quantization Model Weights Codebooks LLM Inference Optimization

Ask PDF