JustQuant: You Don't Need Smoothing, SVD, or Rotation for 4-Bit Activation Quantization

Kaicheng Yang; Kaisen Yang; Chunyu Liu; Xianglong Yan; Haotong Qin; Junyi Wu; Tianao Zhang; Xun Zhang

cs.AISep 27, 2026

JustQuant: You Don't Need Smoothing, SVD, or Rotation for 4-Bit Activation Quantization

Authors: Kaicheng Yang, Kaisen Yang, Chunyu Liu, Xianglong Yan, Haotong Qin, Junyi Wu, Tianao Zhang, Xun Zhang, +3 more

Organizations: Shanghai Jiao Tong University · Tsinghua University · The Hong Kong Polytechnic University

Abstract

Recent generative models have become increasingly powerful, but their inference cost continues to grow. Model quantization offers a promising way to compress these models and accelerate inference. However, at 4 bits, activation quantization is substantially more challenging than weight quantization. Recent post-training quantization (PTQ) and quantization-aware training (QAT) methods have made progress in 4-bit activation quantization by introducing smoothing, SVD branches, rotations, mixed precision, or advanced formats such as NVFP4. These additional operators and data types impose demanding requirements on inference engines and hardware, limiting the broad adoption of low-precision models. Can quantization be achieved using only plain low-bit operators? To answer this question, we propose JustQuant, a simple yet effective framework that moves the complexity of low-bit quantization from deployment-time operators into the training process. We first revisit model quantization from the perspective of knowledge distillation and show that a key reason existing PTQ and QAT methods fail is that they typically exploit supervision at only a single level. We then introduce Theseus QAD, a quantization-aware distillation method that progressively applies multi-level supervision, analogous to the gradual replacement process in the Ship of Theseus. Extensive experiments on DiT and diffusion large language models show two distinct regimes. For smaller models, Theseus QAD can serve as a lightweight warm-up stage that substantially improves subsequent QAT with plain operators, while naive QAD may collapse in the same setting. For larger models, Theseus QAD provides a stronger distillation training path than ordinary QAD. Across both regimes, JustQuant improves low-bit quantization quality while avoiding the complex operators required by many existing PTQ methods.

Figures & tables

Panel of Figure 1 — Figure 1: FLUX-family 12B experiments. JustQuant pushes naive W4A4 quality close to BF16 with only 18 hours on 4 H200 GPUs, while preserving a clean inference path.

Figure 2: The false prosperity of low-bit activation quantization. Across eight recent conferences, $\leq$ 4-bit activation papers increasingly rely on extra operators. This deployment-side complexity can obscure the practical speedups promised by low-bit activations.

	Small model		Large model
Method	Quality	Deploy.	Quality	Deploy.
PTQ	✓	✗	✓	✗
QAT	✓	✗	✗	✓
QAD	✗	✗	✓	✓
JustQuant	✓	✓	✓	✓

Table 1: Different regimes favor different quantization recipes.

Panel of Figure 3 — Figure 3: Naive W1.58A4 ImageNet samples and deployment summary. With a 24-hour GPT5.6-sol kernel-optimization budget, the clean JustQuant inference path realizes latency gains more easily than operator-heavy alternatives, even on a 0.6B DiT where small matrices and non-GEMM time make compute-bound speedup difficult.

Figure 4: Theseus QAD. Theseus QAD expands distillation from local blocks to the full model while allocating training steps geometrically across path lengths.

Figure 5: Why activation quantization is hard to QAT.

Figure 6: Effect of Theseus QAD on W1.58A4 DiT. (a) Intermediate activations after Theseus QAD warm-up and QAT resemble those of the FP model, whereas direct W1.58A4 QAT yields more disordered activations. (b) Theseus QAD substantially improves generation quality and diversity.

1:	$\phi\leftarrow\theta,\quad(w_{0},\ldots,w_{K})\leftarrow\textsc{BinaryMergeWidths}(L)$
2:	$(N_{0},\ldots,N_{K})\leftarrow\textsc{Allocate}(T_{1},\gamma)$ (see text)
3:	for $k=0,\ldots,K$ and $u=1,\ldots,N_{k}$ do
4:	$(x,t)\sim\mathcal{D}$ , $h\leftarrow x,\quad\mathcal{L}_{\mathrm{S1}}\leftarrow 0$
5:	for $g=0,w_{k},2w_{k},\ldots,L-w_{k}$ do
6:	$a\leftarrow h,\quad h\leftarrow B_{g+w_{k}-1}\circ\cdots\circ B_{g}(a)$
7:	$s\leftarrow\widetilde{B}_{g+w_{k}-1}\circ\cdots\circ\widetilde{B}_{g}(a)$

Figure 7: Theseus QAD warm-up and Stage-1 allocation. The left panel shows the online Stage-1 loop: during one FP forward, each reached segment is immediately matched by its quantized counterpart, and the student is updated once after the accumulated loss. The right panel reports the same sweep, allocating 10k Stage-1 steps across replacement widths

(1,2,4,8,16,28)

, followed by 10k-step official QAT.

	Small model	Large model
DiT	DiT-XL/2, 0.6B ( Peebles and Xie, 2023 )	FLUX.1, 12B ( Black Forest Labs, 2024 )
DLLM	ELF-B, 105M ( Hu et al., 2026 )	LLaDA, 8B ( Nie et al., 2025 )

Table 2: Experimental summary.

Setting	Method	W/A	Type/Step	FID $\downarrow$	sFID $\downarrow$	IS $\uparrow$	Additional Operator
10K samples 50 steps	FP ( Peebles and Xie, 2023 )	32/32	-/7000k	6.78	20.56	243.70	–
	SVDQuant ( Li et al., 2025 )	4/4	PTQ/0k	81.28	67.27	29.17	Smooth, SVD
	ConvRot ( Huang et al., 2025a )	4/4	PTQ/0k	18.69	32.06	139.54	Rotation
	QuaRot ( Ashkboos et al., 2024 )	4/4	PTQ/0k	53.31	56.74	53.12	Rotation
	VETA-DiT ( Xu et al., 2025 )	4/4	PTQ/14.4k	9.87	25.52	202.55	Smooth, Rotation
	TreeQ ( Yang et al., 2025 )	4/4	PTQ/36.9k	6.92	20.86	219.66	MP, SVD, Rotation

Table 3: DiT-XL/2 on ImageNet

256\times 256

. Further evaluation details are provided in Appendix B .

W/A	Arm	Qwen3-8B-Base		GPT-2-Large		Add. Op.
W/A	Arm	gen-PPL $\downarrow$ (4seed)	$\Delta$ PPL vs FP	gen-PPL $\downarrow$ (4seed)	$\Delta$ PPL vs FP	Add. Op.
4/4	FP	$14.729\pm 0.152$	–	$19.029\pm 0.305$	–	–
	JustQuant	$14.854\pm 0.344$	$+0.125$	$19.180\pm 0.417$	$+0.152$	None
	RobuQ	$16.751\pm 0.384$	$+2.022$	$22.089\pm 0.529$	$+3.061$	Hadamard, SVD
	Direct QAT	$19.625\pm 0.327$	$+4.896$	$25.011\pm 0.465$	$+5.983$	None
1.58/4	FP	$14.823\pm 0.208$	–	$19.254\pm 0.253$	–	–

Table 5: ELF-B OpenWebText generation. W4A4 and W1.58A4 results are reported with generation perplexity from two judge models, following the evaluation protocol of ELF ( Hu et al., 2026 ) .

Method	LR	Steps	FID $\downarrow$	sFID $\downarrow$	IS $\uparrow$
QAT	1e-4	20k	24.60	28.39	90.01
QAT	1e-6	20k	27.52	35.68	84.70
QAT	2e-6	20k	25.77	36.19	89.09
QAT	5e-6	20k	25.64	34.68	85.60
QAT	1e-5	20k	24.94	33.47	88.06
QAT	2e-5	20k	18.27	29.78	114.82

Table 7: Ablation Study.

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 8 — Figure 8: DiT-XL/2 W1.58A4 qualitative comparisons. The two panels show ImageNet $256\times 256$ samples from the full-precision model and the compared low-bit methods under the W1.58A4 setting.

Figure 9: FLUX.1-schnell W4A4 INT4 qualitative comparison. Each row shows one prompt, and the columns compare FP, ConvRot, SVDQuant, and JustQuant under the W4A4 INT4 setting.

	DiT-XL/2, ImageNet $256^{2}$
Setting	W1.58A32	W4A4	W1.58A4
Stage 1 (Theseus QAD)
Dataset	ImageNet, 128k VAE latents	ImageNet, 128k VAE latents	ImageNet, 128k VAE latents
Learning rate	$1\times 10^{-4}$	$1\times 10^{-4}$	$1\times 10^{-4}$
Batch size	256	128	256
Training steps	10k	10k	10k

Appendix

Table 8: DiT-XL/2 experimental details.

Explore similar work

InfoQuant: Shaping Activation Distributions for Low-Bit LLM Quantization
May 25, 2026Ke Li, Dong An, Xiaoling Zang +6Large Language Model Quantization Distributions
School of Software Technology, Zhejiang University · Ant Group · College of Computer Science and Technology, Zhejiang University of Technology +3
DynamicPTQ: Mitigating Activation Quantization Collapse via Residual-Stream Dynamics
Jun 10, 2026Zimo Zhao, Maolin Wang, Bowen Yu +3Post-Training Quantization Mixed-Precision Quantization
City University of Hong Kong · Hong Kong, China · Zhejiang University of Technology +1
AAAC: Activation-Aware Adaptive Codebooks for 4-bit LLM Weight Quantization
May 9, 2026Beshr IslamBouli, David JinLarge Language Model Quantization Model Weights
Massachusetts Institute of Technology, Cambridge, MA, USA.

May 25, 2026, cs.LGJ/K move · Enter open · S save

InfoQuant: Shaping Activation Distributions for Low-Bit LLM Quantization

Ke Li, Dong An, Xiaoling Zang, Can Ye+5

School of Software Technology, Zhejiang University · Ant Group · College of Computer Science and Technology, Zhejiang University of Technology+3

Large Language Model Quantization Distributions Outliers

Open paper AskPDF arXiv

Low-bit activation quantization remains a major bottleneck in efficient large language model (LLM) deployment. The difficulty is not only that activations contain outliers, but that their distributions are often poorly matched to a low-bit uniform quantizer. Existing post-training quantization (PTQ) methods suppress peaks, balance channels, or minimize reconstruction error, yet they rarely specify what activation distribution is actually easy to discretize. As a result, activations may appear numerically smoother while still incurring large quantization error because the quantization range remains wide or most values collapse into a few levels near the mean. We recast activation transformation as quantizer-facing distribution design and analyze quantization error from an information-theoretic perspective. Our analysis shows that quantization-friendly activations should jointly have a smaller numerical range and sufficient dispersion within that range. Guided by this analysis, we propose InfoQuant, a train-free method that employs Peak Suppression Orthogonal Transformation (PSOT) to shape activations into more quantization-friendly distributions. We further introduce adaptive outlier-token selection to improve the robustness of PSOT during optimization. Across multiple LLM families, InfoQuant consistently outperforms prior PTQ and end-to-end training baselines. Under W4A4KV4, it preserves 97% of floating-point accuracy on average and reduces the LLaMA-2 13B performance gap by 42% over the previous state of the art. Code is available at https://github.com/LLIKKE/InfoQuant

Abstract

Explore similar work

InfoQuant: Shaping Activation Distributions for Low-Bit LLM Quantization

DynamicPTQ: Mitigating Activation Quantization Collapse via Residual-Stream Dynamics

AAAC: Activation-Aware Adaptive Codebooks for 4-bit LLM Weight Quantization