Deflating the Hessian: Rank-4 W4A4 Quantization for Multimodal Diffusion Transformers

Shiwen Wang; Pengxiang Zhao; Xiaoming Yuan

cs.LGOct 8, 2026

Deflating the Hessian: Rank-4 W4A4 Quantization for Multimodal Diffusion Transformers

Authors: Shiwen Wang, Pengxiang Zhao, Xiaoming Yuan

Organizations: The University of Hong Kong · Huawei Technologies Co., Ltd.

Abstract

In diffusion transformers, low-rank branches can mitigate 4-bit weight--activation (W4A4) post-training quantization (PTQ) loss by decomposing each weight into a low-bit residual and a high-precision low-rank component. Existing low-rank PTQ approaches, however, either optimize low-rank compensation and residual quantization separately, often requiring higher ranks, or rely on second-order weight updates without explicitly modeling activation quantization error, which becomes particularly pronounced under 4-bit quantization. To address these limitations, we present \method{}, a unified framework modeling low-rank-assisted W4A4 PTQ as a coupled calibration problem and deriving optimization-based solvers from the joint objective. Eliminating the output-side low-rank factor yields a \emph{deflated Hessian} that discounts residual errors already captured by the low-rank component, while an activation-noise surrogate is incorporated to suppress activation quantization error. Across five diffusion backbones, rank-4 \method{} consistently outperforms rank-4 SVDQuant in PSNR and LPIPS. It further surpasses rank-32 SVDQuant on SANA-1.6B, FLUX.1-schnell, and FLUX.1-dev with an $8\times$ smaller rank and up to $6.25\times$ faster quantization. Furthermore, on the Qwen3-8B LLM, rank-4 \method{} improves MMLU accuracy from 61.50% to 68.17% over rank-32 SVDQuant. Overall, \method{} achieves better W4A4 performance with substantially lower rank and quantization cost.

Figures & tables

Panel of Figure 1 — Figure 1: (a) Residual quantization with Hessian deflation: For a fixed protected subspace ${\bm{L}}_{1}$ , Hessian deflation ( $\widetilde{\bm{H}}_{\perp}{\bm{L}}_{1}=0$ ) focuses residual quantization strictly on errors outside the low-rank subspace. (b) Paired rank efficiency: The paired distributions show consistent gains over rank-4 SVDQuant, and show model-dependent comparisons with rank-32 SVDQuant, with the strongest rank-efficiency gains on SANA and the FLUX variants.

PixArt- $\Sigma$
	MJHQ				sDCI
Method	PSNR $\uparrow$	LPIPS $\downarrow$	CLIP $\uparrow$	IR $\uparrow$	PSNR $\uparrow$	LPIPS $\downarrow$	CLIP $\uparrow$	IR $\uparrow$
H-SVDQuant $r=4$	16.47	0.386	25.995	0.8852	15.36	0.422	26.203	0.9816
SVDQuant $r=4$	14.94	0.478	25.832	0.7489	13.75	0.521	25.604	0.8514
SVDQuant $r=32$	16.27	0.385	25.962	0.8950	15.45	0.413	25.718	0.9002
DiRotQ	16.36	0.389	26.047	0.9165	15.39	0.416	26.239	1.0417

Table 1: W4A4 fidelity and generation quality across modern diffusion transformer backbones. Bold and underline denote the best and second-best results, respectively. All evaluated methods use 4-bit weights and 4-bit activations (W4A4) with group-64.

Figure 2: W4A4 reconstructions across diffusion transformers. Each row compares dense FP16/BF16 against quantized methods under identical prompt and seed. Blue boxes highlight H-SVDQuant ( $r=4$ ). Across all backbones, H-SVDQuant preserves textures, fine structures, and prompt alignment faithful to the 16-bit reference.

Model	Method	Minutes	Speedup vs. SVDQuant r=4	Peak GiB
SANA-1.6B	H-SVDQuant	4.8	$6.25\times$	9.9
	SVDQuant	30.0	–	11.7
FLUX-schnell	H-SVDQuant	282.4	$1.81\times$	52.0
	SVDQuant	511.1	–	50.7
FLUX-dev	H-SVDQuant	285.6	$1.85\times$	52.0
	SVDQuant	527.4	–	51.3

Table 2: Recorded rank-4 PTQ cost on NVIDIA H20. Time is each checkpoint’s complete launcher wall interval; peak memory is the recorded CUDA allocation.

Method	MMLU	ARC-C	ARC-E	HellaSwag	PIQA	Mean
FP16	72.95	56.40	83.54	74.97	76.88	72.95
SVDQuant adaptation $r=4$	61.34	45.31	73.02	67.83	71.76	63.85
SVDQuant adaptation $r=32$	61.50	48.89	74.83	67.84	72.20	65.05
H-SVDQuant $r=4$	68.17	49.83	78.54	70.30	73.34	68.04

Table 3: Qwen3-8B zero-shot evaluation (W4A4, group-128). Mean denotes the unweighted average across all five downstream tasks. SVDQuant refers to the local language-model adaptation; FP16 serves as dense reference.

Table 4: Rank efficiency and activation-error propagation on Qwen3-8B (W4A4, group-128,

\lambda_{A}=0.25

). (a) WikiText-2 perplexity across different protected ranks. (b) Relative activation MSE averaged over 36 transformer blocks using eight 512-token sequences (measured against 16-bit reference activations). Local MSE measures per-block quantization error under reference inputs, while propagated MSE additionally captures errors accumulated from preceding quantized blocks.

Table 5: Activation-noise-weight sensitivity on PixArt-

\Sigma

(group-64), and Qwen3-8B (group-128). (W4A4,

r=4

). We evaluating sensitivity to activation-noise weighting parameter

\lambda_{A}

across vision generation (PixArt-

\Sigma

) and language modeling (Qwen3-8B). Bold denotes best value in each backbone. In Qwen, we disable the optional cached iteration and the outer iteration, the result of

\lambda_{A}=0

matches Table 6 .

${\bm{L}}_{1}$ initialization	Original $\widetilde{\bm{H}}$	Deflated $\widetilde{\bm{H}}_{\perp}$
Plain SVD	10.559	10.539
Calibration-weighted	10.492	10.486

Table 6: Controlled ablation of protected-subspace initialization (in Equation ( 12 ).) and Hessian deflation on Qwen3-8B (W4A4, group-128,

\lambda_{A}=0

) . We keep all other calibration settings fixed. Each row of

\widetilde{\bm{H}}

\widetilde{\bm{H}}_{\perp}

runs shares the same initialized

{\bm{L}}_{1}

and residual. Lower WikiText-2 PPL is better.

${\bm{D}}$ construction	Quant.	PPL $\downarrow$	MMLU $\uparrow$	ARC-C $\uparrow$	ARC-E $\uparrow$	Hella. $\uparrow$	PIQA $\uparrow$	Avg. $\uparrow$
Grid-search ${\bm{D}}$ (SVDQuant)	W4A4	10.895	66.59	51.11	79.21	68.26	74.05	67.84
Closed-form ${\bm{D}}$ (Ours)	W4A4	10.469	67.23	50.77	78.16	70.77	75.24	68.44
Grid-search ${\bm{D}}$ (SVDQuant)	W4A16	10.078	70.83	53.75	82.03	73.80	76.82	71.45
Closed-form ${\bm{D}}$ (Ours)	W4A16	9.976	70.94	54.18	82.37	73.47	77.09	71.61

Table 7: Effect of our closed-form smoothing solution on Qwen3-8B (W4A4/W4A16, group-128,

\lambda_{A}=0.25

) . We keep the H-SVDQuant pipeline fixed and replace the SVDQuant grid-searched

{\bm{D}}

with our closed-form solution in Equation ( 13 ). We disable the optional cached optimization of

{\bm{D}}

and only use its initialization. Avg. denotes the mean accuracy over MMLU, ARC-Challenge, ARC-Easy, HellaSwag, and PIQA. Lower PPL and higher accuracies are better.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

FLUX.1-schnell (4 steps)
	MJHQ				sDCI
Method	PSNR $\uparrow$	LPIPS $\downarrow$	CLIP $\uparrow$	IR $\uparrow$	PSNR $\uparrow$	LPIPS $\downarrow$	CLIP $\uparrow$	IR $\uparrow$
H-SVDQuant $r=4$	17.93	0.260	25.238	0.9029	16.79	0.278	25.838	1.0761
SVDQuant $r=4$	16.18	0.350	25.394	0.8951	15.12	0.363	25.791	1.0844
SVDQuant $r=32$	17.31	0.298	25.223	0.9394	16.23	0.311	25.712	1.0947
DiRotQ	18.56	0.241	24.994	0.9089	17.41	0.257	25.588	1.0951

Appendix

Table 8: W4A4 fidelity and generation quality on accelerated diffusion backbones (FLUX.1-schnell and Qwen-Image Lightning). Both models use 4-step sampling on the predescribed dataset. CLIP and ImageReward (IR) are absolute scorer means, not teacher-relative differences; neither measures teacher fidelity. Bold and underline mark the best and second-best unrounded means per subset and metric. H-SVDQuant use group-64 residuals and the stated protected rank; DiRotQ uses matched A4/group-64.

Figure 3: Qwen-Image base-model reconstructions. The first column is the high-precision reference; the blue outline identifies H-SVDQuant . All methods share the displayed prompt within each row. These qualitative examples complement the complete-subset results in Table 1 .

Figure 4: Four-step Qwen-Image Lightning reconstructions. Column order follows Figure 3 . The accelerated quantitative comparison appears in Table 8 .

Figure 5: Paired teacher-fidelity differences across five diffusion backbones. This expanded version of Figure 1 adds Qwen-Image and compares rank-4 H-SVDQuant against SVDQuant ranks 4 and 32 on identical prompts and seeds. Columns separate baseline rank; rows show PSNR gain and LPIPS reduction. Boxes show the interquartile range, white bars the medians, whiskers the 5th–95th percentiles, and black dots the means. Red denotes MJHQ and blue sDCI; positive values favor H-SVDQuant . The compared checkpoints retain their respective activation quantizers and are complete-recipe comparisons. Qwen-Image images are paired by index under identical prompt, seed, and teacher protocols.

Model	Between blocks	Within block	PSNR	LPIPS
PixArt- $\Sigma$	quantized	sequential	15.523	0.4834
	reference	sequential	15.701	0.4675
	reference	independent	15.833	0.4662
FLUX-schnell	quantized	sequential	17.471	0.3003
	reference	sequential	17.703	0.2961
	reference	independent	17.059	0.3151

Appendix

Table 9: Three-way calibration replay pilot. All rows use rank 4, W4A4/group-64,

\lambda_{A}=0

, and the weight-error code path. Within each model, evaluation uses the same 16 MJHQ prompts and image identifiers. These pilot checkpoints are not substituted into Table 1 .

Explore similar work

KroQuant: Kronecker-Structured Block Transforms for Efficient Post-Training Quantization of Diffusion Transformers
Jul 23, 2026Yann Bouquet, Alireza Khodamoradi, Kristof Denolf +1Diffusion Transformer Diffusion Model Quantization
EPFL, Lausanne, Switzerland. · Advanced Micro Devices, Inc., Longmont, CO, USA.
LoRaQ: Optimized Low Rank Approximation for 4-bit Quantization
Apr 20, 2026Yann Bouquet, Alireza Khodamoradi, Sophie Yáng Shen +2Mixed-Precision Quantization Diffusion Transformer
Research and Advancement Development Team, Advanced Micro Devices, Inc, USA · Computer Vision Laboratory, Ecole Polytechnique F´ed´erale de Lausanne, Lausanne, Switzerland.
DiRotQ: Rotation-Aware Quantization for 4-bit Diffusion Transformers
May 16, 2026Sayeh Sharify, Mahsa Salmani, Hesham MostafaDiffusion Transformer Diffusion Model Quantization
d-Matrix Santa Clara, CA 95054

Jul 23, 2026, cs.LGJ/K move · Enter open · S save

KroQuant: Kronecker-Structured Block Transforms for Efficient Post-Training Quantization of Diffusion Transformers

Yann Bouquet, Alireza Khodamoradi, Kristof Denolf, Mathieu Salzmann

EPFL, Lausanne, Switzerland. · Advanced Micro Devices, Inc., Longmont, CO, USA.

Diffusion Transformer Diffusion Model Quantization 4-Bit Quantization Activation Quantization Post-Training Quantization Neural Network Quantization

Open paper AskPDF arXiv

Post-training quantization (PTQ) of diffusion transformers (DiTs) to W4A4 severely degrades output quality, because activations entering each linear layer contain outliers that 4-bit formats cannot represent. The standard fix applies an invertible linear transform to the activations and its inverse to the weights before quantizing both. Normalization layers between blocks force this transform to run online at every denoising step, making its inference computation cost the binding design constraint. Existing options trade quantization quality for inference cost: per-channel scaling (SmoothQuant) is computationally cheap but impacts the magnitude of the channels, which can harm quantization accuracy; fixed Hadamard transforms yield better quantization accuracy but require large block sizes that incur a high online cost; learned full-

d

invertible transforms calibrate best but entail a prohibitive dense

d \times d

matrix multiplication (GEMM) per layer per step. We propose KroQuant, a PTQ method that applies a learned Kronecker-structured invertible transform to each 32-element block of the activation, storing less than half the parameters of per-channel scaling. The block-local structure runs as small tensor-core GEMMs, and on an MI350 GPU the KroQuant quantizer kernel is up to

14\%

faster than the SmoothQuant kernel. Offline LoRaQ weight calibration then absorbs the residual per-weight quantization error. On PixArt-

Σ

, SANA, and FLUX.1-schnell at W4A4 (MXFP4e2), KroQuant produces outputs closer to the FP reference than SVDQuant and LoRaQ on MJHQ-30K and SDCI, while preserving or improving image quality.

Abstract

Explore similar work

KroQuant: Kronecker-Structured Block Transforms for Efficient Post-Training Quantization of Diffusion Transformers

LoRaQ: Optimized Low Rank Approximation for 4-bit Quantization

DiRotQ: Rotation-Aware Quantization for 4-bit Diffusion Transformers