Chameleon: Dynamic Format Adapter for Efficient Diffusion

Arnab Sanyal; Sandeep Chinchali

cs.LGSep 27, 2026

Chameleon: Dynamic Format Adapter for Efficient Diffusion

Authors: Arnab Sanyal, Sandeep Chinchali

Organizations: UT SWARM Lab, Department of Electrical & Computer Engineering The University of Texas, Austin TX 78712

Abstract

Post-training quantization (PTQ) is the standard way to run modern diffusion models on memory-constrained accelerators, yet every existing diffusion PTQ scheme fixes the $\mathit{number\ format}$ in advance and only tunes the scale, zero point, or per-layer bit-width. At a fixed bit-width the best format depends on the distribution being encoded, and that distribution differs across weight channels, across layers, and along the diffusion timestep, where activation distributions slide from heavy-tailed and noise-dominated to tightly clustered and structured. We propose Chameleon, a PTQ framework that holds the bit-width fixed and treats the format itself as a discrete variable, chosen per weight channel and per (layer, timestep bucket) activation tensor. Activation formats come from {INT8, FP8 E4M3, FP8 E5M2, MXFP8, MXINT8}, selected ahead of time from two cheap statistics (empirical kurtosis and the closed-form diffusion SNR) and stored in a lookup table; weight formats come from {INT8, MXINT8} at 8 bits or {INT4, NF4, FP4 E2M1, MXINT4, MXFP4} at 4 bits, selected offline by reconstruction error. An architectural fork adapts the same selection layer to multi-step UNets, single-step distilled models, and Diffusion Transformers. Across SDXL, SDXL-Turbo, and PixArt- $α$ on COCO-2014, Chameleon achieves the best FID in all six backbone $\times$ bit-width settings, with CLIP within 0.24 of the FP16 reference and the best of all quantized methods at $W_{4}A_{8}$ .

Figures & tables

Panel of Figure 1 — Figure 1: Images generated by each per-architecture PTQ baseline and by Chameleon at 4-bit weights (W 4 A 8 ), from the COCO-2014 caption “An old city fire hydrant has ‘Love was found here’ painted on it.”

Figure 2: Chameleon framework: Offline, a one-pass calibrator accumulates per-(layer, bucket) activation moments for the empirical kurtosis

\kappa_{l}(t)

, while the diffusion

\mathrm{SNR}(t)

is read from the noise schedule. The router applies the rule of Section 4.3 to both and writes the format into an integer lookup table

F\in\{0,\ldots,4\}^{L\times B}

(code 4 is unused); weights are selected offline (Sections 4.5 and 4.6 ). At inference, one fake-quantization module reads the LUT per layer per step on the multi-step paths and dispatches to the matching routine.

Setup	Method	Bits	FID $\downarrow$	CLIP $\uparrow$
SDXL (50 steps, CFG 7.5, $1024\times 1024$ )	FP16 baseline	W 16 A 16	16.16	26.88
	Q-Diffusion	W 8 A 8	14.70	26.77
	PTQ4DM	W 8 A 8	14.65	26.68
	Chameleon *	W 8 A 8	14.23	26.70
	Q-Diffusion	W 4 A 8	21.41	26.27
	PTQ4DM	W 4 A 8	21.82	26.21

Table 1: Combined evaluation on COCO -2014 (24,576 prompts, one image per caption). Clean- FID is computed against the full val2014 set and CLIP -score ( ViT-L/14 , cosine

\times 100

) against the matching captions; best FID per backbone (and CLIP for W 4 A 8 ) in bold . * Ours. In both SDXL-Turbo rows the nine up-block conv_shortcut layers stay FP16 (Section 4.6 ).

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 3: The UNet execution path of Chameleon ( SDXL base 1.0), with the few-step variant used for SDXL-Turbo . The encoder ( DownBlock2D and two CrossAttnDownBlock2D s) descends to a cross-attention mid block and the decoder climbs back up; at every level the concatenation

c

joins the decoder stream with one encoder skip before each of the three up-block ResNets – the encoder block’s two ResNet outputs and the downsampled output of the level above ( conv_in ’s output at the top). Activations follow macro-routing with static, ahead-of-time-calibrated per-bucket scales ( Q-Diffusion style), in contrast to the DiT path’s dynamic per-sample scales; the callout gives the measured share of the 372 quantized layers assigned each format in each bucket of the shipped LUT (Table 7 ). The nine layers matching up_blocks..resnets..conv1 – the

3\times 3

convolution that reads the concatenated tensor, three per level – are routed to MXFP8 E4M3 regardless of the kurtosis/ SNR reading, because that tensor spans a peak-to-median per-channel magnitude range of

58\times

at the median layer and up to

163\times

; the parallel

1\times 1

conv_shortcut reads the same tensor and is routed normally. Weights use per-output-channel format selection (Section 4.5 ); attention Q/K/V projections, conv_in / conv_out and both text encoders stay in FP16 . The few-step variant keeps the topology but runs one step at

512\times 512

: Chameleon supplies the weight palette, activations use MixDQ ’s static scales and mixed-precision assignment, MixDQ ’s BOS -aware bypass is dropped (Table 6 ), and the nine up-block conv_shortcut layers that read the concatenation stay in FP16 , with no MXFP8 pin (Section 4.6 ; Appendix A.4 ).

Figure 4: The DiT execution path of Chameleon ( PixArt -

\mathtt{\alpha}

XL/2 , 28 blocks). There is no shortcut concatenation of the kind the UNet carries, so the topology-driven MXFP8 rule of Section 4.6 is dropped. Weights: every nn.Linear in the self-attention, cross-attention and feed-forward sub-layers, together with the two caption_projection linears (282 in total; proj_out and adaln_single are kept in FP16 ) enters a joint

(g,f)

search over group sizes

g\in\{32,64,128,192,288\}

and formats

f\in\{\texttt{INT4-asym},\texttt{NF4},\texttt{FP4 E2M1}\}

at 4 bits, scored by an input-aware SNR that weights each input channel’s squared error by its activation energy from a 16-prompt pre-pass (it settles on

g=32

everywhere, with INT4-asym for 213 linears and NF4 for 69); the search is per-linear, groups run along the input-channel dimension, and it runs entirely offline. Activations: macro-routing (the per-bucket format LUT of equation 2 ) composed with per-tensor dynamic per-sample scaling. The bucket-to-format assignment shown is the calibrated LUT for this model: FP8 E5M2 wins in

97

–

100\%

of the 282 linears in every bucket and INT8-asym is never selected, so the three-phase progression of Appendix A.3 does not occur on the DiT . Appendix A.9.5 reports the full statistics. The format is fixed at calibration time, but the scale is computed dynamically from the current tensor’s range at inference, following Q-DiT ’s observation that DiT activations shift drastically across samples.

Model	Calib time	Peak GPU mem	LUT size
SDXL UNet	45 m 16 s	11.44 GB	252 KB (act. only) / 18 MB JSON of weight formats ( ${\approx}1$ MB at one byte per channel)
SDXL-Turbo	1 m 01 s	9.63 GB	157 KB (single config file)
PixArt - $\mathtt{\alpha}$	7 m 25 s	18.27 GB	93 KB (per-layer $(g,f)$ + LUT )

Appendix

Table 2: End-to-end calibration cost on a single A100 80 GB in fake-quant mode, with 128 calibration prompts and

B=10

buckets. The reported LUT size includes the per-(layer, bucket) format and scale tables; for SDXL and PixArt -

\mathtt{\alpha}

the larger figure additionally includes the per-output-channel weight-format metadata. SDXL-Turbo is an order of magnitude cheaper because its one-step schedule leaves no timestep axis to route over: calibration reduces to a weight-only format search with no calibration forward passes (Section 4.6 ). The three runs total 53 m 42 s.

Arm	Change	FID $\downarrow$	$\Delta$	CLIP $\uparrow$
both_default	$\kappa$ and SNR (shipped)	19.39	–	26.58
both_perturbed	$\kappa$ gates $(3,5)\to(3.5,5.5)$	19.41	$+0.02$	26.55
kurt_only	SNR gate disabled	19.60	$+0.21$	26.59
snr_only	$\kappa$ frozen at 4 (mid-band)	19.68	$+0.29$	26.38
single_e4m3	no routing, FP8 E4M3 everywhere	20.29	$+0.90$	26.48

Appendix

Table 3: Contribution of each routing statistic ( SDXL W 4 A 8 , 5,000 images).

Arm	Change	FID $\downarrow$	$\Delta$	CLIP $\uparrow$
full_palette	shipped	19.39	–	26.58
drop_nf4	NF4 removed (weights)	19.46	$+0.07$	26.56
drop_mxfp8	MXFP8 removed (shortcut rule off)	19.54	$+0.15$	26.56
drop_mxfp4	MXFP4 E2M1 removed (weights)	19.77	$+0.38$	26.60
single_e4m3	forced E4M3 , shortcuts included	20.29	$+0.90$	26.48
single_int8	forced INT8-asym , shortcuts included	180.51	$+161.12$	22.87

Appendix

Table 4: Contribution of individual formats ( SDXL W 4 A 8 , 5,000 images). drop_mxfp8 removes the only format the shortcut rule of Section 4.6 can select, and therefore disables that rule.

$B$	FID $\downarrow$	CLIP $\uparrow$
1	19.47	26.58
5	19.41	26.57
10 (shipped)	19.35	26.72
20	19.59	26.53

Appendix

Table 5: Sensitivity to the number of timestep buckets ( SDXL W 4 A 8 , 5,000 images). Each

B

is calibrated independently.

Family	Arm	Change	FID $\downarrow$	$\Delta$	CLIP $\uparrow$
SDXL-Turbo W 8 A 8	baq_off	shipped	26.78	–	26.56
SDXL-Turbo W 8 A 8	baq_on	faithful BAQ re-enabled	26.89	$+0.11$	26.61
PixArt - $\mathtt{\alpha}$ W 4 A 8	macro+micro	shipped	29.15	–	25.65
	micro_only	LUT flattened to E4M3	29.84	$+0.69$	25.83
	macro_only	scale frozen after first observation	174.42	$+145.27$	15.91

Appendix

Table 6: The per-family additions of Section 4.6 (5,000 images each). macro_only freezes the per-sample activation scale after the first observation (format LUT still active); micro_only keeps the dynamic scale but flattens the LUT to one format.

	SDXL (scaled-linear)					PixArt - $\mathtt{\alpha}$ (linear)
$b$	SNR	INT8	E4M3	E5M2	MXFP8	SNR	E4M3	E5M2
0	57.7	9.9%	50.3%	37.4%	2.4%	304.5	2.5%	97.5%
1	5.14	12.1%	48.4%	37.1%	2.4%	4.18	2.5%	97.5%
2	2.13	11.6%	49.2%	36.8%	2.4%	1.16	2.8%	97.2%
3	1.05	–	61.3%	36.3%	2.4%	0.41	3.2%	96.8%
4	0.54	–	62.4%	35.2%	2.4%	0.15	–	100%

Appendix

Table 7: Per-bucket activation-format distribution in the shipped LUT s (

B=10

). SDXL : 372 quantized layers; PixArt -

\mathtt{\alpha}

: 282. Bucket mean

\mathrm{SNR}(t)

is computed from each model’s own noise schedule. Bucket

b

is labelled T b+1 in Figures 3 and 4 .

Family	Backbone	FP16	PTQ baseline	Chameleon
UNet diffusion	SDXL	522	605	662
Few-step distilled	SDXL-Turbo	439	605	725
Diffusion transformer	PixArt - $\mathtt{\alpha}$	1287	1030	1057

Appendix

Table 8: High-frequency detail retention, measured as the Variance of the Laplacian ( VL ), a standard image sharpness and focus measure ( Pech-Pacheco et al., 2000 ; Pertuz et al., 2013 ) , over 400 images per configuration. Images are matched by filename across configurations, so every row compares the same prompts under the same seeds. Higher values indicate more high-frequency content; the FP16 column is the unquantized reference for that backbone, not a target to exceed.

Panel of Figure 5 — Figure 5: Qualitative comparison on four prompts at W 4 A 8 . Rows (top to bottom): FP16 reference ( SDXL , SDXL-Turbo , PixArt - $\mathtt{\alpha}$ ); per-architecture PTQ baseline ( Q-Diffusion , MixDQ , Q-DiT respectively); Chameleon. Columns (left to right): model architectures (UNet Diffusion, Few-Step Distilled, Diffusion Transformer). Within each $2\times 2$ block, the four prompts are, in reading order, the cherry-red convertible , the orange tabby cat , the golden retriever puppy , and the black sports car ; the prompts are given verbatim in the released code. Seeds are fixed across rows so that the same prompt index produces the same latent initialization under every quantization configuration.

Explore similar work

PermuQuant: Lowering Per-Group Quantization Error by Reordering Channels for Diffusion Models
May 10, 2026Yongsen Cheng, Kai Liu, Kaiwen Tao +5Quantized Diffusion Models
Shanghai Jiao Tong University · Huawei Noah’s Ark Lab
KroQuant: Kronecker-Structured Block Transforms for Efficient Post-Training Quantization of Diffusion Transformers
Jul 23, 2026Yann Bouquet, Alireza Khodamoradi, Kristof Denolf +1Post-Training Quantization Diffusion Transformers
EPFL, Lausanne, Switzerland. · Advanced Micro Devices, Inc., Longmont, CO, USA.
FourTune: Towards Fully 4-Bit Efficient Post-Training for Diffusion Models
Jul 7, 2026Bowen Xue, Zihan Min, Xingyang Li +8Post-Training Quantization Diffusion Models
Nunchux AI · MIT · Stanford +2

May 10, 2026, cs.CVJ/K move · Enter open · S save

PermuQuant: Lowering Per-Group Quantization Error by Reordering Channels for Diffusion Models

Yongsen Cheng, Kai Liu, Kaiwen Tao, Junxian Li+4

Shanghai Jiao Tong University · Huawei Noah’s Ark Lab

Quantized Diffusion Models Permutation Generative Models Order Matters

Open paper AskPDF arXiv

Large-scale visual generative models have achieved remarkable performance. However, their high computational and memory costs make deployment challenging in resource-constrained scenarios, such as interactive applications and personal single-GPU usage. Post-training quantization (PTQ) offers a practical solution by compressing pretrained models without expensive retraining. However, existing PTQ methods still suffer from severe quality degradation under extremely low-bit settings. In this paper, we identify channel ordering as an important but underexplored factor in per-group quantization. In this setting, each contiguous group shares one quantization scale. When channels with very different statistics are placed in the same group, the scale can be dominated by outliers and cause large quantization errors. Based on this observation, we propose PermuQuant, a simple and effective PTQ framework for low-bit diffusion models. PermuQuant sorts channels by a joint second-moment criterion before per-group quantization, placing channels with similar activation and weight statistics into the same group. It further uses a calibration-based acceptance rule to apply reordering only when the selected permutation reduces quantization error on calibration data. The selected permutations are absorbed into adjacent modules or applied to weights offline, avoiding explicit runtime permutation operations. Extensive experiments on multiple large diffusion models show that PermuQuant consistently reduces quantization error and outperforms existing PTQ baselines. On FLUX.1-dev with an RTX 5090, PermuQuant achieves up to a 1.7

\times

single step speedup and reduces the DiT memory footprint by 3.5

\times

under W4A4 NVFP4 quantization. Code will be available at https://github.com/yscheng04/PermuQuant.

Abstract

Explore similar work

PermuQuant: Lowering Per-Group Quantization Error by Reordering Channels for Diffusion Models

KroQuant: Kronecker-Structured Block Transforms for Efficient Post-Training Quantization of Diffusion Transformers

FourTune: Towards Fully 4-Bit Efficient Post-Training for Diffusion Models