SemanticDialect: Semantic-Aware Mixed-Format Quantization for Video Diffusion Transformers

Wonsuk Jang; Thierry Tambe

cs.CVMar 3, 2026

SemanticDialect: Semantic-Aware Mixed-Format Quantization for Video Diffusion Transformers

Authors: Wonsuk Jang, Thierry Tambe

Organizations: Stanford University

Abstract

Diffusion Transformers (DiTs) achieve state-of-the-art video generation quality, but their substantial memory and computational footprints hinder edge deployment. Quantization can reduce these costs, yet existing methods often degrade video quality due to high activation variation and the difficulty of preserving semantic and temporal coherence. We propose SemanticDialect, which advances block-wise mixed-format quantization. In this framework, each block selects an optimal format (dialect) from a candidate set (formatbook), which is augmented with lookup tables that store quantization errors and quantized indices, enabling efficient per-block format selection and quantization with minimal online overhead. We further introduce attention-guided activation decomposition, which reduces quantization error via residual quantization, and semantic-aware dialect assignment (SeDA), which reduces cross-token quantization inconsistency by enforcing format uniformity among semantically correlated tokens. Experiments demonstrate that SemanticDialect outperforms prior quantization methods and block-wise formats (MXFP4, NVFP4) while approaching FP16 quality on Open-Sora 2.0. We also validate hardware deployability through RTL design and GPU kernel implementation.

Figures & tables

Figure 1: SemanticDialect is a post-training quantization method for video diffusion transformers. It quantizes Open-Sora 2.0 to 4-bit weights and activations while maintaining near-FP16 visual quality.

Figure 2: SemanticDialect overview. (a) LUT-based format selection and quantization. (b) Activation decomposition for quantization-sensitive layers. (c) SeDA for consistent block-wise quantization.

Figure 3: Proposed formatbook and lookup table. (a) Four formatbook construction ground rules. (b) Block-wise normalized value distributions. (c) Lookup table details and data-to-index mapping.

Figure 4: Activation decomposition for vector and matrix activations.

Figure 5: Semantic-Aware Dialect Assignment (SeDA). (a) Anchor/correlated token selection. (b) Sub-formatbook construction. (c) Average anchor-token displacement across denoising timesteps.

Model	Method	Block [-1pt] Size	Eff. bit [-1pt] (A/W)	Aesthetic [-1pt] Quality	Imaging [-2pt] Quality	Motion [-1pt] Smooth.	Dynamic [-2pt] Degree	Subject [-2pt] Consist.	BG. [-1pt] Consist.	Scene [-1pt] Consist.	SS. [-1pt] Consist.
Open-Sora 1.0 factorized attn	FP16	-	16 / 16	58.00	62.03	96.41	54.17	91.51	96.43	36.17	26.14
	MXFP4	16	4.31 / 4.31	33.22	30.78	98.74	82.87	85.24	94.28	4.70	14.39
	MXFP4	32	4.16 / 4.16	34.69	37.80	98.59	84.72	85.99	94.53	5.69	15.00
	NVFP4	16	4.5 / 4.5	51.41	52.46	97.29	56.94	88.77	95.50	16.28	23.17
	NVFP4	32	4.25 / 4.25	46.82	47.26	97.90	50.00	88.80	95.61	13.42	20.88
	Block Dialect	16	4.56 / 4.56	49.03	47.65	98.02	58.80	88.25	95.12	15.72	21.97

Table 1: Text-to-video generation quality on VBench benchmark suite for Open-Sora 1.0 and 2.0.

Model [1pt]	Method [1pt]	Eff. bit [-1pt] (A/W)	Aesthetic [-1pt] Quality	Imaging [-2pt] Quality	Motion [-1pt] Smooth.	Dynamic [-2pt] Degree	Subject [-2pt] Consist.	BG. [-1pt] Consist.	Scene [-1pt] Consist.	SS. [-1pt] Consist.
Open-Sora 1.0	Exact MSE		45.84	50.44	98.40	45.83	90.71	95.78	9.52	19.94
	FB4 (g=32)	4.31/4.31	45.97	50.61	98.37	48.61	90.46	95.63	10.12	19.77
	FB4 (g=8)		45.06	50.15	98.41	44.44	90.68	95.41	9.60	19.73
Open-Sora 2.0	Exact MSE		57.52	64.01	98.67	59.72	92.32	97.01	40.63	27.46
	FB4 (g=32)	4.31/4.31	57.62	63.31	98.67	54.17	92.35	97.03	42.63	27.54
	FB4 (g=8)		57.44	63.80	98.69	58.33	92.07	96.81	40.63	27.54

Table 2: Dialect selection strategy comparison (LUT-based vs. MSE-based). LUT-based selection matches or outperforms MSE-based strategy. We denote num_group by

g

Method	Block size	FVD-FP16 ( $\downarrow$ )	CLIP-Temp	CLIPSIM	VQA-A	VQA-T	$\Delta$ Flow ( $\downarrow$ )
FP16	–	–	0.9987	0.1798	51.66	48.39	–
FB4	16	1.24	0.9983	0.1781	50.72	47.87	0.26
+Decomp.		0.80	0.9989	0.1751	51.01	48.14	0.31
+SeDA		0.76	0.9988	0.1757	51.03	48.10	0.26
FB4	32	1.59	0.9986	0.1778	50.29	47.78	0.84
+Decomp.		1.36	0.9986	0.1758	50.88	48.06	0.55

Table 3: Stepwise ablation on Open-Sora 1.0 (OpenSora t2v_sample prompt set).

Figure 6: Hardware deployment analysis. (a) RTL: MAC and quantization unit comparison. (b) CUDA: Peak memory. (c) CUDA: DiT block inference latency (projected Blackwell-class latency).

Appendix figures & tables17 assets

Supplementary material from the paper’s appendix.

Appendix

	Aesthetic quality	Imaging quality	Motion smooth.	Dynamic degree	Subject consist.	BG. consist.	Scene consist.	SS. consist.
FP16	58.53	65.33	97.29	47.22	93.04	95.98	25.00	25.28
attn_temp.qkv	56.64	59.63	97.90	44.44	93.34	96.26	25.00	25.05
attn_temp.proj	58.47	65.55	97.27	47.22	93.12	95.88	27.90	25.21
attn.qkv	57.32	64.44	97.21	58.33	92.29	95.37	29.09	25.27
attn.proj	58.37	65.15	97.18	45.83	93.07	95.95	25.30	25.62
x_attn.q_linear	58.92	64.99	97.04	47.22	92.44	95.93	28.94	25.16

Appendix

Table 4: Open-Sora 1.0: per-layer sensitivity. Top: FP16 baseline with only one layer quantized. Bottom: FB4 baseline with only one layer set to FP16. Bold : notable degradation or improvement.

	Aesthetic quality	Imaging quality	Motion smooth.	Dynamic degree	Subject consist.	BG. consist.	Scene consist.	SS. consist.
FP16	58.05	64.58	98.83	59.72	94.43	97.80	40.85	27.37
img_attn.qkv	57.74	64.51	98.80	56.94	93.83	97.74	41.07	27.40
img_attn.proj	57.91	64.64	98.83	55.56	94.21	97.83	41.74	27.46
img_mlp.0	57.35	64.81	98.63	59.72	93.51	97.27	41.44	27.61
img_mlp.2	57.41	64.85	98.79	55.56	94.27	97.76	41.89	27.49
img_mod.lin	57.90	65.31	98.71	54.17	93.65	97.57	40.63	27.42

Appendix

Table 5: Open-Sora 2.0: per-layer sensitivity. Top: FP16 baseline with only one layer quantized. Bottom: FB4 baseline with only one layer set to FP16. Bold : notable degradation or improvement.

Method		Aesthetic [-1pt] Quality	Imaging [-2pt] Quality	Motion [-1pt] Smooth.	Dynamic [-2pt] Degree	Subject [-2pt] Consist.	BG. [-1pt] Consist.	Scene [-1pt] Consist.	SS. [-1pt] Consist.
(a)	Act. decomposition scoring
	Incoming (avg. over queries)	50.10	57.01	98.22	47.22	91.93	95.68	11.61	21.70
	Outgoing (avg. over keys)	51.25	60.00	97.62	48.61	90.74	95.04	17.04	21.99
(b)	SeDA anchor/corr. scoring
	Post-softmax	50.97	60.26	97.57	47.22	90.44	94.92	16.74	22.08
	Pre-softmax	51.22	60.40	97.61	47.22	91.17	94.98	20.61	22.32

Appendix

Table 6: Ablation of attention-score choices on Open-Sora 1.0.

Method		Aesthetic [-1pt] Quality	Imaging [-2pt] Quality	Motion [-1pt] Smooth.	Dynamic [-2pt] Degree	Subject [-2pt] Consist.	BG. [-1pt] Consist.	Scene [-1pt] Consist.	SS. [-1pt] Consist.
Target selection	Timestep range
Full tensor	Steps 0–24	47.90	54.30	97.98	41.67	90.28	95.42	18.01	20.66
Full tensor	Steps 75–99	46.19	49.68	98.57	43.06	90.74	95.41	11.46	20.12
Random (25%)	All steps	51.14	58.23	98.10	47.22	91.15	95.74	16.96	21.46
Top-25% magnitude	All steps	50.27	54.57	98.10	36.11	91.65	95.63	13.10	21.74
Top-1 mean attn / $1{\times}4$ tile	All steps	51.25	60.00	97.62	48.61	90.74	95.04	17.04	21.99

Appendix

Table 7: Ablation of salient token selection and attention scoring configurations for activation decomposition (w/o SeDA) on Open-Sora 1.0. “cond” denotes allocating the full salient token budget to the conditional branch only (rather than splitting it across conditional and unconditional branches).

Method			Aesthetic [-1pt] Quality	Imaging [-2pt] Quality	Motion [-2pt] Smooth.	Dynamic [-2pt] Degree	Subject [-2pt] Consist.	BG. [-1pt] Consist.	Scene [-1pt] Consist.	SS. [-1pt] Consist.
(a)	Temporal-axis integration
	Per-frame anchors (2D SeDA)		51.49	60.93	97.55	45.83	91.04	94.74	14.73	22.17
	Cross-frame main anchor (3D SeDA)		51.22	60.40	97.61	47.22	91.17	94.98	20.61	22.32
(b)	Anchor tile	Correlation window
	$4\times 4$	$7\times 7$	50.88	60.11	97.59	48.61	91.15	95.00	18.90	22.17
	$4\times 4$	$3\times 3$	51.07	60.18	97.58	43.06	90.67	94.81	19.12	21.92

Appendix

Table 8: SeDA configuration ablations on Open-Sora 1.0. In the timestep update schedule, the update period denotes the token-update interval in timesteps (1 = every timestep); “

x

(20–89)” means period

x

on timesteps 20–89 and period 1 otherwise.

Prompt	Method	Block size	FVD-FP16 ( $\downarrow$ )	CLIP-Temp	CLIPSIM	VQA-A	VQA-T	$\Delta$ Flow ( $\downarrow$ )
t2v_sample	NVFP4	16	1.12	0.9978	0.1752	50.58	47.97	0.48
	FB4	16	1.24	0.9983	0.1781	50.72	47.87	0.26
	NVFP4	32	2.11	0.9976	0.1775	49.91	47.43	0.21
	FB4	32	1.59	0.9986	0.1778	50.29	47.78	0.84
UCF-101	NVFP4	16	67.08	0.9971	0.2021	49.69	47.31	3.49
	FB4	16	61.80	0.9976	0.2072	49.89	47.54	1.33

Appendix

Table 9: Comparison between NVFP4 and FB4 (w/o activation decomposition, SeDA) on additional metrics on Open-Sora 1.0. num_group is 8 for FB4.

Dataset	Method	FVD-FP16 ( $\downarrow$ )	CLIP-Temp	CLIPSIM	VQA-A	VQA-T	$\Delta$ Flow ( $\downarrow$ )
	FP16	-	0.9973	0.2044	51.11	47.96	-
UCF-101	FB4	85.86	0.9972	0.2061	48.77	46.93	3.91
	+Decomp.	67.04	0.9972	0.2048	49.74	47.45	2.50
	+SeDA	67.46	0.9971	0.2050	49.75	47.46	1.36
	FP16	-	0.9984	0.1953	51.54	48.15	-
VBench	FB4	62.96	0.9980	0.1916	49.48	47.31	2.68

Appendix

Table 10: Stepwise quality improvement across multiple prompt sets (block size 32).

Method	Block [-1pt] Size	Aesthetic [-1pt] Quality	Imaging [-2pt] Quality	Motion [-1pt] Smooth.	Dynamic [-2pt] Degree	Subject [-2pt] Consist.	BG. [-1pt] Consist.	Scene [-1pt] Consist.	SS. [-1pt] Consist.
SemanticDialect	16	54.49	63.72	97.66	34.72	93.28	95.86	22.92	23.47
SemanticDialect +Rotation	16	55.52	64.95	97.40	36.11	93.18	95.51	26.64	23.93
SemanticDialect	32	51.22	60.40	97.61	47.22	91.17	94.98	20.61	22.32
SemanticDialect +Rotation	32	55.07	62.70	97.31	40.28	92.82	95.65	24.40	23.74

Appendix

Table 11: Synergy of rotation-based PTQ methods on Open-Sora 1.0.

Figure 8: Qualitative results on Open-Sora 1.0 for the prompt: “The vibrant beauty of a sunflower field …”. Effective bit-width (A/W): NVFP4 = 4.5/4.5, Ours = 4.76/4.31. NVFP4 shows noisy leaves whose shapes are difficult to recognize throughout the video and exhibits poor structural consistency. In contrast, SemanticDialect preserves the sunflower and leaf structures more clearly and maintains better visual consistency over time.

Figure 9: Qualitative results on Open-Sora 1.0 for the prompt: “A cat eating food out of a bowl”. Effective bit-width (A/W): NVFP4 = 4.5/4.5, Ours = 4.76/4.31. NVFP4 frequently produces frames where the cat and food become entangled, with the food in the bowl exhibiting significant noise and flickering artifacts. FP16 also occasionally produces such entangled frames but remains visually sharp overall. In contrast, SemanticDialect maintains more stable cat and food shapes compared to NVFP4 and achieves better imaging quality.

Figure 10: Qualitative results on Open-Sora 1.0 for the prompt: “A vibrant scene of a snowy mountain landscape …”. Effective bit-width (A/W): NVFP4 = 4.5/4.5, Ours = 4.76/4.31. NVFP4 shows a blurred background due to lower imaging quality. In contrast, SemanticDialect preserves clearer background details and more detailed hot-air balloons, resulting in a more visually faithful scene.

Figure 11: Qualitative results on Open-Sora 1.0 for the prompts “A snowy forest landscape with a dirt road …” and “The video captures the majestic beauty of a waterfall …”. Effective bit-width (A/W): NVFP4 = 4.5/4.5, Ours = 4.76/4.31. SemanticDialect produces higher-quality videos that more closely match FP16 than NVFP4.

Figure 12: Qualitative results on Open-Sora 2.0 for the prompt: “an airplane soaring through a clear blue sky” and "ski slope". Effective bit-width (A/W): NVFP4 = 4.25/4.25, Ours = 4.31/4.31. SemanticDialect shows better subject quality over NVFP4.

Figure 13: Qualitative results on Open-Sora 2.0 for the prompt: “A woman in a red coat and black scarf walks through a crowded night market…". Effective bit-width (A/W): NVFP4 = 4.5/4.5, Ours = 4.63/4.63.

Figure 14: Qualitative results on Open-Sora 2.0 for the prompt: “A man in a navy suit gives a presentation in a modern glass-walled office meeting room…". Effective bit-width (A/W): NVFP4 = 4.5/4.5, Ours = 4.63/4.63.

Figure 15: Qualitative results on Open-Sora 2.0 for the prompt: “A woman in a black dress slowly walks through a contemporary art museum gallery with large paintings…". Effective bit-width (A/W): NVFP4 = 4.5/4.5, Ours = 4.63/4.63.

Diffusion Model Quantization

Diffusion model quantization compresses model weights and activations to reduce inference memory and latency while preserving generated image, video, and speech quality.

See all 39 papers

6Bit-Diffusion: Inference-Time Mixed-Precision Quantization for Video Diffusion Models
Mar 19, 2026Rundong Su, Jintao Zhang, Zhihang Yuan +3Diffusion Model Quantization Mixed-Precision Quantization
Fudan University · Tsinghua University · Shanghai Jiao Tong University
Boundary-Protection W8A8 HiFloat8 Quantization for Large-Scale Text-to-Video Diffusion Transformers
May 31, 2026Yiming ZhaoVideo Diffusion Models Diffusion Model Quantization
University of Chinese Academy of Sciences Beijing, China
DSAQuant: Denoising-Stage-Aligned Quantization-Aware Training for Video Generation
Sep 3, 2026Shuaiting Li, Zelin Gao, Haibin Shen +3Diffusion Model Guidance Video Diffusion Models
1Robbyant · 2ZJU · 4HKUST +1

Mar 19, 2026J/K move · Enter open · S save

6Bit-Diffusion: Inference-Time Mixed-Precision Quantization for Video Diffusion Models

Rundong Su, Jintao Zhang, Zhihang Yuan, Haojie Duanmu+2

Fudan University · Tsinghua University · Shanghai Jiao Tong University

Diffusion Model Quantization Mixed-Precision Quantization Post-Training Quantization Video Diffusion Models Diffusion Model Caching

Open paper AskPDF arXiv

Diffusion transformers have demonstrated remarkable capabilities in generating videos. However, their practical deployment is severely constrained by high memory usage and computational cost. Post-Training Quantization provides a practical way to reduce memory usage and boost computation speed. Existing quantization methods typically apply a static bit-width allocation, overlooking the quantization difficulty of activations across diffusion timesteps, leading to a suboptimal trade-off between efficiency and quality. In this paper, we propose a inference time NVFP4/INT8 Mixed-Precision Quantization framework. We find a strong linear correlation between a block's input-output difference and the quantization sensitivity of its internal linear layers. Based on this insight, we design a lightweight predictor that dynamically allocates NVFP4 to temporally stable layers to maximize memory compression, while selectively preserving INT8 for volatile layers to ensure robustness. This adaptive precision strategy enables aggressive quantization without compromising generation quality. Beside this, we observe that the residual between the input and output of a Transformer block exhibits high temporal consistency across timesteps. Leveraging this temporal redundancy, we introduce Temporal Delta Cache (TDC) to skip computations for these invariant blocks, further reducing the computational cost. Extensive experiments demonstrate that our method achieves 1.92

\times

end-to-end acceleration and 3.32

\times

memory reduction, setting a new baseline for efficient inference in Video DiTs.

Abstract

Diffusion Model Quantization

Explore similar work

6Bit-Diffusion: Inference-Time Mixed-Precision Quantization for Video Diffusion Models

Boundary-Protection W8A8 HiFloat8 Quantization for Large-Scale Text-to-Video Diffusion Transformers

DSAQuant: Denoising-Stage-Aligned Quantization-Aware Training for Video Generation