Not all tokens contribute equally to diffusion learning

Guoqing Zhang; Lu Shi; Wanru Xu; Linna Zhang; Sen Wang; Fangfang Wang; Yigang Cen

cs.CVApr 8, 2026

Not all tokens contribute equally to diffusion learning

Authors: Guoqing Zhang, Lu Shi, Wanru Xu, Linna Zhang, Sen Wang, Fangfang Wang, Yigang Cen

Organizations: State Key Laboratory of Advanced Rail Autonomous Operation, Bejing Jiaotong University, Beijing, China · School of Computer Science and Technology, Bejing Jiaotong University, Beijing, China · Visual Intellgence +X International Cooperation Joint Laboratory of MOE, Bejing Jiaotong University, Beijing, China · School of Mechanical Engineering, Guizhou University, Guizhou, China · Seed, Bytedance, Beijing, China · School of Information Science and Technology, Hangzhou Normal University, Zhejiang, China

Abstract

With the rapid development of conditional diffusion models, significant progress has been made in text-to-video generation. However, we observe that these models often neglect semantically important tokens during inference, leading to biased or incomplete generations under classifier-free guidance. We attribute this issue to two key factors: distributional bias caused by the long-tailed token frequency in training data, and spatial misalignment in cross-attention where semantically important tokens are overshadowed by less informative ones. To address these issues, we propose Distribution-Aware Rectification and Spatial Ensemble (DARE), a unified framework that improves semantic guidance in diffusion models from the perspectives of distributional debiasing and spatial consistency. First, we introduce Distribution-Rectified Classifier-Free Guidance (DR-CFG), which regularizes the training process by dynamically suppressing dominant tokens with low semantic density, encouraging the model to better capture underrepresented semantic cues and learn a more balanced conditional distribution. This design mitigates the risk of the model distribution overfitting to tokens with low semantic density. Second, we propose Spatial Representation Alignment (SRA), which adaptively reweights cross-attention maps according to token importance and enforces representation consistency, enabling semantically important tokens to exert stronger spatial guidance during generation. This mechanism effectively prevents low semantic-density tokens from dominating the attention allocation, thereby avoiding the dilution of the spatial and distributional guidance provided by high semantic-density tokens. Extensive experiments on multiple benchmark datasets demonstrate that DARE consistently improves generation fidelity and semantic alignment, achieving significant gains over existing approaches.

Figures & tables

Panel of Figure 1 — Figure 1: Visualization and Analysis of Conditional Semantic Information Distribution. (a) shows the similarity between conditional semantic representations. The off-diagonal regions indicate that the semantic similarity between different tokens is low, reflecting a discrete distribution. (b) presents the attention scores and corresponding distribution maps of selected underfitted tail tokens generated by Seedance ( Gao et al., 2025 ) before and after the introduction of DARE.

Panel of Figure 2 — Figure 2: Overall Computational Architecture of the Distribution-Aware Rectification and Spatial Ensemble (DARE) Method. By computing token importance weights online and incorporating them into the subsequent DR-CFG and SRA modules, DARE corrects the distributional bias in the model’s fitting to prompt tokens.

Panel of Table 1 — Table 1: Evaluation results of generated outputs under the VBench ( Huang et al., 2024 ; Huang et al., 2025 ) frameworks using the prompts provided by VBench. $\lx@sectionsign$ indicates that the results of the corresponding methods are taken from ( Huang et al., 2024 ) , while all other results are obtained through inference and evaluation conducted by us under the same experimental environment.

Panel of Table 2 — Table 2: Ablation results analyzing the effects of Distribution-Rectification Classifier-Free Guidance (DR-CFG), Spatial Representation Alignment (SRA), and varying proportions of low-semantic tokens on model performance.

Figure 3: Visual evaluation of the effects of varying proportions of low-semantic tokens on model performance.

Panel of Figure 4 — Figure 4: Visualization of Token-Level Guidance Dynamics Before and After DARE from the Perspective of Attention Scores and Distributions.

Figure 5: Visual quality evaluation of generation results based on a self-constructed evaluation dataset and the VBench benchmark.

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 6: The loss curve during training is shown. Introducing $L_{sra}$ accelerates the model’s fitting of the conditional semantic representation, enhancing the semantic consistency between the video and the conditional text. However, in the later stages of training, this leads to oscillations in the loss, causing model instability and a shift in focus towards the consistency between the conditional semantics and individual frame images.

Method	VBench
Method	Imaging Quality(%) $\uparrow$	Dynamic Degree(%) $\uparrow$	Aesthetic Quality(%) $\uparrow$	Subject Consistency(%) $\uparrow$	Background Consistency(%) $\uparrow$	Motion Smoothness(%) $\uparrow$
Wan 2.1	58.01	96.00	45.28	89.61	92.01	98.04
w. DARE (Ours)	63.12	84.00	44.50	95.19	95.64	99.19
Seedance1.0_exp	69.55	94.00	49.56	93.32	94.52	98.44
w. DARE (Ours)	69.29	89.00	49.79	93.39	94.65	98.46

Appendix

Table 3: Evaluation results of generated outputs under the VBench ( Huang et al., 2024 ; Huang et al., 2025 ) and EvalCrafter ( Liu et al., 2024a ) frameworks using self-designed evaluation prompts.

Method	EvalCrafter
Method	CLIP Temp $\uparrow$	SD Score $\uparrow$	CLIP Score $\uparrow$	Face Consistency $\uparrow$	BLIP BLUE $\uparrow$	IS $\uparrow$	IS std $\downarrow$	Flow Score	Warping Error $\downarrow$
Wan 2.1	0.9983	0.7135	0.2098	0.9923	0.2137	12.63	2.147	2.25	124.73
w. DARE (Ours)	0.9988	0.7235	0.2150	0.9929	0.2154	11.11	3.193	1.56	71.32
Seedance1.0_exp	0.9986	0.7167	0.214	0.9909	0.2552	15.02	3.94	1.15	141.798
w. DARE (Ours)	0.9986	0.7195	0.214	0.9907	0.2625	15.33	3.75	1.16	152.028

Appendix

Table 4: Evaluation results of generated outputs under the EvalCrafter ( Liu et al., 2024a ) frameworks using the prompts provided by VBench.

Panel of Figure 7 — Figure 7: Visual comparison and analysis of the attention scores produced by high-density semantic tokens during the inference stage. Using 101 high-density semantic tokens extracted through statistical analysis, (a) illustrates the changes in the attention scores of different tokens before and after introducing DARE; (b) shows the proportion of tokens that exhibit a significant increase in attention scores after incorporating DARE; and (c) presents the overall variation in attention scores before and after applying DARE.

Figure 8: Qualitative comparison of generation results produced by Seedance before and after introducing DARE on the self-constructed dataset and the VBench benchmark.

Figure 9: Qualitative comparison of generation results produced by Wan 2.1 before and after introducing DARE on the self-constructed dataset.

Figure 10: Qualitative comparison of generation results produced by Wan 2.1 before and after introducing DARE on the VBench benchmark.

Explore similar work

Delta Score Matters! Spatial Adaptive Multi Guidance in Diffusion Models
Apr 29, 2026Haosen Li, Wenshuo Chen, Lei Wang +4Diffusion Model Guidance Classifier-Free Guidance
1The Hong Kong University of Science and Technology (Guangzhou) · 2Griffith University & Data61/CSIRO
Guiding Diffusion Models with Semantically Degraded Conditions
Mar 11, 2026Shilong Han, Yuming Zhang, Hongxia WangDiffusion Model Guidance T2I Diffusion Models
College of Science, National University of Defense Technology
Guiding a Diffusion Model by Swapping Its Tokens
Apr 9, 2026Weijia Zhang, Yuehao Liu, Shanyan Guan +4Diffusion Model Guidance Diffusion Guidance
MoE Key Lab of Artificial Intelligence, AI Institute, Shanghai Jiao Tong University · vivo Mobile Communication Co., Ltd.

Apr 29, 2026J/K move · Enter open · S save

Delta Score Matters! Spatial Adaptive Multi Guidance in Diffusion Models

Haosen Li, Wenshuo Chen, Lei Wang, Shaofeng Liang+3

1The Hong Kong University of Science and Technology (Guangzhou) · 2Griffith University & Data61/CSIRO

Diffusion Model Guidance Classifier-Free Guidance Diffusion Model Sampling

Open paper AskPDF arXiv

Diffusion models have achieved remarkable success in synthesizing complex static and temporal visuals, a breakthrough largely driven by Classifier-Free Guidance (CFG). However, despite its pivotal role in aligning generated content with textual prompts, standard CFG relies on a globally uniform scalar. This homogeneous amplification traps models in a well-documented "detail-artifact dilemma": low guidance scales fail to inject intricate semantics, while high scales inevitably cause structural degradation, color over-saturation, and temporal inconsistencies in videos. In this paper, we expose the physical root of this flaw through the lens of differential geometry. By analyzing Tweedie's Formula, we reveal that CFG intrinsically performs a tangential linear extrapolation. Because the natural data manifold is highly curved, this uniform linear step introduces a severe orthogonal deviation. To keep the generation trajectory safely bounded, we formulate a theoretical upper bound for spatial and adaptive guidance. Based on these geometric insights, we propose Spatial Adaptive Multi Guidance (SAMG), a training-free and virtually zero-cost sampling algorithm. SAMG dynamically computes point-wise conditional guidance energy, applying a conservative minimum scale to high-energy boundary regions to preserve delicate micro-textures, while deploying an aggressive maximum scale in low-energy regions to maximize semantic injection. Extensive experiments across diverse image (SD 1.5, SDXL, SD3.5 Medium) and video (CogVideoX, ModelScope) architectures demonstrate that SAMG effectively resolves the detail-artifact dilemma, achieving superior semantic alignment, structural integrity, and temporal smoothness without any computational overhead.

Abstract

Explore similar work

Delta Score Matters! Spatial Adaptive Multi Guidance in Diffusion Models

Guiding Diffusion Models with Semantically Degraded Conditions

Guiding a Diffusion Model by Swapping Its Tokens