Not all tokens contribute equally to diffusion learning
Authors: Guoqing Zhang, Lu Shi, Wanru Xu, Linna Zhang, Sen Wang, Fangfang Wang, Yigang Cen
Organizations: State Key Laboratory of Advanced Rail Autonomous Operation, Bejing Jiaotong University, Beijing, China · School of Computer Science and Technology, Bejing Jiaotong University, Beijing, China · Visual Intellgence +X International Cooperation Joint Laboratory of MOE, Bejing Jiaotong University, Beijing, China · School of Mechanical Engineering, Guizhou University, Guizhou, China · Seed, Bytedance, Beijing, China · School of Information Science and Technology, Hangzhou Normal University, Zhejiang, China
With the rapid development of conditional diffusion models, significant progress has been made in text-to-video generation. However, we observe that these models often neglect semantically important tokens during inference, leading to biased or incomplete generations under classifier-free guidance. We attribute this issue to two key factors: distributional bias caused by the long-tailed token frequency in training data, and spatial misalignment in cross-attention where semantically important tokens are overshadowed by less informative ones. To address these issues, we propose Distribution-Aware Rectification and Spatial Ensemble (DARE), a unified framework that improves semantic guidance in diffusion models from the perspectives of distributional debiasing and spatial consistency. First, we introduce Distribution-Rectified Classifier-Free Guidance (DR-CFG), which regularizes the training process by dynamically suppressing dominant tokens with low semantic density, encouraging the model to better capture underrepresented semantic cues and learn a more balanced conditional distribution. This design mitigates the risk of the model distribution overfitting to tokens with low semantic density. Second, we propose Spatial Representation Alignment (SRA), which adaptively reweights cross-attention maps according to token importance and enforces representation consistency, enabling semantically important tokens to exert stronger spatial guidance during generation. This mechanism effectively prevents low semantic-density tokens from dominating the attention allocation, thereby avoiding the dilution of the spatial and distributional guidance provided by high semantic-density tokens. Extensive experiments on multiple benchmark datasets demonstrate that DARE consistently improves generation fidelity and semantic alignment, achieving significant gains over existing approaches.
Figures & tables
Figure 1: Visualization and Analysis of Conditional Semantic Information Distribution. (a) shows the similarity between conditional semantic representations. The off-diagonal regions indicate that the semantic similarity between different tokens is low, reflecting a discrete distribution. (b) presents the attention scores and corresponding distribution maps of selected underfitted tail tokens generated by Seedance ( Gao et al., 2025 ) before and after the introduction of DARE.
Figure 2: Overall Computational Architecture of the Distribution-Aware Rectification and Spatial Ensemble (DARE) Method. By computing token importance weights online and incorporating them into the subsequent DR-CFG and SRA modules, DARE corrects the distributional bias in the model’s fitting to prompt tokens.
Table 1: Evaluation results of generated outputs under the VBench ( Huang et al., 2024 ; Huang et al., 2025 ) frameworks using the prompts provided by VBench. \lx@sectionsign indicates that the results of the corresponding methods are taken from ( Huang et al., 2024 ) , while all other results are obtained through inference and evaluation conducted by us under the same experimental environment.
Table 2: Ablation results analyzing the effects of Distribution-Rectification Classifier-Free Guidance (DR-CFG), Spatial Representation Alignment (SRA), and varying proportions of low-semantic tokens on model performance.
Figure 3: Visual evaluation of the effects of varying proportions of low-semantic tokens on model performance.
Figure 4: Visualization of Token-Level Guidance Dynamics Before and After DARE from the Perspective of Attention Scores and Distributions.
Figure 5: Visual quality evaluation of generation results based on a self-constructed evaluation dataset and the VBench benchmark.
Appendix figures & tables7 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 6: The loss curve during training is shown. Introducing Lsra accelerates the model’s fitting of the conditional semantic representation, enhancing the semantic consistency between the video and the conditional text. However, in the later stages of training, this leads to oscillations in the loss, causing model instability and a shift in focus towards the consistency between the conditional semantics and individual frame images.
Method
VBench
Imaging Quality(%) ↑
Dynamic Degree(%) ↑
Aesthetic Quality(%) ↑
Subject Consistency(%) ↑
Background Consistency(%) ↑
Motion Smoothness(%) ↑
Wan 2.1
58.01
96.00
45.28
89.61
92.01
98.04
w. DARE (Ours)
63.12
84.00
44.50
95.19
95.64
99.19
Seedance1.0_exp
69.55
94.00
49.56
93.32
94.52
98.44
w. DARE (Ours)
69.29
89.00
49.79
93.39
94.65
98.46
Appendix
Table 3: Evaluation results of generated outputs under the VBench ( Huang et al., 2024 ; Huang et al., 2025 ) and EvalCrafter ( Liu et al., 2024a ) frameworks using self-designed evaluation prompts.
Method
EvalCrafter
CLIP Temp ↑
SD Score ↑
CLIP Score ↑
Face Consistency ↑
BLIP BLUE ↑
IS ↑
IS std ↓
Flow Score
Warping Error ↓
Wan 2.1
0.9983
0.7135
0.2098
0.9923
0.2137
12.63
2.147
2.25
124.73
w. DARE (Ours)
0.9988
0.7235
0.2150
0.9929
0.2154
11.11
3.193
1.56
71.32
Seedance1.0_exp
0.9986
0.7167
0.214
0.9909
0.2552
15.02
3.94
1.15
141.798
w. DARE (Ours)
0.9986
0.7195
0.214
0.9907
0.2625
15.33
3.75
1.16
152.028
Appendix
Table 4: Evaluation results of generated outputs under the EvalCrafter ( Liu et al., 2024a ) frameworks using the prompts provided by VBench.
Figure 7: Visual comparison and analysis of the attention scores produced by high-density semantic tokens during the inference stage. Using 101 high-density semantic tokens extracted through statistical analysis, (a) illustrates the changes in the attention scores of different tokens before and after introducing DARE; (b) shows the proportion of tokens that exhibit a significant increase in attention scores after incorporating DARE; and (c) presents the overall variation in attention scores before and after applying DARE.
Figure 8: Qualitative comparison of generation results produced by Seedance before and after introducing DARE on the self-constructed dataset and the VBench benchmark.
Figure 9: Qualitative comparison of generation results produced by Wan 2.1 before and after introducing DARE on the self-constructed dataset.
Figure 10: Qualitative comparison of generation results produced by Wan 2.1 before and after introducing DARE on the VBench benchmark.
Diffusion models have achieved remarkable success in synthesizing complex static and temporal visuals, a breakthrough largely driven by Classifier-Free Guidance (CFG). However, despite its pivotal role in aligning generated content with textual prompts, standard CFG relies on a globally uniform scalar. This homogeneous amplification traps models in a well-documented "detail-artifact dilemma": low guidance scales fail to inject intricate semantics, while high scales inevitably cause structural degradation, color over-saturation, and temporal inconsistencies in videos. In this paper, we expose the physical root of this flaw through the lens of differential geometry. By analyzing Tweedie's Formula, we reveal that CFG intrinsically performs a tangential linear extrapolation. Because the natural data manifold is highly curved, this uniform linear step introduces a severe orthogonal deviation. To keep the generation trajectory safely bounded, we formulate a theoretical upper bound for spatial and adaptive guidance. Based on these geometric insights, we propose Spatial Adaptive Multi Guidance (SAMG), a training-free and virtually zero-cost sampling algorithm. SAMG dynamically computes point-wise conditional guidance energy, applying a conservative minimum scale to high-energy boundary regions to preserve delicate micro-textures, while deploying an aggressive maximum scale in low-energy regions to maximize semantic injection. Extensive experiments across diverse image (SD 1.5, SDXL, SD3.5 Medium) and video (CogVideoX, ModelScope) architectures demonstrate that SAMG effectively resolves the detail-artifact dilemma, achieving superior semantic alignment, structural integrity, and temporal smoothness without any computational overhead.
Haosen Li, Wenshuo Chen, Lei Wang +4
1The Hong Kong University of Science and Technology (Guangzhou) · 2Griffith University & Data61/CSIRO
Classifier-Free Guidance (CFG) is a cornerstone of modern text-to-image models, yet its reliance on a semantically vacuous null prompt (∅) generates a guidance signal prone to geometric entanglement. This is a key factor limiting its precision, leading to well-documented failures in complex compositional tasks. We propose Condition-Degradation Guidance (CDG), a novel paradigm that replaces the null prompt with a strategically degraded condition, cdeg. This reframes guidance from a coarse "good vs. null" contrast to a more refined "good vs. almost good" discrimination, thereby compelling the model to capture fine-grained semantic distinctions. We find that tokens in transformer text encoders split into two functional roles: content tokens encoding object semantics, and context-aggregating tokens capturing global context. By selectively degrading only the former, CDG constructs cdeg without external models or training. Validated across diverse architectures including Stable Diffusion 3, FLUX, and Qwen-Image, CDG markedly improves compositional accuracy and text-image alignment. As a lightweight, plug-and-play module, it achieves this with negligible computational overhead. Our work challenges the reliance on static, information-sparse negative samples and establishes a new principle for diffusion guidance: the construction of adaptive, semantically-aware negative samples is critical to achieving precise semantic control. Code is available at https://github.com/Ming-321/Classifier-Degradation-Guidance.
Shilong Han, Yuming Zhang, Hongxia Wang
College of Science, National University of Defense Technology
Classifier-Free Guidance (CFG) is a widely used inference-time technique to boost the image quality of diffusion models. Yet, its reliance on text conditions prevents its use in unconditional generation. We propose a simple method to enable CFG-like guidance for both conditional and unconditional generation. The key idea is to generate a perturbed prediction via simple token swap operations, and use the direction between it and the clean prediction to steer sampling towards higher-fidelity distributions. In practice, we swap pairs of most semantically dissimilar token latents in either spatial or channel dimensions. Unlike existing methods that apply perturbation in a global or less constrained manner, our approach selectively exchanges and recomposes token latents, allowing finer control over perturbation and its influence on generated samples. Experiments on MS-COCO 2014, MS-COCO 2017, and ImageNet datasets demonstrate that the proposed Self-Swap Guidance (SSG), when applied to popular diffusion models, outperforms previous condition-free methods in image fidelity and prompt alignment under different set-ups. Its fine-grained perturbation granularity also improves robustness, reducing side-effects across a wider range of perturbation strengths. Overall, SSG extends CFG to a broader scope of applications including both conditional and unconditional generation, and can be readily inserted into any diffusion model as a plug-in to gain immediate improvements.
Weijia Zhang, Yuehao Liu, Shanyan Guan +4
MoE Key Lab of Artificial Intelligence, AI Institute, Shanghai Jiao Tong University · vivo Mobile Communication Co., Ltd.