Which Tokens to Merge? Diffusion Dynamics for Efficient Image Generation

SeungJu Cha; Ye-Chan Kim; HyunGee Kim; Sungho Koh; Dong-Jin Kim

cs.CVOct 15, 2026

Which Tokens to Merge? Diffusion Dynamics for Efficient Image Generation

Authors: SeungJu Cha, Ye-Chan Kim, HyunGee Kim, Sungho Koh, Dong-Jin Kim

Organizations: Hanyang University, South Korea

Abstract

Token merging has emerged as an effective training-free strategy for accelerating diffusion models by reducing redundant token computation while minimizing generation degradation. A key challenge is determining which tokens can be safely merged and which should be preserved for subsequent computation. Existing method addresses this by preserving prompt-aligned foreground regions, thereby protecting semantically relevant structures. However, this text-relevance criterion overlooks background objects, local details that are not explicitly mentioned in the prompt, leading to degraded structural coherence. In this work, we propose Dynamics-aware Token Merging (DaTMe), a training-free framework that grounds token merging in the underlying diffusion dynamics. Our key insight is that token importance should reflect whether a token remains unresolved during denoising and whether it carries structural information. To this end, DaTMe constructs a dynamics- aware importance map from the Tweedie denoised estimate. The map combines a temporal signal, measuring inter-step changes in the predicted clean image, with a structural signal, capturing local high-frequency content such as boundaries and fine details. Using this map, DaTMe adaptively assigns tokens into merging roles under a given compression ratio: important tokens are preserved as anchors, while tokens that are both temporally stabilized and structurally homogeneous are selected as safe merge candidates. Experiments on PixArt-a and FLUX demonstrate that DaTMe maintains visual fidelity across the entire image, highlighting diffusion dynamics as an effective token-level merging decision in efficient generation. Our code is available at https://github.com/SeungJuCha/DaTMe.

Figures & tables

Figure 1: (a) Comparison under different compression ratios. CFG-based merging [ 43 ] often overlooks background details and structures, whereas our method better preserves background integrity. (b) Spatial distribution of top-ranked tokens over denoising timesteps. CFG-based importance remains concentrated on the foreground object with limited temporal variation. In contrast, our score-based signals (Score Slope and Tweedie Diff) progressively shift from object regions to fine-grained spatial details, better matching the pseudo-GT pattern. Here, pseudo-GT is defined as the difference between the final denoised latent and the Tweedie estimate at each timestep. (c) Non-prompt region blind spot. Low-ranked tokens under the CFG-based signal consistently overlap with structurally salient regions in text-irrelevant areas (red).

Figure 2: (a) Cosine similarity and temporal trajectories of top- and bottom-ranked tokens. The Tweedie estimate progressively approaches the final latent over denoising. Meanwhile, tokens with high pseudo-GT magnitude $|x_{0}-\hat{x}_{0}(t)|$ exhibit consistently larger changes than low-ranked tokens. This indicates that the pseudo-GT effectively separates active tokens from converged ones. (b) Step-wise autocorrelation of token-importance signals. Computed between consecutive timesteps, lower values indicate more dynamic importance. The proposed Tweedie-based signal exhibits lower autocorrelation than CFG and score slope. ( c) Correlation with the pseudo-GT change magnitude. The proposed Tweedie difference remains better aligned with actual token changes across timesteps than CFG or score slope. $\tau$ denotes the normalized timestep, scaled to the range $[0,1]$ .

Figure 3: Difference- and Scale- components of the score slope. (Top) Mean magnitude and contribution ratio of the two terms: the scale term becomes dominant in late steps. (Bottom) Spatial distributions: the difference term captures structured token-level changes, whereas the scale term remains spatially unstructured.

Figure 4: Proposed DaTMe pipeline. Using the spatiotemporal axes, DaTMe assigns token roles and performs merging within diffusion layers.

Table 6

Table 7

Appendix figures & tables13 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 10: Additional PixArt- $\alpha$ results under different compression ratios. Unlike prior methods, DaTMe more faithfully preserves the fine details present in the original image.

Figure 11: Additional FLUX results under different compression ratios. Unlike prior methods, DaTMe more faithfully preserves the fine details present in the original image.

Table 12

Table 13

Figure 12: SD 1.5 results under different compression ratios. Unlike prior methods, DaTMe more faithfully preserves the contents of the original image.

Figure 13: SDXL results under different compression ratios. Unlike prior methods, DaTMe more faithfully preserves the contents of the original image especially background details.

Figure 14: Difference in merging maps. Black spots indicate the destinations of merged source tokens. Our method better preserves structurally important details across the entire image.

Table 17

Table 19

Figure 18: Limitations. In highly complex images, aggressive compression adversely affects image quality, leading to loss of structural details and visual coherence.

Figure 19: Example questionnaire for the human preference study. Each evaluation sample includes three questions.

Visual Token Merging

Visual token merging combines redundant image or video tokens to reduce model computation while preserving task-relevant information.

See all 33 papers

DDiT: Dynamic Patch Scheduling for Efficient Diffusion Transformers
Feb 19, 2026Dahye Kim, Deepti Ghadiyaram, Raghudeep GaddeDiffusion Model Inference Acceleration Diffusion-Based Image Generation
Boston University · Amazon
Rethinking Token Reduction for Diffusion Models via Output-Similarity-Awareness
May 21, 2026Hangyeol Lee, Hyojeong Lee, Joo-Young KimDiffusion Transformer Diffusion-Based Image Generation
KAIST, Daejeon, Republic of Korea
MergeTok: Unified Continuous and Discrete Visual Tokenization via Token Merging
May 29, 2026Luyuan Zhang, Siyuan Li, Zedong Wang +7Variational Autoencoders Token Merging
1Tsinghua University · 2Westlake University · 3Zhejiang University +3

Feb 19, 2026J/K move · Enter open · S save

DDiT: Dynamic Patch Scheduling for Efficient Diffusion Transformers

Dahye Kim, Deepti Ghadiyaram, Raghudeep Gadde

Boston University · Amazon

Diffusion Model Inference Acceleration Diffusion-Based Image Generation Diffusion Transformer Video Diffusion Models

Open paper AskPDF arXiv

Diffusion Transformers (DiTs) have achieved state-of-the-art performance in image and video generation, but their success comes at the cost of heavy computation. This inefficiency is largely due to the fixed tokenization process, which uses constant-sized patches throughout the entire denoising phase, regardless of the content's complexity. We propose dynamic tokenization, an efficient test-time strategy that varies patch sizes based on content complexity and the denoising timestep. Our key insight is that early timesteps only require coarser patches to model global structure, while later iterations demand finer (smaller-sized) patches to refine local details. During inference, our method dynamically reallocates patch sizes across denoising steps for image and video generation and substantially reduces cost while preserving perceptual generation quality. Extensive experiments demonstrate the effectiveness of our approach: it achieves up to

3.52\times

and

3.2\times

speedup on FLUX-1.Dev and Wan

2.1

, respectively, without compromising the generation quality and prompt adherence.

Abstract

Visual Token Merging

Explore similar work

DDiT: Dynamic Patch Scheduling for Efficient Diffusion Transformers

Rethinking Token Reduction for Diffusion Models via Output-Similarity-Awareness

MergeTok: Unified Continuous and Discrete Visual Tokenization via Token Merging