Token merging has emerged as an effective training-free strategy for accelerating diffusion models by reducing redundant token computation while minimizing generation degradation. A key challenge is determining which tokens can be safely merged and which should be preserved for subsequent computation. Existing method addresses this by preserving prompt-aligned foreground regions, thereby protecting semantically relevant structures. However, this text-relevance criterion overlooks background objects, local details that are not explicitly mentioned in the prompt, leading to degraded structural coherence. In this work, we propose Dynamics-aware Token Merging (DaTMe), a training-free framework that grounds token merging in the underlying diffusion dynamics. Our key insight is that token importance should reflect whether a token remains unresolved during denoising and whether it carries structural information. To this end, DaTMe constructs a dynamics- aware importance map from the Tweedie denoised estimate. The map combines a temporal signal, measuring inter-step changes in the predicted clean image, with a structural signal, capturing local high-frequency content such as boundaries and fine details. Using this map, DaTMe adaptively assigns tokens into merging roles under a given compression ratio: important tokens are preserved as anchors, while tokens that are both temporally stabilized and structurally homogeneous are selected as safe merge candidates. Experiments on PixArt-a and FLUX demonstrate that DaTMe maintains visual fidelity across the entire image, highlighting diffusion dynamics as an effective token-level merging decision in efficient generation. Our code is available at https://github.com/SeungJuCha/DaTMe.
Figures & tables
Figure 1: (a) Comparison under different compression ratios. CFG-based merging [ 43 ] often overlooks background details and structures, whereas our method better preserves background integrity. (b) Spatial distribution of top-ranked tokens over denoising timesteps. CFG-based importance remains concentrated on the foreground object with limited temporal variation. In contrast, our score-based signals (Score Slope and Tweedie Diff) progressively shift from object regions to fine-grained spatial details, better matching the pseudo-GT pattern. Here, pseudo-GT is defined as the difference between the final denoised latent and the Tweedie estimate at each timestep. (c) Non-prompt region blind spot. Low-ranked tokens under the CFG-based signal consistently overlap with structurally salient regions in text-irrelevant areas (red).
Figure 2: (a) Cosine similarity and temporal trajectories of top- and bottom-ranked tokens. The Tweedie estimate progressively approaches the final latent over denoising. Meanwhile, tokens with high pseudo-GT magnitude ∣x0−x^0(t)∣ exhibit consistently larger changes than low-ranked tokens. This indicates that the pseudo-GT effectively separates active tokens from converged ones. (b) Step-wise autocorrelation of token-importance signals. Computed between consecutive timesteps, lower values indicate more dynamic importance. The proposed Tweedie-based signal exhibits lower autocorrelation than CFG and score slope. ( c) Correlation with the pseudo-GT change magnitude. The proposed Tweedie difference remains better aligned with actual token changes across timesteps than CFG or score slope. τ denotes the normalized timestep, scaled to the range [0,1] .
Figure 3: Difference- and Scale- components of the score slope. (Top) Mean magnitude and contribution ratio of the two terms: the scale term becomes dominant in late steps. (Bottom) Spatial distributions: the difference term captures structured token-level changes, whereas the scale term remains spatially unstructured.
Figure 4: Proposed DaTMe pipeline. Using the spatiotemporal axes, DaTMe assigns token roles and performs merging within diffusion layers.
Figure 5Table 6Table 7Figure 8Figure 9
Appendix figures & tables13 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 10: Additional PixArt- α results under different compression ratios. Unlike prior methods, DaTMe more faithfully preserves the fine details present in the original image.
Figure 11: Additional FLUX results under different compression ratios. Unlike prior methods, DaTMe more faithfully preserves the fine details present in the original image.
Table 12Table 13
Figure 12: SD 1.5 results under different compression ratios. Unlike prior methods, DaTMe more faithfully preserves the contents of the original image.
Figure 13: SDXL results under different compression ratios. Unlike prior methods, DaTMe more faithfully preserves the contents of the original image especially background details.
Figure 14: Difference in merging maps. Black spots indicate the destinations of merged source tokens. Our method better preserves structurally important details across the entire image.
Table 17Figure 18Table 19Figure 20
Figure 18: Limitations. In highly complex images, aggressive compression adversely affects image quality, leading to loss of structural details and visual coherence.
Figure 19: Example questionnaire for the human preference study. Each evaluation sample includes three questions.
Diffusion Transformers (DiTs) have achieved state-of-the-art performance in image and video generation, but their success comes at the cost of heavy computation. This inefficiency is largely due to the fixed tokenization process, which uses constant-sized patches throughout the entire denoising phase, regardless of the content's complexity. We propose dynamic tokenization, an efficient test-time strategy that varies patch sizes based on content complexity and the denoising timestep. Our key insight is that early timesteps only require coarser patches to model global structure, while later iterations demand finer (smaller-sized) patches to refine local details. During inference, our method dynamically reallocates patch sizes across denoising steps for image and video generation and substantially reduces cost while preserving perceptual generation quality. Extensive experiments demonstrate the effectiveness of our approach: it achieves up to 3.52× and 3.2× speedup on FLUX-1.Dev and Wan 2.1, respectively, without compromising the generation quality and prompt adherence.
Diffusion Transformers (DiTs) achieve superior image generation quality but suffer from quadratic computational complexity relative to token count. While various token reduction (TR) methods have been proposed to mitigate this cost, they overlook the primary objective of generative models: minimizing recovery error, which requires reflecting output token similarity. They rely solely on input token similarity inherited from reduction-only ViT paradigms, leading to a fundamental misalignment with this objective. To bridge this gap, we propose DiTo, a novel TR paradigm that shifts the focus toward output-centric token reduction. Based on the observation that output token similarity is consistently preserved across adjacent timesteps, DiTo utilizes prior-step similarities as an effective proxy to establish token correspondences at a Matching timestep, which are then reused across multiple subsequent Reduction timesteps. To optimize this interleaved scheduling, we propose Pair Match Ratio (PMR)-guided Interval Scheduling to determine the optimal matching frequency. Furthermore, to mitigate localized approximation errors and resulting blocking artifacts caused by repeated reuse, we propose Frequency-aware Token Matching by incorporating a selection-frequency penalty. Extensive experiments demonstrate that DiTo consistently outperforms existing TR methods with 1.6-3.9 dB higher PSNR at comparable speedups, achieving a superior Pareto frontier.
Most visual tokenizers for image generation are bifurcated into two families with complementary limitations: continuous VAEs offer high-fidelity reconstruction but suffer from dense, entangled latents that are poorly suited for semantic control, whereas discrete VQ-based models enable autoregressive generation yet struggle with gradient sparsity, unstable training, and codebook collapse. In this work, we introduce MergeTok, a unified tokenizer that jointly optimizes continuous (VAE) and discrete (VQ) tokenizers within a encoder-decoder architecture, leveraging token merging techniques as a semantic bridge. By clustering similar tokens during encoding, MergeTok establishes a structural prior that provides dual supervision signals: (i) it imposes merged-token semantic alignment in the VAE branch, regularizing its latent space toward disentangled, semantic-aware representations; (ii) it derives group-wise constraints, promoting intra-group diversity and inter-group exclusivity that stabilize VQ training. MergeTok shows competitive reconstruction and generation performance on ImageNet-256, with substantially lower rFID than strong VAE and VQ models under matched token budgets, while producing semantically-organized token representations compatible with both autoregressive and diffusion generators. This shows that a single architecture can endow visual tokenizers with robust semantic organization and generator-friendly discreteness.
Luyuan Zhang, Siyuan Li, Zedong Wang +7
1Tsinghua University · 2Westlake University · 3Zhejiang University +3