A training-free framework for high-fidelity appearance transfer via diffusion transformers
Authors: Shengrong Gu, Ye Wang, Song Wu, Rui Ma, Qian Wang, Lanjun Wang, Zili Yi
Organizations: School of Intelligence Science and Technology, Nanjing University, Suzhou, China · School of Artificial Intelligence, Jilin University, Changchun, China · JIUTIAN Research, Bejing, China · School of New Media and Communication, Tianjin University, Tianjin, China · State Key Laboratory of Novel Software Technology, Nanjing University, Nanjing, China
Diffusion Transformers (DiTs) excel at generation, but their global self-attention makes controllable, reference-image-based editing a distinct challenge. Unlike U-Nets, naively injecting local appearance into a DiT can disrupt its holistic scene structure. We address this by proposing the first training-free framework specifically designed to tame DiTs for high-fidelity appearance transfer. Our core is a synergistic system that disentangles structure and appearance. We leverage high-fidelity inversion to establish a rich content prior for the source image, capturing its lighting and micro-textures. A novel attention-sharing mechanism then dynamically fuses purified appearance features from a reference, guided by geometric priors. Our unified approach operates at 1024px and outperforms specialized methods on tasks ranging from semantic attribute transfer to fine-grained material application. Extensive experiments confirm our state-of-the-art performance in both structural preservation and appearance fidelity.
Figures & tables
Figure 1: Our unified, DiT-based framework enables high-fidelity appearance transfer across a wide range of tasks. It can seamlessly transfer semantic attributes (e.g., a tiger’s pattern to a cat) and apply fine-grained materials (e.g., knitted fabric to a cushion), even from multiple references, while precisely preserving the source geometry. The colored boxes indicate the source of the appearance.
Figure 2: Overview of our framework. We perform parallel inversions on both Reference and Source images to extract appearance features (KV) and content priors ( z~n ), respectively. During the Denoising Synthesis (bottom), we initialize with the source noise z~n . For selected DiT layers (highlighted in orange), we inject appearance by expanding the attention context: copying the Reference’s KV features into the Source’s attention blocks. Simultaneously, a global Redux embedding and geometric conditions (Depth/Mask) guide the generation across all steps. This selective fusion strategy ensures the transfer of fine-grained appearance while strictly preserving the source structure.
Figure 3: Comparison of different strategies for our appearance encoding module.
Figure 4: Results of Semantic-Aware Transfer.
Figure 5: Results of Material Transfer.
Model
Semantic-Aware Transfer
Material Transfer
Overall
CLIP-I ↑
DINO ↑
CLIP-T ↑
VQA ↑
DeQA ↑
DiffEditor [ 19 ]
0.7910
0.5269
0.2211
0.6718
3.1191
Cross-Image [ 8 ]
0.8478
0.7499
0.2463
0.6084
3.0488
Eye-for-an-eye [ 9 ]
0.8832
0.7521
0.2428
0.8194
3.6758
ZeST [ 10 ]
0.8381
0.5571
0.2737
0.8725
4.0230
MaterialFusion [ 11 ]
0.8064
0.4247
0.2753
0.8598
3.8984
Table 1: Quantitative comparison on our curated dataset. Best results are in bold , second best are underlined .
Figure 6: Step-by-step ablation of our fidelity enhancement modules. (d) Baseline (geometric guidance + Redux) roughly captures the appearance but lacks detail. (e) Adding Inversion Initialization corrects fine structural details like head/eye proportions by providing a strong content prior. (f) Our full model, with Attention Sharing, successfully transfers nuanced color and texture, demonstrating the necessity of both components.