RATR: Training-Free Material Rebinding via Role Assignment and Temporal Routing
Authors: Jiehui Tang, Yunlong Yu, Zhao Wang, Jungong Han
Organizations: College of Information Science and Electronic Engineering, Zhejiang University, Hangzhou, China . · Ningbo Innovation Center, Zhejiang University, Ningbo, China . · Department of Automation, Tsinghua University, Beijing, China .
Material rebinding aims to transfer the material appearance from a reference image while preserving the target object's geometry and identity. Existing methods typically struggle to balance material fidelity with structural preservation due to the coupled influence of structural and appearance cues in their visual conditions. We identify role entanglement in the target and reference conditions as a contributing factor the target condition carries both geometry and its original appearance while the reference condition contains material cues alongside object-specific structure. To address this issue, we propose RATR, a training-free material rebinding method built on a diffusion-based image editor that assigns complementary roles to the two conditions and coordinates their influence during denoising. To make the target condition primarily convey structure, we introduce an albedo-resolved surface release mechanism, which suppresses the target's original surface appearance while retaining geometric cues, yielding a structure-oriented condition for material transfer. On the reference side, the designed material evidence isolation mechanism emphasizes material cues while reducing interference from object-specific structure, yielding a complementary material-oriented condition.
Figures & tables
Figure 1: Qualitative demonstration of RATR. Top: surface release produces albedo-resolved surface boundaries Cs that preserve target geometry while suppressing the original target appearance. Middle: a fixed target can be rebound to diverse reference materials. Bottom: a single material can transfer across different targets
Figure 2: Overview of RATR. The target and material-reference images are reparameterized into complementary visual inputs: a target-derived surface boundary Cs that suppresses inherited appearance while retaining structural evidence, and a reference-derived material evidence boundary Er . Surface-Material Conditional Direction Routing (SM-CDR) adjusts the contributions of their visual-condition residuals throughout denoising, emphasizing structural evidence and material evidence at different stages under a task-specific schedule.
Figure 3: Qualitative comparison across methods. RATR achieves more faithful material transfer while better preserving target geometry, whereas competing methods often suffer from incomplete material transfer or structural artifacts. More qualitative results are provided in Project .
Method
SSIM ↑
LPIPS ↓
DreamSim ↓
Edge SSIM ↑
Edge MAE ↓
DealMaTe
0.7061
0.3927
0.3887
0.7190
0.0158
Qwen-Edit
0.7051
0.3560
0.3430
0.6948
0.0163
MARBLE
0.6356
0.3957
0.4264
0.6464
0.0204
MaterialFusion
0.7178
0.3419
0.3886
0.6950
0.0162
ZeST
0.6648
0.3781
0.4314
0.6722
0.0186
IP-A. SDXL+CN
0.6152
0.4142
0.4795
0.6463
0.0204
Table 1: Target-side quantitative comparison results of our method and the existing competitors. Higher is better for both SSIM and Edge SSIM; lower is better for LPIPS, DreamSim, and Edge MAE. The best results are marked in bold .
Method
CLIP ↑
DINOv3 ↑
Materialistic ↑
DealMaTe
0.4617
0.5019
0.5974
Qwen-Edit
0.4928
0.5322
0.5323
MARBLE
0.5058
0.5249
0.5643
MaterialFusion
0.4810
0.5313
0.4594
ZeST
0.5247
0.5456
0.5817
IP-A-SDXL-CN
0.5249
0.5437
0.6060
Table 2: Reference-material correspondence comparison. Higher is better for all metrics.
Figure 4: Representative human ratings across four target–material pairs. Each row presents the target, material reference, and nine anonymized outputs (A–I), together with their five-point Material Fidelity, Structure Preservation, and Overall Preference ratings. RATR corresponds to G, D, A, and E from top to bottom. The displayed scores illustrate individual cases, while aggregate evaluation results are reported in Table 3 .
Figure 5: Human-rating distributions and pair-level rankings for RATR. (a) Evaluator-level mean ratings across the three criteria; dots represent individual evaluators, boxes indicate interquartile ranges, and horizontal lines denote medians. (b) Proportions of target–material pairs where RATR ranks strictly first, ties for first, or is not first among the nine methods.
Method
Material Fidelity ↑
Structure Preservation ↑
Overall Preference ↑
DealMaTe
3.009 ± 0.138
3.932 ± 0.622
3.165 ± 0.234
Qwen-Edit
3.239 ± 0.229
3.792 ± 0.512
3.328 ± 0.176
MARBLE
3.221 ± 0.207
3.381 ± 0.415
3.137 ± 0.205
MaterialFusion
2.607 ± 0.337
3.068 ± 0.457
2.626 ± 0.344
ZeST
3.203 ± 0.341
3.158 ± 0.404
3.016 ± 0.285
IP-A. SDXL+CN
3.222 ± 0.312
2.581 ± 0.753
2.531 ± 0.573
Table 3: Human evaluation of material fidelity, target-structure preservation, and overall preference. Scores are reported as the mean ± standard deviation of evaluator-level mean ratings on a 1–5 scale; higher scores indicate better performance.
Routing Strategy
SSIM ↑
LPIPS ↓
DreamSim ↓
Edge SSIM ↑
Edge MAE ↓
CLIP ↑
DINOv3 ↑
Materialistic ↑
Structure-lock
0.7502
0.2949
0.2910
0.7476
0.0135
0.4722
0.5140
0.5062
Default
0.7292
0.3127
0.3414
0.7282
0.0153
0.5251
0.5495
0.6225
Material-forward
0.7098
0.3326
0.3687
0.7087
0.0171
0.5390
0.5633
0.6531
Table 4: Quantitative comparison of SM-CDR routing strategies.
Figure 6: Ablation of target-side role decoupling and SM-CDR.
Figure 7: SM-CDR gain ablation under different material-routing-share profiles pΓ .
Figure 8: Additional qualitative results across diverse target geometries and material appearances.
Figure 9: Additional qualitative results with patterned and object-centric material references.
Figure 10: Representative limitations of RATR. (a) Residual dependence of material reflectance on target illumination. (b) Pair-dependent material adoption under shared routing. (c) Limited reproduction of material-specific geometric relief under target-structure preservation.
Recently, diffusion-based material transfer methods rely on image fine-tuning or complex architectures with auxiliary networks but face challenges such as text dependency, additional computational costs, and feature misalignment. To address these limitations, we propose DealMaTe, using depth, normal, and lighting images for material transfer. DealMaTe is a simplified diffusion framework that eliminates text guidance and reference networks. We design a lightweight 3D information injection method, Multi-Dim 3D Shader LoRA, which, without modifying the base model weights, enables compatible control conditions and achieves harmonious and stable results. Additionally, we optimize the attention mechanism with Shader Causal Mutual Attention and key-value (KV) caching to reduce inference latency caused by multiple conditions, improve computational efficiency, and achieve high-quality material transfer results with low architectural complexity. Extensive experiments covering a wide variety of objects and lighting conditions consistently demonstrate that DealMaTe achieves remarkable high-fidelity material transfer under arbitrary input materials. The code is available at https://github.com/haha-lisa/DealMaTe.
Nisha Huang, Yizhou Lin, Jie Guo +3
Tsinghua University, China · Pengcheng Laboratory, China · National Cheng-Kung University, Taiwan +2
Material replacement is a common interior-design operation: changing the material of a selected surface while preserving its geometry, surroundings, and illumination. Despite its commercial relevance, no public benchmark isolates this task, and evaluating it is challenging. Reference-based metrics penalize valid outputs in this inherently one-to-many setting, favor the style of the reference generator, and cannot fairly compare editors that receive different forms of guidance. We introduce MatReplace, a reference-free benchmark that evaluates edits along four verifiable dimensions: local material correctness, global lighting harmony, outside preservation, and inside structure. It defines three tracks that vary one conditioning signal at a time: (A) instruction only, (B) instruction plus region mask, and (C) material reference image instead of instruction. Our results reveal a clear divide between naming and visually grounding materials. In Track A, leading closed-source editors achieve exemplar-level material rendering and surpass the exemplar anchor under our primary aggregate. In Track B, masks help only mask-compatible models with weak scene preservation, with task-paired, single-seed effects ranging from +0.137 to -0.090 across aligned model families. In Track C, reference-image conditioning degrades every family under both aggregates, by -0.031 to -0.508; in the worst cases, models repaint the reference image itself and perform worse than returning the input unchanged. Thus, named-material rendering is largely solved by the strongest closed editors on this distribution, but grounding materials from pixels remains an open challenge. Expert ratings validate our ranking (Kendall's tau = 0.68) and align with our aggregates more closely than GT-referenced or CLIP-based baselines.
Mingzhe Du, Thong Thanh Nguyen, Nguyen Tran Cong Duy +2
National University of Singapore · Nanyang Technological University · VinUniversity
Recent diffusion-based methods for material transfer rely on image fine-tuning or complex architectures with assistive networks, but face challenges including text dependency, extra computational costs, and feature misalignment. To address these limitations, we propose MaTe, a streamlined diffusion framework that eliminates textual guidance and reference networks. MaTe integrates input images at the token level, enabling unified processing via multi-modal attention in a shared latent space. This design removes the need for additional adapters, ControlNet, inversion sampling, or model fine-tuning. Extensive experiments demonstrate that MaTe achieves high-quality material generation under a zero-shot, training-free paradigm. It outperforms state-of-the-art methods in both visual quality and efficiency while preserving precise detail alignment, significantly simplifying inference prerequisites.
Nisha Huang, Henglin Liu, Yizhou Lin +5
Tsinghua University · PengCheng Laboratory · Lenovo Research +1