MatReplace: A Reference-Free, Conditioning-Aligned Benchmark for Material Replacement in Interior Scenes

Mingzhe Du; Thong Thanh Nguyen; Nguyen Tran Cong Duy; See-Kiong Ng; Luu Anh Tuan

cs.CVAug 25, 2026

MatReplace: A Reference-Free, Conditioning-Aligned Benchmark for Material Replacement in Interior Scenes

Authors: Mingzhe Du, Thong Thanh Nguyen, Nguyen Tran Cong Duy, See-Kiong Ng, Luu Anh Tuan

Organizations: National University of Singapore · Nanyang Technological University · VinUniversity

Abstract

Material replacement is a common interior-design operation: changing the material of a selected surface while preserving its geometry, surroundings, and illumination. Despite its commercial relevance, no public benchmark isolates this task, and evaluating it is challenging. Reference-based metrics penalize valid outputs in this inherently one-to-many setting, favor the style of the reference generator, and cannot fairly compare editors that receive different forms of guidance. We introduce MatReplace, a reference-free benchmark that evaluates edits along four verifiable dimensions: local material correctness, global lighting harmony, outside preservation, and inside structure. It defines three tracks that vary one conditioning signal at a time: (A) instruction only, (B) instruction plus region mask, and (C) material reference image instead of instruction. Our results reveal a clear divide between naming and visually grounding materials. In Track A, leading closed-source editors achieve exemplar-level material rendering and surpass the exemplar anchor under our primary aggregate. In Track B, masks help only mask-compatible models with weak scene preservation, with task-paired, single-seed effects ranging from +0.137 to -0.090 across aligned model families. In Track C, reference-image conditioning degrades every family under both aggregates, by -0.031 to -0.508; in the worst cases, models repaint the reference image itself and perform worse than returning the input unchanged. Thus, named-material rendering is largely solved by the strongest closed editors on this distribution, but grounding materials from pixels remains an open challenge. Expert ratings validate our ranking (Kendall's tau = 0.68) and align with our aggregates more closely than GT-referenced or CLIP-based baselines.

Figures & tables

Figure 1: MatReplace at a glance. Each column is one task, read from the middle out. Middle: the target region in color, outlined over a dimmed grayscale surround (mask $m$ ), reference $r$ inset. Up ( ✓ ): a reference-quality edit (Nano-Banana-2-Lite, Track A). Material changes while geometry, scene, and illumination survive. Down ( ✗ ): one failure per mode. BAGEL-7B (C) repaints the reference and discards the scene (column 1); Qwen-Image-Edit (C) fails to apply the reference and leaves the slate untouched (column 2); OmniGen2 (C) pastes a flat, unshaded slab (column 3); Paste anchor tiles the reference without relighting (column 4). Each failure is caught by the dimension whose contract it violates (Sec. 4 ).

Figure 2: MatReplace Benchmark Construction. One templated generation session per candidate task, an algorithmic validity pass, and a semantic audit by a vision–language agent reduce 2,017 candidate sessions to the frozen 1,421 tasks. The middle bar is drawn to scale, so the admitted and rejected shares are directly comparable and the accounting closes. The 596 rejects ship as a disjoint rejected pool , released for auditing and never used for evaluation.

Figure 3: One failure per dimension: each pair shows the input with the target region outlined, then the model output. SDXL-inpaint paints bare wall instead of navy suede ( $S_{\mathrm{mat}}$ ). InstructPix2Pix floods the whole room with the target emerald ( $S_{\mathrm{pres}}$ ). SD3.5-inpaint re-sculpts the rectangular fireplace into an arched portal ( $S_{\mathrm{struct}}$ ). FLUX-Kontext-ref pastes the reference as flat, unlit patches ( $S_{\mathrm{harm}}$ ).

anchor	mat	pres	struct	harm	overall
gt (exemplar)	0.622	0.964	0.853	0.877	0.829 $\pm$ .004
identity (input)	0.242	1.000	0.999	1.000	0.810 $\pm$ .003
paste (no relight)	0.673	0.951	0.548	0.697	0.717 $\pm$ .005

Table 1: Anchor validation on all 1,421 tasks. Bold marks the dimension each anchor is constructed to violate, and no anchor is penalized elsewhere.

aggregate	identity	paste	gt	$>$ id	$\tau_{\mathrm{raw}}$
raw overall	0.810	0.717	0.829	4/23	1.00
$S_{\mathrm{mat}}$ family-only	0.813	0.749	0.882	9/23	0.91
gate $\mathrm{overall}_{G}$	0.236	0.552	0.729	21/23	0.77

Table 2: Robustness of the aggregate. Columns give the three anchors, the number of the

23

entries scoring above the identity floor, and Kendall

\tau

against the raw ranking. Correlations with the human ranking appear in Table 5 .

Track A: instruction (input $+$ text)
model	family acc.	material	preservation	structure	harmony	raw	gate
Nano-Banana-2-Lite †	0.899	0.640 $\pm$ .011	0.953 $\pm$ .001	0.820 $\pm$ .007	0.857 $\pm$ .007	0.817 $\pm$ .004	0.760
GPT-Image-2 †	0.852	0.616 $\pm$ .012	0.962 $\pm$ .001	0.853 $\pm$ .006	0.878 $\pm$ .007	0.827 $\pm$ .004	0.739
Qwen-Image-Edit	0.887	0.615 $\pm$ .011	0.899 $\pm$ .002	0.710 $\pm$ .010	0.817 $\pm$ .008	0.760 $\pm$ .005	0.677
BAGEL-7B	0.797	0.533 $\pm$ .012	0.917 $\pm$ .002	0.704 $\pm$ .010	0.787 $\pm$ .009	0.735 $\pm$ .004	0.576
FLUX-Kontext	0.780	0.554 $\pm$ .013	0.820 $\pm$ .002	0.613 $\pm$ .011	0.786 $\pm$ .008	0.693 $\pm$ .005	0.540

Table 3: Per-track leaderboard on bench-v1.0 (

1{,}421

tasks, higher is better), ordered by our primary aggregate gate (Sec. 4 ); raw is the unweighted mean, “family acc.” strict SigLIP2 family accuracy, and

\pm

a 95% bootstrap interval. Bold / underline : best/second per column within a track; † closed; ∗ below the gated do-nothing floor (

0.236

). Closed editors lead all three tracks; on Track C every family loses ground and two entries fall below that floor.

Figure 4: Role-binding failure: the model repaints the reference (the oak bench), composites the mask silhouette into it, and discards the scene.

	$\Delta_{\mathrm{mask}}$ (A $\to$ B)		$\Delta_{\mathrm{ref}}$ (B $\to$ C)
family	raw	gate	raw	gate
GPT-Image-2 †	$-0.003^{\mathrm{ns}}$	$-0.016^{\mathrm{ns}}$	$-0.017^{\ddagger}$	$-0.048$
Nano-Banana-2-Lite †	$0.000^{\mathrm{ns}}$	$-0.003^{\mathrm{ns}}$	$-0.010^{\ddagger}$	$-0.031$
Qwen-Image-Edit	$+0.009^{\ddagger}$	$+0.003^{\ddagger}$	$-0.150$	$-0.373$
BAGEL-7B	$-0.019^{\ddagger}$	$-0.006^{\mathrm{ns}}$	$-0.397$	$-0.508$
FLUX-Kontext	$+0.111$	$+0.137$	$-0.141$	$-0.291$

Table 4: Conditioning-aligned cross-track deltas: adding a mask (

\Delta_{\mathrm{mask}}

) and replacing text with the reference (

\Delta_{\mathrm{ref}}

), as task-paired means under both aggregates (per-track scores in Table 3 ). Unmarked entries are Holm-corrected Wilcoxon

p<10^{-3}

; ns not significant; ‡ marks

|\Delta|<0.02

from a single seed, significant over tasks but not seeds (Sec. 4 ). Beyond

\ddagger

, the mask helps one family and hurts one; the reference hurts all six , more under the gate.

	Ours			Established
	gate	raw	fam.	$-$ LPIPS	CLIP-I	CLIPScore	CLIP dir
$\tau_{\mathrm{H}}$	0.72	0.68	0.64	0.55	0.54	0.05	$-0.44$
$\rho_{\mathrm{H}}$	0.89	0.87	0.86	0.75	0.76	0.15	$-0.52$

Table 5: Kendall

\tau_{\mathrm{H}}

and Spearman

\rho_{\mathrm{H}}

against human Bradley–Terry strengths (425 judgments); GT-referenced baselines use the 1,093 clean-exemplar tasks. Ours exceeds every established metric: significantly for LPIPS and CLIPScore (bootstrap

P\geq 0.985

), marginally for CLIP-I (

\Delta\tau=+0.14

, CI

[-0.01,0.28]

Image Editing Evaluation

Image editing evaluation measures whether edits follow instructions while preserving identity, context, and visual quality.

See all 92 papers

Sep 30, 2026

MatLoom: Layered Text-to-Material Generation in a Compact Program Space

Material generation should produce not only an appearance, but also the rules that construct it. We introduce MatLoom, a compact, layer-oriented language for text-to-material generation with pretrained language models. Each program composes alpha-masked layers whose shared spatial expressions define coverage and physically based rendering (PBR) channels, making dependencies between patterns, color, and relief explicit. A standalone interpreter evaluates the program into material maps, while the source retains named fields and layer parameters for subsequent authoring. Without task-specific fine-tuning, our pipeline uses parser-guided repair and preview-based critique to revise material designs, then searches noise seeds while keeping each candidate's remaining source fixed. On a curated benchmark of 141 prompts evaluated with six backbones, our best-performing configuration achieves higher mean scores than three diffusion baselines on all four flat-layout prompt-alignment metrics. Its initial programs already exceed all three baselines on mean BLIPScore, before critique or seed search. Retained programs have a median length of 21 lines when pooled across backbones. In a blind four-way comparison involving 30 participants and 20 prompts, our renders receive 59.2% of choices, compared with 19.3% for the most-preferred baseline. Compact executable programs thus offer a way to generate prompt-aligned materials while retaining their construction as part of the asset.

Anson Y. Lam, Shuqing Li, Michael R. Lyu

Department of Computer Science and Engineering The Chinese University of Hong Kong Hong Kong, China

3D Asset Generation LLM-Based Program Synthesis Physically Based Rendering

Ask PDF

Dec 1, 2025

CoatFusion: Controllable Material Coating in Images

We introduce Material Coating, a novel image editing task that simulates applying a thin material layer onto an object while preserving its underlying coarse and fine geometry. Material coating is fundamentally different from existing "material transfer" methods, which are designed to replace an object's intrinsic material, often overwriting fine details. To address this new task, we construct a large-scale synthetic dataset (110K images) of 3D objects with varied, physically-based coatings, named DataCoat110K. We then propose CoatFusion, a novel architecture that enables this task by conditioning a diffusion model on both a 2D albedo texture and granular, PBR-style parametric controls, including roughness, metalness, transmission, and a key thickness parameter. Experiments and user studies show CoatFusion produces realistic, controllable coatings and significantly outperforms existing material editing and transfer methods on this new task.

Sagie Levy, Elad Aharoni, Matan Levy +2

The Hebrew University of Jerusalem, Israel · Reichman University, Israel

Image Editing Controllable Image Generation Diffusion-Based Image Editing

Ask PDF

May 15, 2026

DealMaTe: Multi-Dimensional Material Transfer via Diffusion Transformer

Recently, diffusion-based material transfer methods rely on image fine-tuning or complex architectures with auxiliary networks but face challenges such as text dependency, additional computational costs, and feature misalignment. To address these limitations, we propose DealMaTe, using depth, normal, and lighting images for material transfer. DealMaTe is a simplified diffusion framework that eliminates text guidance and reference networks. We design a lightweight 3D information injection method, Multi-Dim 3D Shader LoRA, which, without modifying the base model weights, enables compatible control conditions and achieves harmonious and stable results. Additionally, we optimize the attention mechanism with Shader Causal Mutual Attention and key-value (KV) caching to reduce inference latency caused by multiple conditions, improve computational efficiency, and achieve high-quality material transfer results with low architectural complexity. Extensive experiments covering a wide variety of objects and lighting conditions consistently demonstrate that DealMaTe achieves remarkable high-fidelity material transfer under arbitrary input materials. The code is available at https://github.com/haha-lisa/DealMaTe.

Nisha Huang, Yizhou Lin, Jie Guo +3

Tsinghua University, China · Pengcheng Laboratory, China · National Cheng-Kung University, Taiwan +2

Diffusion Transformer Diffusion-Based Image Editing Conditional Diffusion Models

Ask PDF

Abstract

Image Editing Evaluation

Explore similar work

MatLoom: Layered Text-to-Material Generation in a Compact Program Space

CoatFusion: Controllable Material Coating in Images

DealMaTe: Multi-Dimensional Material Transfer via Diffusion Transformer