MatReplace: A Reference-Free, Conditioning-Aligned Benchmark for Material Replacement in Interior Scenes

Mingzhe Du; Thong Thanh Nguyen; Nguyen Tran Cong Duy; See-Kiong Ng; Luu Anh Tuan

cs.CVAug 25, 2026

MatReplace: A Reference-Free, Conditioning-Aligned Benchmark for Material Replacement in Interior Scenes

Authors: Mingzhe Du, Thong Thanh Nguyen, Nguyen Tran Cong Duy, See-Kiong Ng, Luu Anh Tuan

Organizations: National University of Singapore · Nanyang Technological University · VinUniversity

Abstract

Material replacement is a common interior-design operation: changing the material of a selected surface while preserving its geometry, surroundings, and illumination. Despite its commercial relevance, no public benchmark isolates this task, and evaluating it is challenging. Reference-based metrics penalize valid outputs in this inherently one-to-many setting, favor the style of the reference generator, and cannot fairly compare editors that receive different forms of guidance. We introduce MatReplace, a reference-free benchmark that evaluates edits along four verifiable dimensions: local material correctness, global lighting harmony, outside preservation, and inside structure. It defines three tracks that vary one conditioning signal at a time: (A) instruction only, (B) instruction plus region mask, and (C) material reference image instead of instruction. Our results reveal a clear divide between naming and visually grounding materials. In Track A, leading closed-source editors achieve exemplar-level material rendering and surpass the exemplar anchor under our primary aggregate. In Track B, masks help only mask-compatible models with weak scene preservation, with task-paired, single-seed effects ranging from +0.137 to -0.090 across aligned model families. In Track C, reference-image conditioning degrades every family under both aggregates, by -0.031 to -0.508; in the worst cases, models repaint the reference image itself and perform worse than returning the input unchanged. Thus, named-material rendering is largely solved by the strongest closed editors on this distribution, but grounding materials from pixels remains an open challenge. Expert ratings validate our ranking (Kendall's tau = 0.68) and align with our aggregates more closely than GT-referenced or CLIP-based baselines.

Figures & tables

Figure 1: MatReplace at a glance. Each column is one task, read from the middle out. Middle: the target region in color, outlined over a dimmed grayscale surround (mask $m$ ), reference $r$ inset. Up ( ✓ ): a reference-quality edit (Nano-Banana-2-Lite, Track A). Material changes while geometry, scene, and illumination survive. Down ( ✗ ): one failure per mode. BAGEL-7B (C) repaints the reference and discards the scene (column 1); Qwen-Image-Edit (C) fails to apply the reference and leaves the slate untouched (column 2); OmniGen2 (C) pastes a flat, unshaded slab (column 3); Paste anchor tiles the reference without relighting (column 4). Each failure is caught by the dimension whose contract it violates (Sec. 4 ).

Figure 2: MatReplace Benchmark Construction. One templated generation session per candidate task, an algorithmic validity pass, and a semantic audit by a vision–language agent reduce 2,017 candidate sessions to the frozen 1,421 tasks. The middle bar is drawn to scale, so the admitted and rejected shares are directly comparable and the accounting closes. The 596 rejects ship as a disjoint rejected pool , released for auditing and never used for evaluation.

Figure 3: One failure per dimension: each pair shows the input with the target region outlined, then the model output. SDXL-inpaint paints bare wall instead of navy suede ( $S_{\mathrm{mat}}$ ). InstructPix2Pix floods the whole room with the target emerald ( $S_{\mathrm{pres}}$ ). SD3.5-inpaint re-sculpts the rectangular fireplace into an arched portal ( $S_{\mathrm{struct}}$ ). FLUX-Kontext-ref pastes the reference as flat, unlit patches ( $S_{\mathrm{harm}}$ ).

anchor	mat	pres	struct	harm	overall
gt (exemplar)	0.622	0.964	0.853	0.877	0.829 $\pm$ .004
identity (input)	0.242	1.000	0.999	1.000	0.810 $\pm$ .003
paste (no relight)	0.673	0.951	0.548	0.697	0.717 $\pm$ .005

Table 1: Anchor validation on all 1,421 tasks. Bold marks the dimension each anchor is constructed to violate, and no anchor is penalized elsewhere.

aggregate	identity	paste	gt	$>$ id	$\tau_{\mathrm{raw}}$
raw overall	0.810	0.717	0.829	4/23	1.00
$S_{\mathrm{mat}}$ family-only	0.813	0.749	0.882	9/23	0.91
gate $\mathrm{overall}_{G}$	0.236	0.552	0.729	21/23	0.77

Table 2: Robustness of the aggregate. Columns give the three anchors, the number of the

23

entries scoring above the identity floor, and Kendall

\tau

against the raw ranking. Correlations with the human ranking appear in Table 5 .

Track A: instruction (input $+$ text)
model	family acc.	material	preservation	structure	harmony	raw	gate
Nano-Banana-2-Lite †	0.899	0.640 $\pm$ .011	0.953 $\pm$ .001	0.820 $\pm$ .007	0.857 $\pm$ .007	0.817 $\pm$ .004	0.760
GPT-Image-2 †	0.852	0.616 $\pm$ .012	0.962 $\pm$ .001	0.853 $\pm$ .006	0.878 $\pm$ .007	0.827 $\pm$ .004	0.739
Qwen-Image-Edit	0.887	0.615 $\pm$ .011	0.899 $\pm$ .002	0.710 $\pm$ .010	0.817 $\pm$ .008	0.760 $\pm$ .005	0.677
BAGEL-7B	0.797	0.533 $\pm$ .012	0.917 $\pm$ .002	0.704 $\pm$ .010	0.787 $\pm$ .009	0.735 $\pm$ .004	0.576
FLUX-Kontext	0.780	0.554 $\pm$ .013	0.820 $\pm$ .002	0.613 $\pm$ .011	0.786 $\pm$ .008	0.693 $\pm$ .005	0.540

Table 3: Per-track leaderboard on bench-v1.0 (

1{,}421

tasks, higher is better), ordered by our primary aggregate gate (Sec. 4 ); raw is the unweighted mean, “family acc.” strict SigLIP2 family accuracy, and

\pm

a 95% bootstrap interval. Bold / underline : best/second per column within a track; † closed; ∗ below the gated do-nothing floor (

0.236

). Closed editors lead all three tracks; on Track C every family loses ground and two entries fall below that floor.

Figure 4: Role-binding failure: the model repaints the reference (the oak bench), composites the mask silhouette into it, and discards the scene.

	$\Delta_{\mathrm{mask}}$ (A $\to$ B)		$\Delta_{\mathrm{ref}}$ (B $\to$ C)
family	raw	gate	raw	gate
GPT-Image-2 †	$-0.003^{\mathrm{ns}}$	$-0.016^{\mathrm{ns}}$	$-0.017^{\ddagger}$	$-0.048$
Nano-Banana-2-Lite †	$0.000^{\mathrm{ns}}$	$-0.003^{\mathrm{ns}}$	$-0.010^{\ddagger}$	$-0.031$
Qwen-Image-Edit	$+0.009^{\ddagger}$	$+0.003^{\ddagger}$	$-0.150$	$-0.373$
BAGEL-7B	$-0.019^{\ddagger}$	$-0.006^{\mathrm{ns}}$	$-0.397$	$-0.508$
FLUX-Kontext	$+0.111$	$+0.137$	$-0.141$	$-0.291$

Table 4: Conditioning-aligned cross-track deltas: adding a mask (

\Delta_{\mathrm{mask}}

) and replacing text with the reference (

\Delta_{\mathrm{ref}}

), as task-paired means under both aggregates (per-track scores in Table 3 ). Unmarked entries are Holm-corrected Wilcoxon

p<10^{-3}

; ns not significant; ‡ marks

|\Delta|<0.02

from a single seed, significant over tasks but not seeds (Sec. 4 ). Beyond

\ddagger

, the mask helps one family and hurts one; the reference hurts all six , more under the gate.

	Ours			Established
	gate	raw	fam.	$-$ LPIPS	CLIP-I	CLIPScore	CLIP dir
$\tau_{\mathrm{H}}$	0.72	0.68	0.64	0.55	0.54	0.05	$-0.44$
$\rho_{\mathrm{H}}$	0.89	0.87	0.86	0.75	0.76	0.15	$-0.52$

Table 5: Kendall

\tau_{\mathrm{H}}

and Spearman

\rho_{\mathrm{H}}

against human Bradley–Terry strengths (425 judgments); GT-referenced baselines use the 1,093 clean-exemplar tasks. Ours exceeds every established metric: significantly for LPIPS and CLIPScore (bootstrap

P\geq 0.985

), marginally for CLIP-I (

\Delta\tau=+0.14

, CI

[-0.01,0.28]

Image Editing Evaluation

Image editing evaluation measures whether edits follow instructions while preserving identity, context, and visual quality.

See all 92 papers

MatLoom: Layered Text-to-Material Generation in a Compact Program Space
Sep 30, 2026Anson Y. Lam, Shuqing Li, Michael R. Lyu3D Asset Generation LLM-Based Program Synthesis
Department of Computer Science and Engineering The Chinese University of Hong Kong Hong Kong, China
CoatFusion: Controllable Material Coating in Images
Dec 1, 2025Sagie Levy, Elad Aharoni, Matan Levy +2Image Editing Controllable Image Generation
The Hebrew University of Jerusalem, Israel · Reichman University, Israel
DealMaTe: Multi-Dimensional Material Transfer via Diffusion Transformer
May 15, 2026Nisha Huang, Yizhou Lin, Jie Guo +3Diffusion Transformer Diffusion-Based Image Editing
Tsinghua University, China · Pengcheng Laboratory, China · National Cheng-Kung University, Taiwan +2

Sep 30, 2026J/K move · Enter open · S save

MatLoom: Layered Text-to-Material Generation in a Compact Program Space

Anson Y. Lam, Shuqing Li, Michael R. Lyu

Department of Computer Science and Engineering The Chinese University of Hong Kong Hong Kong, China

3D Asset Generation LLM-Based Program Synthesis Physically Based Rendering

Open paper AskPDF arXiv

Material generation should produce not only an appearance, but also the rules that construct it. We introduce MatLoom, a compact, layer-oriented language for text-to-material generation with pretrained language models. Each program composes alpha-masked layers whose shared spatial expressions define coverage and physically based rendering (PBR) channels, making dependencies between patterns, color, and relief explicit. A standalone interpreter evaluates the program into material maps, while the source retains named fields and layer parameters for subsequent authoring. Without task-specific fine-tuning, our pipeline uses parser-guided repair and preview-based critique to revise material designs, then searches noise seeds while keeping each candidate's remaining source fixed. On a curated benchmark of 141 prompts evaluated with six backbones, our best-performing configuration achieves higher mean scores than three diffusion baselines on all four flat-layout prompt-alignment metrics. Its initial programs already exceed all three baselines on mean BLIPScore, before critique or seed search. Retained programs have a median length of 21 lines when pooled across backbones. In a blind four-way comparison involving 30 participants and 20 prompts, our renders receive 59.2% of choices, compared with 19.3% for the most-preferred baseline. Compact executable programs thus offer a way to generate prompt-aligned materials while retaining their construction as part of the asset.

Abstract

Image Editing Evaluation

Explore similar work

MatLoom: Layered Text-to-Material Generation in a Compact Program Space

CoatFusion: Controllable Material Coating in Images

DealMaTe: Multi-Dimensional Material Transfer via Diffusion Transformer