Texture Space Material Diffusion

Jacob Munkberg; Peter Kocsis; Jon Hasselgren

cs.CVSep 29, 2026

Texture Space Material Diffusion

Authors: Jacob Munkberg, Peter Kocsis, Jon Hasselgren

Organizations: NVIDIA

Abstract

We present a method for generating high quality materials for 3D objects entirely in texture space. We finetune a video diffusion transformer for text-guided material generation, multi-view material generation, and material upscaling. Our key insight is to use the known projection from image space to texture space, enabling the diffusion process to generalize across arbitrary geometries and texture parameterizations. This approach also avoids the view consistency issues inherent in video and multi-view diffusion models. Because texture space is two dimensional, we can reuse the strong priors of pretrained video diffusion models. We apply our method to high quality material reconstruction from posed photos captured under unknown lighting, as well as to text- and image guided material generation. Our method can scale to high resolutions (8K), 100+ input views, and neural material representations. In quantitative and qualitative evaluations we show state-of-the-art results for material generation and reconstruction.

Figures & tables

Figure 2: Texture Space Observations. We leverage known geometry with a non-overlapping texture parameterization. Given camera poses and geometry, we project each view back into texture space, which results in partially covered texture space observations. By formulating our diffusion process in the 2D texture space domain, we can leverage powerful image- and video diffusion model priors to synthesize high quality materials.

Figure 3: Method Overview. Our pipeline leverages diffusion transformers in texture space to synthesize high quality materials. We support multiple input modalities: 1) one or more views projected into texture space, which can be photographs, rendered images, or frames generated by diffusion models 2) a text prompt describing the material 3) low-resolution material maps. In all cases, we condition the model on world space positions, normals and a text prompt. Our finetuned DiT successfully demodulates the (unknown) lighting, and creates clean albedo, roughness, and metallicity maps with high frequency details. The materials can be directly applied in standard content creation tools.

Panel of Figure 4 — Figure 4: Reconstruction - Real-World. Material reconstructions from multi-view observations (photographs with known poses) on two examples from the DTC dataset ( Dong et al., 2025 ) . For optimization-based differentiable path tracing (DiffPT) and our method, we leverage known geometry, while LSRM jointly reconstructs geometry and materials. In the right part we show the reconstructed materials under two novel lighting conditions.

		Real recon (136 img)			Synthetic recon (256 img)			Synthetic relight (2k img)
Geom.	Method	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$
Known	DiffPT	30.67	0.959	0.0307	27.50	0.929	0.0739	24.90	0.903	0.0911
(ref.)	Ours	29.03	0.947	0.0331	31.24	0.960	0.0370	30.46	0.950	0.0401
Recon.	LSRM	27.75	0.933	0.0388	22.82	0.872	0.1210	21.82	0.862	0.1241
(LSRM)	Ours (LSRM geo)	27.01	0.936	0.0423	24.10	0.890	0.0896	23.71	0.885	0.0927

Table 1: Multi-view to materials. Comparison on synthetic materials and posed real photographs. Synthetic metrics are averaged over 32 extracted materials rendered from eight views. For the relighting metrics, we render each view with eight different HDR probes. Real metrics are averaged over eight examples from the DTC dataset ( Dong et al., 2025 ) . Materials are reconstructed from 17 views in 2K resolution. We compare against DiffPT ( Hasselgren et al., 2022 ) and LSRM ( Li et al., 2026b ) . DiffPT overfits (reconstruction w/ single lighting) but fails to generalize (relighting).

Panel of Figure 5 — Figure 5: Reconstruction - Generated Frames. Material reconstructions from imperfect views, with views synthesized by an off-the-shelf depth-conditioned video model: Wan2.2-VACE-Fun-A14B ( Jiang et al., 2025 ) . Compared to an optimization-based approach, our method robustly reconstructs materials also from the inconsistent views from the video model.

Figure 6: Generation - Single-View. Materials from our generative method, conditioned on a single image and text description. The visualized view is rotated slightly from the conditioning view (leftmost insets). We use reference geometry and the same conditioning view for all methods.

Single-view to material				Text to material
Method	CLIP-FID ( $\downarrow$ )	CMMD ( $\downarrow$ )	LPIPS ( $\downarrow$ )	Method	CLIP-FID ( $\downarrow$ )	CMMD ( $\downarrow$ )	LPIPS ( $\downarrow$ )
Hunyuan3D 2.1	3.419	0.0437	0.0520
VideoMatGen	2.973	0.0236	0.0527	VideoMatGen	4.725	0.0330	0.0639
Trellis.2	2.227	0.0184	0.0395	VideoMat	4.376	0.0247	0.0652
Ours	1.520	0.0081	0.0325	Ours	3.338	0.0230	0.0542

Table 2: Material generation. Left: single-view to material generation. Right: text-guided material generation. All metrics evaluated on 32 scenes

\times

8 views

\times

8 probes.

Panel of Figure 7 — Figure 7: TEXGen . TEXGen ( Yu et al., 2024 ) only produces albedo textures. Metrics for diffuse-only renderings from 32 scenes $\times$ 8 views $\times$ 8 probes from single-view guidance.

Panel of Figure 8 — Figure 8: Ablation – Inference Scaling. With our inference augmentations, we can scale our results to 8K texture resolution and increase the number of input views from 17 to 117. This enables capturing high-frequency details and accurate specular predictions. We provide average metrics over the first four samples of the real dataset, showing preserved fidelity with slightly improved LPIPS.

Appendix figures & tables14 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 10 — Figure S1: Effect of CRF . We show the effect of the CRF adjustment on the LSRM ( Li et al., 2026b ) prediction of the Gargoyle sample from the DTC dataset ( Dong et al., 2025 ) . Even though the material patterns are mostly captured properly by the baseline, color is shifted due to unknown lighting condition and potential training bias. CRF adjustment helps to rule out this bias and provide a more fair comparison.

		Real recon (136 img)			Synthetic recon (256 img)			Synthetic relight (2k img)
Geom.	Method	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$
Known	DiffPT	27.01	0.959	0.0308	21.99	0.908	0.0854	21.14	0.903	0.0911
(ref.)	Ours	22.66	0.930	0.0430	30.17	0.958	0.0327	29.89	0.954	0.0350
Recon.	LSRM	24.56	0.926	0.0445	20.08	0.848	0.1296	19.80	0.847	0.1330
(LSRM)	Ours (LSRM geo)	22.22	0.923	0.0496	23.61	0.884	0.0860	23.43	0.889	0.0880

Appendix

Table S1: Results Without CRF Adjustment . We provide quantitative results for Table 1 without adjusting the CRF. The tendency is the same, however some metrics become heavily biased due to the decomposition ambiguity.

Method	Resolution	2K recon	8K super-res	Projection	Combined time	Peak VRAM
17-view	2K	130 s	-	-	2 min 10 s	22 GiB
117-view	2K	2610 s	-	-	43 min 30 s	48 GiB
17-view	8K	130 s	267 s	10 s	6 min 47 s	22 GiB
117-view	8K	2610 s	279 s	10 s	48 min 19 s	48 GiB

Appendix

Table S2: Runtime cost . Runtime cost and Peak VRAM usage. Scores are averages over four examples from the DTC dataset, measured on an NVIDIA GB300 GPU.

Panel of Figure 13 — Figure S2: Material components .

Method	Base color		Roughness	Metallicity
	siPSNR ( $\uparrow$ )	PSNR ( $\uparrow$ )	PSNR ( $\uparrow$ )	PSNR ( $\uparrow$ )
Ours	29.19	27.00	24.32	18.24
DiffPT ( Hasselgren et al., 2022 )	25.08	20.21	17.98	13.23
LSRM ( Li et al., 2026b )	23.17	15.46	13.18	14.73

Appendix

Table S3: Material components . We compute metrics on g-buffer renderings for 32 scenes

\times

17 views from multi-view guidance in our BlenderVault test set. We report scale-invariant PSNR for the base color renderings, and standard PSNR scores for base color, roughness, and metallicity g-buffer renderings.

Panel of Figure 15 — Figure S3: Material generations from casually captured photographs, using 17 views from the DTC dataset Dong et al. (2025) . We expect known camera poses and geometry in this test. Top : Input views (two selected views per example). Bottom : Our extracted materials (two seeds per example) re-rendered in novel lighting in Blender.

Panel of Figure 16 — Figure S4: Generated basecolor textures from single-view guidance. Overall, our method generates more consistent albedo maps than TEXGen, demodulates lighting from the conditional view (TEXGen uses a demodulated conditional view), and outputs full PBR materials (not shown here).

Figure S5: Reconstruction - Real-World. Material reconstructions from multi-view observations (photographs with known poses) on three examples from the DTC dataset ( Dong et al., 2025 ) . For optimization-based differentiable path tracing (DiffPT) and our method, we leverage known geometry, while LSRM jointly reconstructs geometry and materials.

	Basecolor		(Rgh, Met)
Method	PSNR ( $\uparrow$ )	LPIPS ( $\downarrow$ )	PSNR ( $\uparrow$ )	LPIPS ( $\downarrow$ )
Our	34.7	0.061	38.0	0.025
PBR-SR ( Chen et al., 2025a )	30.9	0.151	40.6	0.030
ESRGAN ( Wang et al., 2018 )	32.6	0.146	37.9	0.041

Appendix

Table S4: Texture upscaler. Results for

4\times

texture upscaling comparing our method with popular alternatives. Results are quality metric averages for all textures in our synthetic dataset (32 objects).

Figure S6: Attention visualization . We visualize the attention activations for a query point (red circle) for three variants: Top : The RoPE of Wan 2.1, combining pixel $xy$ -coordinates, $\mathbf{p}_{uv}$ , and the frame id $f_{\mathrm{id}}$ . Middle : The Wan 2.1 RoPE and G-buffer world position and normal guides, $G_{\mathrm{buf}}$ . Bottom : Our 3D-aware RoPE + $G_{\mathrm{buf}}$ . We show the attention activations for five layers. We note that our 3D-aware RoPE has stronger activations at similar 3D locations.

Panel of Figure 20 — Figure S7: Attention visualization. We show the attention activations for all layers for two query points on the lion king DTC example. The RoPE of Wan 2.1 combines pixel $xy$ -coordinates, $\mathbf{p}_{uv}$ , and the frame id $f_{\mathrm{id}}$ . We add a G-buffer wpos and normals, $G_{\mathrm{buf}}$ , as conditions (middle column). In the rightmost column, we show our 3D-aware RoPE + $G_{\mathrm{buf}}$ . Please zoom on the PDF image to see details. For reference, the left column shows the world space distance between two points in texture space. Note that our embedding attends strongly to points closer in world space over all layers of the network.

RoPE	Inputs	$G_{\mathrm{buf}}$	CLIP-FID $\downarrow$	CMMD $\downarrow$	LPIPS $\downarrow$
Wan 2.1	$\mathbf{p}_{uv}\!\!+\!\!f_{\mathrm{id}}$	✗	4.036	0.0405	0.1005
Wan 2.1	$\mathbf{p}_{uv}\!\!+\!\!f_{\mathrm{id}}$	✓	3.677	0.0289	0.0971
3D-aware	$\mathbf{p}_{xyz}\!\!+\!\!f_{\mathrm{id}}$	✓	3.561	0.0237	0.0957

Appendix

Table S5: RoPE embedding ablation. We ablate our proposed 3D-aware rotary positional embedding against the standard Wan 2.1 RoPE on the text to material pipeline in Figure 2 of the main paper, evaluated on the same 32 scenes

\times

8 views

\times

8 probes with material textures generated at a resolution of

512\times 512

texels.

Panel of Figure 22 — Figure S8: Stability. Impact of RoPE embedding on a single text to material example with varying random seed. As can be seen, the 3D-aware RoPE encoding better captures the global structure of the object, and produces more consistent materials. The small insets show the base color, roughness, and metallicity maps for each generation.

Figure S9: Generation - Text to material. Materials from our generative method, conditioned on a text prompt and the input geometry (the UV mask, world space positions, and surface normals). We compare against VideoMat ( Munkberg et al., 2025 ) and VideoMatGen ( Hasselgren et al., 2026 ) . Without any image guidance, our model still produces semantically meaningful materials for all examples.

Explore similar work

Sep 28, 2026cs.CV

TaoTex: Boosting Texture Detail Fidelity for Native 3D Material Generation

Recent 3D generation models can produce accurate geometries while still struggling to reconstruct detailed textures. We propose a diffusion-based native 3D material generation model TaoTex, which faithfully recovers intricate textures through tailored strategies and improvements. First, we develop a data construction agent to create high-frequency textured 3D assets to bridge the data gap in public datasets. Training with these data significantly enhances the ability of TaoTex to recover challenging details such as text and patterns. Second, we design a multi-level feature fusion (MLFF) module to adaptively integrate local and global features of the conditional input, providing more complete texture cues for the diffusion model and thereby enhancing reconstruction fidelity. To alleviate VAE reconstruction errors, we adopt a latent-to-pixel space loss transition, further improving the pixel-level details and generation quality. Finally, we scale TaoTex to multi-view inputs by incorporating learnable viewpoint embeddings, achieving accurate and consistent material reconstruction across views. Extensive experiments demonstrate that our method significantly outperforms existing approaches in preserving texture details in both single- and multi-view settings.

Xiuchao Wu, Shuichang Lai, Jiangjing Lyu +1

Alibaba Group

3D Assets Conditional Variational Autoencoder

Ask PDF

May 15, 2026cs.CV

MaTe: Images Are All You Need for Material Transfer via Diffusion Transformer

Recent diffusion-based methods for material transfer rely on image fine-tuning or complex architectures with assistive networks, but face challenges including text dependency, extra computational costs, and feature misalignment. To address these limitations, we propose MaTe, a streamlined diffusion framework that eliminates textual guidance and reference networks. MaTe integrates input images at the token level, enabling unified processing via multi-modal attention in a shared latent space. This design removes the need for additional adapters, ControlNet, inversion sampling, or model fine-tuning. Extensive experiments demonstrate that MaTe achieves high-quality material generation under a zero-shot, training-free paradigm. It outperforms state-of-the-art methods in both visual quality and efficiency while preserving precise detail alignment, significantly simplifying inference prerequisites.

Nisha Huang, Henglin Liu, Yizhou Lin +5

Tsinghua University · PengCheng Laboratory · Lenovo Research +1

Diffusion Transformers Diffusion Models Cross-Modal Attention Model Fine-Tuning

Ask PDF

Apr 26, 2024cs.CV

MaPa: Text-driven Photorealistic Material Painting for 3D Shapes

This paper aims to generate materials for 3D meshes from text descriptions. Unlike existing methods that synthesize texture maps, we propose to generate segment-wise procedural material graphs as the appearance representation, which supports high-quality rendering and provides substantial flexibility in editing. Instead of relying on extensive paired data, i.e., 3D meshes with material graphs and corresponding text descriptions, to train a material graph generative model, we propose to leverage the pre-trained 2D diffusion model as a bridge to connect the text and material graphs. Specifically, our approach decomposes a shape into a set of segments and designs a segment-controlled diffusion model to synthesize 2D images that are aligned with mesh parts. Based on generated images, we initialize parameters of material graphs and fine-tune them through the differentiable rendering module to produce materials in accordance with the textual description. Extensive experiments demonstrate the superior performance of our framework in photorealism, resolution, and editability over existing methods. Project page: https://zju3dv.github.io/MaPa

Shangzhan Zhang, Sida Peng, Tao Xu +7

Zhejiang University · Ant Group · Shenzhen University

Differentiable Rendering Materials Generative Models

Ask PDF