MatLat: Material Latent Space for PBR Texture Generation

Kyeongmin Yeo; Yunhong Min; Jaihoon Kim; Minhyuk Sung

cs.CVDec 19, 2025

MatLat: Material Latent Space for PBR Texture Generation

Authors: Kyeongmin Yeo, Yunhong Min, Jaihoon Kim, Minhyuk Sung

Organizations: KAIST

Abstract

We propose a generative framework for producing high-quality PBR textures on a given 3D mesh. As large-scale PBR texture datasets are scarce, our approach focuses on effectively leveraging the embedding space and diffusion priors of pretrained latent image generative models while learning a material latent space, MatLat, through targeted fine-tuning. Unlike prior methods that freeze the embedding network, which leads to distribution shifts when encoding additional PBR channels and hinders subsequent diffusion training, we fine-tune the pretrained VAE so that new material channels can be incorporated with minimal latent distribution deviation. We further show that correspondence-aware attention alone is insufficient for cross-view consistency unless the latent-to-image mapping preserves locality. To enforce this locality, we introduce a regularization in the VAE fine-tuning that crops latent patches, decodes them, and aligns the corresponding image regions to maintain strong pixel-latent spatial correspondence. Ablation studies and comparison with previous baselines demonstrate that our framework improves PBR texture fidelity and that each component is critical for achieving state-of-the-art performance.

Figures & tables

Figure 2: Overview of Frozen VAE [ 18 ] . Zero-padded roughness–metallic maps and albedo image are encoded by the frozen VAE to produce $(\bm{\mu}_{\text{rm}},\bm{\sigma}_{\text{rm}})$ and $(\bm{\mu}_{\text{a}},\bm{\sigma}_{\text{a}})$ , respectively.

Figure 3: Comparison of PBR Material Encoder Schemes. (a) LayerDiffuse [ 58 ] uses residual prediction but only predicts the latent mean and enforces identity consistency via $\mathcal{L}_{\text{id}}$ . (b) Orchid [ 29 ] uses direct prediction to output the full latent parameters $(\bm{\mu}_{\text{full}},\bm{\sigma}_{\text{full}})$ and regularizes them with $\mathcal{L}_{\text{reg}}$ . (c) MatVAE (ours) synergistically integrates residual prediction and KL regularization , enabling effective incorporation of PBR material images while preserving the pretrained latent space.

Figure 4: Illustration of Correspondence-Aware Attention (CAA) and Locality Regularization. (a) CAA restricts attention to geometrically corresponding tokens across views. (b) Locality regularizer enforces patch-wise reconstruction such that image pixels are decoded from spatially aligned latent tokens. Together, CAA and locality regularization enable multi-view consistent PBR texture generation.

Id	Methods	Shaded			Albedo			Rough.	Metal.	c-PSNR $\uparrow$	Time $\downarrow$
	Methods	$\mathrm{FID}_{\text{CLIP}}\downarrow$	$\mathrm{KID}\downarrow$	$\mathrm{CLIP}\uparrow$	$\mathrm{FID}_{\text{CLIP}}\downarrow$	$\mathrm{KID}\downarrow$	$\mathrm{CLIP}\uparrow$	$\mathrm{RMSE}\downarrow$	$\mathrm{RMSE}\downarrow$	c-PSNR $\uparrow$	Time $\downarrow$
1	Frozen VAE [ 18 ]	3.419	1.052	0.318	4.926	1.560	0.315	0.193	0.251	19.869	111s
2	Res. Pred. + $\mathcal{L}_{\text{reg}}$ (Ours)	3.083	1.327	0.318	4.599	1.574	0.314	0.158	0.134	21.934	34s
3	$\llcorner$ Res. Pred. + $\mathcal{L}_{\text{id}}$ [ 58 ]	3.210	1.090	0.317	4.871	1.525	0.312	0.165	0.128	20.977	34s
4	$\llcorner$ Direct Pred. + $\mathcal{L}_{\text{reg}}$ [ 29 ]	3.192	1.146	0.319	4.768	1.472	0.312	0.161	0.170	21.468	34s
5	$\llcorner$ w/o $\mathcal{L}_{\text{local}}$	3.419	1.611	0.316	5.873	4.122	0.312	0.154	0.182	19.437	34s

Table 1: Quantitative Results of Ablation Study. Best scores are bold , and the runner-up scores are underlined .

Panel of Figure 5 — Figure 5: Qualitative Comparisons for Each Component. Combining residual prediction with $\mathcal{L}_{\text{reg}}$ effectively preserves the pretrained latent distribution and yields high-quality PBR textures.

Method	Type	Shaded			Albedo			Rough.	Metal.	Time $\downarrow$
Method	Type	$\mathrm{FID}_{\text{CLIP}}\downarrow$	$\mathrm{KID}\downarrow$	$\mathrm{CLIP}\uparrow$	$\mathrm{FID}_{\text{CLIP}}\downarrow$	$\mathrm{KID}\downarrow$	$\mathrm{CLIP}\uparrow$	$\mathrm{RMSE}\downarrow$	$\mathrm{RMSE}\downarrow$	Time $\downarrow$
MeshGen [ 7 ]	SC	8.637	11.105	0.282	11.322	16.193	0.282	0.143	0.201	195s
TexGaussian [ 52 ]	SC	6.025	3.571	0.301	12.119	9.381	0.299	0.145	0.243	73s
Paint-it [ 55 ]	SDS	8.547	5.382	0.309	13.063	10.665	0.299	0.168	0.200	1260s
DreamMat [ 61 ]	SDS	5.422	2.668	0.311	9.621	6.002	0.311	0.167	0.165	2400s
FlashTex [ 10 ]	SDS	7.119	5.354	0.305	12.320	10.441	0.298	0.143	0.186	285s

Table 2: Quantitative Comparisons of PBR Texture Generation Methods. Best scores are bold , and the runner-up scores are underlined .

Panel of Figure 6 — Figure 6: Qualitative Results of MatLat . MatLat produces realistic textures with strong text-alignment across diverse material types.

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 7: Direct vs. Residual Prediction. We show the albedo, roughness, and metallic reconstruction errors at an early training stage (10k iterations) for direct and residual prediction. Residual prediction preserves the pretrained latent representation, achieving superior albedo reconstruction quality (darker regions indicate lower errors).

Methods	Shaded	Albedo
Methods	$\mathrm{KID}_{\text{CLIP}}\downarrow$	$\mathrm{KID}_{\text{CLIP}}\downarrow$
Frozen VAE [ 18 ]	2.123	3.547
Res. Pred. + $\mathcal{L}_{\text{reg}}$ (Ours)	1.496	2.855
Res. Pred. + $\mathcal{L}_{\text{id}}$ [ 58 ]	1.977	3.710
Direct Pred. + $\mathcal{L}_{\text{reg}}$ [ 29 ]	1.917	3.254
w/o $\mathcal{L}_{\text{local}}$	2.660	8.820

Appendix

Table 3: Evaluation on CLIP-based KID. We additionally measure KID in the CLIP feature space. Our proposed VAE fine-tuning scheme yields superior performance, with both our full model and the variant without CAA consistently outperforming the other baselines.

Figure 8: Visualization of Reconstruction Error Maps on Cropped Patches. We show albedo, roughness, and metallic reconstruction error maps for MatVAE trained with and without $\mathcal{L}_{\text{local}}$ . Applying $\mathcal{L}_{\text{local}}$ significantly reduces patch reconstruction errors, indicating improved latent–image spatial alignment.

Crop range	$\mathrm{FID}^{\text{CLIP}}_{\text{shaded}}$	$\mathrm{FID}^{\text{CLIP}}_{\text{albedo}}$	c-PSNR
$[3,32]$	3.083	4.599	21.934
$[24,48]$	3.208	5.027	20.571
$[32,64]$	3.339	4.719	20.980

Appendix

Table 4: Crop-range ablation for locality regularization. We vary the crop-size range used in

\mathcal{L}_{\text{local}}

on a

64\times 64

latent grid. Smaller crop ranges yield the best overall performance, while all cropped settings outperform training without cropping.

$\mathrm{SWD}(\mathbf{z},\mathbf{z}_{\mathrm{a}})$	$\mathrm{SWD}(\mathbf{z},\mathbf{z}_{\mathrm{rm}})$	$\mathrm{SWD}(\mathbf{z},\mathbf{z}_{\mathrm{ours}})$
0.539	0.897	0.560

Appendix

Table 5: Sliced Wasserstein distances to the pretrained diffusion prior. Lower values indicate that the encoded latents are better aligned with the pretrained latent space.

Panel of Figure 9 — Figure 9: Relighting results. Example objects generated by MatLat and rendered under four different environment maps.

Figure 10 : Additional Qualitative Results of Ablation Studies. Extended qualitative results of the ablation studies presented in Fig. 5 : Frozen VAE, Res. Pred. +

\mathcal{L}_{\text{id}}

, Direct Pred. +

\mathcal{L}_{\text{reg}}

, and Res. Pred. +

\mathcal{L}_{\text{reg}}

(Ours). Best viewed in video.

Figure 11 : Additional Qualitative Comparisons with Previous Methods. Each column shows the shaded output rendered under identical lighting conditions. Our method produces high-quality, physically plausible materials with superior text-visual alignment and detail preservation. Best viewed in video.

Figure 12 : Additional Qualitative Results of MatLat . Gallery of assets textured by MatLat for various text prompts and environment maps. Best viewed in video.

Appendix

Table 14

Texture Synthesis

Texture synthesis creates and edits realistic visual patterns and surface materials, from image patches to coherent, view-consistent 3D assets.

See all 34 papers

Mar 17, 2026

VideoMatGen: PBR Materials through Joint Generative Modeling

We present a method for generating physically-based materials for 3D shapes based on a video diffusion transformer architecture. Our method is conditioned on input geometry and a text description, and jointly models multiple material properties (base color, roughness, metallicity, height map) to form physically plausible materials. We further introduce a custom variational auto-encoder which encodes multiple material modalities into a compact latent space, which enables joint generation of multiple modalities without increasing the number of tokens. Our pipeline generates high-quality materials for 3D shapes given a text prompt, compatible with common content creation tools.

Jon Hasselgren, Zheng Zeng, Milos Hasan +1

NVIDIA

3D Texture Generation 3D Asset Generation Multimodal Diffusion Models

Ask PDF

Sep 29, 2026

Texture Space Material Diffusion

We present a method for generating high quality materials for 3D objects entirely in texture space. We finetune a video diffusion transformer for text-guided material generation, multi-view material generation, and material upscaling. Our key insight is to use the known projection from image space to texture space, enabling the diffusion process to generalize across arbitrary geometries and texture parameterizations. This approach also avoids the view consistency issues inherent in video and multi-view diffusion models. Because texture space is two dimensional, we can reuse the strong priors of pretrained video diffusion models. We apply our method to high quality material reconstruction from posed photos captured under unknown lighting, as well as to text- and image guided material generation. Our method can scale to high resolutions (8K), 100+ input views, and neural material representations. In quantitative and qualitative evaluations we show state-of-the-art results for material generation and reconstruction.

Jacob Munkberg, Peter Kocsis, Jon Hasselgren

NVIDIA

3D Asset Generation Texture Synthesis

Ask PDF

Jun 26, 2026

Monocular Avatar Reconstruction via Cascaded Diffusion Priors and UV-Space Differentiable Shading

Reconstructing high-fidelity, relightable 3D avatars from a single in-the-wild image is a challenging ill-posed problem, primarily hindered by the scarcity of high-quality PBR data and the complexity of disentangling illumination from intrinsic materials. In this paper, we present a data-efficient framework that leverages the robust priors of a unified pre-trained diffusion backbone to sequentially address texture completion, delighting, and material decomposition. Unlike existing methods that rely on fragmented pipelines or extensive proprietary datasets, we utilize cascaded Low-Rank Adaptations (LoRAs) to adapt the strong generative prior of the diffusion model for each sub-task in UV space. Specifically, we first employ an Inpainting LoRA to complete missing UV textures caused by occlusion, leveraging the model's semantic understanding to generate semantically and photometrically coherent details. Subsequently, a Light-Homogenization LoRA and a novel Cross-Intrinsic Attention mechanism are introduced to remove baked-in lighting and collaboratively synthesize pixel-aligned PBR maps (Albedo, Normal, Roughness, Specular, and Displacement). To ensure physical plausibility, we impose a UV-space differentiable BRDF shading loss during the decomposition stage, forcing the generative process to adhere to the rendering equation without the artifacts typical of rasterization-based supervision. Extensive experiments demonstrate that our method, trained on fewer than 100 real 3D scans, generates comprehensive, 4K-resolution PBR assets with superior realism and generalization compared to state-of-the-art methods, and all training code and model weights will be released upon acceptance.

Hong Li, Minqi Meng, Yanjun Liang +10

Beihang University, China · Great Wall Motors, China · CUHK-Shenzhen, China +6

Inverse Rendering Intrinsic Image Decomposition 3D Reconstruction Monocular 3D Reconstruction Texture Synthesis Diffusion Models

Ask PDF

Abstract

Texture Synthesis

Explore similar work

VideoMatGen: PBR Materials through Joint Generative Modeling

Texture Space Material Diffusion

Monocular Avatar Reconstruction via Cascaded Diffusion Priors and UV-Space Differentiable Shading