We propose a generative framework for producing high-quality PBR textures on a given 3D mesh. As large-scale PBR texture datasets are scarce, our approach focuses on effectively leveraging the embedding space and diffusion priors of pretrained latent image generative models while learning a material latent space, MatLat, through targeted fine-tuning. Unlike prior methods that freeze the embedding network, which leads to distribution shifts when encoding additional PBR channels and hinders subsequent diffusion training, we fine-tune the pretrained VAE so that new material channels can be incorporated with minimal latent distribution deviation. We further show that correspondence-aware attention alone is insufficient for cross-view consistency unless the latent-to-image mapping preserves locality. To enforce this locality, we introduce a regularization in the VAE fine-tuning that crops latent patches, decodes them, and aligns the corresponding image regions to maintain strong pixel-latent spatial correspondence. Ablation studies and comparison with previous baselines demonstrate that our framework improves PBR texture fidelity and that each component is critical for achieving state-of-the-art performance.
Figures & tables
Figure 2: Overview of Frozen VAE [ 18 ] . Zero-padded roughness–metallic maps and albedo image are encoded by the frozen VAE to produce (μrm,σrm) and (μa,σa) , respectively.
Figure 3: Comparison of PBR Material Encoder Schemes. (a) LayerDiffuse [ 58 ] uses residual prediction but only predicts the latent mean and enforces identity consistency via Lid . (b) Orchid [ 29 ] uses direct prediction to output the full latent parameters (μfull,σfull) and regularizes them with Lreg . (c) MatVAE (ours) synergistically integrates residual prediction and KL regularization , enabling effective incorporation of PBR material images while preserving the pretrained latent space.
Figure 4: Illustration of Correspondence-Aware Attention (CAA) and Locality Regularization. (a) CAA restricts attention to geometrically corresponding tokens across views. (b) Locality regularizer enforces patch-wise reconstruction such that image pixels are decoded from spatially aligned latent tokens. Together, CAA and locality regularization enable multi-view consistent PBR texture generation.
Id
Methods
Shaded
Albedo
Rough.
Metal.
c-PSNR ↑
Time ↓
FIDCLIP↓
KID↓
CLIP↑
FIDCLIP↓
KID↓
CLIP↑
RMSE↓
RMSE↓
1
Frozen VAE [ 18 ]
3.419
1.052
0.318
4.926
1.560
0.315
0.193
0.251
19.869
111s
2
Res. Pred. + Lreg (Ours)
3.083
1.327
0.318
4.599
1.574
0.314
0.158
0.134
21.934
34s
3
└ Res. Pred. + Lid [ 58 ]
3.210
1.090
0.317
4.871
1.525
0.312
0.165
0.128
20.977
34s
4
└ Direct Pred. + Lreg [ 29 ]
3.192
1.146
0.319
4.768
1.472
0.312
0.161
0.170
21.468
34s
5
└ w/o Llocal
3.419
1.611
0.316
5.873
4.122
0.312
0.154
0.182
19.437
34s
Table 1: Quantitative Results of Ablation Study. Best scores are bold , and the runner-up scores are underlined .
Figure 5: Qualitative Comparisons for Each Component. Combining residual prediction with Lreg effectively preserves the pretrained latent distribution and yields high-quality PBR textures.
Method
Type
Shaded
Albedo
Rough.
Metal.
Time ↓
FIDCLIP↓
KID↓
CLIP↑
FIDCLIP↓
KID↓
CLIP↑
RMSE↓
RMSE↓
MeshGen [ 7 ]
SC
8.637
11.105
0.282
11.322
16.193
0.282
0.143
0.201
195s
TexGaussian [ 52 ]
SC
6.025
3.571
0.301
12.119
9.381
0.299
0.145
0.243
73s
Paint-it [ 55 ]
SDS
8.547
5.382
0.309
13.063
10.665
0.299
0.168
0.200
1260s
DreamMat [ 61 ]
SDS
5.422
2.668
0.311
9.621
6.002
0.311
0.167
0.165
2400s
FlashTex [ 10 ]
SDS
7.119
5.354
0.305
12.320
10.441
0.298
0.143
0.186
285s
Table 2: Quantitative Comparisons of PBR Texture Generation Methods. Best scores are bold , and the runner-up scores are underlined .
Figure 6: Qualitative Results of MatLat . MatLat produces realistic textures with strong text-alignment across diverse material types.
Appendix figures & tables7 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 7: Direct vs. Residual Prediction. We show the albedo, roughness, and metallic reconstruction errors at an early training stage (10k iterations) for direct and residual prediction. Residual prediction preserves the pretrained latent representation, achieving superior albedo reconstruction quality (darker regions indicate lower errors).
Methods
Shaded
Albedo
KIDCLIP↓
KIDCLIP↓
Frozen VAE [ 18 ]
2.123
3.547
Res. Pred. + Lreg (Ours)
1.496
2.855
Res. Pred. + Lid [ 58 ]
1.977
3.710
Direct Pred. + Lreg [ 29 ]
1.917
3.254
w/o Llocal
2.660
8.820
Appendix
Table 3: Evaluation on CLIP-based KID. We additionally measure KID in the CLIP feature space. Our proposed VAE fine-tuning scheme yields superior performance, with both our full model and the variant without CAA consistently outperforming the other baselines.
Figure 8: Visualization of Reconstruction Error Maps on Cropped Patches. We show albedo, roughness, and metallic reconstruction error maps for MatVAE trained with and without Llocal . Applying Llocal significantly reduces patch reconstruction errors, indicating improved latent–image spatial alignment.
Crop range
FIDshadedCLIP
FIDalbedoCLIP
c-PSNR
[3,32]
3.083
4.599
21.934
[24,48]
3.208
5.027
20.571
[32,64]
3.339
4.719
20.980
Appendix
Table 4: Crop-range ablation for locality regularization. We vary the crop-size range used in Llocal on a 64×64 latent grid. Smaller crop ranges yield the best overall performance, while all cropped settings outperform training without cropping.
SWD(z,za)
SWD(z,zrm)
SWD(z,zours)
0.539
0.897
0.560
Appendix
Table 5: Sliced Wasserstein distances to the pretrained diffusion prior. Lower values indicate that the encoded latents are better aligned with the pretrained latent space.
Figure 9: Relighting results. Example objects generated by MatLat and rendered under four different environment maps.
Figure 10 : Additional Qualitative Results of Ablation Studies. Extended qualitative results of the ablation studies presented in Fig. 5 : Frozen VAE, Res. Pred. + Lid , Direct Pred. + Lreg , and Res. Pred. + Lreg (Ours). Best viewed in video.
Figure 11 : Additional Qualitative Comparisons with Previous Methods. Each column shows the shaded output rendered under identical lighting conditions. Our method produces high-quality, physically plausible materials with superior text-visual alignment and detail preservation. Best viewed in video.
Figure 12 : Additional Qualitative Results of MatLat . Gallery of assets textured by MatLat for various text prompts and environment maps. Best viewed in video.