Generating physically based rendering (PBR) texture maps for a 3D mesh from a single reference image is highly ambiguous, as the observed appearance entangles material identity, surface reflectance, and illumination. While current generative methods produce realistic albedo maps, they still struggle to infer accurate metallic and roughness attributes for relighting. We present UltraTex, a cascaded multi-view framework that moves beyond appearance matching by integrating physical knowledge into multi-view diffusion-based PBR generation. UltraTex first jointly predicts albedo and world-space shading normals to generate more detailed albedo maps, and then predicts metallic-roughness (MR) maps conditioned on the generated albedo, reference image, and material semantic priors. To inject material semantics, a lightweight adapter transforms hidden-layer features of a vision-language model (VLM) into intrinsic material representation tokens, providing part-level knowledge anchors for MR estimation. To model material-light interaction, photometric representation alignment guides intermediate diffusion features toward ground-truth material renderings under controlled lighting. This makes metallic and roughness variations observable through highlight intensity, specular sharpness, and local reflectance contrast, providing stronger guidance for fine-grained material prediction without explicit lighting supervision. Extensive experiments show that UltraTex generates high-quality textures and outperforms existing open-source and commercial methods in visual fidelity, material accuracy, and physical plausibility.
Figures & tables
Figure 1. PBR texture gallery generated by UltraTex. Our method produces detailed and physically plausible PBR materials across diverse assets under environmental lighting. In the first row, the right column shows the generated albedo, metallic, and roughness maps from top to bottom. In the second row, a close-up render (right) of the boxed region (left) reveals fine-grained material details. Project page: https://ultratex.vercel.app/ .
Figure 2. Overall pipeline of UltraTex. UltraTex adopts a cascaded two-stage multi-view diffusion pipeline. Stage 1 jointly predicts albedo maps and world-space shading normals from the reference image and multi-view geometry normals. Stage 2 predicts metallic-roughness (MR) maps (where purple and green represent roughness and metallic, respectively) conditioned on the generated albedo, reference-image features, and VLM-derived intrinsic material tokens. For MR training, photometric representation alignment uses a frozen visual encoder to extract representations from controlled-light material renderings, and an MLP aligns intermediate diffusion features to this photometric representation space via LREPA .
Figure 3. VLM-guided intrinsic material adapter. A frozen VLM extracts hidden-layer visual features from rendered reference images, and a trainable adapter projects them into 1D intrinsic material tokens. With the image diffusion model frozen, the adapter is trained to reconstruct ground-truth metallic and roughness maps by injecting these tokens into the denoising blocks. The learned tokens encode part-aware material semantic priors and are used to guide MR generation.
Figure 4. Effectiveness of joint albedo and shading-normal estimation. (a) Tangent-space normal or bump maps depend on local UV parameterization; continuous 3D surfaces may be unwrapped into disjoint UV patches with different orientations, leading to inconsistent tangent frames. (b) By jointly estimating world-space shading normals with albedo, our model avoids such UV-dependent artifacts and produces richer surface details. We compare against a model trained without joint shading-normal estimation, where the normal visualization corresponds to the input geometry normals.
Method
FID ↓
CLIP-FID ↓
CMMD ↓
CLIP-I ↑
LPIPS ↓
MSE M↓
MSE R↓
Paint-it
270.7
31.65
2.694
0.8809
0.2514
57.64
41.43
DreamMat
272.3
30.90
2.592
0.8868
0.2464
107.4
39.84
Hunyuan3D-2.1
189.2
18.33
1.758
0.9302
0.1585
50.04
32.73
LumiTex
159.5
14.98
1.431
0.9439
0.1286
54.32
25.96
TRELLIS.2
158.7
16.29
1.351
0.9382
0.1295
50.54
28.23
Ours (w/o joint)
145.3
14.32
1.188
0.9454
0.1198
53.03
16.49
Table 1. Quantitative comparison with open-source PBR texture generation methods . All textures are evaluated at resolution 10242 for a fair comparison. “M” and “R” denote “metallic” and “roughness”, respectively.
Method
Average ↑
Rendering ↑
Albedo ↑
Metallic ↑
Rough. ↑
Hyper3D v2.0 ( Hyper3D, 2026 )
2.19
1.97
1.93
2.39
2.47
Hunyuan3D v3.1 ( Team, 2025b )
2.85
3.01
3.01
2.85
2.55
TripoAI v3.1 (4K) ( TripoAI, 2026 )
2.95
3.17
3.03
3.04
2.53
Meshy 6 (HD) ( Meshy, 2026 )
3.13
3.36
3.41
2.96
2.83
Seed3D 2.0 ( Seed3D, 2026 )
3.20
2.96
3.29
3.36
3.17
Ours
4.31
4.55
4.33
4.21
4.11
Table 2. User study results. Comparison of rendering, albedo quality, and PBR material accuracy against the latest commercial methods.
Figure 5. The result of the user study . We provide the pairwise comparison result with the latest commercial methods.
Figure 8
Figure 8. Comparison of local details. Our model excels in generating ultra-fine albedo details with high fidelity, even for challenging surfaces such as oxidized metals and those characterized by intricate micro-textures.
Figure 9. Relighting Results. Our generated PBR textures exhibit physically plausible and consistent material responses under novel illumination conditions, faithfully reflecting changes in highlights and reflections.
Figure 10. Results on real-world captures. Our method produces accurate and physically plausible PBR textures from casually captured images and scanned meshes, demonstrating robustness to real-world inputs.
Figure 11. Qualitative comparison with ground-truth PBR maps . Given the same clay mesh and reference image, our method more faithfully recovers the ground-truth material decomposition than the state-of-the-art multi-view and 3D-native baselines, with finer base-color details, cleaner metallic separation, and more accurate spatially varying roughness. Each object has: the albedo on the left, the metallic on the top right, and the roughness on the bottom right.
Method
FID ↓
CLIP-FID ↓
CMMD ↓
CLIP-I ↑
LPIPS ↓
TRELLIS.2
158.4
22.89
1.8896
0.9049
0.1213
LumiTex
144.4
17.87
1.6943
0.9249
0.1110
Ours
111.2
15.08
1.3575
0.9370
0.0988
Table 3. Quantitative comparison of environment relighting results . We evaluate 152 objects across four views under sampled environment maps and compare the relighting results against ground-truth renderings.
Figure 12. Impact of semantic and photometric alignment . The first row visualizes our VLM adapter capturing intrinsic material semantics. The following rows compare metallic and roughness maps of ablated variants and our full model. Without VLM-derived tokens, the network incorrectly predicts non-metallic surfaces (e.g., painted fish, plastic TV casing) as metallic. Removing photometric alignment yields weaker metallic contrast and fails to recover structural details like gears, rivets, and screws.
Figure 13. VLM-token attention visualization . We visualize the attention map from VLM material tokens to noisy image tokens. The attention maps exhibit clear spatial localization over material-relevant object parts, such as metallic ornaments and hardware, indicating that the 1D material tokens capture part-aware material correspondence.
Method
MSE M ↓
MSE R ↓
Ours (w/o visual conditions)
55.46
28.23
Ours (w/o VLM)
54.85
22.50
Ours (w/o photometric alignment)
52.32
18.45
Ours
46.77
16.57
Table 4. Quantitative ablation study. We evaluate the contribution of each core component by removing visual conditions, VLM guidance, or photometric alignment. MSE values are reported in units of 10−3 .
Encoder
MSEM↓
MSER↓
LPIPS (ORM) ↓
Image VAE
58.04
34.01
0.4085
DINOv2
43.76
32.09
0.3592
VLM (Qwen3.5)
27.93
15.79
0.2065
Table 5. Quantitative comparison of different encoder designs . For each encoder, we train only the adapter while keeping the image diffusion model frozen, and evaluate the resulting MR map generation. MSE values are reported in units of 10−3 . “ORM” denotes the channel combination of “one”, “roughness” and “metallic”.
PSNR ↑
SSIM ↑
LPIPS ↓
VAE Recon.
38.656
0.9839
0.0051
Table 6. VAE reconstruction quality. We evaluate reconstruction fidelity on 366 MR view-maps using the RGB-pretrained VAE.
Figure 14. Multi-view input pipeline . Our framework naturally extends to multi-view references by independently encoding each input view and pairing its image features with the corresponding geometry tokens in the Multi-Modal DiT, while leaving the remaining generation pipeline unchanged.
Figure 15. Normal map transformation . Our predicted shading normals are converted to a tangent-space normal map, adding fine-scale surface details to the original clay mesh without modifying its geometry.
Figure 16. Failure Cases . Highly stylized assets may use MR for artistic rather than physical effects, leading to ambiguous decomposition, as seen in the joker’s hair and the character’s garments. In addition, effects beyond the opaque metallic–roughness model, such as the vehicle’s transparent windshield, cannot be faithfully represented.
Figure 17. More Qualitative Results. We present additional PBR comparisons with other methods (from left to right: albedo, metallic, roughness). Zooming in for details.
Figure 18. More results. We provide additional multi-view renderings with their corresponding generated PBR maps. Zooming in is recommended to observe the ultra-fine albedo and material details.
High-quality material generation for physically based rendering (PBR) remains challenging because datasets and synthesis pipelines lack standardized representations. Differences in material definitions, rendering conventions, and parameterizations lead to unstable and physically implausible results. We propose MIKU, a unified framework for standardized PBR asset representation and physically consistent material generation across diverse 3D content domains. Its core is a unified processing pipeline that standardizes the workflow from data preprocessing to final texture synthesis, maintaining consistent geometry and texture representations. A normalization scheme harmonizes heterogeneous material parameterizations in a common representation space, enabling robust cross-dataset training and inference. Building on this formulation, we train a novel physically aware diffusion model for PBR texture generation within a two-stage multi-view synthesis framework. First, geometry-aware albedo generation synthesizes multi-view albedo maps under explicit geometric guidance to ensure cross-view appearance consistency. Second, metallic-roughness generation uses the diffusion model to predict physically plausible roughness and metallic maps conditioned on the generated albedo and unified representation. Trained end-to-end on standardized data, the model learns to generate materials that respect category-level material priors and semantic part boundaries. By decoupling representation standardization from appearance synthesis, MIKU substantially improves material generation stability and physical consistency. Extensive experiments demonstrate state-of-the-art texture quality and semantic consistency across object parts, advancing automated material generation for 3D content creation.
Yajie Bao, Jiaye Fu, Youtian Lin +5
Nanjing University, China · DreamTech, China · Peking University, China +1
We propose a generative framework for producing high-quality PBR textures on a given 3D mesh. As large-scale PBR texture datasets are scarce, our approach focuses on effectively leveraging the embedding space and diffusion priors of pretrained latent image generative models while learning a material latent space, MatLat, through targeted fine-tuning. Unlike prior methods that freeze the embedding network, which leads to distribution shifts when encoding additional PBR channels and hinders subsequent diffusion training, we fine-tune the pretrained VAE so that new material channels can be incorporated with minimal latent distribution deviation. We further show that correspondence-aware attention alone is insufficient for cross-view consistency unless the latent-to-image mapping preserves locality. To enforce this locality, we introduce a regularization in the VAE fine-tuning that crops latent patches, decodes them, and aligns the corresponding image regions to maintain strong pixel-latent spatial correspondence. Ablation studies and comparison with previous baselines demonstrate that our framework improves PBR texture fidelity and that each component is critical for achieving state-of-the-art performance.
Single image 3D generation aims to dress a given 3D mesh with relightable materials while faithfully preserving the visual details of a reference image. Existing methods either introduce artifacts when fusing inconsistent multi-view predictions or struggle to preserve fine details through implicit image-feature conditioning. To resolve these challenges, we introduce SUPRA-Tex, built on the principle of decoupling visible-material estimation, surface alignment, and observation-guided 3D completion. This division of labor allows the 3D generator, trained on limited 3D data, to focus its capacity on preserving visible evidence and completing unobserved surfaces. Specifically, we first train an MLLM-based image editing model to predict lighting-separated PBR maps in the reference view. We then project these maps onto the visible surface and encode them with the pretrained material VAE encoder into native observation latents that share the generation target's latent space. An interleaved flow transformer holds these latents as fixed, read-only anchors while alternating unobserved-region and full-surface updates to generate complete material latents. This design achieves super-alignment with the reference, delivering What You See Is What You Get fidelity on visible regions and coherent completion on unobserved surfaces. Trained solely on public datasets, SUPRA-Tex achieves SOTA performance in both geometry-conditioned texturing and end-to-end asset generation, matching or even exceeding leading commercial systems on in-the-wild images. Project page: https://supra-3d.github.io/supratex/
Chen Yang, Jiemin Fang, Fanpeng Meng +5
Huawei Inc. · Shanghai Jiao Tong University · Huazhong University of Science and Technology