High-quality material generation for physically based rendering (PBR) remains challenging because datasets and synthesis pipelines lack standardized representations. Differences in material definitions, rendering conventions, and parameterizations lead to unstable and physically implausible results. We propose MIKU, a unified framework for standardized PBR asset representation and physically consistent material generation across diverse 3D content domains. Its core is a unified processing pipeline that standardizes the workflow from data preprocessing to final texture synthesis, maintaining consistent geometry and texture representations. A normalization scheme harmonizes heterogeneous material parameterizations in a common representation space, enabling robust cross-dataset training and inference. Building on this formulation, we train a novel physically aware diffusion model for PBR texture generation within a two-stage multi-view synthesis framework. First, geometry-aware albedo generation synthesizes multi-view albedo maps under explicit geometric guidance to ensure cross-view appearance consistency. Second, metallic-roughness generation uses the diffusion model to predict physically plausible roughness and metallic maps conditioned on the generated albedo and unified representation. Trained end-to-end on standardized data, the model learns to generate materials that respect category-level material priors and semantic part boundaries. By decoupling representation standardization from appearance synthesis, MIKU substantially improves material generation stability and physical consistency. Extensive experiments demonstrate state-of-the-art texture quality and semantic consistency across object parts, advancing automated material generation for 3D content creation.
Figures & tables
Figure 1. The entire scene is assembled from AIGC-generated meshes, where both the surface textures and PBR material maps are consistently synthesized by our method. A scene assembled from generated 3D meshes, showing consistent textures and physically based material maps under varied illumination.
Figure 2. Illustration of the discrepancy attenuation in image-supervised PBR learning. Even with fixed geometry, albedo, camera view, and illumination, substantially different MR conditions may result in only minor shading differences after rendering.
Figure 3. Visualization of our color-preserving illumination strategy. Compared with chromatic illumination, which introduces color casts and appearance ambiguity, achromatic illumination preserves intrinsic albedo colors while providing shading cues. A pipeline illustrating deterministic diffusion-based multi-view PBR material estimation with packed inputs, conditional guidance, and LoRA fine-tuning.
Figure 4. Overview of Our Multi-Stage Pipeline for High-Quality PBR Material Dataset Construction and Refinement. This figure illustrates our end-to-end pipeline for constructing a high-quality PBR material dataset. We first collect multi-source data from the public TexVerse dataset and private asset collections. We then define physically based material properties and render multi-view observations under controlled settings. Next, we apply albedo extraction and PBR-aware filtering to remove low-quality samples. Vision-Language Models (VLMs) with SAM3 are used for material recognition and semantic segmentation to enable fine-grained decomposition. Finally, we enhance texture details via affine transformations and high-frequency boosting, followed by human validation to ensure data quality. A pipeline illustrating deterministic diffusion-based multi-view PBR material estimation with packed inputs, conditional guidance, and LoRA fine-tuning.
Material Class
Metallic
Roughness
Skin
0.0
0.40–0.50
Eye
0.0
0.02–0.12
Hair
0.0
0.35–0.45
Metal
1.0
0.00–0.10
Teeth
0.0
0.28–0.34
Lips
0.0
0.30–0.36
Table 1. Metallic–roughness (MR) definitions for the 20 predefined semantic material categories and the four optional fine-grained metal subclasses under our unified PBR specification. Each category is associated with a representative MR range that serves as a consistent material prior during data standardization.
Figure 5. Overview of our conditional diffusion framework for multi-view albedo generation. Given a noisy input, a reference image, and geometric guidance, the model progressively denoises the input to synthesize consistent multi-view albedo maps. A pipeline illustrating deterministic diffusion-based multi-view PBR material estimation with packed inputs, conditional guidance, and LoRA fine-tuning.
Figure 6. Overview of our deterministic diffusion framework for multi-view PBR material estimation. Given a packed albedo input xrgb and a shaded reference image xref , we formulate a conditional diffusion process to predict the target material representation xmr . A pipeline illustrating deterministic diffusion-based multi-view PBR material estimation with packed inputs, conditional guidance, and LoRA fine-tuning.
Figure 7. Qualitative comparison of texture and PBR material generation across different methods. Each example is arranged in two rows: the first row shows the rendered results under a fixed illumination setting, while the second row presents the predicted material maps, including the MR map, i.e., metallic and roughness channels, and the generated albedo texture. Compared with existing methods, our approach produces sharper and more detailed albedo textures, while also yielding cleaner, more spatially coherent, and semantically consistent MR predictions across diverse samples.
Figure 8. Visualization results of PBR materials and texture maps generated for two given meshes under image guidance. The rendered results under diverse lighting environments demonstrate that the model effectively adapts to varying illumination conditions. In particular, metallic regions exhibit physically plausible reflections, accurately capturing environmental lighting and producing realistic appearance. A pipeline illustrating deterministic diffusion-based multi-view PBR material estimation with packed inputs, conditional guidance, and LoRA fine-tuning.
Figure 9. Qualitative comparison of texture generation using geometry-based classifier-free guidance (CFG) versus alternative CFG strategies. The results show that geometry-guided CFG produces sharper textures that more closely align with the conditioning image, leading to improved structural fidelity and overall visual consistency. A pipeline illustrating deterministic diffusion-based multi-view PBR material estimation with packed inputs, conditional guidance, and LoRA fine-tuning.
Figure 10. User-study results comparing perceptual quality and reference-image consistency across different methods for multi-view image generation. The bar chart reports the average scores for each method, with higher values indicating better perceived quality and consistency with the reference image. A pipeline illustrating deterministic diffusion-based multi-view PBR material estimation with packed inputs, conditional guidance, and LoRA fine-tuning.
Figure 11. Visual comparison between models trained on original artist-defined PBR data and our unified normalized PBR data. The first column shows the reference image, the second column presents results from the model trained on the original artist-defined PBR data, and the third column shows results from the model trained on our unified normalized PBR data. A pipeline illustrating deterministic diffusion-based multi-view PBR material estimation with packed inputs, conditional guidance, and LoRA fine-tuning.
Method
FID ↓
CMMD ↓
LPIPS ↓
CLIP-I ↑
CLIP-FID ↓
Inference Time (s) ↓
Albedo Resolution ↑
MR Resolution ↑
MvAdapter
165.49
2.2235
0.2383
0.9097
22.911
36
768
None
UniTex
161.27
2.1898
0.2174
0.9052
22.623
220
512
None
Hunyuan3d-2.1
154.88
2.2625
0.2167
0.9132
20.132
48
768
768
LumiTex
144.61
1.8677
0.1992
0.9215
16.182
110+110
512
512
Ours
121.25
1.4652
0.1612
0.9411
13.358
40+12
1024
512
Table 2. Quantitative Comparison with State-of-the-Art Methods on PBR Texture Generation Quality and Efficiency.
Method
Setting
FID ↓
CMMD ↓
LPIPS ↓
CLIP-I ↑
CLIP-FID ↓
Ours
w/o CFG
128.55
1.5439
0.1688
0.9342
15.172
w/ CFG
121.25
1.4652
0.1612
0.9411
13.358
Table 3. Ablation study on the effect of classifier-free guidance (CFG) in our framework. Enabling CFG significantly improves generation quality and semantic consistency across all evaluation metrics.
Figure 12. Visualization results of arbitrary image-guided texture generation and mesh stylization. The leftmost images show the reference guidance images, while the rendered results on the right are obtained by generating textures conditioned on each reference image and applying them back onto the same underlying mesh. Despite the substantial appearance differences among the guidance images, our method produces coherent and harmonious stylized textures that are well integrated with the shared geometry, demonstrating strong image-guided stylization capability.
Figure 13. User study on perceptual PBR material quality. Left: Pairwise preferences between our model trained on the curated, standardized annotations and that trained on the original annotations, where win, tie, and loss are defined from the perspective of the former. Right: Mean five-point ratings of Hunyuan3D-V2.1 under the pretrained setting and after fine-tuning on either the raw collected data or our curated data. A pipeline illustrating deterministic diffusion-based multi-view PBR material estimation with packed inputs, conditional guidance, and LoRA fine-tuning.
Figure 14. Visualization of text and logo fidelity under reference guidance. For each example, the leftmost image denotes the reference input containing distinctive textual or logo-like patterns. The two rendered results on the right show the textured mesh after back-projecting the generated maps, where the left rendering presents the frontal view and the right rendering presents the back view. The results demonstrate that our method can preserve fine-grained text and logo structures from the reference image while maintaining coherent appearance across different viewpoints.
Figure 15. Additional qualitative results on diverse inputs. Our method demonstrates strong generalization across a wide range of mesh geometries, styles, and input variations, consistently producing high-quality textures along with semantically accurate and physically plausible PBR materials.
Training Data
PBR Consistency MSE ↓
Artist-defined Materials
0.0513
Standardized Materials
0.0258
Table 4. Ablation study on the effect of PBR material standardization. The standardized material specification improves prediction consistency under semantic-preserving appearance variations.
We present a method for generating physically-based materials for 3D shapes based on a video diffusion transformer architecture. Our method is conditioned on input geometry and a text description, and jointly models multiple material properties (base color, roughness, metallicity, height map) to form physically plausible materials. We further introduce a custom variational auto-encoder which encodes multiple material modalities into a compact latent space, which enables joint generation of multiple modalities without increasing the number of tokens. Our pipeline generates high-quality materials for 3D shapes given a text prompt, compatible with common content creation tools.
We propose a generative framework for producing high-quality PBR textures on a given 3D mesh. As large-scale PBR texture datasets are scarce, our approach focuses on effectively leveraging the embedding space and diffusion priors of pretrained latent image generative models while learning a material latent space, MatLat, through targeted fine-tuning. Unlike prior methods that freeze the embedding network, which leads to distribution shifts when encoding additional PBR channels and hinders subsequent diffusion training, we fine-tune the pretrained VAE so that new material channels can be incorporated with minimal latent distribution deviation. We further show that correspondence-aware attention alone is insufficient for cross-view consistency unless the latent-to-image mapping preserves locality. To enforce this locality, we introduce a regularization in the VAE fine-tuning that crops latent patches, decodes them, and aligns the corresponding image regions to maintain strong pixel-latent spatial correspondence. Ablation studies and comparison with previous baselines demonstrate that our framework improves PBR texture fidelity and that each component is critical for achieving state-of-the-art performance.
This paper aims to generate materials for 3D meshes from text descriptions. Unlike existing methods that synthesize texture maps, we propose to generate segment-wise procedural material graphs as the appearance representation, which supports high-quality rendering and provides substantial flexibility in editing. Instead of relying on extensive paired data, i.e., 3D meshes with material graphs and corresponding text descriptions, to train a material graph generative model, we propose to leverage the pre-trained 2D diffusion model as a bridge to connect the text and material graphs. Specifically, our approach decomposes a shape into a set of segments and designs a segment-controlled diffusion model to synthesize 2D images that are aligned with mesh parts. Based on generated images, we initialize parameters of material graphs and fine-tune them through the differentiable rendering module to produce materials in accordance with the textual description. Extensive experiments demonstrate the superior performance of our framework in photorealism, resolution, and editability over existing methods. Project page: https://zju3dv.github.io/MaPa
Shangzhan Zhang, Sida Peng, Tao Xu +7
Zhejiang University · Ant Group · Shenzhen University