We present a method for generating physically-based materials for 3D shapes based on a video diffusion transformer architecture. Our method is conditioned on input geometry and a text description, and jointly models multiple material properties (base color, roughness, metallicity, height map) to form physically plausible materials. We further introduce a custom variational auto-encoder which encodes multiple material modalities into a compact latent space, which enables joint generation of multiple modalities without increasing the number of tokens. Our pipeline generates high-quality materials for 3D shapes given a text prompt, compatible with common content creation tools.
Figures & tables
Figure 2: Our method starts from a known 3D model and a text prompt. We first render videos of normal maps and world space positions. Next, these conditions are encoded into latent space, using a pretrained encoder, E , to produce latent conditions, zI . These are concatenated with noisy latents, zτmat , representing material modalities, along the channel dimension. The latents and text prompt are then passed to our finetuned video model, which generates a denoised latent, z^mat . The denoised latent is decoded into videos of the intrinsic material channels: base color, roughness, metallicity, and height, using a custom VAE decoder Dpbr which decodes all material properties jointly. Finally, we project the generated views into texture space to extract high quality, standard PBR materials.
Figure 3: Material generation. We compare against Hunyuan3D Paint 2.1 [ 18 ] (image and text guided versions) and VideoMat [ 37 ] (text) on three example meshes from the BlenderVault [ 33 ] dataset. We encourage the reader to zoom in and compare the quality of the intrinsics (base color, roughness, metallicity), as well as to see the supplementary materials.
Method
Mode
CLIP-FID ( ↓ )
CMMD ( ↓ )
LPIPS ( ↓ )
TRELLIS.2 [ 58 ]
image
3.913
0.030
0.101
Hunyuan3D 2.1 [ 18 ]
image
4.197
0.039
0.102
MVPainter [ 47 ]
image
6.583
0.112
0.136
VideoMatGen (ours)
image
4.032
0.028
0.109
Hunyuan3D 2.1 [ 18 ]
text
6.694
0.046
0.137
MVPainter [ 47 ]
text
7.313
0.096
0.149
Table 1: Quantitative metrics for material generation. The mode column indicates if the method is image or text guided.
Figure 4: Left : Our method predicts a height (bump) map, which improves the visual richness of the generated material. Right : corresponding rendering without bump map.
Figure 5: We generate three materials from the same text prompt (see supplemental), each with a unique random seed. This results in subtle variations of materials for the two examples.
Figure 6: We show relit results, using three HDR probes [ 67 ] , of the generated materials for Hunyuan3D-Paint (image-guided), VideoMat, and our method (both text-guided). Our generated materials produce convincing details in three different lighting scenarios.
Figure 7: We compare our text- and image-guided models for six examples, and note that the image-guided version more closely resembles the materials of the dataset entry. We deliberately chose a view with 45∘ rotation from the conditioning view.
Base color
HRM
Method
PSNR ( ↑ )
LPIPS ( ↓ )
PSNR ( ↑ )
Cosmos Tokenizer
38.8
0.046
35.1
VAEpbr
38.3
0.043
33.8
Table 2: VAE finetuning evaluation on material maps from our test set and the MatSynth test set. We report PSNR (dB) and LPIPS scores for base color and only PSNR (dB) scores for HRM (height, roughness, metallicity), as perceptual metrics are not applicable. The VAEpbr offers 2 × additional compression but has similar visual quality as the Cosmos Tokenizer.
Figure 8: Our text-to-PBR material generations (renderings in Blender) on a collection of TRELLIS.2-generated base geometry.
Figure 9: We compare two versions of joint prediction. Our version uses the latent space of VAEpbr and predicts a video with 16 frames. Frame concatenation doubles the number of tokens by treating the material prediction as diffusing a video with twice the number of frames [16 × base color, 16 × HRM]. The resulting quality is similar, arguably with more coherent material predictions in our approach. Please zoom to see the details.
Method
CLIP-FID ( ↓ )
CMMD ( ↓ )
LPIPS ( ↓ )
VideoMatGen ( VAEpbr )
5.638
0.035
0.126
VideoMatGen (FCat)
5.712
0.039
0.129
Table 3: Quantitative metrics for text-to-material generation. We compare two variants of joint prediction, frame concatenation (FCat) which doubles the number of video frames/latent tokens, and our version with VAEpbr .
This paper aims to generate materials for 3D meshes from text descriptions. Unlike existing methods that synthesize texture maps, we propose to generate segment-wise procedural material graphs as the appearance representation, which supports high-quality rendering and provides substantial flexibility in editing. Instead of relying on extensive paired data, i.e., 3D meshes with material graphs and corresponding text descriptions, to train a material graph generative model, we propose to leverage the pre-trained 2D diffusion model as a bridge to connect the text and material graphs. Specifically, our approach decomposes a shape into a set of segments and designs a segment-controlled diffusion model to synthesize 2D images that are aligned with mesh parts. Based on generated images, we initialize parameters of material graphs and fine-tune them through the differentiable rendering module to produce materials in accordance with the textual description. Extensive experiments demonstrate the superior performance of our framework in photorealism, resolution, and editability over existing methods. Project page: https://zju3dv.github.io/MaPa
Shangzhan Zhang, Sida Peng, Tao Xu +7
Zhejiang University · Ant Group · Shenzhen University
We present a method for generating high quality materials for 3D objects entirely in texture space. We finetune a video diffusion transformer for text-guided material generation, multi-view material generation, and material upscaling. Our key insight is to use the known projection from image space to texture space, enabling the diffusion process to generalize across arbitrary geometries and texture parameterizations. This approach also avoids the view consistency issues inherent in video and multi-view diffusion models. Because texture space is two dimensional, we can reuse the strong priors of pretrained video diffusion models. We apply our method to high quality material reconstruction from posed photos captured under unknown lighting, as well as to text- and image guided material generation. Our method can scale to high resolutions (8K), 100+ input views, and neural material representations. In quantitative and qualitative evaluations we show state-of-the-art results for material generation and reconstruction.
Material generation should produce not only an appearance, but also the rules that construct it. We introduce MatLoom, a compact, layer-oriented language for text-to-material generation with pretrained language models. Each program composes alpha-masked layers whose shared spatial expressions define coverage and physically based rendering (PBR) channels, making dependencies between patterns, color, and relief explicit. A standalone interpreter evaluates the program into material maps, while the source retains named fields and layer parameters for subsequent authoring. Without task-specific fine-tuning, our pipeline uses parser-guided repair and preview-based critique to revise material designs, then searches noise seeds while keeping each candidate's remaining source fixed. On a curated benchmark of 141 prompts evaluated with six backbones, our best-performing configuration achieves higher mean scores than three diffusion baselines on all four flat-layout prompt-alignment metrics. Its initial programs already exceed all three baselines on mean BLIPScore, before critique or seed search. Retained programs have a median length of 21 lines when pooled across backbones. In a blind four-way comparison involving 30 participants and 20 prompts, our renders receive 59.2% of choices, compared with 19.3% for the most-preferred baseline. Compact executable programs thus offer a way to generate prompt-aligned materials while retaining their construction as part of the asset.
Anson Y. Lam, Shuqing Li, Michael R. Lyu
Department of Computer Science and Engineering The Chinese University of Hong Kong Hong Kong, China