VideoMatGen: PBR Materials through Joint Generative Modeling

Jon Hasselgren; Zheng Zeng; Milos Hasan; Jacob Munkberg

cs.CVMar 17, 2026

VideoMatGen: PBR Materials through Joint Generative Modeling

Authors: Jon Hasselgren, Zheng Zeng, Milos Hasan, Jacob Munkberg

Organizations: NVIDIA

Abstract

We present a method for generating physically-based materials for 3D shapes based on a video diffusion transformer architecture. Our method is conditioned on input geometry and a text description, and jointly models multiple material properties (base color, roughness, metallicity, height map) to form physically plausible materials. We further introduce a custom variational auto-encoder which encodes multiple material modalities into a compact latent space, which enables joint generation of multiple modalities without increasing the number of tokens. Our pipeline generates high-quality materials for 3D shapes given a text prompt, compatible with common content creation tools.

Figures & tables

Figure 2: Our method starts from a known 3D model and a text prompt. We first render videos of normal maps and world space positions. Next, these conditions are encoded into latent space, using a pretrained encoder, $\mathcal{E}$ , to produce latent conditions, $\textbf{z}^{\mathbf{I}}$ . These are concatenated with noisy latents, $\textbf{z}_{\tau}^{\mathbf{mat}}$ , representing material modalities, along the channel dimension. The latents and text prompt are then passed to our finetuned video model, which generates a denoised latent, $\hat{\textbf{z}}^{\mathbf{mat}}$ . The denoised latent is decoded into videos of the intrinsic material channels: base color, roughness, metallicity, and height, using a custom VAE decoder $\mathcal{D}_{\mathrm{pbr}}$ which decodes all material properties jointly. Finally, we project the generated views into texture space to extract high quality, standard PBR materials.

Panel of Figure 3 — Figure 3: Material generation. We compare against Hunyuan3D Paint 2.1 [ 18 ] (image and text guided versions) and VideoMat [ 37 ] (text) on three example meshes from the BlenderVault [ 33 ] dataset. We encourage the reader to zoom in and compare the quality of the intrinsics (base color, roughness, metallicity), as well as to see the supplementary materials.

Method	Mode	CLIP-FID ( $\downarrow$ )	CMMD ( $\downarrow$ )	LPIPS ( $\downarrow$ )
TRELLIS.2 [ 58 ]	image	3.913	0.030	0.101
Hunyuan3D 2.1 [ 18 ]	image	4.197	0.039	0.102
MVPainter [ 47 ]	image	6.583	0.112	0.136
VideoMatGen (ours)	image	4.032	0.028	0.109
Hunyuan3D 2.1 [ 18 ]	text	6.694	0.046	0.137
MVPainter [ 47 ]	text	7.313	0.096	0.149

Table 1: Quantitative metrics for material generation. The mode column indicates if the method is image or text guided.

Figure 4: Left : Our method predicts a height (bump) map, which improves the visual richness of the generated material. Right : corresponding rendering without bump map.

Panel of Figure 5 — Figure 5: We generate three materials from the same text prompt (see supplemental), each with a unique random seed. This results in subtle variations of materials for the two examples.

Panel of Figure 6 — Figure 6: We show relit results, using three HDR probes [ 67 ] , of the generated materials for Hunyuan3D-Paint (image-guided), VideoMat, and our method (both text-guided). Our generated materials produce convincing details in three different lighting scenarios.

Panel of Figure 7 — Figure 7: We compare our text- and image-guided models for six examples, and note that the image-guided version more closely resembles the materials of the dataset entry. We deliberately chose a view with $45^{\circ}$ rotation from the conditioning view.

	Base color		HRM
Method	PSNR ( $\uparrow$ )	LPIPS ( $\downarrow$ )	PSNR ( $\uparrow$ )
Cosmos Tokenizer	38.8	0.046	35.1
$\mathrm{VAE}_{\mathrm{pbr}}$	38.3	0.043	33.8

Table 2: VAE finetuning evaluation on material maps from our test set and the MatSynth test set. We report PSNR (dB) and LPIPS scores for base color and only PSNR (dB) scores for HRM (height, roughness, metallicity), as perceptual metrics are not applicable. The

\mathrm{VAE}_{\mathrm{pbr}}

offers 2

\times

additional compression but has similar visual quality as the Cosmos Tokenizer.

Panel of Figure 8 — Figure 8: Our text-to-PBR material generations (renderings in Blender) on a collection of TRELLIS.2-generated base geometry.

Panel of Figure 9 — Figure 9: We compare two versions of joint prediction. Our version uses the latent space of $\mathrm{VAE}_{\mathrm{pbr}}$ and predicts a video with 16 frames. Frame concatenation doubles the number of tokens by treating the material prediction as diffusing a video with twice the number of frames [16 $\times$ base color, 16 $\times$ HRM]. The resulting quality is similar, arguably with more coherent material predictions in our approach. Please zoom to see the details.

Method	CLIP-FID ( $\downarrow$ )	CMMD ( $\downarrow$ )	LPIPS ( $\downarrow$ )
VideoMatGen ( $\mathrm{VAE}_{\mathrm{pbr}}$ )	5.638	0.035	0.126
VideoMatGen (FCat)	5.712	0.039	0.129

Table 3: Quantitative metrics for text-to-material generation. We compare two variants of joint prediction, frame concatenation (FCat) which doubles the number of video frames/latent tokens, and our version with

\mathrm{VAE}_{\mathrm{pbr}}

3D Texture Generation

See all 19 papers

MaPa: Text-driven Photorealistic Material Painting for 3D Shapes
Apr 26, 2024Shangzhan Zhang, Sida Peng, Tao Xu +73D Asset Generation Text-to-3D Generation
Zhejiang University · Ant Group · Shenzhen University
Texture Space Material Diffusion
Sep 29, 2026Jacob Munkberg, Peter Kocsis, Jon Hasselgren3D Asset Generation Texture Synthesis
NVIDIA
MatLoom: Layered Text-to-Material Generation in a Compact Program Space
Sep 30, 2026Anson Y. Lam, Shuqing Li, Michael R. Lyu3D Asset Generation LLM-Based Program Synthesis
Department of Computer Science and Engineering The Chinese University of Hong Kong Hong Kong, China

Apr 26, 2024J/K move · Enter open · S save

MaPa: Text-driven Photorealistic Material Painting for 3D Shapes

Shangzhan Zhang, Sida Peng, Tao Xu, Yuanbo Yang+6

Zhejiang University · Ant Group · Shenzhen University

3D Asset Generation Text-to-3D Generation

Open paper AskPDF arXiv

This paper aims to generate materials for 3D meshes from text descriptions. Unlike existing methods that synthesize texture maps, we propose to generate segment-wise procedural material graphs as the appearance representation, which supports high-quality rendering and provides substantial flexibility in editing. Instead of relying on extensive paired data, i.e., 3D meshes with material graphs and corresponding text descriptions, to train a material graph generative model, we propose to leverage the pre-trained 2D diffusion model as a bridge to connect the text and material graphs. Specifically, our approach decomposes a shape into a set of segments and designs a segment-controlled diffusion model to synthesize 2D images that are aligned with mesh parts. Based on generated images, we initialize parameters of material graphs and fine-tune them through the differentiable rendering module to produce materials in accordance with the textual description. Extensive experiments demonstrate the superior performance of our framework in photorealism, resolution, and editability over existing methods. Project page: https://zju3dv.github.io/MaPa

Abstract

3D Texture Generation

Explore similar work

MaPa: Text-driven Photorealistic Material Painting for 3D Shapes

Texture Space Material Diffusion

MatLoom: Layered Text-to-Material Generation in a Compact Program Space