Using conditional image generators, texture artists can explore many single-view looks for an existing 3D shape. Despite impressive progress, state-of-the-art generative methods still struggle to generate a full object texture while closely adhering to fine scale geometric detail and single view references, leaving little room for artists guidance. Furthermore, current automatic models lack the flexibility for artist to explore multiple textures from varied sources in an interactive and controllable manner. Unlike methods trained on large 3D datasets that generate full object textures from global guidance, our work explores a local and less data-hungry approach to texture with explicit artist control. We leverage the geometric self-similarity and geometry-texture correlation existing in many natural and man-made shapes; and train a shape-specific local texture generation and completion model. This model learns from existing image model priors and a single 3D shape, and is guided by attending to a set of geometry-aware reference patches. The trained shape-specific network can transfer any novel reference to the full target object texture through patchwise inpainting. We show improved or comparable quality to strong image-conditioned texture generation baselines, suggesting local texturing as a promising research direction. Our model also enables local geometry-conditioned texture inpainting, guided by artist-selected references, and generalizes to PBR materials and unseen meshes for texture transfer. We piloted our novel texture fill capability as a Blender addon with several 3D texturing professionals who reported positive feedback on the model's controllability, practical usefulness, and creative affordances.
Figures & tables
Figure 1. We train a GLOSS network on many single-view generated images of one untextured 3D shape (left), allowing it to learn how to texture local shape regions by attending to local geometry (such as scales) and faithfully following conditional texture patches. Once trained, artists can use GLOSS to mix and match any number of references for interactive texture completion of the target shape (right), enabled through reference-guided local texture fill. We also show fully automated texturing, PBR channel completion, and transfer results across shapes.
Figure 2. Data and Training : we use off-the-shelf models to generate training data (§ 3.2 ) for our local texture inpainting model (§ 3.3 , § 3.4 ). Steps A to F show the patch data generation pipeline. Then we show how the generated data is used for model input as well as the training for the model.
Figure 3. Model Inference Applications: Reference brush creation from (A) a single view image or (B) any PBR texture map. The model can be used interactively to complete select faces, or fully automatically to complete the whole mesh.
Figure 4. Interactive texturing with multiple source single views on various parts of the object. We show the reference texture patch from the single view along with the normal, and where the texture brush created with that reference was applied to inpaint on the existing painted texture.
Figure 5. Interactive texturing : our model supports faithful and controllable texture fills from multiple sources.
Figure 6. Interactive applications, transfer, comparisons : our model supports a wide range of texturing applications.
Figure 7. Automatic texturing with fine-tuned models for the fish and reptile categories.
Figure 8. An example of Automatic texturing visual comparisons.
Method
Per-mesh
Fine-tune
LPIPS ↓
FID ↓
DS ↓
CMMD ↓
LPIPS ↓
FID ↓
DS ↓
CMMD ↓
TexGen
-
106.917
0.436
0.883
-
124.460
0.538
1.143
MV-Adapter
0.358
76.98
0.370
0.666
0.366
122.865
0.473
0.961
Hunyuan2.1
0.383
73.702
0.437
0.494
0.299
93.045
0.356
0.570
Trellis2
0.356
102.22
0.411
0.545
0.307
110.773
0.371
0.777
Ours
0.302
80.258
0.344
0.478
0.282
104.108
0.352
0.694
Table 1. Quantitative Results: Overall (DreamSim) and patch-level (LPIPS, FID, CMMD) metrics on automatic texture generation for per-mesh and fine-tuned models. Best in bold.
Variant
Croissant
Koi Fish
Sea Shell
Overall
LPIPS ↓
FID ↓
LPIPS ↓
FID ↓
LPIPS ↓
FID ↓
LPIPS ↓
FID ↓
No batch-attn.
0.4277
100.93
0.2248
57.09
0.2923
81.06
0.3262
79.70
No geom. cond.
0.3838
64.62
0.2512
79.73
0.2810
90.48
0.3175
78.27
No img. loss
0.3006
62.31
0.2027
53.25
0.2628
76.76
0.2517
64.11
No fine-tuning
0.3020
59.85
0.2011
51.77
0.2623
78.53
0.2518
63.38
Full model
0.2830
59.45
0.2009
55.54
0.2673
77.11
0.2420
64.03
Table 2. Ablations on 3 shapes and a truncated training run using our validation set suggest that batch-attention and geometric conditioning have the greatest effect on local texture quality.
Appendix figures & tables14 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 9. Color leak effect from training model using nearest neighbor interpolation for latent mask downsampling. Top two rows show masked albedo and corresponding mask as model inputs; the third row is ground truth composited with green; the last row shows the model output.
Metric
Method
Overall
by Example
Brick
Cabbage
Croissant
Tire
Fire Hydrant
Gourd
Koi Fish
Barrel
Sea Shell
Turtle
LPIPS ↓
Paint3D
0.378
0.580
0.501
0.370
0.379
0.265
0.432
0.337
0.328
0.348
0.242
MV-Adapter
0.358
0.650
0.485
0.340
0.352
0.226
0.429
0.286
0.295
0.302
0.211
Hunyuan2.1
0.383
0.577
0.498
0.352
0.370
0.272
0.604
0.303
0.314
0.328
0.214
Trellis2.0
0.356
0.547
0.482
0.341
0.358
0.283
0.407
0.299
0.307
0.327
0.213
Ours
0.302
0.462
0.386
0.256
0.319
0.208
0.361
0.281
0.283
0.283
0.181
Appendix
Table 3. Quantitative Results: Overall (DreamSim) and patch-level (LPIPS, FID, CMMD) metrics on texture generation task conditioned on a single view, across 10 shapes with 50 views each, reveal competitive performance of our method, despite using no textured 3D training data for training.
Metric
Method
Overall
by Example
Cabbage1
Cabbage3
Cantaloup
Fish1
Fish3
Fish4
Toad
Tortoise
Turtle
LPIPS ↓
MV-Adapter
0.366
0.319
0.299
0.598
0.301
0.409
0.247
0.399
0.387
0.334
Hunyuan2.1
0.299
0.287
0.266
0.440
0.253
0.335
0.187
0.327
0.318
0.279
Trellis2.0
0.307
0.284
0.279
0.471
0.268
0.379
0.188
0.321
0.306
0.269
Ours
0.282
0.278
0.255
0.407
0.248
0.308
0.172
0.315
0.287
0.268
FID ↓
TexGen
124.460
116.061
84.111
290.178
130.762
120.826
107.264
52.956
113.996
103.989
Appendix
Table 4. Quantitative Results on Transfer Meshes: Overall and per-example LPIPS, FID, DreamSim, and CMMD on the texture-transfer evaluation set (9 sub-meshes spanning 3 base shapes). Best per column in bold.
Figure 10. We show a visual comparison of our model ablation experiments. We show reference albedo and geometry patches on the left and each model variant’s output.
Experiment
LPIPS ↓
DreamSim ↓
FID ↓
guidance_2 (final)
0.3031
0.3441
80.24
wo_syncmvd
0.3040
0.3393
74.81
guidance_4
0.3044
0.3514
86.64
guidance_10
0.3169
0.3829
120.97
overlap_0.4
0.3099
0.3644
97.85
wo_color_correct
0.3044
0.3497
84.41
Appendix
Table 5. Quantitative evaluation of textures generated with variations of final completion experiment settings. We report average LPIPS, DreamSim, and FID metrics over 10 objects.
Cabbages
Fishes
Turtles
Overall
LPIPS ↓
FID ↓
LPIPS ↓
FID ↓
LPIPS ↓
FID ↓
LPIPS ↓
FID ↓
v100–10k
0.260
75.87
0.176
111.95
0.263
76.32
0.233
88.05
v100–20k
0.253
80.78
0.169
109.63
0.253
76.33
0.225
88.91
v100–50k
0.255
80.16
0.170
104.27
0.268
88.24
0.231
90.89
Appendix
Table 6. Step-count ablation on the 100 view variant. LPIPS / FID at 10k, 20k, and 50k fine-tuning steps. We show that finetuning results are best at an early stage, enabling faster convergence.
Cabbages
Fishes
Turtles
Overall
LPIPS ↓
FID ↓
LPIPS ↓
FID ↓
LPIPS ↓
FID ↓
LPIPS ↓
FID ↓
50 views
0.255
72.80
0.170
108.66
0.257
78.77
0.227
86.74
100 views
0.255
80.16
0.170
104.27
0.268
88.24
0.231
90.89
200 views
0.255
83.40
0.172
110.26
0.268
82.68
0.232
92.11
Appendix
Table 7. View-count ablation comparing v50, v100, and v200 fine-tunes. We show that we can finetune with a small number of views.
Figure 11. Batch multi-attention visualization comparing two batches with the same reference images but a different target image (last in the batch). The overlaid heatmap shows how the center patch token in the target image attends to all patch tokens across the batch.
Figure 12. On the left we show examples of semantic texture failure. On the right we show examples of failure cases due to low geometry and texture correlation.
Figure 13. Additional result on texturing applications.
Figure 15. Additional results on the baseline comparison, both global view and zoomed-in views.
Figure 16. Additional results on the baseline comparison, both global view and zoomed-in views.
Figure 17. We show more automatic texture completion results with GLOSS. We show views of the 3D model that are opposite to the conditional view. We also show the textured 3D model from a view other than these two.