Generating high-fidelity voxels while keeping their lightweight nature remains an open problem in computer vision. Current generators are either high-fidelity but computationally heavy, or lightweight but of lower fidelity. Hierarchical methods keep the lightweight nature of voxels. However, their coarse levels, which decide where all finer cells can appear, are generated with the least information about the shape and are never revisited, so they are often plausible but wrong. Our key insight is to let finer cells take part in deciding coarse cells. To this end, we propose JoVox, a discrete flow matching framework that learns the joint distribution of all levels of the voxel hierarchy, instead of a separate distribution for each level, so that coarse cells can be predicted with finer cells already generated nearby. Within this framework, we further introduce a two-level prediction, which outputs the cells two levels finer at once instead of one level finer, halving the number of levels that the network processes and yielding voxels of higher fidelity. We also introduce an unmasking strategy that revisits early decisions once finer cells are generated and corrects inconsistencies in masked diffusion models. Our framework outperforms the baselines on both unconditional and image-conditioned voxel generation. Compared with the current state-of-the-art TRELLIS.2, we achieve 8x higher training throughput, 1/15 of the FLOPs in a forward pass, train on 4 vs. 32 GPUs, and obtain higher quality with an 11% lower Chamfer distance.
Figures & tables
Method
Hierarchy
No autoencoder
Parallel
Joint levels
TRELLIS, TRELLIS.2
✗
✗
✓
–
LAS-Diffusion
✗
✓
✓
–
DVD
✗
✓
✓
–
XCube
✓
✗
✓
✗
Nexus, OctFusion
✓
✓
✓
✗
Octree Transformer, OAT
✓
✓
✗
✗
Table 1: Comparison of voxel generation methods.
Figure 2: (a) Two-level prediction. For each cell u at level l , the network predicts the occupancy codes of the eight children of u at level l+1 . The occupancy code of the j -th child consists of the values aj,1,…,aj,8 of its eight children at level l+2 , where a value is 1 for an occupied cell, 0 for an empty one and ∅ for a cell whose occupancy is unknown. When this occupancy code is masked, the value aj of the j -th child and aj,1,…,aj,8 are all ∅ . (b) The network. Each cell u at level 0 and each occupied cell u at even levels has one token. The token h embeds the values aj and aj,b , the level l and the position r . A transformer processes the tokens of all levels together, conditioned on c and on the fractions t of unmasked occupancy codes, and outputs for u eight distributions over the 256 values of an occupancy code.
Voxels
Meshes
Method
CD V↓
IoU ↑
LPIPS V↓
DINO V↑
CD shell ( ×10−3 ) ↓
LPIPS N↓
CLIP N↑
Uni3D ↑
TripoSR ∗‡
0.01663
0.3199
0.1837
0.7813
0.781
0.2291
0.7431
0.3414
TripoSG ∗‡
0.01325
0.3722
0.1626
0.8523
0.525
0.1698
0.7592
0.3849
Direct3D-S2 ‡
0.01246
0.3694
0.1709
0.8550
0.539
0.1939
0.7600
0.3824
DVD
0.01973
0.2786
0.2119
0.7914
1.070
0.2412
0.7538
0.3718
TRELLIS
0.01328
0.3595
0.1641
0.8385
0.524
0.1899
0.7583
0.3799
Table 2: Image-conditioned generation on Toys4k ( ∗ voxels from voxelized meshes; ‡ evaluated on their own meshes). The best and second-best values of each column are in bold and underlined, respectively.
Figure 3: An example of image-conditioned generation on Toys4k. The top row shows the generated 643 voxels and the bottom row the meshes obtained from them. Each shape is aligned to its ground truth (GT) for display. More examples are in the appendix.
Method
1-NNA
COV ↑
MMD ↓
FID V↓
FID VD↓
KID VD↓
Four categories
LAS-Diffusion
0.782
0.365
2.08
63.3
82.5
16.7
OctFusion †
0.805
0.290
2.23
46.2
184.5
41.5
OctFusion (mesh)
0.779
0.310
2.23
11.1
43.0
4.99
OctGPT †
0.773
0.388
2.22
43.2
186.6
42.5
OctGPT (mesh)
0.736
0.412
2.25
10.1
61.0
10.4
Table 3: Unconditional generation on ShapeNet, averaged over the four categories, and over chair and airplane for the last two rows. For 1-NNA, closer to 0.5 is better. MMD is in units of 10−3 and KID VD in units of 10−2 ( † 323 occupancy output; “mesh” meshes voxelized at 643 ). The best and second-best values within each block are in bold and underlined, respectively.
Figure 4: Unconditional generation of chairs by our framework on ShapeNet.
Figure 5: Peak GPU memory and training throughput of our network and of the voxel stage of TRELLIS.2, with the same backbone, at each batch size on one A100 GPU with 80 GB. A cross marks the batch size at which TRELLIS.2 runs out of memory.
Variant
CD V↓
IoU ↑
CD shell↓
LPIPS N↓
Uni3D ↑
Single-level
0.02150
0.238
1.280
0.2566
0.3508
Two-level
0.01941
0.264
1.096
0.2428
0.3575
Ours
0.01506
0.334
0.744
0.2019
0.3820
Table 4: Ablation of two-level prediction and joint hierarchical modeling on Toys4k. The first two variants generate the hierarchy level by level. CD shell is in units of 10−3 . The best value of each column is in bold.
Sampler
CD V↓
IoU ↑
MDLM [ 21 ]
0.01162
0.4035
ReMDM [ 25 ]
0.01115
0.4085
ReMDM-conf [ 25 ]
0.01116
0.4139
MaskGIT [ 5 ]
0.01243
0.3987
Ours
0.01107
0.4158
Table 5: Ablation of the unmasking strategy on Toys4k. The best and second-best values of each column are in bold and underlined, respectively.
Figure 6: Image-conditioned generation at 2563 . Each input image is shown in the box at the top left of the generated 2563 voxels, which are colored by the texture stage of TRELLIS.2.
Ours
TRELLIS.2
Batch
Time (s)
Memory (GB)
Time (s)
Memory (GB)
8
2.02
6.1
2.30
5.7
16
1.77
7.9
2.24
11.2
32
1.80
11.3
2.21
22.2
64
1.67
17.8
2.20
44.2
128
1.62
28.9
OOM
Table 6: Time per shape and peak GPU memory of generating the voxels for image-conditioned generation on one A100 GPU with 80 GB, for our framework and the voxel stage of TRELLIS.2 (OOM, out of memory). The time includes image encoding, with 80 sampling steps for our framework and the default 12 for TRELLIS.2.
Chair
Airplane
Method
1-NNA
COV ↑
MMD ↓
FID V↓
FID VD↓
KID VD↓
1-NNA
COV ↑
MMD ↓
FID V↓
FID VD↓
KID VD↓
LAS-Diffusion
0.697
0.432
3.40
30.67
57.6
8.77
0.881
0.357
1.06
73.07
130.3
28.54
OctFusion †
0.689
0.406
3.59
22.49
106.8
15.63
0.902
0.261
1.17
51.04
305.2
67.86
OctFusion (mesh)
0.671
0.412
3.68
5.01
29.5
2.70
0.863
0.268
1.00
12.64
51.2
6.41
OctGPT †
0.715
0.397
3.74
25.38
139.6
26.30
0.872
0.365
1.09
47.83
312.3
68.88
OctGPT (mesh)
0.697
0.401
3.90
9.36
77.8
14.74
0.821
0.361
0.93
12.78
27.9
2.21
Table 7: Unconditional generation on each ShapeNet category (test splits). For 1-NNA, closer to 0.5 is better. MMD is in units of 10−3 and KID VD in units of 10−2 . Rows marked † use the 323 occupancy the method outputs, and rows marked “mesh” use the method’s meshes voxelized at 643 . XCube releases models only for chair and airplane. The best and second-best values of each column within each category are in bold and underlined, respectively.
Aug.
CD V↓
IoU ↑
CD shell↓
LPIPS N↓
CLIP N↑
Uni3D ↑
0.01280
0.3923
0.568
0.1858
0.7588
0.3844
✓
0.01271
0.3790
0.524
0.1797
0.7608
0.3847
Table 8: Training with and without augmentation, compared after the same number of training steps on the first 200 objects of Toys4k. CD shell is in units of 10−3 . The better value of each column is in bold.
Voxels
Meshes
Method
CD V↓
IoU ↑
LPIPS V↓
DINO V↑
CD shell ( ×10−3 ) ↓
LPIPS N↓
CLIP N↑
Uni3D ↑
TRELLIS
0.01328
0.3595
0.1641
0.8385
0.524
0.1899
0.7583
0.3799
Ours, about two days ( 80 K steps)
0.01221
0.3893
0.1485
0.8614
0.497
0.1737
0.7600
0.3851
TRELLIS.2
0.01250
0.4017
0.1524
0.8513
0.524
0.1666
0.7613
0.3865
Ours, about two and a half days ( 104 K steps)
0.01137
0.4035
0.1424
0.8692
0.421
0.1668
0.7612
0.3859
Table 9: Our framework after about two days and two and a half days of training on four H100 GPUs, compared with TRELLIS and TRELLIS.2 on Toys4k. Each of our rows is compared with the baseline above it. Steps are counted in batches of 256 shapes.
Figure 7: Image-conditioned generation on Toys4k. For each input image, the top row shows the generated 643 voxels and the bottom row the meshes obtained from them as in Section 5.1 . The ground truth (GT) shows the voxels and the mesh of the shape. Each shape is aligned to its ground truth for display.
Figure 8: Image-conditioned generation on Toys4k, continued from Figure 7 .
Figure 9: Image-conditioned generation on Toys4k, continued from Figure 8 .