Direct3D-V3: Latent-Free Diffusion Transformers in the Native Voxel Domain
Authors: Shuang Wu, Jiachen Qian, Youtian Lin, Feihu Zhang, Philip Torr, Yao Yao, Xun Cao
Organizations: Nanjing University, Nanjing, China · DreamTech, Shenzhen, China · The University of Hong Kong, Hong Kong, Hong Kong · Nanjing University, Suzhou, China · University of Oxford, Oxford, United Kingdom
Native 3D generation has largely been built around latent diffusion, with recent advances concentrating on stronger variational autoencoders (VAE) and increasingly elaborate latent representations, from vecset and triplanes to structured sparse voxels. Despite their architectural differences, these methods all share the same premise: geometry is first compressed into a latent space and only then modeled by the generator. We take a different route and propose Direct3D-V3, a latent-free diffusion transformer (DiT) framework that discards the VAE altogether and operates in the native voxel domain, directly modeling raw SDF voxel grids during both training and sampling. To make this direct voxel formulation geometrically reliable, we introduce an implicit manifold regularization over neighboring voxels to suppress non-watertight surfaces, and convert predicted SDF grids into meshes via Marching Cubes so that rendered normal losses can supervise the model directly from its predicted SDF. To improve alignment between the conditioning image and generated geometry, Direct3D-V3 also predicts the camera parameters of the conditioning image as an auxiliary task, encouraging the DiT to learn the mapping from image observations to 3D shape. This latent-free design removes the reconstruction bottleneck, brings training under an objective in the native voxel domain, and yields meshes with substantially richer local detail. The predicted camera and native SDF voxel output further make Direct3D-V3 extensible to generative reconstruction, where diffusion is interleaved with normal-guided refinement to improve alignment with the observed view while the generative prior completes unseen regions. Project page: https://dreamtechai.github.io/Direct3D-V3.
Figures & tables
Figure 1. Direct3D-V3 generates diverse, high-fidelity 3D meshes by modeling raw SDF voxel grids directly with latent-free diffusion transformers, removing the VAE bottleneck used by latent diffusion pipelines.
Figure 2. Framework of Direct3D-V3. Rather than compressing geometry into a learned latent representation, Direct3D-V3 trains and samples directly on raw SDF voxel grids. During training, we extract the SDF voxel grid and surface patch SDF values from the ground-truth mesh, and add Gaussian noise to the surface patch SDF values to form the DiT input xt . During inference, the first-stage model is used only to provide the surface patch indices; sampling then starts from Gaussian noise on these sparse patches, and the latent-free DiT predicts the clean SDF values directly in the native voxel domain. The patch-level DiT uses the noisy SDF signal, DINOv2 image tokens, and timestep embedding to reason efficiently over global shape semantics, while a lightweight voxel-level DiT refines voxel-level SDF values within each selected patch. In parallel, a camera prediction branch combines patch-level features with image tokens to regress normalized intrinsic and extrinsic parameters, encouraging the generated shape to remain spatially aligned with the conditioning image.
Figure 3. The architecture of the dual-granularity DiT.
Figure 4. Illustration of the Implicit Manifold Regularization (IMR). Without the decoder-induced geometric prior used in latent-based pipelines, a latent-free DiT can predict locally inconsistent TSDF fields, leading to abrupt zero-level crossings and non-watertight meshes after Marching Cubes. Since the surface manifold is implicitly defined by the zero level set of the TSDF, we regularize this manifold by penalizing neighboring voxels whose predicted SDF values differ by more than one voxel length η . This local constraint encourages coherent TSDF transitions around the surface, reducing holes and producing a more watertight extracted mesh while preserving the generated shape.
Figure 5. Generative reconstruction pipeline. Given the input images, we use pre-trained predictors to estimate the corresponding foreground mask and normal map as auxiliary observations. The key reconstruction mechanism lies in how these cues are coupled with the latent-free DiT output: the mask calibrates the predicted camera, while the normal map drives direct refinement of the predicted SDF voxel grid. At each guided sampling step, the DiT proposes a clean SDF grid together with an image-view camera. We first align the rendered mask with the estimated one to correct the camera, and then compare rendered surface normals with the estimated normal map to update the SDF values. The refined SDF is re-noised and passed back to the sampler, coupling observation-driven correction with the learned generative prior for the final reconstruction.
Resolution
Learning rate
Batch size
Training Steps
2563
1e-4
8 × 16
400K
5123
1e-4
4 × 16
50K
10243
5e-5
1 × 16
50K
15363
5e-5
1 × 16
10K
Table 1. Progressive training schedule for Direct3D-V3.
Method
Uni3D ↑
ULIP-2 ↑
Michelangelo
0.3404
0.3605
Hunyuan3D 2.1
0.4187
0.4424
Direct3D-S2
0.4190
0.4485
UltraShape
0.4185
0.4510
TRELLIS2
0.4172
0.4405
Ours
0.4198
0.4535
Table 2. Quantitative comparison with recent image-to-3D methods. Higher is better for all metrics, and the best result is shown in bold. Ours ∗ denotes Direct3D-V3 trained with the public training data plus approximately 200K additional private meshes.
Figure 6. User study results for image-to-3D generation. Scores range from 1 to 5, and higher is better.
Figure 7. Qualitative comparison with recent open-source image-to-3D generation methods.
Method
CD ↓
F1 0.95 ↑
F1 0.99 ↑
Vol. IoU ↑
TRELLIS
0.0153
0.9586
0.5123
0.3315
CUPID
0.0141
0.9550
0.5728
0.3744
ReconViaGen
0.0194
0.9239
0.4585
0.2830
TRELLIS2
0.0147
0.9510
0.5934
0.3884
Ours
0.0131
0.9600
0.6043
0.3815
Table 3. Quantitative comparison on GSO generative reconstruction. The table reports Chamfer Distance (CD), F1-score at 0.95 and 0.99, and Volume IoU. The best result is shown in bold.
Figure 8. Qualitative comparison for single-view generative reconstruction. The figure compares reconstructed meshes from the same challenging input images.
Figure 9. Qualitative comparison for multi-view generative reconstruction on scenes from the Tanks and Temples dataset.
Patch DiT
Voxel DiT
IMR
Normal Sup.
Normal-FID ↓
✓
-
-
-
92.96
✓
✓
-
-
75.80
✓
✓
✓
-
73.24
✓
✓
✓
✓
68.97
Table 4. Core component ablations on the held-out Objaverse-XL subset. Each checkmark indicates that the corresponding module is enabled. Lower is better for Normal-FID, and the best result is shown in bold.
Resolution
Method
Avg. boundary-edge count ↓
2563
w/o IMR
2,462
w/ IMR
196
10243
w/o IMR
25,353
w/ IMR
898
Table 5. Effect of IMR on boundary artifacts. Lower boundary-edge count indicates fewer open mesh boundaries.
Figure 10. Qualitative ablation of the core components. The patch-level DiT alone produces coarse and incomplete geometry. Adding the voxel-level DiT restores finer details but still leaves many holes; IMR greatly reduces these artifacts, and normal-rendering supervision further improves the quality. Qualitative comparison of core component ablations, showing improvements from patch-level DiT to voxel-level DiT, IMR, and normal-rendering supervision.
Figure 11. Qualitative ablation of the camera prediction branch. Adding the branch improves alignment between the generated mesh and the conditioning image, especially for structures visible from the conditioning view. Qualitative comparison showing that the camera prediction branch improves alignment between the conditioning image and generated mesh.
Patch size
Normal-FID ↓
p=4
57.63
p=8 (default)
68.97
p=12
76.21
Table 6. Patch-size ablation for the latent-free DiT. Smaller patches improve Normal-FID but increase the number of patch tokens; we use p=8 as the default patch size.
First stage
Normal-FID ↓
Dense latent-free DiT
80.61
Hunyuan3D 2.1
80.43
Ours (VecSet DiT)
76.62
Table 7. First-stage choices evaluated on the Objaverse-XL validation set at 5123 resolution. Lower is better for Normal-FID.
Figure 12. Ablation of first-stage surface patch indices at 5123 resolution. Comparable results across a pre-trained VecSet DiT, a dense 643 index predictor, and Hunyuan3D 2.1 indicate that the latent-free DiT is robust to different sources of surface patch indices.
Figure 13. Failure case of single-view generative reconstruction. When the first-stage surface indices are poorly aligned with the conditioning image, normal-guided SDF refinement operates on inaccurate sparse support and can distort the mesh. A failure case visualization showing distorted geometry caused by normal-guided SDF refinement on inaccurate first-stage surface indices.
Appendix figures & tables3 assets
Supplementary material from the paper’s appendix.
Appendix
Res.
Patch
Avg. Patch Tokens
Memory (MB)
Time/step (s)
2563
4
7,980
4,386
0.31
8
1,629
3,716
0.12
12
721
3,798
0.12
5123
4
35,576
8,082
1.55
8
7,340
5,716
0.37
12
2,986
5,080
0.25
Appendix
Table 8. Computational cost and scaling across resolutions and patch sizes. Memory and per-step inference time are measured on one A800 GPU.
Figure 14. Additional visual results generated by Direct3D-V3. Additional generated meshes from Direct3D-V3, illustrating diverse shapes and surface details.
Figure 15. Additional qualitative comparisons with recent image-to-3D generation methods. Additional qualitative comparisons for image-to-3D generation, showing the input images and generated meshes from different methods.
We introduce Discrete Voxel Diffusion (DVD), a discrete diffusion framework to generate, assess, and edit sparse voxels for SLat (Structured LATent) based 3D generative pipelines. Although discrete diffusion has not generally displaced continuous diffusion in image-like generation, we show that it can be an effective first-stage prior for sparse voxel scaffolds. By treating voxel occupancy as a native discrete variable, DVD avoids continuous-to-discrete thresholding and provides a simple framework for voxel generation, uncertainty estimation, and editing. Beyond quality gains, DVD provides more interpretable generation dynamics through explicit categorical modeling. Furthermore, we leverage the predictive entropy as a robust uncertainty metric to identify ambiguous voxel regions and complicated samples, facilitating tasks such as data filtering and quality assessment. Finally, we propose a lightweight fine-tuning strategy using block-structured perturbation patterns. This approach empowers the model to inpaint and edit voxels within a single sampling round, requiring negligible auxiliary computation and no additional model evaluations. Code is available at https://github.com/TeCai/DVD.
Zhengrui Xiang, Jiaqi Wu, Fupeng Sun +2
Imperial College London · Math Magic, Hitem3D · Math Magic
Recent native 3D generators achieve high-quality asset creation by modeling structured 3D latents with flow-based transformers, but their high-resolution generation remains computationally expensive due to costly iterative sampling. In this work, we introduce a few-step distillation framework for native 3D generation, enabled by continuous self-consistency training with a custom JVP FlashAttention kernel and an analytic velocity regression objective for sparse structured 3D latents. To stabilize joint optimization, we further introduce an adaptive loss weighting strategy that automatically balances them without manual tuning. Beyond acceleration, we reveal a previously overlooked failure mode of few-step native 3D generation: sampling-induced latent distribution drift preserves geometric correctness but disrupts downstream PBR texture synthesis, causing texture artifacts on geometrically valid assets. To address this issue, we propose Channel Match, a training-free adaptive per-channel latent calibration method that restores the latent distribution without modifying either generator. We validate our approach on TRELLIS.2 at the 10243 resolution setting. Our two-step model achieves 4.5× faster geometry sampling while maintaining geometry quality comparable to the many-step teacher, and Channel Match further improves rendering quality by recovering the latent interface between geometry and appearance generation.
In this paper, we introduce LATO, a novel topology-preserving latent representation that enables scalable, flow matching-based synthesis of explicit 3D meshes. LATO represents a mesh as a Vertex Displacement Field (VDF) anchored on surface, incorporating a sparse voxel Variational Autoencoder (VAE) to compress this explicit signal into a structured, topology-aware voxel latent. To decapsulate the mesh, the VAE decoder progressively subdivides and prunes latent voxels to instantiate precise vertex locations. In the end, a dedicated connection head queries the voxel latent to predict edge connectivity between vertex pairs directly, allowing mesh topology to be recovered without isosurface extraction or heuristic meshing. For generative modeling, LATO adopts a two-stage flow matching process, first synthesizing the structure voxels and subsequently refining the voxel-wise topology features. Compared to prior isosurface/triangle-based diffusion models and autoregressive generation approaches, LATO generates meshes with complex geometry, well-formed topology while being highly efficient in inference.
Tianhao Zhao, Youjia Zhang, Hang Long +7
Huazhong University of Science and Technology · Peking University · Independent Researcher +1