Coarse-to-fine 3D-native generation relies heavily on the first-stage sparse structure, which determines the global orientation and overall geometry for subsequent refinement. However, existing coarse generators suffer from two key limitations: their orientation is difficult to control due to ambiguous semantic canonicalization, and their structures are often inconsistent with input images due to weak 2D-3D grounding. We propose Coarse3D, a sparse-structure generation model that produces orientation-controllable and structurally faithful voxel geometry from arbitrary unposed images. Coarse3D introduces three designs: a first-frame anchored canonical space that aligns generation with the first input view while preserving object-centric normalization; a dual-tower DiT that couples generation with auxiliary feed-forward reconstruction to learn spatially grounded features; and spatial-prior guided attention that regularizes cross-attention with pixel-to-voxel correspondence. Coarse3D can directly replace the coarse stage of existing pipelines. Experiments show that it substantially improves sparse structure quality and consistently enhances final 3D generation, achieving state-of-the-art performance.
Figures & tables
Figure 1: Coarse3D generates orientation-controllable and structurally faithful sparse structures. Trellis2, ReconViaGen and Pixal3D use their own coarse and refinement stages while Ours+Trellis2 replaces only the coarse stage with Coarse3D and reuses the Trellis2 refinement stage. Each cell shows the coarse voxel structure ( left ) and the final shape it yields ( right ). Coarse3D consistently follows the first-view orientation and preserves structural fidelity across diverse objects, directly translating into higher-quality final shapes through the coarse-to-fine pipeline.
Figure 2: Comparison of three coordinate definitions: canonical space uses a fixed semantic orientation that is ambiguous; pixel-aligned camera space is aligned with the input view but not object-centric; ours is aligned with the first input view and object-centric.
Figure 3: Overview of Coarse3D. (A) First-frame Anchoring rotates the canonical geometry to zero the first view’s azimuth, turning orientation prediction into a well-defined, first-frame-conditioned task. (B) Dual-tower DiT couples an Image-Tower, supervised by auxiliary tasks ( Lrecon ) to encode cross-view geometry, with a Geometry-Tower predicting the velocity field ( Lgeo ) via layer-wise cross-attention, trained end-to-end. (C) Spatial-prior-guided Attention regularizes cross-attention with a rasterized voxel-to-pixel prior ( Lattn ), strengthening 2D-3D correspondence.
GSO
Toys4K
Method
IoU ↑
PA-IoU ↑
CD ↓
IoU ↑
PA-IoU ↑
CD ↓
TRELLIS-MV
0.2243
0.2909
0.0320
0.3516
0.3785
0.0246
TRELLIS2-MV
0.3098
0.3655
0.0274
0.3840
0.4412
0.0221
ReconViaGen
0.2323
0.2984
0.0293
0.2818
0.3228
0.0271
Pixal3D-MV *
0.4419
0.4420
0.0181
0.5079
0.5080
0.0133
Coarse3D (Ours)
0.5101
0.5364
0.0124
0.6315
0.6368
0.0091
Table 1: Coarse structure generation on GSO and Toys4K. IoU is orientation-sensitive, PA-IoU factors out global orientation, and CD is computed after rigid alignment. Note that Pixal3D-MV * relies on ground-truth camera poses to align and fuse features across the multiple input views, whereas the other methods operate on fully unposed images. Best in bold.
Method
GSO
Toys4k
CD ↓
F-Score ↑
ULIP ↑
Uni3D ↑
CD ↓
F-Score ↑
ULIP ↑
Uni3D ↑
TRELLIS-MV
0.0772
0.4021
0.2112
0.2557
0.0397
0.6157
0.2373
0.3610
+ Ours
0.0128
0.8413
0.2255
0.2688
0.0131
0.8600
0.2461
0.3690
TRELLIS2-MV
0.0613
0.5425
0.2073
0.2607
0.0360
0.6522
0.2373
0.3652
+ Ours
0.0127
0.8576
0.2183
0.2650
0.0099
0.9128
0.2391
0.3684
ReconViaGen
0.0785
0.3835
0.2245
0.2707
0.0490
0.5096
0.2519
0.3687
Table 2: Final shape quality when only the coarse stage is replaced by Coarse3D ( + Ours ). CD and F-Score measure geometric accuracy, while ULIP and Uni3D measure semantic similarity to the input images. Bold marks the better result of each pair. Note that Pixal3D-MV * uses GT cameras.
Figure 4: Qualitative comparison across three baselines (TRELLIS, TRELLIS2, and ReconViaGen). For each baseline, we show the original result ( Baseline ) and the result after replacing its first stage with our Coarse3D ( + Ours ). Anchored by the first frame of the multi-view input (leftmost), our method yields (i) more controllable orientations , (ii) higher structural fidelity , and (iii) seamless plug-and-play integration without modifying subsequent stages.
Method
GSO
Toys4k
RRE ↓
RTE ↓
CD ↓
RRE ↓
RTE ↓
CD ↓
VGGT
10.54
0.2010
0.0226
16.11
0.2352
0.0160
ReconViaGen
4.09
0.0934
0.0203
8.15
0.1293
0.0144
RecGen3D *
2.16
0.0338
–
3.67
0.0637
–
Ours
2.02
0.0319
0.0174
3.43
0.0370
0.0128
Table 3: Auxiliary reconstruction accuracy. RRE and RTE are relative camera rotation and translation errors, and CD is measured on the predicted point maps after rigid alignment. * numbers are copied from the original paper. Best in bold.
Setting
Coarse Generation
Reconstruction
IoU ↑
PA-IoU ↑
CD ↓
RRE ↓
RTE ↓
Baseline
0.4041
0.4677
0.0280
–
–
+ Dual-DiT under canonical space
0.4652
0.5329
0.0117
5.88
0.0692
+ First-frame anchored canonical space
0.5586
0.5704
0.0104
3.42
0.0413
+ Spatial-Prior guided Attention
0.6392
0.6550
0.0087
3.10
0.0399
Table 4: Cumulative ablation on the three key components of Coarse3D. Each row adds one component to the row above. The baseline has no image tower and therefore no reconstruction metrics.
Figure 5: Effect of spatial-prior guided attention. For a representative voxel query, we visualize the attention map over pixel tokens of four input views. Top : spatial prior Asp derived from geometric projection. Middle : base model attention Adense , which spreads diffusely across the object. Bottom : our guided model concentrates on the pixels the query voxel actually projects to.
Appendix figures & tables2 assets
Supplementary material from the paper’s appendix.
Appendix
Space Type
View-dependent
Alignment of Orientation Angles
Standard Canonical Space
✗
none
Pixel-aligned Camera Space
✓
all (azimuth, elevation and roll)
First-frame Anchored Canonical Space (ours)
✓
only azimuth
Appendix
Table 5: Summary of three types of 3D generative spaces.
GSO
Toys4K
# Views
IoU ↑
PA-IoU ↑
CD ↓
IoU ↑
PA-IoU ↑
CD ↓
1
0.3129
0.3495
0.0285
0.3876
0.4148
0.0236
2
0.4200
0.4534
0.0187
0.5234
0.5375
0.0163
3
0.4834
0.5075
0.0168
0.6045
0.6149
0.0131
4
0.5101
0.5364
0.0124
0.6315
0.6368
0.0091
5
0.5244
0.5449
0.0115
0.6419
0.6465
0.0087
Appendix
Table 6: Ablation on the number of input views. We vary the number of input images from 1 to 6 and report the coarse structure reconstruction quality of Coarse3D on GSO and Toys4K.
Scaling image-to-3D generation to ultra-high resolutions requires controlling rapidly growing computational costs without sacrificing fine geometric detail. We present Filigree3D, a sparse latent flow-matching framework that generates 3D geometry from a single image at voxel resolutions up to 20483, with straightforward extensibility to 40963. To make training tractable, we introduce Structure-Aware Sparse Scaling, which combines spatial bounding with alternating local-global attention to constrain token growth while preserving both fine-scale details and long-range structural context. To enhance detail reconstruction, we curate training samples based on their high-resolution geometric gains and inject multi-scale image features into a sparse 3D DiT, effectively coupling structural semantics with fine-grained visual cues. Furthermore, a visibility-aware voxel regularization strategy improves robustness against sparse perturbations and facilitates the completion of unobserved geometry. Under our default configuration, Filigree3D maintains peak GPU memory consumption within practical limits for contemporary hardware, enabling the generation of highly intricate 3D geometry in approximately one minute. Extensive experiments demonstrate that our method yields substantial improvements in overall geometric fidelity and fine-detail preservation compared to existing baselines, validating practical, detail-preserving 3D generation at unprecedented resolutions.
We present SparseGen, a novel framework for efficient image-to-3D generation, which exhibits low input-view bias while being significantly faster. Unlike traditional approaches that rely on dense volumetric grids, triplanes, or pixel-aligned primitives, we model scenes with a compact sparse set of learned 3D anchor queries and a learned expansion operator that decodes each transformed query into a small local set of 3D Gaussian primitives. Trained under a rectified-flow reconstruction objective without 3D supervision, our model learns to allocate representation capacity where geometry and appearance matter, achieving significant reductions in memory and inference time while preserving multi-view fidelity. We introduce quantitative measures of input-view bias and utilization to show that sparse queries reduce overfitting to conditioning views while being representationally efficient. Our results argue that sparse set-latent expansion is a principled, practical alternative for efficient 3D generative modeling.
Generating complete 3D assets from sparse, unposed images requires preserving observed geometry while completing unseen regions. Joint reconstruction and generation offers a way to address both goals: reconstruction recovers the geometry observed in the input images, while generation contributes learned shape priors. However, combining camera-relative reconstruction with canonical-space generation introduces a coordinate mismatch that makes direct geometric interaction more difficult. Moreover, conditioning each 3D location on all views without accounting for their relevance may mix useful geometric evidence with unrelated image content, hindering local detail recovery. We present UniForge3D, which jointly models reconstruction and generation in a shared anchor-guided space, resolving their coordinate mismatch. This space follows the anchor view's azimuth while keeping the object upright. The two branches exchange features bidirectionally to predict per-view point maps and a complete sparse voxel structure. We further condition point-map prediction on the generated structure to establish pixel-to-voxel correspondences and select the most informative input view for each voxel. Features from the selected views then guide detailed shape generation, helping preserve observed geometric details. Our method achieves state-of-the-art appearance and geometry results on Toys4K and GSO, generating complete 3D assets with improved fidelity. Project webpage: https://xishuxishu.github.io/uniforge3d-project/
Wenjue Chen, Wenyue Chen, Heliang Zheng +3
Peking University · Math Magic · The Hong Kong University of Science and Technology