Rethinking Image-to-3D Generation with Sparse Queries: Efficiency, Capacity, and Input-View Bias

Zhiyuan Xu; Jiuming Liu; Yuxin Chen; Masayoshi Tomizuka; Chenfeng Xu; Chensheng Peng

cs.CVApr 15, 2026

Rethinking Image-to-3D Generation with Sparse Queries: Efficiency, Capacity, and Input-View Bias

Authors: Zhiyuan Xu, Jiuming Liu, Yuxin Chen, Masayoshi Tomizuka, Chenfeng Xu, Chensheng Peng

Organizations: UC Berkeley · University of Cambridge · UT Austin

Abstract

We present SparseGen, a novel framework for efficient image-to-3D generation, which exhibits low input-view bias while being significantly faster. Unlike traditional approaches that rely on dense volumetric grids, triplanes, or pixel-aligned primitives, we model scenes with a compact sparse set of learned 3D anchor queries and a learned expansion operator that decodes each transformed query into a small local set of 3D Gaussian primitives. Trained under a rectified-flow reconstruction objective without 3D supervision, our model learns to allocate representation capacity where geometry and appearance matter, achieving significant reductions in memory and inference time while preserving multi-view fidelity. We introduce quantitative measures of input-view bias and utilization to show that sparse queries reduce overfitting to conditioning views while being representationally efficient. Our results argue that sparse set-latent expansion is a principled, practical alternative for efficient 3D generative modeling.

Figures & tables

Figure 2: Overview of SparseGen . Given $V$ input views (clean and/or noisy) with known camera poses, an image encoder (with adaLN timesteps) and a 3D position encoder generate position-aware image features. A sparse set of learnable 3D anchor queries attends to these fused features in a transformer-based expansion network and is decoded into a compact set of 3D Gaussians. Finally, the generated Gaussians are rendered for target views via differentiable splatting, enabling fast, high-quality 3D generation and rendering.

Figure 3: 3D Position-Aware Encoder. Pixels are unprojected along the camera frustum to fixed depths and mapped by a small Conv–ReLU–Conv into per-pixel 3D features. These 3D features are merged with DINO-extracted image features and flattened into 3D position-aware tokens for the following transformer.

Method	PSNR $\uparrow$	LPIPS $\downarrow$	SSIM $\uparrow$	FID $\downarrow$	Reconstruction Time $\downarrow$	3D Representation Size $\downarrow$
OpenLRM [ 9 ]	18.286	0.134	0.815	51.421	0.301s	3,840KB (12,288 Triplane Cells)
Splatter Image [ 33 ]	23.933	0.077	0.922	44.908	0.042s	1,472KB (16,384 Gaussians)
Viewset Diffusion [ 32 ]	22.688	0.096	0.891	39.807	16.32s	8,192KB (32,768 Voxel Cells)
SparseGen	24.018	0.081	0.913	23.595	0.027s	280KB (5,120 Gaussians)

Table 1: Single-view reconstruction on ShapeNet-SRN Cars. One conditioning view is provided; metrics are averaged over the remaining ground-truth views. Higher is better for PSNR/SSIM; lower is better for LPIPS/FID. “Reconstruction Time” is the wall-clock time to produce the 3D representation on a single NVIDIA L40; “3D Representation Size” reports memory footprint and the number/type of primitives. Best results are in bold while runners-up are underlined . SparseGen uses 5,120 Gaussians and achieves the best PSNR and FID with the fastest runtime and smallest footprint. Note that Splatter Image uses degree 1 spherical harmonics for color representation, while SparseGen uses RGB colors, and therefore the number of parameters per Gaussian is different.

Subset	Method	PSNR $\uparrow$	LPIPS $\downarrow$	SSIM $\uparrow$
Hydrant	Viewset Diffusion	19.664	0.232	0.693
Hydrant	SparseGen	20.366	0.192	0.724
Teddybear	Viewset Diffusion	15.473	0.405	0.492
Teddybear	SparseGen	19.005	0.353	0.568

Table 2: Single-view reconstruction on CO3D subsets. One input view per object; metrics computed on a held-out target view. Results are reported per category. SparseGen improves PSNR/LPIPS/SSIM over Viewset Diffusion on both Hydrant and Teddybear.

Method	PSNR $\uparrow$	LPIPS $\downarrow$	SSIM $\uparrow$	FID $\downarrow$
w/o rectified flow	23.069	0.092	0.895	28.415
w/o 3d pos embed	20.757	0.108	0.861	27.946
w/o learnable queries	17.159	0.209	0.807	178.129
SparseGen	24.018	0.081	0.913	23.595

Table 3: Ablations on ShapeNet-SRN Cars. Each row removes one component from SparseGen . Rectified flow improves realism and fidelity; 3D positional embeddings inject essential geometry; learnable 3D anchor queries are critical—removing them causes a quality drop and a large FID increase.

Method	PSNR $\uparrow$	LPIPS $\downarrow$	SSIM $\uparrow$	FID $\downarrow$	Reconstruction Time $\downarrow$	Representation Size $\downarrow$
Viewset Diffusion [ 32 ]	25.053	0.079	0.919	35.934	21.26s	1 $\times$
Splatter Image [ 33 ]	23.257	0.075	0.916	40.957	0.052s	2 $\times$
SparseGen	25.982	0.065	0.935	19.536	0.029s	1 $\times$

Table 4: Two-view reconstruction on ShapeNet-SRN Cars. Higher is better for PSNR/SSIM; lower is better for LPIPS/FID. “Representation Size” indicates the relative size of the 3D representation compared to, when using one view as input. We evaluated baselines with their released checkpoints. Best results are in bold .

Method	$\Delta$ PSNR	$\Delta$ LPIPS	$\Delta$ SSIM
OpenLRM	8.819	-0.103	0.138
Splatter Image	14.821	-0.072	0.073
Viewset Diffusion	8.445	-0.061	0.084
SparseGen	3.502	-0.025	0.041

Table 5: Input-view bias analysis. Gap between conditioning and novel views on ShapeNet-SRN Cars. Larger absolute values indicate stronger bias.

Figure 4: Back-view conditioning qualitative example. The input view observes the object from the back, providing limited information about the front. SparseGen produces plausible novel views, while deterministic feed-forward baselines fail.

Figure 5: Qualitative results on ShapeNet-SRN Cars. We show examples under one-view conditioning. SparseGen yields sharper details, cleaner boundaries, with fastest generation speed.

Panel of Figure 6 — Figure 6: Scalability with respect to the number of queries. Increasing the number of learned 3D anchor queries ( $M$ ) consistently improves reconstruction quality, supporting the view that queries provide a principled, low-waste representation bottleneck.

Figure 7: Opacity/density utilization. Histogram comparison across methods. Splatter Image produces many near-transparent Gaussians, whereas SparseGen yields predominantly non-trivial opacities. Viewset Diffusion has many zero-density voxels. For Viewset Diffusion, densities are passed through a sigmoid and rescaled to [0,1] for visualization.

Panel of Figure 8 — Figure 8: Qualitative visualization of query utilization and locality. (a) Projection of per-query mean decoded Gaussian centers onto the image plane. (b) Corresponding RGB image. (c) A subset of decoded Gaussian centers, colored by the anchor query that generated them, illustrating query-induced locality.

Method	GPU Device	Training Time
Viewset Diffusion	L40	3 Days
Splatter Image	A6000	7 Days
LRM	A100	300 Days
LVSM	A100	200 Days
SparseGen	L40	3 Days

Table 6: Training efficiency. Approximate training cost in GPU-days on the indicated hardware. SparseGen is comparable to Viewset Diffusion and Splatter Image, yet far less expensive than large models (e.g., LRM, LVSM).

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Parameter	Value	Description
M	512	Number of anchor queries
K	10	Gaussians per query (expansion factor)
d	512	Transformer hidden dimension
$H,W$	128	Input image resolution (pixels)
$d_{th}$	64	Depth samples per ray
$s_{max}$	0.1	Maximum Gaussian scale

Appendix

Table 7: Default hyperparameters used in our experiments.

Panel of Figure 9 — Figure 9: Qualitative results on CO3D subsets. We show examples under one-view conditioning settings. Compared to other methods, SparseGen yields better visual quality, with significantly faster speed and smaller representation size.

Figure 10: Additional qualitative results on the ShapeNet-SRN dataset. Each group features one input image (left) and two novel view renderings (right) from all tested methods. Generally, deterministic feed-forward methods (i.e., Splatter Image [ 33 ] ) tend to produce unsatisfactory details on regions not well observed in the input view, while generative methods with iterative diffusion (i.e., Viewset Diffusion [ 32 ] ) can synthesize plausible details but sometimes introduce artifacts. Our SparseGen effectively generates high-quality novel views with ultra-fast speed.

Method	PSNR $\uparrow$	LPIPS $\downarrow$	SSIM $\uparrow$
OpenLRM	14.526	0.199	0.741
Splatter Image	21.065	0.111	0.878
SparseGen	21.427	0.160	0.850

Appendix

Table 8: Quantitative evaluation on the GSO dataset. Best results in bold , second best underlined . Our method achieves the highest PSNR while maintaining competitive perceptual quality.

Image-to-3D Scene Generation

See all 24 papers

Sparse auto-regressive modeling for scene generation from multi-view images
Sep 3, 2026Thomas Lucas, Maxime Pietrantoni, Philippe Weinzaepfel +4Sparse-View 3D Reconstruction 3D Scene Generation
1NAVER LABS Europe, France · 2NAVER LABS, South Korea · 3Carnegie Mellon University, USA
FILIGREE3D: Scaling Sparse Latent Flow Matching for Ultra-High-Resolution Image-to-3D Generation
Sep 28, 2026Hongjie Li, Xinran Yang, Xiuchao Wu +2Flow Matching 3D ViTs
AlibabaGroup
ROAR-3D: Routing Arbitrary Views for High-Fidelity 3D Generation
May 20, 2026Hanxiao Sun, Mingxin Yang, Shuhui Yang +53D Asset Generation 3D Shape Generation
The Hong Kong University of Science and Technology · Tencent Hunyuan

Sep 3, 2026J/K move · Enter open · S save

Sparse auto-regressive modeling for scene generation from multi-view images

Thomas Lucas, Maxime Pietrantoni, Philippe Weinzaepfel, Wonjune Cho+3

1NAVER LABS Europe, France · 2NAVER LABS, South Korea · 3Carnegie Mellon University, USA

Sparse-View 3D Reconstruction 3D Scene Generation 3D Scene Representation Autoregressive Generation 3D Scene Completion Autoregressive 3D Scene Generation

Open paper AskPDF arXiv

Generating complete 3D scenes from sparse, unconstrained views is a fundamental challenge in 3D vision which requires reasoning beyond observed content while remaining computationally tractable. Existing feed-forward reconstruction methods are inherently limited to content visible in the input images, while 3D generative modeling is hindered by the high computational cost of dense volumetric representations and the scarcity of large-scale 3D supervision. We introduce SPAR3S, a sparse voxel-aligned 3D latent generative model for conditional scene completion without requiring ground-truth 3D data for supervision. Our key insight is to formulate 3D scene generation in a structured, compact, voxel-aligned 3D latent space where only occupied voxels are represented. We learn this sparse latent space directly from multi-view images using photometric supervision via differentiable 3D Gaussian Splatting. Given a partial set of observed voxels encoded from sparse input views, scene completion reduces to predicting the missing latent tokens and their spatial support within the voxel grid. To this end, we train a masked autoregressive transformer that jointly models voxel occupancy and latent token values, enabling efficient and spatially consistent generation of unseen regions. We demonstrate the effectiveness of our method on synthetic indoor scenes, achieving higher novel-view quality than prior work. We further validate its generalization on RealEstate10k, highlighting its applicability to real-world data.

Abstract

Image-to-3D Scene Generation

Explore similar work

Sparse auto-regressive modeling for scene generation from multi-view images

FILIGREE3D: Scaling Sparse Latent Flow Matching for Ultra-High-Resolution Image-to-3D Generation

ROAR-3D: Routing Arbitrary Views for High-Fidelity 3D Generation