Latent 3D representations have significantly expanded the capabilities of modern shape modeling, enabling compact encoding of 3D scenes and supporting powerful 3D generative models. Despite this progress, current representations struggle to capture the fine details of complex objects, fundamentally limited by the capacity of their latents. However, scaling latent capacity at training time is computationally expensive and often infeasible under memory limits, and naively increasing the number of latents at inference fails to make effective use of the added capacity. In this paper, we address both challenges with split-and-scale, a method that effectively expands latent capacity at inference while keeping training cost tractable. Our key observation is that by representing 3D scenes as continuous functions of 3D coordinates, a single pretrained tokenizer can be applied at inference to arbitrarily small sub-regions of a scene, allocating the same latent capacity to a smaller spatial extent and thus capturing far higher detail. We validate this insight by conducting extensive ablations to study the trade-offs between compute, memory, latent capacity, and reconstruction quality, demonstrating our inference-time scaling method achieves quality competitive with much larger models that exceed single-GPU memory limits. We further demonstrate that split-and-scale supports high-quality generative modeling, training an autoregressive 3D generator that performs competitively with state-of-the-art baselines.
Figures & tables
Figure 1: Split-and-scale reuses the same tokenizer across spatial scales to effectively increase latent capacity at inference time. Recommend viewing the figure on monitor, zoomed in. Asset credit ( Cherchye, 2016 ) .
Figure 2: Naive scaling vs . split-and-scale. (Left) Naive scaling increases number of latents by using more queries to the tokenizer. As the tokenizer is not trained with the higher number of queries, the quality improvement is limited. (Right) Split-and-scale encodes/decodes subregions of the scene with the same tokenizer/decoder and the designed number of queries, effectively increasing latent capacity. Asset credit ( Studio, 2022 ) .
Figure 3: Quantitative comparison versus naive token count scaling. We train a split-and-scale model that outputs 1024 latents per voxel. We compare with LiTo-1024, LiTo-8192, LiTo-32768 that use 1024, 8192, 32768 tokens during training, respectively. We also evaluate these models with naive inference-time scaling. Note that training LiTo-32768 requires larger memory than all other methods, and its training takes 2.7× more time. All models are trained to the same number of iterations. As shown, the performance of the proposed method (blue curve) increases consistently as we increase number of tokens during inference, whereas naive scaling quickly hits bottleneck.
Figure 4: Visualization of our proposed generation procedure. To reduce memory requirements, we propose autoregressive voxel-based generation with flow matching-based sampling and cross-attention conditioning. Given an input image, we start by first generating coarse 43 occupancy. For the occupied voxels, we then autoregressively sample voxel latents conditioned on previously-generated latents, until all voxels are filled. Asset credit ( Studio, 2022 ) .
Figure 5: Qualitative reconstruction comparison. For comparison, we visualize a LiTo model trained with 8192 tokens (“LiTo-8192”), that can be trained under similar compute limits as our split-and-scale model. Note that the metrics for TRELLIS and TRELLIS-2 are quite poor due to color shift. Asset credits ( okajima, 2019 ; MrsM, 2020 ; andrea.notarstefano, 2020 ) .
Figure 6: Qualitative generation comparison. A generative model trained on split-and-scale can reproduce fine details. In contrast, competitor methods suffer from either poor detail or poor material estimation. Asset credits ( wribeast, 2020 ; slasmir, 2020 ; bobrik00, 2020 ) .
Figure 7: Failure cases. (a) In tokenization, because each voxel is independently processed, subtle seams can appear in areas with significant view-dependent appearance. (b) Because our proposed generation model is autoregressive, sometimes it can experience the structural flaws common to such models. This manifests in the form of repeated geometry or poor symmetry. Asset credits ( 3Dji, 2025 ; Sohaibakr, 2019 ; Nikkip, 2021 ) .
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
Table 4: Ablation study. We ablate our proposed designs for reconstruction as well as generation. In (b), both LiTo-32768 variants cannot be trained on an 80GB GPU, for both the representation and generative model.
Figure 8: Quantitative geometry comparison versus naive token count scaling. We compare the quality of sampled point clouds from split-and-scale versus comparable LiTo models ( Chang et al., 2026 ) .
Figure 9: Quantitative comparison versus naive token count scaling. Following the recipe in Section 3.2 , we train a split-and-scale model that outputs 1024 latents per voxel at inference. For comparison, we train 1024 and 8192 token LiTo models ( Chang et al., 2026 ) that use a similar amount of memory/compute during training. In both (a) equal compute time and (b) equal iteration scenarios, split-and-scale can be seen to scale with token count better than the baseline models. Although it cannot be trained within the memory limits of an 80GB GPU, we include a 32768 token variant of LiTo ( Chang et al., 2026 ) as a performance upper-bound; split-and-scale is still competitive.
Figure 10: Upper-bound qualitative reconstruction comparison. Building off Figure 5 , although it cannot be trained within the memory limits of an 80GB GPU, we show a 32768 token variant of LiTo (“LiTo-32768”) as an upper-bound for performance; split-and-scale is still competitive. Asset credits ( okajima, 2019 ; MrsM, 2020 ; andrea.notarstefano, 2020 ) .
Method
LPIPS
PSNR
original TRELLIS ( Xiang et al., 2025 )
0.212
20.86
TRELLIS, n=2
0.254
19.63
TRELLIS, n=2 + overlapping
0.284
18.69
TRELLIS, n=4
0.295
17.86
Appendix
Table 6: TRELLIS tiling experiment. We evaluate whether the vanilla TRELLIS ( Xiang et al., 2025 ) tokenizer can be applied in a similar tile-based fashion as split-and-scale. Quantitatively, performance can be seen to drop with the amount of tiling, even with overlapping reconstructions across different voxels.
We present SparseGen, a novel framework for efficient image-to-3D generation, which exhibits low input-view bias while being significantly faster. Unlike traditional approaches that rely on dense volumetric grids, triplanes, or pixel-aligned primitives, we model scenes with a compact sparse set of learned 3D anchor queries and a learned expansion operator that decodes each transformed query into a small local set of 3D Gaussian primitives. Trained under a rectified-flow reconstruction objective without 3D supervision, our model learns to allocate representation capacity where geometry and appearance matter, achieving significant reductions in memory and inference time while preserving multi-view fidelity. We introduce quantitative measures of input-view bias and utilization to show that sparse queries reduce overfitting to conditioning views while being representationally efficient. Our results argue that sparse set-latent expansion is a principled, practical alternative for efficient 3D generative modeling.
Scene-level 3D generation has long been dominated by 2D multi-view or video diffusion models. Typically, these 2D-based approaches perform scene generation in a 2D latent space, which introduces two fundamental issues: (i) representing 3D scenes via 2D views leads to significant representation redundancy, and (ii) latent space rooted in 2D inherently limits the spatial consistency of the generated scenes. In this paper, we propose to perform 3D scene generation directly within a high-dimensional implicit 3D latent space derived from powerful 2D foundation models (e.g., DINOv2, SigLIP2, DA3). Then we tame diffusion transformers to perform diffusion modeling directly in the 3D latent space, enabling 3D-grounded scene generation. In our experiments, we not only demonstrate the advantages of our method over 2D-based approaches in terms of efficiency and spatial consistency, but also comprehensively analyze the impact of 3D latent representation on 3D scene generation performance. Furthermore, we validate the favorable scalability of our method with respect to representation and model capacity.
Dongxu Wei, Qi Xu, Zhiqi Li +6
Westlake University · Afari Intelligent Drive · Zhejiang University
Recent video world models have increasingly adopted 3D memory mechanisms to integrate historical observations into a lightweight and persistent scene representation. This naturally raises a complementary question: if 3D representations can serve as an effective mechanism for memorizing observed scenes, can they also be directly used as the latent space for reconstructing and generating scenes? Motivated by this idea, we represent a scene as a compact and unstructured 3D-anchored latent space (SceneAE). Each latent combines an explicit 3D position with an implicit content feature and is spatially grounded to describe its local scene region, forming a sparse latent point cloud. Unlike conventional video-VAE latents that describe how a scene appears from particular viewpoints, our representation directly describes the scene itself, making it naturally applicable to both reconstruction and generation. For reconstruction, once the scene has been generated, our persistent 3D representation can be projected and decoded into RGB-D observations by different target cameras, without the need to regenerate a video for each new camera trajectory. For generation, we leverage the geometry-content separation to first generate camera-controlled 3D geometry and then synthesize content conditioned on it, reducing camera-appearance entanglement and improving camera controllability. Experiments show that this 3D latent achieves superior generation quality, computational cost, and camera controllability.
Shuofeng Sun, Wenzhao Zheng, Yunpeng Zhang +3
Beijing University of Posts and Telecommunications · University of California, Berkeley · PhiGent Robotics +1