We present SceneTok, a novel tokenizer for encoding view sets of scenes into a compressed and diffusable set of unstructured tokens. Existing approaches for 3D scene representation and generation commonly use 3D data structures or view-aligned fields. In contrast, we introduce the first method that encodes scene information into a small set of permutation-invariant tokens that is disentangled from the spatial grid. The scene tokens are predicted by a multi-view tokenizer given many context views and rendered into novel views by employing a light-weight rectified flow decoder. We show that the compression is 1-3 orders of magnitude stronger than for other representations while still reaching state-of-the-art reconstruction quality. Further, our representation can be rendered from novel trajectories, including ones deviating from the input trajectory, and we show that the decoder gracefully handles uncertainty. Finally, the highly-compressed set of unstructured latent scene tokens enables simple and efficient scene generation in 5 seconds, achieving a much better quality-speed trade-off than previous paradigms.
Figures & tables
Figure 2: Method Overview. (a) The SceneTok autoencoder encodes view sets into a set of compressed, unstructured scene tokens by chaining a VA-VAE image compressor and a perceiver module. The tokens can be rendered from novel views with a generative decoder based on rectified flows. (b) A latent diffusion transformer can perform scene generation by generating compressed scene tokens. Scene generation can be conditioned on a single or a few images and a set of anchor poses, defining the spatial scene extent.
Figure 3: A single scene perceiver block. The scene perceiver module consists of L blocks, alternating self-attention between scene queries Q and cross-attention to the multi-view encoder branch, which is AdaLN-modulated with ray embeddings.
12 Context Views
5 Context Views
Repr. Size (#Floats)
PSNR ↑
LPIPS ↓
SSIM ↑
rFVD ↓
rFID ↓
Repr. Size (#Floats)
PSNR ↑
LPIPS ↓
SSIM ↑
rFVD ↓
rFID ↓
Explicit Representation
MVSplat [ 9 ]
46.40M
20.13
0.255
0.755
354.46
29.89
19.33M
22.82
0.152
0.856
144.91
15.85
MVSplat360 [ 10 ]
74.71M
21.09
0.247
0.769
294.82
25.64
31.12M
23.88
0.174
0.828
136.52
18.15
DepthSplat [ 56 ]
46.40M
21.55
0.202
0.809
204.52
21.35
19.33M
24.45
0.124
0.884
109.74
12.89
Latent Representation
Table 1: Quantitative comparison of NVS quality on RE10K. SceneTok achieves superior performance compared to previous works achieving significant improvements in terms of rFVD and rFID alongside with PSNR and LPIPS while having a significantly smaller representation size.
Figure 4: Qualitative NVS. ( , , ) denote the context views (only four is shown), the target rendering of SceneTok and the ground-truth target view (only one is shown) respectively. While the baselines suffer from blur artifacts even without compression, SceneTok produces cleaner renderings and better details from a small set of scene tokens. More examples in Appendix Sec. D .
DL3DV-140
Zero-Shot ACID
Repr. Size (#Floats)
PSNR ↑
LPIPS ↓
SSIM ↑
rFVD ↓
rFID ↓
Repr. Size (#Floats)
PSNR ↑
LPIPS ↓
SSIM ↑
rFVD ↓
rFID ↓
Requires Target Views
RayZer [ 18 ]
2.36M
26.89
0.154
0.834
87.41
14.75
-
-
-
-
-
-
Explicit Representation
MVSplat360 [ 10 ]
174.32M
18.54
0.278
0.629
314.28
21.95
74.71M
21.75
0.280
0.701
294.86
43.24
DepthSplat [ 56 ]
108.27M
21.86
0.192
0.769
423.91
20.90
46.40M
21.52
0.216
0.772
351.97
29.20
Table 2: Quantitative comparison of NVS quality on DL3DV-140 and ACID : SceneTok achieves better performance in most metrics as compared to baselines with explicit and latent representations on DL3DV-140 and superior generalization on zero-shot ACID. Note that the models used in ACID were trained on RealEstate10K (c.f. Tab. 1 ).
R. Acc. ↑
T. Acc. ↑
AUC ↑
10∘
20∘
30∘
10∘
20∘
30∘
10∘
20∘
30∘
RayZer [ 18 ]
26.16
42.50
54.45
07.82
23.75
41.59
0.007
0.020
0.096
LVSM [ 19 ]
48.07
65.60
74.56
14.25
33.02
49.60
0.032
0.103
0.178
SceneTok
75.81
86.47
89.86
62.44
80.31
86.48
0.307
0.492
0.593
Table 3: Transferability on scene pairs from DL3DV-140. We compare the transferability of novel camera trajectories by swapping target poses of scene pairs and quantify using TPS [ 33 ] .
MEt3RRE10K↓
MEt3RDL3DV↓
MEt3RACID↓
MVSplat360 [ 10 ]
0.0220
0.0579
0.0196
LVSM [ 19 ]
0.0162
0.0541
0.0126
SceneTok
0.0149
0.0538
0.0133
Table 4: Multi-View Consistency. SceneTok achieves good pairwise consistency with MEt3R [ 1 ] across all dataset.
Figure 5: Qualitative single-view generation comparison. ( , , ) represents the input view, the rendered target views from the generated scene, and the ground-truth views respectively. DFM produces blurry results and struggles to generate complete structures. Although our method falls short in generating high-quality, high-frequency details when compared to DFoT, which performs pixel-space diffusion, and the foundational generative model SEVA, it is significantly more efficient (c.f. Tab. 5 ). More examples in Appendix Sec. D .
Figure 6: RGB and per-pixel variance vs mask-ratio. We show a single RGB rendering with its variance map (over 10 samples) on increasing mask-ratios of 0→100% . Note that we scale the variance map by a factor of 1.5.
PSNR ↑
LPIPS ↓
SSIM ↑
gFVD ↓
gFID ↓
Inference (s) ↓
MEt3R ↓
DFM [ 48 ]
15.86
0.468
0.529
566.71
52.64
630
0.0225
DFoT [ 44 ]
14.76
0.433
0.469
220.36
35.40
146
0.0194
SEVA [ 71 ]
14.41
0.442
0.450
133.00
17.69
1620
0.0164
SceneGen
15.12
0.415
0.493
157.89
18.90
26 (11 + 16)
0.0156
Table 5: Single-View Generation Comparison. We test SceneGen and the baselines on 200 scenes, each with 192 frames. SceneGen performs comparably to previous works in metrics but is significantly faster in generation ( ≈11s ).
Figure 7: Depth decoding on frozen SceneTok on DL3DV-140. We show two scenes with the pseudo-GT from UniDepthV2 [ 39 ] and the predicted depth renderings with our finetuned depth decoder along with depth uncertainty and the RGB rendering. More examples are provided in Fig. 16 in the appendix.
Figure 8: RGB and per-pixel variance vs context views. ( , , ) denote different target views and denotes the 12 context views. As we increase n , information in the tokens increases, and the variance reduces accordingly for the newly observed regions. Note that we scale the variance map by a factor of 2.
Appendix figures & tables27 assets
Supplementary material from the paper’s appendix.
Appendix
Model Configurations
Encoder
Decoder
Overall size
242M
595M
Hidden dimensions dH
768
1024
Depth L
12
24
Number of heads
12
16
MLP ratio
4.0
4.0
2D RoPE
✓
✗
Appendix
Table 6: Summary of SceneTok . We show the model, input, training and sampling our scene autoencoder. Both the encoder and the decoder using the transformer modules from LightningDiT [ 58 ] with additional modifications (c.f. Fig. 3 and Sec. 3 ).
Figure 9: Decoder block of SceneTok .
Figure 10: Noise-level shift. We plot the resulting timestep t^ for different amount of s . For s>1 , we shift more towards higher noise and vice versa.
Model Configurations
Overall size
1.3B
Hidden dimensions dH
1536
Depth L
24
Number of heads
24
MLP ratio
4.0
Norm type
RMSNorm
Appendix
Table 7: Summary of SceneGen . We show the model, input, training and sampling configurations for both versions of latent scene generation model. We use the transformer module from LightningDiT [ 58 ] with an additional cross attention module for input conditioning (c.f. Sec. 3.4 ).
Figure 11: UMAP plot of scene tokens across 8 scenes from RealEstate10K. We compute the UMAP across all 512 tokens for 8 different scenes out of which 4 are outdoor ( + ) and indoor ( ∙ ) scenes (c.f. Fig 12 ). We see that the majority of the tokens that belongs to the outdoor scenes are grouped closely together as compared to tokens from indoor scenes. This shows that our latent token space is discriminative in nature which is helpful for downstream generation task [ 59 ] .
Figure 12: Examples of outdoor and indoor scenes. ( , ) denote outdoor and indoor scenes respectively. We show the RGB renderings of the scenes used in Fig. 11 . Note that the scenes are ordered from top-to-bottom with indices 0→7 .
Figure 13: Comparison of 3D vs 2D RoPE in the encoder. denotes the input context views, ( , ) denotes the rendered targets with 3D RoPE [ 30 ] and 2D RoPE [ 45 ] respectively. We show 12 target views (RGB and variance map) from left-to-right where the first 6 columns shows the rendered views for the forward tajectory which aligns with the input order of the context and the remaining 6 views shows the backward trajectory. We observe higher variance for the backward trajectory for 3D RoPE and this indicates temporal bias of the scene tokens Z . Note that we scale our variance map by a factor of 2.
Compression (Spatial, Temp.)
PSNR ↑
LPIPS ↓
SSIM ↑
rFVD ↓
rFID ↓
VideoDCAE [ 38 ]
×32 , ×4
21.95
0.219
0.660
246.51
19.12
WanVAE [ 50 ] )
×16 ,( ×4+1)
19.81
0.254
0.561
189.74
14.30
Appendix
Table 8: Ablation on pretrained Video VAEs in SceneTok decoder. We achieve better high frequency details with the SceneTok renderer trained with WanVAE [ 50 ] . This partly due to smaller spatial compression as compared to VideoDCAE. From this, we conclude that SceneTok performance is bounded by the performance of the video VAEs and therefore, exploring better alternatives can potentially improve SceneTok .
Guidance Scale
PSNR ↑
LPIPS ↓
SSIM ↑
rFVD ↓
rFID ↓
1.0
24.22
0.157
0.784
68.39
9.13
2.0
24.35
0.151
0.792
69.52
9.69
3.0
24.05
0.158
0.787
79.60
11.25
4.0
23.71
0.166
0.778
89.11
12.75
5.0
23.49
0.174
0.771
101.24
14.19
6.0
23.28
0.183
0.764
112.04
16.00
Appendix
Table 9: Ablation on CFG scales in SceneTok . We show the quantitative results for increasing guidance scale when rendering novel views. The experiment is performed on the wide-view baseline (c.f. Sec. 4.1 ) using 12 context views and 128 target views. We clearly see that using a smaller guidance scale performs better.
Figure 14: PSNR vs Number of Tokens on DL3DV. We train four variant of our method on different number of tokens while keeping the channel dimensions fixed.
Timestep Shift (s)
gFID ↓
1.0
19.99
4.0
19.04
12.0
18.90
Appendix
Table 10: Ablation on Timestep Shift in SceneGen . We show the gFID three different timestep shifts where we train a separate model for each shift amount and also apply it during inference. We found that shifting towards higher noise results in better global structures as more steps are allocated in the low frequency regime.
Figure 15: Average per-pixel variance. We plot the per-pixel variance averaged across the spatial and temporal dimension against increasing number of context views and masked tokens (out of 512 tokens). We show the plot for a single example scene.
Figure 16: Depth decoding on frozen SceneTok on DL3DV-140. We show two scenes with the pseudo-GT from UniDepthV2 [ 39 ] and the predicted depth renderings with our finetuned depth decoder along with depth uncertainty and the RGB rendering.
Figure 17: Qualitative NVS on Transfer Trajectories. ( , ) denotes the context views of the source scene, and the renderings of the source scene with trajectory from the transfer scene with SceneTok . Note that we can expect artifacts in regions that are not present in the context views and both SceneTok , and LVSM [ 19 ] are not trained for extrapolation. RayZer on the other hand fails to follow the given trajectory and only renders the original trajectory of the source scene.
Figure 18: Qualitative NVS on Transfer Trajectories. ( , ) denotes the context views of the source scene, and the renderings of the source scene with trajectory from the transfer scene with SceneTok . Note that we can expect artifacts in regions that are not present in the context views and both SceneTok , and LVSM [ 19 ] are not trained for extrapolation. RayZer on the other hand fails to follow the given trajectory and only renders the original trajectory of the source scene.
Figure 19: Qualitative NVS on RealEstate10K. ( , , ) denotes the 5 input context views, the target renderings from SceneTok and the corresponding ground-truth target views respectively (we show six views from top-to-bottom). * denotes output of MVSplat360 taken before the refinement step with a video diffusion model.
Figure 20: Qualitative NVS on RealEstate10K. ( , , ) denotes the 12 input context views, the target renderings from SceneTok and the corresponding ground-truth target views respectively (we show six views from top-to-bottom). * denotes output of MVSplat360 taken before the refinement step with a video diffusion model.
Figure 21: Qualitative NVS on RealEstate10K. ( , , ) denotes the 12 input context views, the target renderings from SceneTok and the corresponding ground-truth target views respectively (we show six views from top-to-bottom). * denotes output of MVSplat360 taken before the refinement step with a video diffusion model.
Figure 22: Qualitative NVS on RealEstate10K. ( , , ) denotes the 12 input context views, the target renderings from SceneTok and the corresponding ground-truth target views respectively (we show six views from top-to-bottom). * denotes output of MVSplat360 taken before the refinement step with a video diffusion model.
Figure 23: Qualitative NVS on DL3DV-140. ( , , ) denotes the input context views (only 8 are shown), the target renderings from SceneTok (* denote decoder version trained with WanVAE [ 50 ] ) and the corresponding ground-truth target views respectively (we show eight views from top-to-bottom).
Figure 24: Qualitative NVS on DL3DV-140. ( , , ) denotes the input context views (only 8 are shown), the target renderings from SceneTok (* denote decoder version trained with WanVAE [ 50 ] ) and the corresponding ground-truth target views respectively (we show eight views from top-to-bottom).
Figure 25: Qualitative NVS on DL3DV-140. ( , , ) denotes the input context views (only 8 are shown), the target renderings from SceneTok (* denote decoder version trained with WanVAE [ 50 ] ) and the corresponding ground-truth target views respectively (we show eight views from top-to-bottom).
Figure 26: Qualitative NVS on DL3DV-140. ( , , ) denotes the input context views (only 8 are shown), the target renderings from SceneTok (* denote decoder version trained with WanVAE [ 50 ] ) and the corresponding ground-truth target views respectively (we show eight views from top-to-bottom).
Figure 27: Qualitative NVS on DL3DV-140. ( , , ) denotes the input context views (only 8 are shown), the target renderings from SceneTok (* denote decoder version trained with WanVAE [ 50 ] ) and the corresponding ground-truth target views respectively (we show eight views from top-to-bottom).
Figure 28: Qualitative NVS on DL3DV-140. ( , , ) denotes the input context views (only 8 are shown), the target renderings from SceneTok (* denote decoder version trained with WanVAE [ 50 ] ) and the corresponding ground-truth target views respectively (we show eight views from top-to-bottom).
Figure 29: Qualitative Single-View Generation Comparison. ( , , ) denotes the input view, the target renderings from SceneGen and the corresponding ground-truth target views respectively.
Figure 30: Qualitative Single-View Generation Comparison. ( , , ) denotes the input view, the target renderings from SceneGen and the corresponding ground-truth target views respectively.
We present CoTL, a framework that learns a compact set of tokens that preserves scene geometry and appearance from unposed multi-view images. Our encoder compresses the input images into small per-view sets of latent tokens that aggregate information across views to form a view-consistent scene representation. Each additional view expands the representation with only a small, resolution-independent set of tokens, allowing its capacity to grow with the available observations. To force this compact bottleneck to retain sufficient scene information, a query-based Gaussian decoder uses scene-independent queries to attend to the learned tokens and recover an explicit, renderable set of 3D Gaussian primitives. Novel-view rendering supervision trains the compact representation end to end through the reconstructed Gaussian scene. Experiments on RealEstate10K and ACID show that the resulting compact, unstructured token set supports competitive novel-view synthesis and can serve as the latent space of a generative model for 3D Gaussian scene generation. Together, these results show that 3D Gaussian reconstruction is an effective objective for compact scene representation learning and the learned token space provides a basis for broader downstream tasks in 3D vision.
3D scene generation has long been dominated by 2D multi-view or video diffusion models. This is due not only to the lack of scene-level 3D latent representation, but also to the fact that most scene-level 3D visual data exists in the form of multi-view images or videos, which are naturally compatible with 2D diffusion architectures. Typically, these 2D-based approaches degrade 3D spatial extrapolation to 2D temporal extension, which introduces two fundamental issues: (i) representing 3D scenes via 2D views leads to significant representation redundancy, and (ii) latent space rooted in 2D inherently limits the spatial consistency of the generated 3D scenes. In this paper, we propose, for the first time, to perform 3D scene generation directly within an implicit 3D latent space to address these limitations. First, we repurpose frozen 2D representation encoders to construct our 3D Representation Autoencoder (3DRAE), which grounds view-coupled 2D semantic representations into a view-decoupled 3D latent representation. This enables representing 3D scenes observed from arbitrary numbers of views--at any resolution and aspect ratio--with fixed complexity and rich semantics. Then we introduce 3D Diffusion Transformer (3DDiT), which performs diffusion modeling in this 3D latent space, achieving remarkably efficient and spatially consistent 3D scene generation while supporting diverse conditioning configurations. Moreover, since our approach directly generates a 3D scene representation, it can be decoded to images and optional point maps along arbitrary camera trajectories without requiring per-trajectory diffusion sampling pass, which is common in 2D-based approaches.
Dongxu Wei, Qi Xu, Zhiqi Li +6
Westlake University · Afari Intelligent Drive · Zhejiang University
Compact token sequences are essential for efficient 3D generation. However, existing 3D tokenizers typically organize latent representations either over spatial regions or as fixed-size sets of global tokens, both suffering sharp reconstruction degradation when compressed to extremely low token budgets. In this paper, we present ZipTok3D, a 3D tokenizer designed for high-fidelity reconstruction from extremely short token sequences. Its key idea is to organize object geometry into progressively informative global-token prefixes and unfold these compact representations through iterative decoding. Specifically, nested dropout randomly truncates the latent sequence after encoding during training and requires each retained prefix to reconstruct the complete object, thereby prioritizing essential geometric information in the leading tokens. The decoder then repeatedly applies a parameter-shared Transformer block to recover fine-grained geometry from each prefix without a separate generative sampling stage. With the same token dimension, ZipTok3D achieves reconstruction quality comparable to the 32-token COD-VAE baseline using only one token on ShapeNet and four on TRELLIS, yielding 32× and 8× shorter token sequences, respectively.
Mingda Lin, Weijie Wang, Zeyu Zhang +7
1Zhejiang University · 2Monash University · University of Adelaide