SceneTok: A Compressed, Diffusable Token Space for 3D Scenes

Mohammad Asim; Christopher Wewer; Jan Eric Lenssen

cs.CVFeb 21, 2026

SceneTok: A Compressed, Diffusable Token Space for 3D Scenes

Authors: Mohammad Asim, Christopher Wewer, Jan Eric Lenssen

Organizations: Max Planck Institute for Informatics, Saarland Informatics Campus

Abstract

We present SceneTok, a novel tokenizer for encoding view sets of scenes into a compressed and diffusable set of unstructured tokens. Existing approaches for 3D scene representation and generation commonly use 3D data structures or view-aligned fields. In contrast, we introduce the first method that encodes scene information into a small set of permutation-invariant tokens that is disentangled from the spatial grid. The scene tokens are predicted by a multi-view tokenizer given many context views and rendered into novel views by employing a light-weight rectified flow decoder. We show that the compression is 1-3 orders of magnitude stronger than for other representations while still reaching state-of-the-art reconstruction quality. Further, our representation can be rendered from novel trajectories, including ones deviating from the input trajectory, and we show that the decoder gracefully handles uncertainty. Finally, the highly-compressed set of unstructured latent scene tokens enables simple and efficient scene generation in 5 seconds, achieving a much better quality-speed trade-off than previous paradigms.

Figures & tables

Panel of Figure 2 — Figure 2: Method Overview. (a) The SceneTok autoencoder encodes view sets into a set of compressed, unstructured scene tokens by chaining a VA-VAE image compressor and a perceiver module. The tokens can be rendered from novel views with a generative decoder based on rectified flows. (b) A latent diffusion transformer can perform scene generation by generating compressed scene tokens. Scene generation can be conditioned on a single or a few images and a set of anchor poses, defining the spatial scene extent.

Figure 3: A single scene perceiver block. The scene perceiver module consists of

L

blocks, alternating self-attention between scene queries

\mathbf{Q}

and cross-attention to the multi-view encoder branch, which is AdaLN-modulated with ray embeddings.

	12 Context Views						5 Context Views
	Repr. Size (#Floats)	PSNR $\uparrow$	LPIPS $\downarrow$	SSIM $\uparrow$	rFVD $\downarrow$	rFID $\downarrow$	Repr. Size (#Floats)	PSNR $\uparrow$	LPIPS $\downarrow$	SSIM $\uparrow$	rFVD $\downarrow$	rFID $\downarrow$
Explicit Representation
MVSplat [ 9 ]	46.40M	20.13	0.255	0.755	354.46	29.89	19.33M	22.82	0.152	0.856	144.91	15.85
MVSplat360 [ 10 ]	74.71M	21.09	0.247	0.769	294.82	25.64	31.12M	23.88	0.174	0.828	136.52	18.15
DepthSplat [ 56 ]	46.40M	21.55	0.202	0.809	204.52	21.35	19.33M	24.45	0.124	0.884	109.74	12.89
Latent Representation

Table 1: Quantitative comparison of NVS quality on RE10K. SceneTok achieves superior performance compared to previous works achieving significant improvements in terms of rFVD and rFID alongside with PSNR and LPIPS while having a significantly smaller representation size.

Panel of Figure 4 — Figure 4: Qualitative NVS. $($ , , $)$ denote the context views (only four is shown), the target rendering of SceneTok and the ground-truth target view (only one is shown) respectively. While the baselines suffer from blur artifacts even without compression, SceneTok produces cleaner renderings and better details from a small set of scene tokens. More examples in Appendix Sec. D .

	DL3DV-140						Zero-Shot ACID
	Repr. Size (#Floats)	PSNR $\uparrow$	LPIPS $\downarrow$	SSIM $\uparrow$	rFVD $\downarrow$	rFID $\downarrow$	Repr. Size (#Floats)	PSNR $\uparrow$	LPIPS $\downarrow$	SSIM $\uparrow$	rFVD $\downarrow$	rFID $\downarrow$
Requires Target Views
RayZer [ 18 ]	2.36M	26.89	0.154	0.834	87.41	14.75	-	-	-	-	-	-
Explicit Representation
MVSplat360 [ 10 ]	174.32M	18.54	0.278	0.629	314.28	21.95	74.71M	21.75	0.280	0.701	294.86	43.24
DepthSplat [ 56 ]	108.27M	21.86	0.192	0.769	423.91	20.90	46.40M	21.52	0.216	0.772	351.97	29.20

Table 2: Quantitative comparison of NVS quality on DL3DV-140 and ACID : SceneTok achieves better performance in most metrics as compared to baselines with explicit and latent representations on DL3DV-140 and superior generalization on zero-shot ACID. Note that the models used in ACID were trained on RealEstate10K (c.f. Tab. 1 ).

	R. Acc. $\uparrow$			T. Acc. $\uparrow$			AUC $\uparrow$
	$10^{\circ}$	$20^{\circ}$	$30^{\circ}$	$10^{\circ}$	$20^{\circ}$	$30^{\circ}$	$10^{\circ}$	$20^{\circ}$	$30^{\circ}$
RayZer [ 18 ]	26.16	42.50	54.45	07.82	23.75	41.59	0.007	0.020	0.096
LVSM [ 19 ]	48.07	65.60	74.56	14.25	33.02	49.60	0.032	0.103	0.178
SceneTok	75.81	86.47	89.86	62.44	80.31	86.48	0.307	0.492	0.593

Table 3: Transferability on scene pairs from DL3DV-140. We compare the transferability of novel camera trajectories by swapping target poses of scene pairs and quantify using TPS [ 33 ] .

	$\text{MEt3R}_{RE10K}\downarrow$	$\text{MEt3R}_{DL3DV}\downarrow$	$\text{MEt3R}_{ACID}\downarrow$
MVSplat360 [ 10 ]	0.0220	0.0579	0.0196
LVSM [ 19 ]	0.0162	0.0541	0.0126
SceneTok	0.0149	0.0538	0.0133

Table 4: Multi-View Consistency. SceneTok achieves good pairwise consistency with MEt3R [ 1 ] across all dataset.

Figure 5: Qualitative single-view generation comparison. $($ , , $)$ represents the input view, the rendered target views from the generated scene, and the ground-truth views respectively. DFM produces blurry results and struggles to generate complete structures. Although our method falls short in generating high-quality, high-frequency details when compared to DFoT, which performs pixel-space diffusion, and the foundational generative model SEVA, it is significantly more efficient (c.f. Tab. 5 ). More examples in Appendix Sec. D .

Figure 6: RGB and per-pixel variance vs mask-ratio. We show a single RGB rendering with its variance map (over 10 samples) on increasing mask-ratios of $0\rightarrow 100\%$ . Note that we scale the variance map by a factor of 1.5.

	PSNR $\uparrow$	LPIPS $\downarrow$	SSIM $\uparrow$	gFVD $\downarrow$	gFID $\downarrow$	Inference (s) $\downarrow$	MEt3R $\downarrow$
DFM [ 48 ]	15.86	0.468	0.529	566.71	52.64	630	0.0225
DFoT [ 44 ]	14.76	0.433	0.469	220.36	35.40	146	0.0194
SEVA [ 71 ]	14.41	0.442	0.450	133.00	17.69	1620	0.0164
SceneGen	15.12	0.415	0.493	157.89	18.90	26 (11 + 16)	0.0156

Table 5: Single-View Generation Comparison. We test SceneGen and the baselines on 200 scenes, each with 192 frames. SceneGen performs comparably to previous works in metrics but is significantly faster in generation (

\approx 11s

Figure 7: Depth decoding on frozen SceneTok on DL3DV-140. We show two scenes with the pseudo-GT from UniDepthV2 [ 39 ] and the predicted depth renderings with our finetuned depth decoder along with depth uncertainty and the RGB rendering. More examples are provided in Fig. 16 in the appendix.

Figure 8: RGB and per-pixel variance vs context views. $($ , , $)$ denote different target views and denotes the 12 context views. As we increase $n$ , information in the tokens increases, and the variance reduces accordingly for the newly observed regions. Note that we scale the variance map by a factor of 2.

Appendix figures & tables27 assets

Supplementary material from the paper’s appendix.

Appendix

Model Configurations	Encoder	Decoder
Overall size	$242M$	$595M$
Hidden dimensions $d_{H}$	$768$	$1024$
Depth $L$	$12$	$24$
Number of heads	$12$	$16$
MLP ratio	$4.0$	$4.0$
2D RoPE	✓	✗

Appendix

Table 6: Summary of SceneTok . We show the model, input, training and sampling our scene autoencoder. Both the encoder and the decoder using the transformer modules from LightningDiT [ 58 ] with additional modifications (c.f. Fig. 3 and Sec. 3 ).

Figure 9: Decoder block of SceneTok .

Figure 10: Noise-level shift. We plot the resulting timestep

\hat{t}

for different amount of

s

. For

s>1

, we shift more towards higher noise and vice versa.

Model Configurations
Overall size	$1.3B$
Hidden dimensions $d_{H}$	$1536$
Depth $L$	$24$
Number of heads	$24$
MLP ratio	$4.0$
Norm type	RMSNorm

Appendix

Table 7: Summary of SceneGen . We show the model, input, training and sampling configurations for both versions of latent scene generation model. We use the transformer module from LightningDiT [ 58 ] with an additional cross attention module for input conditioning (c.f. Sec. 3.4 ).

Figure 11: UMAP plot of scene tokens across 8 scenes from RealEstate10K. We compute the UMAP across all 512 tokens for 8 different scenes out of which 4 are outdoor (

+

) and indoor (

\bullet

) scenes (c.f. Fig 12 ). We see that the majority of the tokens that belongs to the outdoor scenes are grouped closely together as compared to tokens from indoor scenes. This shows that our latent token space is discriminative in nature which is helpful for downstream generation task [ 59 ] .

Figure 12: Examples of outdoor and indoor scenes. ( , ) denote outdoor and indoor scenes respectively. We show the RGB renderings of the scenes used in Fig. 11 . Note that the scenes are ordered from top-to-bottom with indices $0\rightarrow 7$ .

Figure 13: Comparison of 3D vs 2D RoPE in the encoder. denotes the input context views, $($ , $)$ denotes the rendered targets with 3D RoPE [ 30 ] and 2D RoPE [ 45 ] respectively. We show 12 target views (RGB and variance map) from left-to-right where the first 6 columns shows the rendered views for the forward tajectory which aligns with the input order of the context and the remaining 6 views shows the backward trajectory. We observe higher variance for the backward trajectory for 3D RoPE and this indicates temporal bias of the scene tokens $\mathcal{Z}$ . Note that we scale our variance map by a factor of 2.

	Compression (Spatial, Temp.)	PSNR $\uparrow$	LPIPS $\downarrow$	SSIM $\uparrow$	rFVD $\downarrow$	rFID $\downarrow$
VideoDCAE [ 38 ]	$\times 32$ , $\times 4$	21.95	0.219	0.660	246.51	19.12
WanVAE [ 50 ] )	$\times 16$ ,( $\times 4+1)$	19.81	0.254	0.561	189.74	14.30

Appendix

Table 8: Ablation on pretrained Video VAEs in SceneTok decoder. We achieve better high frequency details with the SceneTok renderer trained with WanVAE [ 50 ] . This partly due to smaller spatial compression as compared to VideoDCAE. From this, we conclude that SceneTok performance is bounded by the performance of the video VAEs and therefore, exploring better alternatives can potentially improve SceneTok .

Guidance Scale	PSNR $\uparrow$	LPIPS $\downarrow$	SSIM $\uparrow$	rFVD $\downarrow$	rFID $\downarrow$
1.0	24.22	0.157	0.784	68.39	9.13
2.0	24.35	0.151	0.792	69.52	9.69
3.0	24.05	0.158	0.787	79.60	11.25
4.0	23.71	0.166	0.778	89.11	12.75
5.0	23.49	0.174	0.771	101.24	14.19
6.0	23.28	0.183	0.764	112.04	16.00

Appendix

Table 9: Ablation on CFG scales in SceneTok . We show the quantitative results for increasing guidance scale when rendering novel views. The experiment is performed on the wide-view baseline (c.f. Sec. 4.1 ) using 12 context views and 128 target views. We clearly see that using a smaller guidance scale performs better.

Figure 14: PSNR vs Number of Tokens on DL3DV. We train four variant of our method on different number of tokens while keeping the channel dimensions fixed.

Timestep Shift (s)	gFID $\downarrow$
1.0	19.99
4.0	19.04
12.0	18.90

Appendix

Table 10: Ablation on Timestep Shift in SceneGen . We show the gFID three different timestep shifts where we train a separate model for each shift amount and also apply it during inference. We found that shifting towards higher noise results in better global structures as more steps are allocated in the low frequency regime.

Figure 15: Average per-pixel variance. We plot the per-pixel variance averaged across the spatial and temporal dimension against increasing number of context views and masked tokens (out of 512 tokens). We show the plot for a single example scene.

Figure 16: Depth decoding on frozen SceneTok on DL3DV-140. We show two scenes with the pseudo-GT from UniDepthV2 [ 39 ] and the predicted depth renderings with our finetuned depth decoder along with depth uncertainty and the RGB rendering.

Panel of Figure 17 — Figure 17: Qualitative NVS on Transfer Trajectories. $($ , ) denotes the context views of the source scene, and the renderings of the source scene with trajectory from the transfer scene with SceneTok . Note that we can expect artifacts in regions that are not present in the context views and both SceneTok , and LVSM [ 19 ] are not trained for extrapolation. RayZer on the other hand fails to follow the given trajectory and only renders the original trajectory of the source scene.

Panel of Figure 18 — Figure 18: Qualitative NVS on Transfer Trajectories. $($ , ) denotes the context views of the source scene, and the renderings of the source scene with trajectory from the transfer scene with SceneTok . Note that we can expect artifacts in regions that are not present in the context views and both SceneTok , and LVSM [ 19 ] are not trained for extrapolation. RayZer on the other hand fails to follow the given trajectory and only renders the original trajectory of the source scene.

Figure 19: Qualitative NVS on RealEstate10K. $($ , , $)$ denotes the 5 input context views, the target renderings from SceneTok and the corresponding ground-truth target views respectively (we show six views from top-to-bottom). * denotes output of MVSplat360 taken before the refinement step with a video diffusion model.

Figure 20: Qualitative NVS on RealEstate10K. $($ , , $)$ denotes the 12 input context views, the target renderings from SceneTok and the corresponding ground-truth target views respectively (we show six views from top-to-bottom). * denotes output of MVSplat360 taken before the refinement step with a video diffusion model.

Figure 21: Qualitative NVS on RealEstate10K. $($ , , $)$ denotes the 12 input context views, the target renderings from SceneTok and the corresponding ground-truth target views respectively (we show six views from top-to-bottom). * denotes output of MVSplat360 taken before the refinement step with a video diffusion model.

Figure 22: Qualitative NVS on RealEstate10K. $($ , , $)$ denotes the 12 input context views, the target renderings from SceneTok and the corresponding ground-truth target views respectively (we show six views from top-to-bottom). * denotes output of MVSplat360 taken before the refinement step with a video diffusion model.

Figure 23: Qualitative NVS on DL3DV-140. $($ , , $)$ denotes the input context views (only 8 are shown), the target renderings from SceneTok (* denote decoder version trained with WanVAE [ 50 ] ) and the corresponding ground-truth target views respectively (we show eight views from top-to-bottom).

Figure 24: Qualitative NVS on DL3DV-140. $($ , , $)$ denotes the input context views (only 8 are shown), the target renderings from SceneTok (* denote decoder version trained with WanVAE [ 50 ] ) and the corresponding ground-truth target views respectively (we show eight views from top-to-bottom).

Figure 25: Qualitative NVS on DL3DV-140. $($ , , $)$ denotes the input context views (only 8 are shown), the target renderings from SceneTok (* denote decoder version trained with WanVAE [ 50 ] ) and the corresponding ground-truth target views respectively (we show eight views from top-to-bottom).

Figure 26: Qualitative NVS on DL3DV-140. $($ , , $)$ denotes the input context views (only 8 are shown), the target renderings from SceneTok (* denote decoder version trained with WanVAE [ 50 ] ) and the corresponding ground-truth target views respectively (we show eight views from top-to-bottom).

Figure 27: Qualitative NVS on DL3DV-140. $($ , , $)$ denotes the input context views (only 8 are shown), the target renderings from SceneTok (* denote decoder version trained with WanVAE [ 50 ] ) and the corresponding ground-truth target views respectively (we show eight views from top-to-bottom).

Figure 28: Qualitative NVS on DL3DV-140. $($ , , $)$ denotes the input context views (only 8 are shown), the target renderings from SceneTok (* denote decoder version trained with WanVAE [ 50 ] ) and the corresponding ground-truth target views respectively (we show eight views from top-to-bottom).

Panel of Figure 29 — Figure 29: Qualitative Single-View Generation Comparison. $($ , , $)$ denotes the input view, the target renderings from SceneGen and the corresponding ground-truth target views respectively.

Panel of Figure 30 — Figure 30: Qualitative Single-View Generation Comparison. $($ , , $)$ denotes the input view, the target renderings from SceneGen and the corresponding ground-truth target views respectively.

3D Scene Representation

3D scene representation studies how to encode geometry, appearance, semantics, and motion for reconstruction, rendering, generation, and interaction.

See all 235 papers

CoTL: Compact Token Learning for 3D Scene Representation
Oct 13, 2026Han Liang, Sibo Wu, Anpei Chen +13D Scene Representation Multi-View 3D Reconstruction
TU Munich · MCML · Westlake University
Any 3D Scene is Worth 1K Tokens: 3D-Grounded Representation for Scene Generation at Scale
Apr 13, 2026Dongxu Wei, Qi Xu, Zhiqi Li +63D Scene Generation 3D Diffusion Models
Westlake University · Afari Intelligent Drive · Zhejiang University
ZipTok3D: High-Fidelity 3D Tokenization with Compact Token Prefixes
Sep 1, 2026Mingda Lin, Weijie Wang, Zeyu Zhang +73D Reconstruction 3D Representation Learning
1Zhejiang University · 2Monash University · University of Adelaide

Oct 13, 2026J/K move · Enter open · S save

CoTL: Compact Token Learning for 3D Scene Representation

Han Liang, Sibo Wu, Anpei Chen, Rüdiger Westermann

TU Munich · MCML · Westlake University

3D Scene Representation Multi-View 3D Reconstruction Novel View Synthesis

Open paper AskPDF arXiv

We present CoTL, a framework that learns a compact set of tokens that preserves scene geometry and appearance from unposed multi-view images. Our encoder compresses the input images into small per-view sets of latent tokens that aggregate information across views to form a view-consistent scene representation. Each additional view expands the representation with only a small, resolution-independent set of tokens, allowing its capacity to grow with the available observations. To force this compact bottleneck to retain sufficient scene information, a query-based Gaussian decoder uses scene-independent queries to attend to the learned tokens and recover an explicit, renderable set of 3D Gaussian primitives. Novel-view rendering supervision trains the compact representation end to end through the reconstructed Gaussian scene. Experiments on RealEstate10K and ACID show that the resulting compact, unstructured token set supports competitive novel-view synthesis and can serve as the latent space of a generative model for 3D Gaussian scene generation. Together, these results show that 3D Gaussian reconstruction is an effective objective for compact scene representation learning and the learned token space provides a basis for broader downstream tasks in 3D vision.

Abstract

3D Scene Representation

Explore similar work

CoTL: Compact Token Learning for 3D Scene Representation

Any 3D Scene is Worth 1K Tokens: 3D-Grounded Representation for Scene Generation at Scale

ZipTok3D: High-Fidelity 3D Tokenization with Compact Token Prefixes