cs.CVMar 11, 2026

LiTo: Surface Light Field Tokenization

Authors: Jen-Hao Rick Chang, Xiaoming Zhao, Dorian Chan, Oncel Tuzel

Organizations: Apple

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

We propose a 3D latent representation that jointly models object geometry and view-dependent appearance. Most prior works focus on either reconstructing 3D geometry or predicting view-independent diffuse appearance, and thus struggle to capture realistic view-dependent effects. Our approach leverages that RGB-depth images provide samples of a surface light field. By encoding random subsamples of this surface light field into a compact set of latent vectors, our model learns to represent both geometry and appearance within a unified 3D latent space. This representation reproduces view-dependent effects such as specular highlights and Fresnel reflections under complex lighting. We further train a latent flow matching model on this representation to learn its distribution conditioned on a single input image, enabling the generation of 3D objects with appearances consistent with the lighting and materials in the input. Experiments show that our approach achieves higher visual quality and better input fidelity than existing methods.

Figures & tables

Figure 1: LiTo tokenizes surface light fields into a latent representation. It models 3D geometry and view-dependent appearance such as specular reflection. The figure shows reconstructions (first 3 columns) and single-image-to-3D results (last two columns). Mesh credit: Anthony Schmidt (2016) ; @sanyabeast (2021) ; LLOYDO (2019) ; brysew (2015) ; Osho (2018) . See more on the project page .
Figure 2: Overview of the 3D latent representation. Given samples of the surface light field of the scene, we learn a latent representation that reconstruct the full surface light field information. The encoder (pink block) condenses input information into the latent representation. We jointly supervise the latent representation to contain full 3D geometry and view-dependent radiance information beyond the input samples. In the architectures, we design localized attention pattern to improve efficiency and support 1 million input tokens.
Figure 3: 3D patchification
Figure 4: Reconstruction results on various lighting conditions. Boxes on ground-truth highlight specular and Fresnel reflection. Please refer to Tab. 1 for quantitative results. Mesh credit: DigitalSouls (2019) ; 3Dji (2025) ; Virtual Museums of Małopolska (2020) .
Figure 5: Single image to 3D results. The input image is shown at the center of each set with black border. The rendering at the input view is shown with the input image. Please refer to Tab. 3 for quantitative results. Mesh credit: Eleanie (2025) ; Rigsters (2017) ; 3d-coat (2015) ; 3Dji (2025) .
MethodSimple, Camera Radius [3, 4]Hard, Camera Radius [1, 3]
PSNR ↑\uparrowSSIM ↑\uparrowLPIPS ↓\downarrowPSNR ↑\uparrowSSIM ↑\uparrowLPIPS ↓\downarrow
TRELLIS31.12 ±\pm 3.390.974 ±\pm 0.0220.034 ±\pm 0.02227.57 ±\pm 3.380.941 ±\pm 0.0500.090 ±\pm 0.055
Ours34.16 ±\pm 3.390.985 ±\pm 0.0160.023 ±\pm 0.01832.36 ±\pm 3.770.967 ±\pm 0.0400.055 ±\pm 0.046
Table 1: Reconstruction on Toys4k. We provide input needed by individual methods. TRELLIS ( Xiang et al., 2025 ) takes the ground-truth mesh and 150 sphere-distributed renderings. Ours uses RGBD images from 150 evenly distributed views. For appearance evaluation, we render each model’s output from 100 random cameras, varying difficulty by adjusting camera radius. Please refer to Fig. 4 for qualitative results and Sec. C for comprehensive quantitative results. The better one is highlighted.
 MethodAppearanceLatent sizePBR-ObjaverseToys4kGSO
0GT––82.49 ±\pm 21.3976.03 ±\pm 24.3087.50 ±\pm 21.04
Requires coarse geometry oracle:  
1TripoSF ( He et al., 2025 )✗≈\approx 244k ×\times 1183.11 ±\pm 21.4576.99 ±\pm 24.9587.63 ±\pm 21.67
2TRELLIS ( Xiang et al., 2025 )✓≈\approx 20k ×\times 1195.16 ±\pm 20.7792.21 ±\pm 25.99105.5 ±\pm 22.44
3-13DTopia-XL ( Chen et al., 2025b )✓2048 ×\times 64412.5 ±\pm 1129.153.9 ±\pm 305.898.50 ±\pm 19.66
3-2(worst 10% removed)✓2048 ×\times 64135.4 ±\pm 95.7590.61 ±\pm 23.9693.62 ±\pm 12.86
Table 2: Geometric reconstruction evaluation . We report Chamfer distances multiplied by 10410^{4} for readability, computed using 100k sampled points each from ground-truth and reconstruction. As 3DTopia-XL ( Chen et al., 2025b ) and TripoSG ( Li et al., 2025a ) can be sensitive to input geometry, we also list variants with their 10% worst-performing objects removed. We separate our tested approaches based on those that require ground-truth coarse geometry for decoding the latent representation, and those that do not utilize this information. Our method outputs the best geometry among the approaches in the latter category, and it is competitive with the techniques in the former while using a 10x smaller latent space. Best and 2nd-Best methods in each category are highlighted.
MethodCLIP ↑\uparrowConditioning ViewNovel View
FID ↓\downarrowKID ↓\downarrowFID dino{}_{\text{dino}} ↓\downarrowKID dino{}_{\text{dino}} ↓\downarrowFID ↓\downarrowKID ↓\downarrowFID dino{}_{\text{dino}} ↓\downarrowKID dino{}_{\text{dino}} ↓\downarrow
TRELLIS0.899 ±\pm 0.04512.840.08884.6922.3117.6000.10067.4583.166
Ours0.905 ±\pm 0.0416.2190.00941.6211.3336.2160.05866.5303.522
Table 3: Single-image-to-3D generation on Toys4k. KID is reported by ×100\times 100 . CFG scale for both models are 3.0. The best is highlighted. See Fig. 5 , 6 for qualitative results.
Figure 6: Fidelity to input view. Our image-to-3d generative model respects the coordinate system of the input view. In contrast, existing state-of-the-art techniques, e.g . , TRELLIS ( Xiang et al., 2025 ) , do not. Mesh credit: Virtual Museums of Małopolska (2016) ; animanyarty (2022) .
Appendix figures & tables13 assets

Supplementary material from the paper’s appendix.

Appendix

Figure S1: Mesh comparisons. We demonstrate the qualities of our mesh decoder results to TRELLIS. As highlighted, our produced mesh maintains more details. Mesh credit: alzarac (2019) ; Alienor.org, Conseil des musées (2016) ; 1812panorama (2019) ; AdamJonesCGD (2020) ; nastasyas (2019) ; a108082046 (2022) ; GJ2012 (2013) .
 MethodSH DegEnc RayPred OccMeshSimple, Camera Radius [3, 4]Hard, Camera Radius [1, 3]CD (100k) ↓\downarrow
PSNR ↑\uparrowSSIM ↑\uparrowLPIPS ↓\downarrowPSNR ↑\uparrowSSIM ↑\uparrowLPIPS ↓\downarrow
Uniform Lighting
1-1TRELLIS0✗-✓28.17 ±\pm 4.090.970 ±\pm 0.0240.039 ±\pm 0.02424.63 ±\pm 4.010.934 ±\pm 0.0540.098 ±\pm 0.05995.19 ±\pm 26.41
1-2Ours0✗✗✗34.40 ±\pm 3.620.984 ±\pm 0.0170.025 ±\pm 0.01932.19 ±\pm 3.950.965 ±\pm 0.0420.059 ±\pm 0.04790.22 ±\pm 25.48
1-3Ours0✓✗✗34.44 ±\pm 3.470.984 ±\pm 0.0170.026 ±\pm 0.02032.18 ±\pm 3.790.964 ±\pm 0.0420.060 ±\pm 0.04889.24 ±\pm 25.34
1-4Ours1✓✗✗35.12 ±\pm 3.390.986 ±\pm 0.0150.023 ±\pm 0.01733.17 ±\pm 3.760.968 ±\pm 0.0400.054 ±\pm 0.04488.42 ±\pm 25.21
Appendix
Table S2: Reconstruction on Toys4k. For 3D assets, we adapt inputs per model. TRELLIS ( Xiang et al., 2025 ) takes the ground-truth mesh and 150 sphere-distributed renderings. Ours uses RGBD images from 150 evenly distributed views. For appearance evaluation, we render each model’s output from 100 random cameras, varying difficulty by adjusting camera radius. Each model is further evaluated under three distinct lighting conditions. Importantly, no separate models are trained; all evaluations are conducted on the same model. As a result, we conduct evaluations at the scale of over 3000 (objects) ×\times 100 (views) ×\times 2 (difficulties) ×\times 3 (lightings) ≈\approx 1.8 million images . We report Chamfer distances multiplied by 10410^{4} for readability, computed using 100k sampled points each from the ground truth and reconstruction. We report in the format of mean±std\texttt{mean}{\pm\texttt{std}} , where the standard deviation is computed across objects.
 MethodSH DegEnc RayPred OccMeshSimple, Camera Radius [3, 4]Hard, Camera Radius [1, 3]CD (100k) ↓\downarrow
PSNR ↑\uparrowSSIM ↑\uparrowLPIPS ↓\downarrowPSNR ↑\uparrowSSIM ↑\uparrowLPIPS ↓\downarrow
Uniform Lighting
1-1TRELLIS0✗-✓27.34 ±\pm 3.820.947 ±\pm 0.0360.053 ±\pm 0.02923.72 ±\pm 3.660.883 ±\pm 0.0680.139 ±\pm 0.065108.2 ±\pm 22.56
1-2Ours0✗✗✗34.27 ±\pm 3.250.975 ±\pm 0.0220.034 ±\pm 0.02531.39 ±\pm 3.610.937 ±\pm 0.0460.093 ±\pm 0.055101.4 ±\pm 22.21
1-3Ours0✓✗✗34.04 ±\pm 3.230.974 ±\pm 0.0220.034 ±\pm 0.02531.15 ±\pm 3.590.935 ±\pm 0.0480.093 ±\pm 0.055100.0 ±\pm 21.41
1-4Ours1✓✗✗34.55 ±\pm 3.180.976 ±\pm 0.0210.031 ±\pm 0.02331.75 ±\pm 3.600.939 ±\pm 0.0450.087 ±\pm 0.05399.21 ±\pm 21.44
Appendix
Table S3: Reconstruction on GSO. For 3D assets, we adapt inputs per model. TRELLIS ( Xiang et al., 2025 ) takes the ground-truth mesh and 150 sphere-distributed renderings. Ours uses RGBD images from 150 evenly distributed views. For appearance evaluation, we render each model’s output from 100 random cameras, varying difficulty by adjusting camera radius. Each model is further evaluated under three distinct lighting conditions. Importantly, no separate models are trained; all evaluations are conducted on the same model. As a result, we conduct evaluations at the scale of over 1000 (objects) ×\times 100 (views) ×\times 2 (difficulties) ×\times 3 (lightings) ≈\approx 600 thousand images . We report Chamfer distances multiplied by 10410^{4} for readability, computed using 100k sampled points each from the ground truth and reconstruction. We report in the format of mean±std\texttt{mean}{\pm\texttt{std}} , where the standard deviation is computed across objects.
 MethodSH DegEnc RayPred OccMeshSimple, Camera Radius [3, 4]Hard, Camera Radius [1, 3]CD (100k) ↓\downarrow
PSNR ↑\uparrowSSIM ↑\uparrowLPIPS ↓\downarrowPSNR ↑\uparrowSSIM ↑\uparrowLPIPS ↓\downarrow
Uniform Lighting
1-1TRELLIS0✗-✓28.63 ±\pm 3.090.955 ±\pm 0.0280.046 ±\pm 0.02525.06 ±\pm 2.930.902 ±\pm 0.0570.121 ±\pm 0.06298.09 ±\pm 22.21
1-2Ours0✗✗✗32.95 ±\pm 2.870.974 ±\pm 0.0180.033 ±\pm 0.02030.07 ±\pm 3.020.939 ±\pm 0.0420.087 ±\pm 0.05195.08 ±\pm 22.61
1-3Ours0✓✗✗33.14 ±\pm 2.680.974 ±\pm 0.0180.034 ±\pm 0.02130.21 ±\pm 2.850.937 ±\pm 0.0420.089 ±\pm 0.05394.16 ±\pm 22.55
1-4Ours1✓✗✗34.35 ±\pm 2.370.978 ±\pm 0.0160.028 ±\pm 0.01831.67 ±\pm 2.670.947 ±\pm 0.0380.076 ±\pm 0.04693.48 ±\pm 22.55
Appendix
Table S4: Reconstruction on PBR-Objaverse. For 3D assets, we adapt inputs per model. TRELLIS ( Xiang et al., 2025 ) takes the ground-truth mesh and 150 sphere-distributed renderings. Ours uses RGBD images from 150 evenly distributed views. For appearance evaluation, we render each model’s output from 100 random cameras, varying difficulty by adjusting camera radius. Each model is further evaluated under three distinct lighting conditions. Importantly, no separate models are trained; all evaluations are conducted on the same model. As a result, we conduct evaluations at the scale of 200 (objects) ×\times 100 (views) ×\times 2 (difficulties) ×\times 3 (lightings) ≈\approx 120 thousand images . We report Chamfer distances multiplied by 10410^{4} for readability, computed using 100k sampled points each from the ground truth and reconstruction. We report in the format of mean±std\texttt{mean}{\pm\texttt{std}} , where the standard deviation is computed across objects.
 MethodSimple, Camera Radius [3, 4]Hard, Camera Radius [1, 3]CD (100k) ↓\downarrow
PSNR ↑\uparrowSSIM ↑\uparrowLPIPS ↓\downarrowPSNR ↑\uparrowSSIM ↑\uparrowLPIPS ↓\downarrow
150 input views
1TRELLIS31.559 ±\pm 3.5090.9740 ±\pm 0.02240.0361 ±\pm 0.021727.948 ±\pm 3.5390.9408 ±\pm 0.05080.0928 ±\pm 0.053990.65 ±\pm 25.13
2Ours33.909 ±\pm 3.1570.9841 ±\pm 0.01620.0260 ±\pm 0.018932.073 ±\pm 3.5210.9658 ±\pm 0.04030.0585 ±\pm 0.045880.54 ±\pm 27.62
120 input views
3TRELLIS31.518 ±\pm 3.5090.9738 ±\pm 0.02250.0363 ±\pm 0.021827.912 ±\pm 3.5410.9404 ±\pm 0.05100.0932 ±\pm 0.054190.88 ±\pm 25.19
Appendix
Table S5: Ablation on number of input views for reconstruction during inference. We choose TRELLIS lighting setup on Toys4k dataset. Our model is the same as “ours” in Tab. 1 . Both TRELLIS and ours are trained with 150 views. For appearance evaluation, we render each model’s output from 100 random cameras, varying difficulty by adjusting camera radius. We report in the format of mean±std\texttt{mean}{\pm\texttt{std}} , where the standard deviation is computed across objects. We report Chamfer distances multiplied by 10410^{4} for readability, computed using 100k sampled points each from the ground truth and reconstruction. Note, we re-render the evaluation data for this ablation, thus row 1 (row 2) differs slightly from row 2-1 (row 2-9) in Tab. S2 .
 MethodTrain w/ RayInfer w/ GT RayTrain ItersCLIP ↑\uparrowConditioning ViewNovel View
FID ↓\downarrowKID ↓\downarrowFID dino{}_{\text{dino}} ↓\downarrowKID dino{}_{\text{dino}} ↓\downarrowFID ↓\downarrowKID ↓\downarrowFID dino{}_{\text{dino}} ↓\downarrowKID dino{}_{\text{dino}} ↓\downarrow
1TRELLIS✗-400k0.899 ±\pm 0.04512.840.08884.6922.3117.6000.10067.4583.166
2-1Ours✗-280k0.906 ±\pm 0.0408.1930.01248.1170.4616.6480.06475.8144.321
2-2Ours✗-400k0.906 ±\pm 0.0417.7410.01044.5550.3926.4130.06471.4363.997
2-3Ours✗-600k0.905 ±\pm 0.0416.2190.00941.6211.3336.2160.05866.5303.522
3Ours✓✗290k0.900 ±\pm 0.04010.780.06665.6442.2818.0760.10192.9156.698
Appendix
Table S6: Single-image-conditioned generation on Toys4k with TRELLIS lighting. KID is reported by ×100\times 100 . CFG scale is 3.0. The best is highlighted.
 Occ PredData TypeMethodStepCLIP ↑\uparrowConditioning ViewNovel View
FID ↓\downarrowKID ↓\downarrowFID dino{}_{\text{dino}} ↓\downarrowKID dino{}_{\text{dino}} ↓\downarrowFID ↓\downarrowKID ↓\downarrowFID dino{}_{\text{dino}} ↓\downarrowKID dino{}_{\text{dino}} ↓\downarrow
1✗float32Heun1000.905 ±\pm 0.0416.2190.00941.6211.3336.2160.05866.5303.522
2✓float32Heun1000.905 ±\pm 0.0416.6220.02142.1971.3916.2700.06466.6993.534
3✓bfloat16Heun1000.905 ±\pm 0.0416.6610.02043.9921.7416.2700.06368.0253.906
4✓bfloat16Heun500.905 ±\pm 0.0416.6590.02045.5332.1056.2660.06268.3194.185
5✓bfloat16Heun250.904 ±\pm 0.0416.6440.01954.2314.0116.2510.06077.1485.879
Appendix
Table S7: Ablation on DiT sampler for single-image-conditioned generation. The experiments are conducted on Toys4k with TRELLIS lighting. The generative model is trained for 600k iterations. Note, row 1 is copied from ”ours” in Tab. 3 . KID is reported by ×100\times 100 . CFG scale is 3.0. Our generative model’s performance is robust across various numbers of sampling steps and numerical integration algorithms.
 Cond Proc (ms)Structure Gen (s)Latent Gen (s)Occ Pred (ms)3DGS Dec (ms)Mesh Dec (ms)Total (s)Memory (GB)
NVIDIA A100-SXM4-80GB
TRELLIS68.90 ±0.49\pm 0.494.89 ±0.80\pm 0.807.720 ±5.10\pm 5.10–18.70 ±5.46\pm 5.4667.33 ±13.98\pm 13.9812.7612.70
Ours68.78 ±0.41\pm 0.41–17.32 ±1.50\pm 1.5036.07 ±3.67\pm 3.6735.32 ±14.2\pm 14.290.78 ±29.75\pm 29.7517.5515.95
NVIDIA H100 80GB HBM3
TRELLIS31.01 ±0.55\pm 0.553.95 ±1.17\pm 1.177.868 ±4.06\pm 4.06–15.03 ±6.19\pm 6.1946.81 ±13.31\pm 13.3111.9112.69
Ours22.58 ±10.3\pm 10.3–9.266 ±0.38\pm 0.3827.16 ±6.87\pm 6.8730.96 ±14.3\pm 14.379.15 ±31.71\pm 31.719.42615.93
Appendix
Table S8: Generative model runtime analysis. All results are reported with torch.profiler across three runs. TRELLIS uses 50 Euler steps for both its sparse structure and structured latent generations. We use 50 Euler steps for generating the latents, corresponding to row 7 in Tab. S7 .
Figure 18
Figure 19
Figure 20
Figure S8: Rendering with various spherical harmonics degrees in reconstruction. When restricted to zeroth-order spherical harmonics, our 3D Gaussians produce a view-independent appearance and avoid the over-exposed regions observed in TRELLIS’s renderings. As we progressively incorporate higher-order spherical harmonics, our method yields increasingly pronounced view-dependent effects. Mesh credit: DigitalSouls (2019) .
Figure S9: Rendering with various spherical harmonics degrees in generation. When restricted to zeroth-order spherical harmonics, our 3D Gaussians produce a view-independent appearance and avoid the over-exposed regions observed in TRELLIS’s renderings. As we progressively incorporate higher-order spherical harmonics, our method yields increasingly pronounced view-dependent effects. Mesh credit: Eleanie (2025) .
More in this topic

3D Representation Learning

3D representation learning builds geometric features for recognizing, reconstructing, generating, and reasoning about objects and scenes.

See all 187 papers

Explore similar work

CardsList

  1. LumiTokens: 3D Relighting via Token-Space Lighting Transformation

    Aug 18, 2026Yiwen Chen, Matheus Gadelha, Huaizu Jiang3D Scene RelightingImage Relighting

    Northeastern University · Adobe Research

  2. SceneAE: Scene Generation and Reconstruction in Queryable 3D-Anchored Latent Space

    Oct 14, 2026Shuofeng Sun, Wenzhao Zheng, Yunpeng Zhang +33D Scene Generation3D Scene Representation

    Beijing University of Posts and Telecommunications · University of California, Berkeley · PhiGent Robotics +1

  3. Luce: Relightable Gaussians for 3D Asset Generation

    Aug 25, 2026Mayank Singh, Michele Stoppa, Alvise Memo +73D Gaussian Splatting3D Asset Generation

    Apple