Tokenization is a fundamental technique in the generative modeling of various modalities. In particular, it plays a critical role in autoregressive (AR) models, which have recently emerged as a compelling option for 3D generation. However, optimal tokenization of 3D shapes remains an open question. State-of-the-art (SOTA) methods primarily rely on geometric level-of-detail (LoD) hierarchies, originally designed for rendering and compression. These spatial hierarchies are often token-inefficient and lack semantic coherence for AR modeling. We propose Level-of-Semantics Tokenization (LoST), which orders tokens by semantic salience, such that early prefixes decode into complete, plausible shapes that possess principal semantics, while subsequent tokens refine instance-specific geometric and semantic details. To train LoST, we introduce Relational Inter-Distance Alignment (RIDA), a novel 3D semantic alignment loss that aligns the relational structure of the 3D shape latent space with that of the semantic DINO feature space. Experiments show that LoST achieves SOTA reconstruction, surpassing previous LoD-based 3D shape tokenizers by large margins on both geometric and semantic reconstruction metrics. Moreover, LoST achieves efficient, high-quality AR 3D generation and enables downstream tasks like semantic retrieval, while using only 0.1%-10% of the tokens needed by prior AR models.
Figures & tables
Figure 2: Overview of LoST . Left: LoST maps 3D shape latents into a token sequence ordered by semantic salience, where early prefixes capture coarse semantics and later tokens refine instance-specific detail. A conditional generative DiT decoder reconstructs the complete latent from any prefix. Right: The semantic extractor is pretrained with Relational Inter-Distance Alignment (RIDA), which aligns relationships in 3D latent space with DINO feature relationships to provide semantics-aware supervision.
Figure 3: For each 3D shape (in blue), we visualize the shapes (in yellow) decoded from the learned LoST token sequences. Even as few as 1 token based generations result in semantically similar shapes while more tokens help to capture both semantic and geometric details.
OctGPT [ 34 ]
VertexRegen [ 43 ]
LoST (ours)
Num Tokens →
∼ 219
∼ 3615
∼ 15031
∼ 61962
∼ 239004
∼ 2730
∼ 2790
∼ 2910
∼ 3450
∼ 7530
1
4
16
64
512
CD ( ×10−2 ) ↓
16.923
1.759
0.850
0.533
0.470
4.290
1.865
0.809
0.209
0.034
2.271
1.328
0.723
0.382
0.234
FID ↓
341.174
265.774
184.252
100.781
88.483
186.611
186.454
176.137
151.393
86.098
31.649
29.255
26.565
21.133
13.591
DINO ↑
0.382
0.470
0.535
0.619
0.695
0.463
0.485
0.518
0.602
0.791
0.731
0.765
0.814
0.880
0.921
Table 1: Tokenizer reconstruction. We compare the reconstruction of LoST to recent LoD based tokenizers (OctGPT and VertexRegen) using varying number of tokens. We report the Chamfer Distance (CD) for geometric, and FID and DINO similarity for semantic accuracy. LoST outperforms baselines while using significantly fewer tokens. The top-performing score for each decoding level is in bold .
Figure 4: AR generation comparison. We compare image/text based autoregressive generation methods. LoST achieves superior performance in high-quality and faithful generation.
Figure 5: Given a query shape (top), we show shape retrieval results using triplane, DINO, and RIDA features. While original triplane features focus on geometric similarity, RIDA mapped triplane features capture semantic alignment similar to DINO. In this example, we use a confusing query of a submarine shaped like a fish.
Figure 6: LoST-GPT trained with our tokenizer can generate varying number of token sequences that can be decoded into complete, level-of-semantics shapes enabling efficient early stopping. For example, the model predicts a treasure chest without coins and a cargo ship without containers at the 1-token level, with these finer semantic and geometric details added in later levels.
Method
Num Tokens
FID ↓
DINO ↑
OctGPT [ 34 ]
∼ 50,000
66.926
✗
Llama-Mesh [ 33 ]
∼ 3758
118.576
✗
ShapeLLM-Omni [ 38 ]
1024
48.702
0.680
LoST (ours)
128
34.251
0.758
Table 2: AR generation. We report shading-based FID along with DINO similarity for image-conditioned methods. We use renderings of the generated evaluation dataset as ground truth reference.
Out-of-Distribution Set
In-Distribution Set
Metric
Triplane
Regression
RIDA (ours)
Triplane
Regression
RIDA (ours)
Top-3 Retrieval (K=3)
Recall@3
20.20%
20.63%
32.03%
19.07%
27.00%
48.50%
mAP@3
17.47%
17.28%
28.28%
16.42%
22.90%
44.28%
Jaccard
13.73%
13.91%
23.25%
13.60%
19.19%
38.36%
Top-5 Retrieval (K=5)
Table 3: Shape Retrieval Evaluation. We evaluate RIDA against raw triplane latents and a direct regression baseline that is trained to predict DINOv2 features. Ground truth neighbors are defined by DINOv2 similarity. RIDA (ours) outperforms the geometric baseline on both the in-distribution validation set and the out-of-distribution evaluation set (generated using Step1X-3D ), confirming that RIDA effectively captures semantics.
w/o RIDA
w/ RIDA (ours)
Num Tokens →
1
4
16
64
512
1
4
16
64
512
DINO ↑
0.720
0.758
0.821
0.876
0.904
0.731
0.765
0.814
0.880
0.921
DINOv2 ↑
0.528
0.590
0.693
0.763
0.867
0.556
0.612
0.694
0.805
0.875
Table 4: Ablation tokenizer reconstruction. We compare LoST trained without the proposed RIDA semantic alignment across multiple decoding levels. RIDA consistently improves semantic reconstruction quality, as measured by DINO and DINOv2 similarity, with the largest gains appearing in the low-token regime where semantic guidance is most critical. This confirms that RIDA helps short token prefixes encode more semantically meaningful structure. The best score at each decoding level is highlighted in bold .
Figure 7: LoST in the TRELLIS 3D VAE latent space . LoST applied to TRELLIS Stage-1 latents demonstrate that the proposed tokenization generalizes beyond the Direct3D triplane representation. These results highlight the flexibility of LoST as a representation-agnostic tokenizer for variable-length 3D generation.
Num Tokens
CD ( ×10−2 ) ↓
DINO ↑
OctGPT
∼ 219
17.925
0.529
∼ 15,031
1.210
0.611
∼ 239,004
0.123
0.729
VertexRegen
∼ 3,521
3.612
0.545
∼ 3,701
1.593
0.595
∼ 8,321
0.625
0.753
Table 5: Tokenization Evaluation on Objaverse. We evaluated 128 high-quality watertight Objaverse assets (filtered via Step1X-3D). These results are consistent with our results on our evaluation set in Table 1 . We note that all evaluation is computed on untextured renderings, which focuses on geometry alone (best results are in bold, second best are underlined).
Num Tokens
Token Dimension
Total
OctGPT [ 34 ]
∼ 50,000
1
50,000
Llama-Mesh [ 33 ]
∼ 3758
4096
∼ 15,392,768
ShapeLLM-Omni [ 38 ]
1024
3584
3,670,016
MeshGPT [ 28 ]
1200 - 4800
192
230,400 - 921,600
LoST GPT (ours)
128
32
4,096
Table 6: We compare total token dimension cost against other autoregressive methods. We note ShapeLLM-Omni uses 32-dimensional tokens for representation but this increases to 3584 due to LLM usage.
Compact token sequences are essential for efficient 3D generation. However, existing 3D tokenizers typically organize latent representations either over spatial regions or as fixed-size sets of global tokens, both suffering sharp reconstruction degradation when compressed to extremely low token budgets. In this paper, we present ZipTok3D, a 3D tokenizer designed for high-fidelity reconstruction from extremely short token sequences. Its key idea is to organize object geometry into progressively informative global-token prefixes and unfold these compact representations through iterative decoding. Specifically, nested dropout randomly truncates the latent sequence after encoding during training and requires each retained prefix to reconstruct the complete object, thereby prioritizing essential geometric information in the leading tokens. The decoder then repeatedly applies a parameter-shared Transformer block to recover fine-grained geometry from each prefix without a separate generative sampling stage. With the same token dimension, ZipTok3D achieves reconstruction quality comparable to the 32-token COD-VAE baseline using only one token on ShapeNet and four on TRELLIS, yielding 32× and 8× shorter token sequences, respectively.
Mingda Lin, Weijie Wang, Zeyu Zhang +7
1Zhejiang University · 2Monash University · University of Adelaide
Autoregressive multimodal large language models (MLLMs) enable 3D generation but struggle to scale to high-resolution shapes due to inadequate 3D tokenizations. Compact set-based representations discard deterministic spatial ordering, leading to ambiguous sequence prediction, while uniform or octree-based voxel grids preserve ordering at the cost of severe redundancy and excessively long sequences. This structural trade-off limits stable and efficient autoregressive 3D generation. We present SuperVoxelGPT, a representation-first framework that resolves this tension through adaptive and deterministically ordered supervoxel tokenization. Given a prompt, we first predict a coarse geometric saliency distribution and construct a shape-adaptive supervoxel partition using saliency-guided centroidal Voronoi tessellation, allocating fine-grained cells to complex regions and larger cells to smooth regions. Conditioned on this prompt and ordered supervoxel layout, we introduce a SuperVoxelVAE and fine-tune a pretrained MLLM to autoregressively generate supervoxel tokens. Experiments using Trellis-500K data show that SuperVoxelGPT reduces token sequence length to 12.8% of uniform voxel tokenization while achieving state-of-the-art generation quality and an average 10x speedup over prior methods.
Yuan Li, Congyi Zhang, Xifeng Gao +1
University of Texas at Dallas, USA · LightSpeed, USA
3D point clouds generation requires capturing both part-level layout and surface detail while preserving permutation equivariance. Existing latent generative models often encode an entire shape into a single vector, which limits their ability to retain part-level structure. We propose TokenFlow, a hierarchical flow matching framework that represents each shape as a set of latent tokens and couples a rectified flow over these tokens with a conditional flow over points. A bottleneck of M≪N learnable tokens aggregates global context through cross-attention at O(NM) cost. On ShapeNet, TokenFlow achieves the best MMD and coverage across airplane, car, and chair, while obtaining 1-NNA comparable to the strongest baselines. On ModelNet, its learned latent set reaches the best reported linear-evaluation accuracy on ModelNet10. These results show that representing a shape as a set of latent tokens, aggregated at O(NM) cost, provides an effective representation for 3D point cloud generation.