Octrees as an Explicit 3D Language

Ran Dan; Si-Tong Wei; Pengfei Xiong; Wei Zhang; Yadong Mu; Peng-Shuai Wang

cs.CVOct 1, 2026

Octrees as an Explicit 3D Language

Authors: Ran Dan, Si-Tong Wei, Pengfei Xiong, Wei Zhang, Yadong Mu, Peng-Shuai Wang

Organizations: Peking University · Independent Researcher

Abstract

Existing 3D large language models (LLMs) compromise on two fronts: they compress shapes into latent codebook indices or coordinate text, which removes spatial structure from what the model observes, and they acquire the 3D modality by fine-tuning the backbone, which overwrites its general language ability. We present OctLLM, which addresses both limitations. Geometry enters as an explicit 3D sequence of octree occupancy tokens. However, full octree sequences grow rapidly with depth; OctLLM therefore randomly empties penultimate-level nodes and omits descendants while preserving shape, yielding a shorter coordinate- and depth-anchored Sparse Octree (S-Octree) for position-aware mask-modeling generation and 3D understanding. On the other front, existing methods introduce a new modality with full fine-tuning or LoRA, but full fine-tuning is costly, LoRA limits 3D capacity, and both modify the language pathway. OctLLM instead adds 3D capacity in parameters separate from the pretrained ones: mesh tokens are routed through independent trainable branches in a subset of blocks while text and image tokens retain the frozen vision-language pathway, and the two streams interact through shared self-attention. It trains far fewer parameters than full fine-tuning, yet sets a new state of the art among unified multimodal LLMs, lowering image-to-3D FID by $17.4\%$ and raising render-grounded captioning by $28.7$ points over ShapeLLM-Omni, while matching the backbone on general language benchmarks.

Figures & tables

Figure 1: OctLLM unifies 3D generation and 3D understanding in a single LLM. Left: 3D assets generated by OctLLM from text and image conditions. Right: natural-language descriptions it produces for 3D inputs. Bottom right: a shape written as an S-Octree, the coarse-to-fine occupancy sequence OctLLM reads and writes, shown at octree depths 3 to 6. Generation and understanding share this explicit 3D language. Zoom in for a better view.

Figure 2: Overview of OctLLM. For generation, OctLLM autoregressively predicts a position-aware S-Octree sequence from a text or image condition, which a 3D U-Net completes into dense occupancy before a flow-based decoder, conditioned on the occupied voxels, produces the mesh. For 3D understanding, the same S-Octree is read with a text prompt as the 3D input language.

Figure 3: Token-routed dual-stream block. Text and image tokens retain the frozen modules, while S-Octree bytes and predictive <MASK> tokens use trainable 3D embeddings, projections, feed-forward branches, and a dedicated mesh head. Both streams interact through shared causal self-attention, so language, images, and geometry can exchange information.

Panel of Figure 4 — Figure 4: Position-aware mask modeling. Each inserted <MASK> token shares the 3D rotary coordinates and depth embedding of the mesh-byte token that follows it, so the hidden state at the mask queries a known 3D location before predicting occupancy. Historical masks are suppressed as attention keys; text, boundary, and previous mesh-byte tokens stay visible.

Figure 5: Image-to-3D generation. Each asset is shown from two viewpoints. OctLLM follows the structure and part layout of the conditioning image, while ShapeLLM-Omni and SAR3D produce geometry of lower quality.

Image-to-3D
Method	FID (Inc.) $\downarrow$	KID (Inc.) $\downarrow$	FID (DINOv2) $\downarrow$	KID (DINOv2) $\downarrow$	CLIP $\uparrow$
Task-specific 3D generators
TRELLIS ( Xiang et al., 2025 )	18.15	0.13	48.47	2.06	92.86
SAR3D ( Chen et al., 2025c )	67.42	3.16	206.79	20.56	84.46
Multimodal LLMs
ShapeLLM-Omni ( Ye et al., 2025 )	38.21	1.02	145.14	9.98	86.75

Table 1: Image-to-3D and text-to-3D generation on Toys4K. FID/KID are computed on rendered views, with KID scaled by

\times 100

, and CLIP measures condition alignment. Each modality block lists only the methods supporting that conditioning, split into task-specific 3D generators and multimodal LLMs; within a block, the best and second-best result per column are highlighted.

Table 2: 3D understanding and language ability. (a) Captioning on the 200-object PointLLM benchmark, judged by Qwen3.8-Max; the rule separates task-specific understanding models from multimodal LLMs, and the best and second-best result per column are highlighted; Table S2 adds PointLLM-13B and

n

-gram metrics. (b) General language benchmarks under greedy decoding, with the backbone in gray as a reference rather than a competitor and the best result per column highlighted. Higher is better throughout.

Figure 6: Text-to-3D generation. Prompts are truncated for space and describe how parts are composed; Section C.1 provides the complete conditions. “N/A” marks a prompt for which LLaMA-Mesh returned no valid mesh.

Appendix figures & tables15 assets

Supplementary material from the paper’s appendix.

Appendix

Figure S1: Text-conditioned generation at inference time. The prompt is answered with a short natural-language preamble that leads into <mesh_bos> , after which each mesh byte is produced by first attaching the next node’s 3D position to a transient <MASK> query and then sampling only the occupancy value at that position. The partial octree grows coarse to fine, and the completed grid is passed to the completion network and the flow-based decoder.

Setting	Value
Input limits	8,192 tokens and 262,144 image pixels
OctLLM schedule	AdamW, peak learning rate $1\times 10^{-4}$ , cosine decay, and $10\%$ linear warmup
Batch and precision	2 samples/GPU, 2 gradient-accumulation steps, and bfloat16
Validation	$0.1\%$ of the training mixture
Completion schedule	AdamW with cosine decay from $1\times 10^{-4}$ to $1\times 10^{-6}$

Appendix

Table S1: Optimization settings not specified in Section 4 .

Task-specific 3D understanding models
	$n$ -gram overlap								Semantic		Judge
Method	BLEU-1	BLEU-2	BLEU-3	BLEU-4	ROUGE-1	ROUGE-2	ROUGE-L	METEOR	S-BERT	SimCSE	GPT-ref	GPT-img
PointLLM-7B ( Xu et al., 2024 )	5.43	1.59	0.74	0.48	10.46	0.89	7.67	13.47	48.05	48.63	46.06	45.63
PointLLM-13B ( Xu et al., 2024 )	5.45	1.63	0.75	0.48	10.46	0.96	7.64	13.61	48.85	49.78	48.09	48.06
ShapeLLM-7B ( Qi et al., 2024a )	9.29	2.86	1.57	1.15	12.83	1.04	11.09	11.71	37.27	37.10	15.92	30.23
Multimodal LLMs

Appendix

Table S2: Complete PointLLM-200 metrics. All values are scaled by 100; the best result within each group is highlighted.

Figure S2: 3D understanding. Descriptions of two held-out assets, each shown from two viewpoints and truncated for space. ShapeLLM-Omni produces terse descriptions and sometimes misses the object category, while PointLLM-13B emphasizes appearance over structure. OctLLM gives more complete descriptions of the visible parts and their spatial arrangement; Section C.1 reports its complete outputs.

Figure S3: Additional image-to-3D generation results. Each row shows one image condition followed by the generated mesh from six viewpoints.

Figure S4: Additional text-to-3D generation results. Each row pairs a text condition, truncated in the figure, with five viewpoints of the generated mesh.

Method	Output tokens	Structure tokens	AR latency (s) $\downarrow$
ShapeLLM-Omni ( Ye et al., 2025 )	1,026.00	1,024.00	19.92
AR3D-R1 ( Tang et al., 2025b )	1,026.00	1,024.00	23.50
LLaMA-Mesh ( Wang et al., 2024c )	4,443.50	4,442.53	85.30
OctGPT ( Wei et al., 2025 )	10,033.44	2,757.48	52.53
OctLLM (Ours)	1,277.06	1,262.06	37.59

Appendix

Table S3: Text-to-3D token count and autoregressive latency on a fixed 200-prompt Toys4K subset. Values are means on one RTX 5090. Token units are method-native and differ across representations.

	Generation		Understanding
Representation	FID $\downarrow$	KID $\downarrow$	Sentence-BERT $\uparrow$	SimCSE $\uparrow$
VQVAE tokens (1,024)	33.10	0.80	67.15	68.87
S-Octree (Ours)	26.31	0.80	69.47	70.28

Appendix

Table S4: Ablation on the 3D representation. All settings except the representation are fixed; KID is scaled by 100. Generation metrics are computed on the image-to-3D task.

Method	MMLU $\uparrow$	HellaSwag $\uparrow$	GSM8K $\uparrow$	IFEval $\uparrow$
Frozen Qwen2.5-VL-7B-Instruct (reference)	64.65	67.44	83.54	75.41
LoRA + UltraChat	59.58	49.34	69.97	59.33
OctLLM (Ours)	64.65	67.44	83.09	75.41

Appendix

Table S5: Language preservation after ShapeNet-airplane training. The LoRA control includes UltraChat; the frozen backbone is shown as a gray reference.

Method	FID $\downarrow$	KID $\downarrow$
w/o 3D Position Embedding	32.84	1.00
w/o Mask Token	61.08	3.87
w/o Mask Suppression	28.03	0.94
Full model	26.31	0.80

Appendix

Table S6: S-Octree component ablations for image-to-3D generation. KID is scaled by 100.

	Generation				Understanding
Placement	FID (image) $\downarrow$	KID (image) $\downarrow$	FID (text) $\downarrow$	KID (text) $\downarrow$	Sentence-BERT $\uparrow$	SimCSE $\uparrow$
First 8/28 layers	36.63	1.70	38.97	1.62	72.26	74.37
Last 8/28 layers	34.03	1.31	37.09	1.49	73.65	73.71
Uniform 8/28 layers (Ours)	26.31	0.80	30.48	1.07	69.47	70.28

Appendix

Table S7: Placement of eight routed 3D branches. FID/KID (scaled by 100) evaluate generation; Sentence-BERT and SimCSE evaluate understanding.

	3D Generation				Language Ability (reference)
Method	FID (image) $\downarrow$	KID (image) $\downarrow$	FID (text) $\downarrow$	KID (text) $\downarrow$	MMLU $\uparrow$	HellaSwag $\uparrow$
Dual-stream branches (Ours)	26.31	0.80	30.48	1.07	64.65	67.44
Token-routed LoRA (rank 256)	32.44	1.06	30.93	0.98	64.65	67.44

Appendix

Table S8: Full-rank dual-stream branches versus token-routed LoRA on ShapeNet airplanes. FID/KID evaluate image- and text-conditioned 3D generation (KID scaled by 100); language scores are reference values because both variants match the frozen backbone.

Figure S6: Full octree versus S-Octree on the same asset, shown at depths 3 to 6 with the corresponding mesh-token sequence beneath each level. Pruning part of the second-to-last level, and with it the finest-level descendants, shortens the sequence the LLM must write without altering the coarse structure it predicts.

Panel of Figure 22 — Figure S7: Practical benefits of the S-Octree representation on five selected ShapeNet categories. Compared with full octrees, S-Octrees shift the mesh-token length distribution toward shorter sequences and reduce both peak GPU memory and training time per epoch.

Completion design	FID $\downarrow$	KID $\downarrow$
Encoder alignment + MSE	58.93	3.49
3D U-Net + skip connections + BCE	52.14	2.92
3D U-Net + Dice	38.89	1.87
3D U-Net + BCE (Ours)	36.86	1.60

Appendix

Table S9: Ablation on the sparse voxel completion network for image-to-3D generation. KID is scaled by 100.

3D Shape Generation

3D shape generation creates and edits 3D objects from learned geometric priors and text, images, sketches, or structural constraints.

See all 106 papers

Beyond Single Object: Learning 3D Relations with Large Language Models
Aug 16, 2026Kohsuke Ide, Ryousuke Yamada, Yue Qiu +43D VLMs 3D Spatial Reasoning
AIST · University of Tsukuba · University of Technology Nuremberg +1
3D-PLOT-LLM: Part-Level Object Tokens for 3D Large Language Models
Jun 18, 2026Jintang Xue, Xinyu Wang, Yixing Wu +23D Representation Learning Multimodal Foundation Models
University of Southern California Los Angeles, California, USA · Ohio State University Columbus, Ohio, USA
Point Cloud as a Foreign Language for Multi-modal Large Language Model
Mar 10, 2026Sneha Paul, Zachary Patterson, Nizar Bouguila3D VLMs Point Cloud Learning
Concordia University, Canada

Aug 16, 2026J/K move · Enter open · S save

Beyond Single Object: Learning 3D Relations with Large Language Models

Kohsuke Ide, Ryousuke Yamada, Yue Qiu, Xianzheng Ma+3

AIST · University of Tsukuba · University of Technology Nuremberg+1

3D VLMs 3D Spatial Reasoning 3D Representation Learning

Open paper AskPDF arXiv

We address a fundamental gap in 3D-LLMs: existing models focus on single-object/scene description, struggling with detailed, inter-object comparison. We propose a framework for detailed object-level reasoning across multiple objects with three components: (1) MO3D (Multi-Object in 3D), an instruction dataset requiring fine-grained multi-object comparison; (2) Multi-3DLLM, using a minimal Patch-Interaction Transformer (PIT) that models inter-/intra-object relationships while preserving local geometry; (3) Mini-apps, two application-driven benchmarks (Shape Mating, Change Captioning) that probe geometric understanding for practical use. Recent 3D-LLMs and 2D-VLMs perform poorly on these tasks, lacking both comparison-centric design and geometric awareness. In contrast, Multi-3DLLM trained on our mixture data learns geometric reasoning, surpasses all baselines on MO3D, and provides positive transfer to single-object classification.

Abstract

3D Shape Generation

Explore similar work

Beyond Single Object: Learning 3D Relations with Large Language Models

3D-PLOT-LLM: Part-Level Object Tokens for 3D Large Language Models

Point Cloud as a Foreign Language for Multi-modal Large Language Model