Multi-modal large language models (MLLMs) have shown remarkable progress in integrating visual and linguistic understanding. Recent efforts have extended these capabilities to 3D understanding through encoder-based architectures that rely on pre-trained 3D encoders to extract geometric features. However, such approaches suffer from semantic misalignment between geometric and linguistic spaces, resolution sensitivity, and substantial computational overhead. In this work, we present SAGE, the first end-to-end 3D MLLM that directly processes raw point clouds without relying on a pre-trained 3D encoder. Our approach introduces a lightweight 3D tokenizer that combines geometric sampling and neighbourhood aggregation with vector quantization to convert point clouds into discrete tokens--treating 3D data as a foreign language that naturally extends the LLM's vocabulary. Furthermore, to enhance the model's reasoning capability on complex 3D tasks, we propose a preference optimization training strategy with a semantic alignment-based reward, specifically designed for open-ended 3D question answering where responses are descriptive. Extensive experiments across diverse 3D understanding benchmarks demonstrate that our end-to-end approach outperforms existing encoder-based methods while offering significant advantages in computational efficiency, generalization across LLM backbones, and robustness to input resolution variations. Code is available at: github.com/snehaputul/SAGE3D.
Figures & tables
Figure 1: Our proposed encoder-free 3D Multimodal Large Language Model efficiently captures 3D information from point clouds without relying on any pretrained 3D encoder. The figure on the left illustrates the overall architecture, while the figure on the right shows an example conversation about an object generated by our model.
Figure 2: Architecture of our proposed method, encoder-free 3D Multimodal Large Language Model.
Figure 3: The proposed training pipeline of our model. The model is trained in three stages — each stage focusing on a specific training objective.
Model
Captioning
Cls.
VQA
GPT-4
Sentence-BERT
SimCSE
BLEU-1
ROUGE-L
METEOR
GPT-4
GPT-4
InstructBLIP-7B [ 15 ]
43.54
47.41
48.48
4.27
8.28
12.99
43.50
–
InstructBLIP-13B [ 15 ]
44.97
45.90
48.86
4.65
8.85
13.23
44.25
–
LLaVA-7B [ 38 ]
46.71
45.61
47.10
3.64
7.70
12.14
50.00
–
LLaVA-13B [ 38 ]
38.28
46.37
45.90
4.02
8.15
12.58
51.75
47.90
PointLLM-7B [ 68 ]
44.85
47.47
48.55
3.87
7.30
11.92
53.00
41.20
Table 1: Performance comparison on various 3D downstream tasks. Here, the Objaverse dataset is used for 3D object captioning and recognition tasks, and the MM-Vet dataset is used for 3D VQA. In addition to our full model (SAGE) , we provide the results without the preference optimization (SAGE ∗ ) . Additional comparisons with existing methods on different datasets are provided in the Appendix A3 .
Model
Latency
Throughput
(ms) ↓
(Sample/s) ↑
PointLLM-7B
239
4.2
SAGE-7B
100
10.0
Table 2: Runtime Complexity Comparison. Inference latency (ms) and memory (GB) on H100 GPU, 8K points, and Objaverse dataset.
Figure 4: Performance of SAGE on diverse ranges of point cloud resolution on 3D captioning task on Objaverse dataset.
Samples 1, 2
Uid
0ea33b6617174530b97d6b7a92c275fb
de8ec2a724f14fc4b54624512f80f13e
Prompt
What is this?
This is an object of
Human
A cartoon green and red like a fruit
A black insect
InstructBLIP [ 15 ]
an appleavatar 3d model
26
LLaVA [ 38 ]
This is a computer-generated image (CGI) or a 3D model of an apple with a leaf on it.
This is an image of a large, close-up, and dark-colored insect, possibly a big cockroach, with long antennae.
3D-LLM [ 25 ]
A 3D model of a red apple.
A small, black spider with a long tail.
Table 3: Qualitative results on Objaverse. We adopt this table from [ 68 ] , and compare our proposed model’s generated description for the specific 3D object samples.
Stage
Hyperparam.
Value
β
0.25
λ
0.50
Codebook size
8192
Warm-up ratio
0.03
Weight decay
0.05
Number of sampled points, Ns
512
Table A1 : List of model-specific hyperparameters.
ScanQA (val)
Scan2Cap (val)
Method
Modality
B-4 ↑
M ↑
R ↑
B-4 ↑
M ↑
R ↑
Specialist Models
ScanQA [ 1 ]
PC
10.1
13.1
33.3
-
-
-
3D-VLP [ 33 ]
PC
11.2
13.5
34.5
32.3
24.8
51.5
3D-VisTA [ 81 ]
PC
10.4
13.9
45.7
34.1
26.8
55.0
Scan2Cap [ 11 ]
PC
-
-
-
23.3
22.0
44.8
Table A2 : Performance comparison with existing methods. We adopt the following table from [ 17 ] and follow the notations and categorization defined by them. Here, “Specialist Model” refers to models specifically designed for individual tasks such as 3D question answering, 3D dense captioning, or referring segmentation. “Finetuned 3D MLLMs” denotes models that are jointly trained and subsequently fine-tuned on each dataset before evaluation. “3D MLLMs” represents models trained on multiple tasks without task-specific fine-tuning. “PC” stands for point cloud, and “I” denotes multi-view images. Note that the results of LEO [ 28 ] on ScanQA are shown in gray and excluded from direct comparison, as the model uses a different setting that accesses ground-truth objects related to the questions. All models use the 7B parameter configuration.
Figure A1 : Fig. (Left) Impact of the number of LLM trainable layers during the stage 1 training. Fig. (Middle) Impact of the number of LLM trainable layers during stage 1 training. Fig(Right) Impact of group normalization coefficient on performance of SAGE-7B.
Model
Training time (h)
PointLLM
26.1
SAGE ∗
18.0
SAGE
27.4
Table A3: Total training time.
Method
ModelNet40
ScanObjectNN
Top1
Top3
Top5
Top1
Top3
Top5
2D Inference without 3D Training
PointCLIP [ 73 ]
19.3
28.6
34.8
10.5
20.8
30.6
PointCLIPv2 [ 80 ]
63.6
77.9
85.0
42.2
63.3
74.5
Trained on ShapeNet
ReCon [ 47 ]
61.2
73.9
78.1
42.3
62.5
75.6
Table A4 : Performance comparison on Zero-shot 3D classification on ModelNet40 [ 65 ] and ScanObjectNN [ 60 ] . Here, Ensembled [ 40 ] reprsents pretraining with four datasets, ShapeNet [ 6 ] , ABO [ 14 ] and 3D-FUTURE [ 20 ] . † : Uni3D employs a larger EVA-CLIP-E [ 52 ] teacher, while other methods employ OpenCLIP-bigG [ 30 ] .
Model
Captioning
Cls.
VQA
GPT-4
Sentence-BERT
SimCSE
BLEU-1
ROUGE-L
METEOR
GPT-4
GPT-4
SAGE-7B ∗
49.05
49.23
48.56
7.41
10.25
14.35
55.71
46.38
SD
0.09
0.04
0.05
0.04
0.02
0.02
0.03
0.04
SAGE-7B
50.98
50.11
49.70
9.50
12.66
16.95
57.11
49.53
SD
0.08
0.03
0.04
0.05
0.04
0.03
0.03
0.05
Table A5 : Performance comparison across three runs on (SAGE) and (SAGE ∗ ) on 7B parameters. Here, the Objaverse dataset is used for 3D object captioning and recognition tasks, and the MM-Vet dataset is used for 3D VQA.
Table A6 : Sensitivity study on different model-specific parameters on ModelNet40.
Embedding type
S-BERT ↑
Continuous (w/o codebook)
47.67
Discrete (with codebook)
50.11
Table A7: Impact of discrete vs. continuous point embeddings.
LLM
Model
S-BERT ↑
LLaMA-3.1
PointLLM
51.23
SAGE
55.89
Qwen-2.5
PointLLM
52.35
SAGE
56.91
Table A8: Impact of different LLM backbones.
Figure A2 : Dialogues between PointLLM and a human user.
Samples 1, 2
Uid
059a7936ed89419ba9eae3153753ae86
82413f74efb84bc3b872a7a8f26cca81
Prompt
Caption this 3D model in detail.
Caption this 3D model in detail.
Human
A blue cartoon
A white fish with orange/red spots
InstructBLIP [ 15 ]
There is a cute blue and white pony in a still, looking cheerful and cheerful. It’s standing next to a larger blue-colored object, potentially a barrier or structure, which provides the backdrop for this scene. The overall image is designed to create a fun, whimsical setting for the pony.
There is a fish hanging on a string in front of a black background. The fish is swimming in the middle of the image and is a little bit up from the bottom of the screen. The fish has a red spots on the left and right sides, and is positioned slightly towards the top of the screen.
LLaVA [ 38 ]
The image features a 3D model of a cute, colorful cartoon unicorn with a blue and white body and a hint of pink. The unicorn has a magical aura and is set against a black background. Its long, flowing mane and tail add to its whimsical charm. The unicorn appears to be in motion, possibly flying or prancing, as it captures the viewer’s imagination.
The image features a 3D model of a fish floating in a dark, empty space. The fish is positioned towards the left side of the image and appears to be swimming upwards. The fish is large and occupies a significant portion of the image, giving the impression of a life-like, computer-generated representation. The dark background and lack of other objects in the scene emphasize the fish’s presence and create a sense of depth and dimension.
3D-LLM [ 25 ]
A 3D model of a purple and pink pony with long hair, standing on its hind legs.
A small fish with a long tail is visible in the image. It has no distinguishing features or markings, and its color appears to be brown.
Table A9 : Qualitative results on the Objaverse dataset.
Understanding 3D point clouds through language remains a fundamental challenge in computer graphics and visual computing, due to the irregular structure of point cloud data and the lack of explicit reasoning in existing 3D multimodal models. While Chain-of-Thought (CoT) reasoning has shown strong effectiveness in LLMs and image-based MLLMs, its extension to 3D understanding remains largely underexplored. In this paper, we propose a data-centric framework for constructing large-scale CoT supervision tailored to 3D point cloud understanding. Our framework consists of a two-stage pipeline that first refines point-text instruction data via vision-language-model-based quality evaluation and reference-guided refinement, and then synthesizes high-quality reasoning paths through Human-in-the-Loop Prompt Optimization (HiLPO). Using this approach, we build PoCoTI, a CoT-enhanced point-text instruction-following dataset containing 55K samples with explicit reasoning paths. Fine-tuning PointLLM on PoCoTI yields PointLLM-R, a reasoning-capable 3D multimodal language model. Extensive experiments on generative 3D classification and captioning demonstrate that PointLLM-R achieves state-of-the-art performance and generalizes robustly to real-world scanned point clouds and multi-turn dialogue scenarios.
Chaoqi Chen, Qile Xu, Wenjun Zhou +1
Visual Computing Research Center (VCC), College of Computer Science and Software Engineering (CSSE), Shenzhen University, China · VCC, CSSE, Shenzhen University, China
While Multimodal Large Language Models (MLLMs) demonstrate proficiency in 2D scenes, extending their perceptual intelligence to 3D point cloud understanding remains a significant challenge. Current approaches focus primarily on aligning 3D features with pre-trained models. However, they typically treat geometric reasoning as an implicit mapping process. These methods bypass intermediate logical steps and consequently suffer from geometric hallucinations. They confidently generate plausible responses that fail to ground in precise structural details. To bridge this gap, we present PointCoT, a novel framework that empowers MLLMs with explicit Chain-of-Thought (CoT) reasoning for 3D data. We advocate for a Look, Think, then Answer paradigm. In this approach, the model is supervised to generate geometry-grounded rationales before predicting final answers. To facilitate this, we construct Point-Reason-Instruct, a large-scale benchmark comprising ∼86k instruction-tuning samples with hierarchical CoT annotations. By leveraging a dual-stream multi-modal architecture, our method synergizes semantic appearance with geometric truth. Extensive experiments demonstrate that PointCoT achieves state-of-the-art performance on complex reasoning tasks.
Dongxu Zhang, Yiding Sun, Pengcheng Li +12
Xi’an Jiaotong University · Shanghai AI Laboratory · Tsinghua University +6
Large Multimodal Models (LMMs) that process 3D data typically rely on heavy, pre-trained visual encoders to extract geometric features. While recent 2D LMMs have begun to eliminate such encoders for efficiency and scalability, extending this paradigm to 3D remains challenging due to the unordered and large-scale nature of point clouds. This leaves a critical unanswered question: How can we design an LMM that tokenizes unordered 3D data effectively and efficiently without a cumbersome encoder? We propose Fase3D, the first efficient encoder-free Fourier-based 3D scene LMM. Fase3D tackles the challenges of scalability and permutation invariance with a novel tokenizer that combines point cloud serialization and the Fast Fourier Transform (FFT) to approximate self-attention. This design enables an effective and computationally minimal architecture, built upon three key innovations: First, we represent large scenes compactly via structured superpoints. Second, our space-filling curve serialization followed by an FFT enables efficient global context modeling and graph-based token merging. Lastly, our Fourier-augmented LoRA adapters inject global frequency-aware interactions into the LLMs at a negligible cost. Fase3D achieves performance comparable to encoder-based 3D LMMs while being significantly more efficient in computation and parameters. Project website: https://tev-fbk.github.io/Fase3D.
Guofeng Mei, Wei Lin, Luigi Riz +3
Fondazione Bruno Kessler, Italy · JKU Linz, Austria · CSIRO, Australia