cs.CVMar 10, 2026
SavePoint Cloud as a Foreign Language for Multi-modal Large Language Model
Organizations: Concordia University, Canada
Abstract
Multi-modal large language models (MLLMs) have shown remarkable progress in integrating visual and linguistic understanding. Recent efforts have extended these capabilities to 3D understanding through encoder-based architectures that rely on pre-trained 3D encoders to extract geometric features. However, such approaches suffer from semantic misalignment between geometric and linguistic spaces, resolution sensitivity, and substantial computational overhead. In this work, we present SAGE, the first end-to-end 3D MLLM that directly processes raw point clouds without relying on a pre-trained 3D encoder. Our approach introduces a lightweight 3D tokenizer that combines geometric sampling and neighbourhood aggregation with vector quantization to convert point clouds into discrete tokens--treating 3D data as a foreign language that naturally extends the LLM's vocabulary. Furthermore, to enhance the model's reasoning capability on complex 3D tasks, we propose a preference optimization training strategy with a semantic alignment-based reward, specifically designed for open-ended 3D question answering where responses are descriptive. Extensive experiments across diverse 3D understanding benchmarks demonstrate that our end-to-end approach outperforms existing encoder-based methods while offering significant advantages in computational efficiency, generalization across LLM backbones, and robustness to input resolution variations. Code is available at: github.com/snehaputul/SAGE3D.
Figures & tables
Figure 1: Our proposed encoder-free 3D Multimodal Large Language Model efficiently captures 3D information from point clouds without relying on any pretrained 3D encoder. The figure on the left illustrates the overall architecture, while the figure on the right shows an example conversation about an object generated by our model.
Figure 2: Architecture of our proposed method, encoder-free 3D Multimodal Large Language Model.
Figure 3: The proposed training pipeline of our model. The model is trained in three stages — each stage focusing on a specific training objective.
| Model | Captioning | Cls. | VQA | |||||
|---|---|---|---|---|---|---|---|---|
| GPT-4 | Sentence-BERT | SimCSE | BLEU-1 | ROUGE-L | METEOR | GPT-4 | GPT-4 | |
| InstructBLIP-7B [ 15 ] | 43.54 | 47.41 | 48.48 | 4.27 | 8.28 | 12.99 | 43.50 | – |
| InstructBLIP-13B [ 15 ] | 44.97 | 45.90 | 48.86 | 4.65 | 8.85 | 13.23 | 44.25 | – |
| LLaVA-7B [ 38 ] | 46.71 | 45.61 | 47.10 | 3.64 | 7.70 | 12.14 | 50.00 | – |
| LLaVA-13B [ 38 ] | 38.28 | 46.37 | 45.90 | 4.02 | 8.15 | 12.58 | 51.75 | 47.90 |
| PointLLM-7B [ 68 ] | 44.85 | 47.47 | 48.55 | 3.87 | 7.30 | 11.92 | 53.00 | 41.20 |
Table 1: Performance comparison on various 3D downstream tasks. Here, the Objaverse dataset is used for 3D object captioning and recognition tasks, and the MM-Vet dataset is used for 3D VQA. In addition to our full model (SAGE) , we provide the results without the preference optimization (SAGE ∗ ) . Additional comparisons with existing methods on different datasets are provided in the Appendix A3 .
| Model | Latency | Throughput |
|---|---|---|
| (ms) | (Sample/s) | |
| PointLLM-7B | 239 | 4.2 |
| SAGE-7B | 100 | 10.0 |
Table 2: Runtime Complexity Comparison. Inference latency (ms) and memory (GB) on H100 GPU, 8K points, and Objaverse dataset.
Figure 4: Performance of SAGE on diverse ranges of point cloud resolution on 3D captioning task on Objaverse dataset.
| Samples 1, 2 | ||
|---|---|---|
| Uid | 0ea33b6617174530b97d6b7a92c275fb | de8ec2a724f14fc4b54624512f80f13e |
| Prompt | What is this? | This is an object of |
| Human | A cartoon green and red like a fruit | A black insect |
| InstructBLIP [ 15 ] | an appleavatar 3d model | 26 |
| LLaVA [ 38 ] | This is a computer-generated image (CGI) or a 3D model of an apple with a leaf on it. | This is an image of a large, close-up, and dark-colored insect, possibly a big cockroach, with long antennae. |
| 3D-LLM [ 25 ] | A 3D model of a red apple. | A small, black spider with a long tail. |
Table 3: Qualitative results on Objaverse. We adopt this table from [ 68 ] , and compare our proposed model’s generated description for the specific 3D object samples.
| Stage | Hyperparam. | Value |
| 0.25 | ||
| 0.50 | ||
| Codebook size | 8192 | |
| Warm-up ratio | 0.03 | |
| Weight decay | 0.05 | |
| Number of sampled points, | 512 |
Table A1 : List of model-specific hyperparameters.
| ScanQA (val) | Scan2Cap (val) | ||||||
| Method | Modality | B-4 | M | R | B-4 | M | R |
| Specialist Models | |||||||
| ScanQA [ 1 ] | PC | 10.1 | 13.1 | 33.3 | - | - | - |
| 3D-VLP [ 33 ] | PC | 11.2 | 13.5 | 34.5 | 32.3 | 24.8 | 51.5 |
| 3D-VisTA [ 81 ] | PC | 10.4 | 13.9 | 45.7 | 34.1 | 26.8 | 55.0 |
| Scan2Cap [ 11 ] | PC | - | - | - | 23.3 | 22.0 | 44.8 |
Table A2 : Performance comparison with existing methods. We adopt the following table from [ 17 ] and follow the notations and categorization defined by them. Here, “Specialist Model” refers to models specifically designed for individual tasks such as 3D question answering, 3D dense captioning, or referring segmentation. “Finetuned 3D MLLMs” denotes models that are jointly trained and subsequently fine-tuned on each dataset before evaluation. “3D MLLMs” represents models trained on multiple tasks without task-specific fine-tuning. “PC” stands for point cloud, and “I” denotes multi-view images. Note that the results of LEO [ 28 ] on ScanQA are shown in gray and excluded from direct comparison, as the model uses a different setting that accesses ground-truth objects related to the questions. All models use the 7B parameter configuration.
Figure A1 : Fig. (Left) Impact of the number of LLM trainable layers during the stage 1 training. Fig. (Middle) Impact of the number of LLM trainable layers during stage 1 training. Fig(Right) Impact of group normalization coefficient on performance of SAGE-7B.
| Model | Training time (h) |
|---|---|
| PointLLM | 26.1 |
| SAGE ∗ | 18.0 |
| SAGE | 27.4 |
Table A3: Total training time.
| Method | ModelNet40 | ScanObjectNN | ||||
|---|---|---|---|---|---|---|
| Top1 | Top3 | Top5 | Top1 | Top3 | Top5 | |
| 2D Inference without 3D Training | ||||||
| PointCLIP [ 73 ] | 19.3 | 28.6 | 34.8 | 10.5 | 20.8 | 30.6 |
| PointCLIPv2 [ 80 ] | 63.6 | 77.9 | 85.0 | 42.2 | 63.3 | 74.5 |
| Trained on ShapeNet | ||||||
| ReCon [ 47 ] | 61.2 | 73.9 | 78.1 | 42.3 | 62.5 | 75.6 |
Table A4 : Performance comparison on Zero-shot 3D classification on ModelNet40 [ 65 ] and ScanObjectNN [ 60 ] . Here, Ensembled [ 40 ] reprsents pretraining with four datasets, ShapeNet [ 6 ] , ABO [ 14 ] and 3D-FUTURE [ 20 ] . † : Uni3D employs a larger EVA-CLIP-E [ 52 ] teacher, while other methods employ OpenCLIP-bigG [ 30 ] .
| Model | Captioning | Cls. | VQA | |||||
|---|---|---|---|---|---|---|---|---|
| GPT-4 | Sentence-BERT | SimCSE | BLEU-1 | ROUGE-L | METEOR | GPT-4 | GPT-4 | |
| SAGE-7B ∗ | 49.05 | 49.23 | 48.56 | 7.41 | 10.25 | 14.35 | 55.71 | 46.38 |
| SD | 0.09 | 0.04 | 0.05 | 0.04 | 0.02 | 0.02 | 0.03 | 0.04 |
| SAGE-7B | 50.98 | 50.11 | 49.70 | 9.50 | 12.66 | 16.95 | 57.11 | 49.53 |
| SD | 0.08 | 0.03 | 0.04 | 0.05 | 0.04 | 0.03 | 0.03 | 0.05 |
Table A5 : Performance comparison across three runs on (SAGE) and (SAGE ∗ ) on 7B parameters. Here, the Objaverse dataset is used for 3D object captioning and recognition tasks, and the MM-Vet dataset is used for 3D VQA.
Table A6 : Sensitivity study on different model-specific parameters on ModelNet40.
| Embedding type | S-BERT |
|---|---|
| Continuous (w/o codebook) | 47.67 |
| Discrete (with codebook) | 50.11 |
Table A7: Impact of discrete vs. continuous point embeddings.
| LLM | Model | S-BERT |
|---|---|---|
| LLaMA-3.1 | PointLLM | 51.23 |
| SAGE | 55.89 | |
| Qwen-2.5 | PointLLM | 52.35 |
| SAGE | 56.91 |
Table A8: Impact of different LLM backbones.
Figure A2 : Dialogues between PointLLM and a human user.
| Samples 1, 2 | ||
|---|---|---|
| Uid | 059a7936ed89419ba9eae3153753ae86 | 82413f74efb84bc3b872a7a8f26cca81 |
| Prompt | Caption this 3D model in detail. | Caption this 3D model in detail. |
| Human | A blue cartoon | A white fish with orange/red spots |
| InstructBLIP [ 15 ] | There is a cute blue and white pony in a still, looking cheerful and cheerful. It’s standing next to a larger blue-colored object, potentially a barrier or structure, which provides the backdrop for this scene. The overall image is designed to create a fun, whimsical setting for the pony. | There is a fish hanging on a string in front of a black background. The fish is swimming in the middle of the image and is a little bit up from the bottom of the screen. The fish has a red spots on the left and right sides, and is positioned slightly towards the top of the screen. |
| LLaVA [ 38 ] | The image features a 3D model of a cute, colorful cartoon unicorn with a blue and white body and a hint of pink. The unicorn has a magical aura and is set against a black background. Its long, flowing mane and tail add to its whimsical charm. The unicorn appears to be in motion, possibly flying or prancing, as it captures the viewer’s imagination. | The image features a 3D model of a fish floating in a dark, empty space. The fish is positioned towards the left side of the image and appears to be swimming upwards. The fish is large and occupies a significant portion of the image, giving the impression of a life-like, computer-generated representation. The dark background and lack of other objects in the scene emphasize the fish’s presence and create a sense of depth and dimension. |
| 3D-LLM [ 25 ] | A 3D model of a purple and pink pony with long hair, standing on its hind legs. | A small fish with a long tail is visible in the image. It has no distinguishing features or markings, and its color appears to be brown. |
Table A9 : Qualitative results on the Objaverse dataset.
More in this topic
3D VLMs
Three-dimensional vision-language models connect language with spatially grounded visual data for 3D understanding, reasoning, and action.
See all 211 papers