Organizations: Xi’an Jiaotong University · Shanghai AI Laboratory · Tsinghua University · Shanghai Jiao Tong University · Zhejiang University · Nanyang Technological University · Harbin Institute of Technology, Shenzhen · Institute of Automation, CASIA · Taiyuan University of Technology
While Multimodal Large Language Models (MLLMs) demonstrate proficiency in 2D scenes, extending their perceptual intelligence to 3D point cloud understanding remains a significant challenge. Current approaches focus primarily on aligning 3D features with pre-trained models. However, they typically treat geometric reasoning as an implicit mapping process. These methods bypass intermediate logical steps and consequently suffer from geometric hallucinations. They confidently generate plausible responses that fail to ground in precise structural details. To bridge this gap, we present PointCoT, a novel framework that empowers MLLMs with explicit Chain-of-Thought (CoT) reasoning for 3D data. We advocate for a Look, Think, then Answer paradigm. In this approach, the model is supervised to generate geometry-grounded rationales before predicting final answers. To facilitate this, we construct Point-Reason-Instruct, a large-scale benchmark comprising ∼86k instruction-tuning samples with hierarchical CoT annotations. By leveraging a dual-stream multi-modal architecture, our method synergizes semantic appearance with geometric truth. Extensive experiments demonstrate that PointCoT achieves state-of-the-art performance on complex reasoning tasks.
Figures & tables
Figure 1: Existing 3D-LLMs (left) treat geometric reasoning as a direct mapping process, often suffering from Geometric Hallucination, failing to perceive the missing leg and incorrectly judging the chair as stable. In contrast, our proposed PointCoT (right) introduces an explicit Look-Think-Answer paradigm. By generating a geometry-grounded rationale (detecting the missing rear-left leg) before the final conclusion, our method significantly reduces hallucinations and enables interpretable 3D reasoning.
Figure 2: The Data Construction Pipeline of Point-Reason-Instruct. The pipeline consists of three stages: (1) Dual-Stream Preprocessing, where objects are sampled into point clouds and rendered into 8 spherical views; (2) Multi-Task Reasoning Generation, where the Qwen2.5-VL teacher agent generates hierarchical CoT rationales covering geometric attributes, spatial relations, and functionality; and (3) Quality Filtering, which validates the rationales against metadata to eliminate hallucinations.
Figure 3: The overall architecture of PointCoT. The framework operates in a Look-Think-Answer paradigm. In the Look Stage, a dual-stream encoder extracts geometric and visual features, which are fused into a Tri-Modal Manifold z via Geometry-Guided Cross-Modal Attention. During the Think Stage, a VLM autoregressively generates an explicit rationale R , while its hidden states ht are strictly grounded to Hgeo via an InfoNCE loss Lanchor to mitigate spatial hallucinations. Finally, in the Answer Stage, the final answer A is deduced conditionally on both z and R . The entire pipeline is trained through a progressive dual-stage optimization using Lgen and Lpred .
Model
Modality
Backbone (Encoder / LLM)
Overall
Geo.
Spat.
Func.
General-Purpose 2D VLMs (Zero-shot*)
GPT-4V [ 1 ]
Img
Proprietary / GPT-4
65.4
58.2
68.5
71.2
Qwen2-VL-7B [ 3 ]
Img
ViT-L / Qwen2
63.8
54.5
64.2
69.5
LLaVA-1.5 [ 18 ]
Img
ViT-L / Vicuna-7B
54.1
47.2
51.8
58.9
Specialized 3D-LLMs (Fine-tuned)
Chat-3D v2 [ 10 ]
PC
ViT-g / Vicuna-7B
66.1
72.4
63.2
62.8
Table 1: Quantitative results on the Point-Reason-Instruct benchmark. We report the Accuracy (%) across three cognitive dimensions. Geo. evaluates fundamental structural perception; Spat. assesses relative positioning and connectivity; Func. measures high-level deductive reasoning regarding physical utility.
LLM Backbone
Point Encoder
Overall Acc (%)
Reasoning Score
Vicuna-7B
PointBERT
73.4
7.9
Mistral-7B-v0.3
PointBERT
76.8
8.3
Qwen2.5-7B (Ours)
PointBERT
78.5
8.5
Qwen2.5-7B
PointNeXt
79.6
8.7
Table 2: Evaluation of architectural agnosticism. We report the Overall Accuracy (%) and GPT-4 based Reasoning Score (1-10) for PointCoT instantiated with varying LLM backbones and 3D point encoders. The results demonstrate steady scalability and robustness across diverse network configurations. Best results are highlighted in bold.
Method
Training Data (Source)
Data Scale
ScanQA †
Objaverse
3D-LLM [ 9 ]
300k (3D-Data)
4.3x
24.5
49.2
Point-LLM [ 29 ]
660k (Cap3D)
9.6x
22.4
45.1
PointCoT (Ours)
∼ 69k (Ours)
1.0x
23.4
51.8 (+2.6)
Table 3: Zero-Shot Transfer Evaluation. We assess generalization capability on unseen benchmarks. ScanQA evaluates scene-level spatial reasoning (BLEU-4), while Objaverse evaluates open-vocabulary classification (Accuracy). † indicates evaluation on localized instance crops to align with our object-centric training regime.
Method
Correctness
Logic
Grounding
Average
Point-LLM (CoT-adapted)
7.1
6.8
6.2
6.7
GPT-4V
8.1
8.4
7.6
8.0
PointCoT (Ours)
8.4
8.6
8.9
8.6
Table 4: Automated Evaluation of Rationale Quality via GPT-4. Scores range from 1 to 10. The evaluator is strictly conditioned on ground-truth 3D metadata to ensure objective geometric verification.
Table 5: Ablation of input modalities and reasoning strategies. (a) Hybrid modality synergizes semantic and geometric precision. (b) Explicit CoT significantly suppresses geometric hallucinations compared to black-box mapping.
Figure 4: Qualitative comparison on the Point-Reason-Instruct benchmark. While the baseline relies on implicit semantic priors and suffers from geometric hallucinations, PointCoT follows a Look-Think-Answer paradigm. By actively selecting decisive viewpoints and explicitly grounding rationales in local 3D geometric structures, PointCoT effectively mitigates hallucinations and yields interpretable predictions.
Understanding 3D point clouds through language remains a fundamental challenge in computer graphics and visual computing, due to the irregular structure of point cloud data and the lack of explicit reasoning in existing 3D multimodal models. While Chain-of-Thought (CoT) reasoning has shown strong effectiveness in LLMs and image-based MLLMs, its extension to 3D understanding remains largely underexplored. In this paper, we propose a data-centric framework for constructing large-scale CoT supervision tailored to 3D point cloud understanding. Our framework consists of a two-stage pipeline that first refines point-text instruction data via vision-language-model-based quality evaluation and reference-guided refinement, and then synthesizes high-quality reasoning paths through Human-in-the-Loop Prompt Optimization (HiLPO). Using this approach, we build PoCoTI, a CoT-enhanced point-text instruction-following dataset containing 55K samples with explicit reasoning paths. Fine-tuning PointLLM on PoCoTI yields PointLLM-R, a reasoning-capable 3D multimodal language model. Extensive experiments on generative 3D classification and captioning demonstrate that PointLLM-R achieves state-of-the-art performance and generalizes robustly to real-world scanned point clouds and multi-turn dialogue scenarios.
Chaoqi Chen, Qile Xu, Wenjun Zhou +1
Visual Computing Research Center (VCC), College of Computer Science and Software Engineering (CSSE), Shenzhen University, China · VCC, CSSE, Shenzhen University, China
Multi-modal large language models (MLLMs) have shown remarkable progress in integrating visual and linguistic understanding. Recent efforts have extended these capabilities to 3D understanding through encoder-based architectures that rely on pre-trained 3D encoders to extract geometric features. However, such approaches suffer from semantic misalignment between geometric and linguistic spaces, resolution sensitivity, and substantial computational overhead. In this work, we present SAGE, the first end-to-end 3D MLLM that directly processes raw point clouds without relying on a pre-trained 3D encoder. Our approach introduces a lightweight 3D tokenizer that combines geometric sampling and neighbourhood aggregation with vector quantization to convert point clouds into discrete tokens--treating 3D data as a foreign language that naturally extends the LLM's vocabulary. Furthermore, to enhance the model's reasoning capability on complex 3D tasks, we propose a preference optimization training strategy with a semantic alignment-based reward, specifically designed for open-ended 3D question answering where responses are descriptive. Extensive experiments across diverse 3D understanding benchmarks demonstrate that our end-to-end approach outperforms existing encoder-based methods while offering significant advantages in computational efficiency, generalization across LLM backbones, and robustness to input resolution variations. Code is available at: github.com/snehaputul/SAGE3D.
While Multimodal Large Language Models (MLLMs) have achieved remarkable success in 2D visual understanding, their ability to reason about 3D space remains limited. To address this gap, we introduce geometrically referenced 3D scene representations (GR3D). Given a set of input images, GR3D annotates objects in the images with unique IDs and encodes their 3D geometric attributes as textual references indexed by these IDs. This representation enables MLLMs to interpret 3D cues using their advanced language-based skills in mathematical reasoning, while concurrently analyzing 2D visual features in a tightly coupled way. We present a simple yet effective approach based on GR3D, which requires no additional training and is readily applicable to different MLLMs. Implemented in a zero-shot setting, our approach yields substantial improvements on challenging spatial reasoning benchmarks, boosting GPT-5 performance by 9% on VSI-Bench and 12% on MindCube. Qualitative studies further demonstrate that GR3D empowers MLLMs to perform complex spatial reasoning with highly sparse input views.