Organizations: Xi’an Jiaotong University · Shanghai AI Laboratory · Tsinghua University · Shanghai Jiao Tong University · Zhejiang University · Nanyang Technological University · Harbin Institute of Technology, Shenzhen · Institute of Automation, CASIA · Taiyuan University of Technology
While Multimodal Large Language Models (MLLMs) demonstrate proficiency in 2D scenes, extending their perceptual intelligence to 3D point cloud understanding remains a significant challenge. Current approaches focus primarily on aligning 3D features with pre-trained models. However, they typically treat geometric reasoning as an implicit mapping process. These methods bypass intermediate logical steps and consequently suffer from geometric hallucinations. They confidently generate plausible responses that fail to ground in precise structural details. To bridge this gap, we present PointCoT, a novel framework that empowers MLLMs with explicit Chain-of-Thought (CoT) reasoning for 3D data. We advocate for a Look, Think, then Answer paradigm. In this approach, the model is supervised to generate geometry-grounded rationales before predicting final answers. To facilitate this, we construct Point-Reason-Instruct, a large-scale benchmark comprising ∼86k instruction-tuning samples with hierarchical CoT annotations. By leveraging a dual-stream multi-modal architecture, our method synergizes semantic appearance with geometric truth. Extensive experiments demonstrate that PointCoT achieves state-of-the-art performance on complex reasoning tasks.
Figures & tables
Figure 1: Existing 3D-LLMs (left) treat geometric reasoning as a direct mapping process, often suffering from Geometric Hallucination, failing to perceive the missing leg and incorrectly judging the chair as stable. In contrast, our proposed PointCoT (right) introduces an explicit Look-Think-Answer paradigm. By generating a geometry-grounded rationale (detecting the missing rear-left leg) before the final conclusion, our method significantly reduces hallucinations and enables interpretable 3D reasoning.
Figure 2: The Data Construction Pipeline of Point-Reason-Instruct. The pipeline consists of three stages: (1) Dual-Stream Preprocessing, where objects are sampled into point clouds and rendered into 8 spherical views; (2) Multi-Task Reasoning Generation, where the Qwen2.5-VL teacher agent generates hierarchical CoT rationales covering geometric attributes, spatial relations, and functionality; and (3) Quality Filtering, which validates the rationales against metadata to eliminate hallucinations.
Figure 3: The overall architecture of PointCoT. The framework operates in a Look-Think-Answer paradigm. In the Look Stage, a dual-stream encoder extracts geometric and visual features, which are fused into a Tri-Modal Manifold z via Geometry-Guided Cross-Modal Attention. During the Think Stage, a VLM autoregressively generates an explicit rationale R , while its hidden states ht are strictly grounded to Hgeo via an InfoNCE loss Lanchor to mitigate spatial hallucinations. Finally, in the Answer Stage, the final answer A is deduced conditionally on both z and R . The entire pipeline is trained through a progressive dual-stage optimization using Lgen and Lpred .
Model
Modality
Backbone (Encoder / LLM)
Overall
Geo.
Spat.
Func.
General-Purpose 2D VLMs (Zero-shot*)
GPT-4V [ 1 ]
Img
Proprietary / GPT-4
65.4
58.2
68.5
71.2
Qwen2-VL-7B [ 3 ]
Img
ViT-L / Qwen2
63.8
54.5
64.2
69.5
LLaVA-1.5 [ 18 ]
Img
ViT-L / Vicuna-7B
54.1
47.2
51.8
58.9
Specialized 3D-LLMs (Fine-tuned)
Chat-3D v2 [ 10 ]
PC
ViT-g / Vicuna-7B
66.1
72.4
63.2
62.8
Table 1: Quantitative results on the Point-Reason-Instruct benchmark. We report the Accuracy (%) across three cognitive dimensions. Geo. evaluates fundamental structural perception; Spat. assesses relative positioning and connectivity; Func. measures high-level deductive reasoning regarding physical utility.
LLM Backbone
Point Encoder
Overall Acc (%)
Reasoning Score
Vicuna-7B
PointBERT
73.4
7.9
Mistral-7B-v0.3
PointBERT
76.8
8.3
Qwen2.5-7B (Ours)
PointBERT
78.5
8.5
Qwen2.5-7B
PointNeXt
79.6
8.7
Table 2: Evaluation of architectural agnosticism. We report the Overall Accuracy (%) and GPT-4 based Reasoning Score (1-10) for PointCoT instantiated with varying LLM backbones and 3D point encoders. The results demonstrate steady scalability and robustness across diverse network configurations. Best results are highlighted in bold.
Method
Training Data (Source)
Data Scale
ScanQA †
Objaverse
3D-LLM [ 9 ]
300k (3D-Data)
4.3x
24.5
49.2
Point-LLM [ 29 ]
660k (Cap3D)
9.6x
22.4
45.1
PointCoT (Ours)
∼ 69k (Ours)
1.0x
23.4
51.8 (+2.6)
Table 3: Zero-Shot Transfer Evaluation. We assess generalization capability on unseen benchmarks. ScanQA evaluates scene-level spatial reasoning (BLEU-4), while Objaverse evaluates open-vocabulary classification (Accuracy). † indicates evaluation on localized instance crops to align with our object-centric training regime.
Method
Correctness
Logic
Grounding
Average
Point-LLM (CoT-adapted)
7.1
6.8
6.2
6.7
GPT-4V
8.1
8.4
7.6
8.0
PointCoT (Ours)
8.4
8.6
8.9
8.6
Table 4: Automated Evaluation of Rationale Quality via GPT-4. Scores range from 1 to 10. The evaluator is strictly conditioned on ground-truth 3D metadata to ensure objective geometric verification.
Table 5: Ablation of input modalities and reasoning strategies. (a) Hybrid modality synergizes semantic and geometric precision. (b) Explicit CoT significantly suppresses geometric hallucinations compared to black-box mapping.
Figure 4: Qualitative comparison on the Point-Reason-Instruct benchmark. While the baseline relies on implicit semantic priors and suffers from geometric hallucinations, PointCoT follows a Look-Think-Answer paradigm. By actively selecting decisive viewpoints and explicitly grounding rationales in local 3D geometric structures, PointCoT effectively mitigates hallucinations and yields interpretable predictions.
Visual Computing Research Center (VCC), College of Computer Science and Software Engineering (CSSE), Shenzhen University, China · VCC, CSSE, Shenzhen University, China