Organizations: Tsinghua Shenzhen International Graduate School, Tsinghua University, Shenzhen, China · College of Computer Science and Software Engineering, Shenzhen University, Shenzhen, China
3D MLLMs recognize objects via pattern matching, which fails when geometrically similar objects require functional reasoning to disambiguate (e.g., dresser vs. wardrobe). Our per-sample Geometric Confusion Score (GCS) exposes this failure: baseline accuracy is 33.7% on ModelNet40's most ambiguous 20% versus 84.0% on its least ambiguous 20%. We identify a function-as-annotation pattern, in which functional language decorates a known identity instead of supporting inference, and hypothesize that reversing this information flow should concentrate improvements on the most ambiguous samples. PFIR (Part-level Functional Inference for Recognition) reverses that flow. Built on PartNeXt part segmentation, its data pipeline produces 507K QA pairs with functional CoTs (geometry → function → identity), and its progressive curriculum culminates in dual-objective GRPO: strict functional reasoning and descriptive grounding for Objaverse's diverse distribution. The two RL objectives yield nearly orthogonal task vectors (cos=0.056), enabling composition via Task Arithmetic. Under a strict dual OOD protocol (Objaverse → ModelNet40; XYZRGB → XYZ-only), PFIR raises zero-shot accuracy from 60.5% to 66.7%, with gains concentrated 2.6× more on geometrically ambiguous samples---supporting part-level functional reasoning as an effective route to more generalizable 3D recognition. Code: https://github.com/this233/pfir-3d.
Figures & tables
Figure 1. Motivation. Pattern matching collapses on geometrically ambiguous objects (a) . A per-sample GCS reveals that the hardest 20% reach only 33.7% baseline accuracy. PFIR addresses this via part-level functional reasoning, with gains concentrated 2.6× more on the most confused quintile (b) , by shifting from function-as-annotation to function-as- evidence (c) . Teaser figure: confusion-prone 3D object pairs, GCS-stratified accuracy comparison, and information-flow contrast between function-as-annotation and function-as-evidence paradigms.
Figure 2. Overview of the proposed PFIR framework. The part-level functional pipeline produces 507K QA pairs with function-as-evidence reasoning chains. The three-stage curriculum progressively builds function-guided recognition: Stage 1 aligns part-aware representations, Stage 2 teaches the reasoning format via SFT, and Stage 3 trains two complementary RL experts (functional reasoning + descriptive grounding) composed via Task Arithmetic. Overview of the PFIR framework: part-level functional data pipeline and three-stage training curriculum.
Method
LLM
3D Enc.
Train Data
MN40 Cls. (%)
Obj. Cls. (%)
Obj. Caption (Img-Based)
(I)
(C)
Avg
(I)
(C)
Avg
Corr
Hall ↓
Prec
Point-Bind LLM ( Guo et al., 2023 )
LLaMA-7B
I2P-MAE
SN
48.74
37.60
43.17
6.00
4.50
5.25
0.61
3.77
13.91
PointLLM-7B ( Xu et al., 2024 )
Vicuna-7B
Point-BERT
Obj.
51.54
50.61
51.08
61.00
63.50
62.25
2.43
1.87
56.57
PointLLM-13B ( Xu et al., 2024 )
Vicuna-13B
Point-BERT
Obj.
50.65
50.93
50.79
64.50
64.00
64.25
2.48
1.82
57.84
ShapeLLM-7B ( Qi et al., 2024a )
Vicuna-7B
ReCon++
Obj.+GAP
53.08
53.08
53.08
54.50
54.50
54.50
1.05
1.36
53.22
ShapeLLM-13B ( Qi et al., 2024a )
Vicuna-13B
ReCon++
Obj.+GAP
52.96
52.96
52.96
54.00
54.00
54.00
1.67
1.44
53.76
Table 1. Generative zero-shot 3D classification and captioning. (I)/(C): instruction-/completion-typed prompt. Captioning: GPT-4o image-based scoring on 200 Objaverse objects following ( Xu et al., 2025 ) . ModelNet40 ( Wu et al., 2015 ) is OOD and XYZ-only for all methods. Best in bold , second best underlined . 3 3 3 Dataset abbreviations: Obj. = Objaverse ( Deitke et al., 2023 ) , SN = ShapeNet ( Chang et al., 2015 ) , GAP = GAPartNet ( Geng et al., 2023 ) .
Samples 1, 2
Point Cloud Curvature Vis.
Prompt
What is this?
Briefly caption this 3D model.
Ground Truth
Dresser
A cartoon black monster like a dragon
InstructBLIP-13B ( Dai et al., 2023 )
3d model.
a black lizard with a sharp tooth in a dark room
LLaVA-13B ( Liu et al., 2023a )
The image features square box or cube.
A 3D model of a dark, menacing dragon.
3D-LLM ( Hong et al., 2023 )
-
A black and white tiger with long legs, standing on its hind leg.
Point-Bind LLM ( Guo et al., 2023 )
-
The 3D model features a large, ornate gargoyle with a horned helmet, sitting on top of a building.
Table 2. Qualitative comparisons. The classification and caption results of models on ModelNet40 and Objaverse are presented. Correct answers and fine-grained functional descriptions are highlighted in green bold .
Path
Obj.
MN40
GCS quintile (easy → hard)
OB-I
Q1
Q2
Q3
Q4
Q5
S2 start
–
61.6
81.9
73.8
66.7
51.5
34.3
55.0
S1 → RL
RFR
65.5
88.0
79.7
70.8
52.9
36.1
65.4
S2 → RL
RFR
68.2
88.4
79.7
71.6
57.4
44.2
65.5
S2 → RL
RDG
52.4
83.0
63.1
47.7
41.4
27.0
79.2
S2 joint
RFR⋅RDG
65.4
86.4
76.1
68.2
55.8
40.7
63.0
Table 3. GCS-aware ablation of RL strategies (%). PLLM-V2 denotes PointLLM-V2 ( Xu et al., 2025 ) ; † marks our reproduction. S1/S2 are checkpoints; S1 → RL skips Stage 2 CoT SFT. ⊕ denotes Task Arithmetic, OB-I is Objaverse instruction accuracy, and Δ is PFIR minus PLLM-V2. Best/second-best scores are bold/underlined; bold Δ marks the largest quintile gain.
Configuration
Stage 1
Stage 2
Stage 3
MN40-I
PointLLM-V2 baseline
–
–
–
60.5
+ Multi-source alignment
✓
–
–
59.9
+ CoT SFT
✓
✓
–
61.6
+ Dual-obj. RL + Merge
✓
✓
✓
66.7
Table 4. Ablation on training stages. Each row adds one stage to the previous configuration. ModelNet40 instruction-prompt accuracy (%) is reported.
Re-annot.
Part-loc.
PF direct
MN40-I
✓
–
–
47.4
✓
✓
–
49.5
✓
✓
✓
59.9
Table 5. Ablation on Stage 1 data sources. MN40-I: ModelNet40 instruction-prompt accuracy (%) evaluated after Stage 1 alignment (projector-only training), before SFT and RL stages.