cs.CVAug 16, 2026
SaveBeyond Single Object: Learning 3D Relations with Large Language Models
Organizations: AIST · University of Tsukuba · University of Technology Nuremberg · University of Oxford
Abstract
We address a fundamental gap in 3D-LLMs: existing models focus on single-object/scene description, struggling with detailed, inter-object comparison. We propose a framework for detailed object-level reasoning across multiple objects with three components: (1) MO3D (Multi-Object in 3D), an instruction dataset requiring fine-grained multi-object comparison; (2) Multi-3DLLM, using a minimal Patch-Interaction Transformer (PIT) that models inter-/intra-object relationships while preserving local geometry; (3) Mini-apps, two application-driven benchmarks (Shape Mating, Change Captioning) that probe geometric understanding for practical use. Recent 3D-LLMs and 2D-VLMs perform poorly on these tasks, lacking both comparison-centric design and geometric awareness. In contrast, Multi-3DLLM trained on our mixture data learns geometric reasoning, surpasses all baselines on MO3D, and provides positive transfer to single-object classification.
Figures & tables
Figure 1: Overview of our proposed multi-object 3D understanding framework. Left: existing models fail at complex, multi-object 3D tasks. Standard 3D-LLMs cannot perform fine-grained comparisons, and 2D-VLMs are geometrically-unaware, failing tasks like complementarity. Right: our Multi-3DLLM, trained on our MO3D and Mini-apps datasets, addresses these challenges, enabling robust comparison (top) and geometric understanding (bottom).
Figure 2: Overview of our three proposed benchmarks. MO3D tasks models with multi-object comparison, including Positional , Comparative , and Holistic understanding. Shape Mating is a geometrically-intensive task requiring the model to identify the single correct complementary pair from a set of three components. Change Captioning tasks the model with generating a natural language description of the geometric edit that transforms an anchor object (left chair) into a target object (right chair).
Figure 3: Multi-3DLLM architecture. point clouds are encoded independently by the Point Encoder and Projector. A PIT Block then performs self-attention over the concatenated point tokens, injecting cross-object relations before the tokens are passed to the LLM.
| Model | MO3D | Shape Mating | Change Captioning | ||||||
| Positional (%) | Comparative (%) | Holistic (%) | Selection (%) | Verify (%) | Delta Caption (%) | ||||
| M | M | B | R | S | R | B | R | M | |
| LLaVA-7B (n images) | 25.8 | 11.7 | 61.1 | 41.9 | 20.4 | 2.0 | 48.2 | 29.6 | 7.6 |
| LLaVA-7B (2*n images) | 21.3 | 4.8 | 59.9 | 31.0 | 21.9 | 2.5 | 49.1 | 34.6 | 9.3 |
| Molmo-7B (n images) | 16.5 | 10.5 | 61.8 | 41.6 | 20.7 | 0.2 | 27.8 | 29.6 | 0.0 |
| Molmo-7B (2*n images) | 16.8 | 6.9 | 63.3 | 36.5 | 22.7 | 0.3 | 34.3 | 28.1 | 0.0 |
Table 1: Comprehensive comparison of Multi-3DLLM against SOTA 3D-LLM and 2D-VLM baselines across all proposed tasks: MO3D, Shape Mating, and Change Captioning. Metrics are reported as: M = Semantic Accuracy, B = Binary Accuracy, R = Reasoning Accuracy, and S = Selection Accuracy (all in %). For 3D-LLM baselines, “Concat.” denotes the point cloud concatenation strategy. For 2D-VLM baselines, “(n images)” and “(2*n images)” refer to providing 1 view and 2 opposing views per object, respectively.
| 1-Obj | Multi-Obj Task (%) | |||||
| Task | 2 Inputs | 3 Inputs | ||||
| Model | (%) | M=1 | M=2 | M=1 | M=2 | M=3 |
| PointLLM [ 53 ] | 50.7 | - | - | - | - | - |
| PointLLM [ 53 ] w/ MI | - | 7.4 | 31.2 | 2.1 | 8.7 | 27.6 |
| Ours w/o PIT block | 53.4 | 53.9 | 52.1 | 52.9 | 51.7 | 50.2 |
| Ours | 54.2 | 54.7 | 51.1 | 5 2.5 | 51.2 | 51.5 |
Table 2: Zero-shot classification (ModelNet40) results for single-object (1-Obj) versus multi-object (Multi-Obj) inputs. ‘M’ denotes the ordinal position of the queried object in 2-input and 3-input scenes, ‘MI’ indicates multiple 3D point cloud inputs, and ‘Ours’ employs Multi-3DLLM architecture.
| Training Strategy | MO3D | Shape Mating | Change Captioning | ||||||
| Positional (%) | Comparative (%) | Holistic (%) | Selection (%) | Verify (%) | Delta Caption (%) | ||||
| M | M | B | R | S | R | B | R | M | |
| Multi-3DLLM (MO3D-only) | 45.5 | 21.9 | 84.6 | 53.0 | - | - | - | - | - |
| Multi-3DLLM (ShapeMating-only) | - | - | - | - | 34.4 | 36.7 | - | - | - |
| Multi-3DLLM (ChangeCaptioning-only) | - | - | - | - | - | - | 49.7 | 34.0 | 48.2 |
| Multi-3DLLM (All Tasks) | 56.3 | 33.8 | 81.7 | 57.2 | 37.1 | 36.8 | 51.2 | 37.3 | 51.0 |
Table 3: Ablation study on the training data mixture. We compare our final model (trained on a uniform mixture of all three datasets) against specialized models trained only on a single downstream task. This analysis measures the effect of co-training with our main MO3D dataset on the performance of the mini-applications.
| Model | Sparsity (Gini) | Entropy | Top-10% Conc. | Top-20% Conc. | Effective Rank |
| Multi-3DLLM (MO3D-Only) | 0.605 | 0.822 | 0.538 | 0.652 | 2209 |
| Multi-3DLLM | 0.726 | 0.716 | 0.675 | 0.763 | 1088 |
Table 4: Quantitative comparison of attention patterns between our Multi-3DLLM model and a model trained only on MO3D baseline. Metrics are computed on 266 object instances from the MO3D test set. Sparsity (Gini) measures attention inequality (1 = max concentration). Entropy measures attention uniformity (0 = max concentration). Top-K% Conc indicates proportion of attention mass on the top K% of points. Effective Rank approximates the number of points receiving substantial attention, computed as where is the entropy.
Figure 4: Attention visualization comparison. Attention weights overlaid on a 3D point cloud from the MO3D test set. Left (MO3D Only): The task-specific model shows distributed, diffuse attention across the entire object. Right (MO3D + Mini-apps): Our multitask model exhibits concentrated and sparse attention. Attention weights are mapped to colors, with yellow indicating high attention and darker colors indicating low attention. The multitask model achieves competitive performance while attending to significantly fewer points (see Table 4 ).
Appendix figures & tables33 assets
Supplementary material from the paper’s appendix.
Appendix
| Parameter | Phase 1 | Phase 2 |
| Feature Alignment | Holistic Task-Mixture | |
| Batch Size | 16 | 14 |
| Learning Rate | 2e-3 | 2e-5 |
| Weight Decay | 0.0 | 0.0 |
| Number of Epochs | 3 | 3 |
| LR Scheduler | Cosine w/ Warmup | Cosine w/ Warmup |
Table 5: Training Hyperparameters.
Figure 5: Category Distribution and Weighted Balancing. We transform the naturally occurring, unbalanced attribute distribution (left) into a curated target distribution (middle). This process explicitly prioritizes geometric and structural categories ( e.g . , Geometry, Material) while suppressing superficial visual cues ( e.g . , Color) to enforce 3D understanding. The stacked bars (right) verify that this balanced distribution is consistently applied across all three task types: Positional, Comparative, and Holistic QA.
| Total Samples | Task Types | Group Types | |||
| Positional | Comparative | Holistic | Positive | Negative | |
| 69,996 | 23,332 | 23,332 | 23,332 | 34,926 | 35,070 |
| (100%) | (33.3%) | (33.3%) | (33.3%) | (49.9%) | (50.1%) |
Table 6: Overall Statistics of the MO3D Dataset. The dataset maintains a perfectly balanced distribution across the three core tasks and a near-even split between positive and negative object groupings.
| Category | Target Weight | Actual Count (%) |
| Geometry / Structure | 30% | 21,264 (30.4%) |
| Material | 20% | 14,316 (20.5%) |
| Function | 15% | 10,578 (15.1%) |
| Taxonomy | 15% | 10,200 (14.6%) |
| Style / Aesthetics | 15% | 10,206 (14.6%) |
| Color | 5% | 0 3,432 0 (4.9%) |
Table 7: Distribution of Question Categories. We compare our design target weights with the actual distribution in the final dataset. The resulting distribution closely matches our goal of prioritizing geometric and structural comparison.
| Core Category | Description and Example Question Focus | Examples of Merged Raw Categories |
| geometry/structure | The shape, size, number of parts, and structural complexity of an object. e.g., “Does it have more than four legs?” | shape , size , components , part , feature , features , detail , base , form |
| taxonomy | The general class, type, or identity of an object. e.g., “Is this object a piece of furniture?” | objecttype , type , object_type , object , theme , category |
| function | The intended purpose, use, or action associated with an object. e.g., “What is this object used for?” | function , action , purpose , usage , activity |
| material | The physical substance an object is made of. This is distinct from color. e.g., “Is the frame made of metal or wood?” | material , substance , composition |
| style/aesthetics | The visual style, era, pattern, decoration, or contextual setting of an object. e.g., “Does this object have a writing on the surface?” | style , decoration , pattern , design , era , location , context |
| color | The surface color or hue of an object or its parts. e.g., “Is the main color of the object brown?” | color , hue , finish |
Table 8: The six core reasoning categories curated from our data-driven analysis. We define each category and provide examples of the raw, LLM-discovered themes that were merged into it.
Figure 6: Intrinsic Ambiguity. Disagreements in human verification often arise from subjective perception ( e.g . abstract shapes or occlusions)
| Dataset | Task Type | Instruction | Response | ||
| Mean | Max | Mean | Max | ||
| MO3D | Positional | 10.8 | 21 | 10.0 | 25 |
| Comparative | 13.3 | 30 | 18.1 | 33 | |
| Holistic | 9.9 | 18 | 12.8 | 31 | |
| Shape Mating | Selection | 31.8 | 34 | 73.5 | 130 |
| Change Cap. | Verify | 64.3 | 191 | 25.4 | 83 |
Table 9: Linguistic Complexity Statistics. We report the average and maximum length in words for instructions and responses. Shape Mating features long responses due to the requirement of geometric rationales.
Figure 7: Point cloud generation pipeline for Shape Mating. Starting from a raw mesh (Left), we apply the cut shell operation to split the object into two complementary halves (Middle). Unlike solid cuts, this operation preserves the hollow, surface-only structure of the object. Finally, each part is uniformly sampled into 8,192 points (Right) to serve as the input for our model.
Figure 8: Hard Negative Sampling for Change Captioning. We construct contrastive triplets where the Negative candidate (red captions) is not random, but a “hard negative” sharing the same source Anchor as the Positive target (green captions). This Negative sample is crucial for the Verification task to generate challenging “No” instances
| Model | Pos. | Comp. | Hol. (B) | Hol. (R) | |
| Scene-level | Chat-Scene | 10.8 | 3.0 | 49.7 | 31.6 |
| LL3DA (Click) | 17.8 | 4.5 | 48.8 | 26.5 | |
| LL3DA (Bbox) | 18.2 | 2.9 | 48.8 | 25.9 | |
| Text-only | Vicuna-7B (GT captions) | 52.0 | 28.9 | 70.5 | 50.3 |
| Ours (Multi-3DLLM) | 56.3 | 33.8 | 81.7 | 57.2 | |
Table 10: Additional results on MO3D: scene-level and text-only baselines.
| Interaction Mechanism | MO3D | Shape Mating | Change Captioning | ||||||
| Positional (%) | Comparative (%) | Holistic (%) | Selection (%) | Verify (%) | Delta Caption (%) | ||||
| M | M | B | R | S | R | B | R | M | |
| No-Interaction (w/o PIT) | 45.5 | 21.8 | 81.3 | 53.0 | 34.4 | 36.7 | 49.1 | 37.1 | 48.0 |
| Object-Level (Mean) | 52.9 | 32.3 | 81.0 | 49.7 | 25.0 | 23.7 | 51.7 | 34.7 | 49.6 |
| Object-Level (Max) | 56.6 | 36.5 | 81.3 | 45.1 | 23.9 | 22.5 | 50.9 | 37.2 | 50.0 |
| Micro-Token ( ) | 56.9 | 35.3 | 80.1 | 44.0 | 24.5 | 21.8 | 50.6 | 34.6 | 50.0 |
Table 11: Extended ablation on interaction mechanics. We compare different pooling strategies (Mean vs. Max) and granularities (Object vs. Micro vs. Patch). Micro-Token uses 32 representative tokens per object. w/ PIT uses full patch-level interaction.
| Model | ScanObjectNN (Real) | OmniObject3D (Real) |
| LLaVA-7B (Zero-shot) | 21.6 | 26.0 |
| Ours (Zero-shot) | 29.0 | 36.3 |
| Ours (Fine-tune) | 48.0 | 62.0 |
Table 12: Results on real-world scanned datasets. Metric is Selection Accuracy (S, %) following the main paper. Chance rate is 25% for both.
| Model & Training Strategy | MO3D | Shape Mating | Change Captioning | ||||||
| Positional (%) | Comparative (%) | Holistic (%) | Selection (%) | Verify (%) | Delta Caption (%) | ||||
| M | M | B | R | S | R | B | R | M | |
| No-Interaction (1-Stage) | 47.6 | 21.0 | 78.9 | 41.9 | 23.8 | 22.0 | 49.1 | 39.4 | 42.0 |
| No-Interaction (2-Stage) | 45.5 | 21.8 | 81.3 | 53.0 | 34.4 | 36.7 | 49.1 | 37.1 | 48.0 |
| w/ PIT (1-Stage) | 47.6 | 24.5 | 78.9 | 44.8 | 66.6 | 64.1 | 50.3 | 38.4 | 56.0 |
| w/ PIT (2-Stage) | 56.3 | 33.8 | 81.7 | 57.2 | 37.1 | 36.8 | 51.2 | 37.3 | 51.0 |
Table 13: Ablation on training stages across architectures. We compare the 1-Stage and 2-Stage (Alignment Holistic Tuning) strategies for both the No-Interaction baseline and full model w/ PIT . The results investigate whether the initial feature alignment (Phase 1) is universally beneficial or specifically critical for our patch-interaction mechanism.
| Model | Positional QA | Comparative QA | Holistic QA | |||||||||
| B-4 | R-L | MET | Sim | B-4 | R-L | MET | Sim | B-4 | R-L | MET | Sim | |
| LLaVA (1-view) | 23.97 | 49.84 | 47.43 | 62.31 | 3.97 | 26.48 | 28.66 | 58.46 | 2.19 | 20.03 | 23.05 | 57.68 |
| LLaVA (2-view) | 23.79 | 49.55 | 47.48 | 61.62 | 3.59 | 26.18 | 27.52 | 58.16 | 2.29 | 20.32 | 23.51 | 58.19 |
| Molmo (1-view) | 3.88 | 24.27 | 31.26 | 55.04 | 3.35 | 21.26 | 27.87 | 63.21 | 1.01 | 15.15 | 19.92 | 54.22 |
| Molmo (2-view) | 3.40 | 24.20 | 30.70 | 55.06 | 3.40 | 21.23 | 27.43 | 62.79 | 0.99 | 14.92 | 19.17 | 53.47 |
| MiniGPT-3D | 15.99 | 44.01 | 50.36 | 63.04 | 13.73 | 42.01 | 40.52 | 64.59 | 4.20 | 24.38 | 24.28 | 58.46 |
Table 14: Standard NLP Metrics for MO3D. Detailed scores for Positional, Comparative, and Holistic QA tasks. SimCSE scores evaluate semantic similarity.
| Model | Shape Mating | Change Captioning (Verify) | Change Captioning (Delta) | |||||||||
| B-4 | R-L | MET | Sim | B-4 | R-L | MET | Sim | B-4 | R-L | MET | Sim | |
| LLaVA (1-view) | 1.17 | 21.99 | 15.31 | 67.48 | 4.53 | 30.05 | 23.57 | 56.27 | 3.58 | 26.69 | 19.85 | 55.86 |
| LLaVA (2-view) | 1.17 | 22.23 | 15.42 | 67.81 | 4.52 | 28.51 | 22.76 | 55.39 | 3.33 | 26.26 | 18.77 | 56.28 |
| Molmo (1-view) | 2.59 | 24.66 | 20.60 | 57.78 | 1.98 | 21.11 | 24.29 | 55.84 | 1.52 | 18.26 | 21.49 | 52.71 |
| Molmo (2-view) | 2.85 | 23.26 | 20.60 | 60.45 | 2.02 | 21.50 | 24.77 | 55.76 | 1.49 | 18.24 | 21.73 | 52.78 |
| MiniGPT-3D | 1.91 | 14.93 | 15.78 | 45.86 | 2.60 | 22.12 | 15.87 | 45.47 | 0.98 | 11.95 | 7.76 | 33.75 |
Table 15: Standard NLP Metrics for Mini-Applications. Detailed scores for Shape Mating, Change Captioning (Verify), and Change Captioning (Delta).
Figure 9: System Prompt for MO3D QA Generation. To ensure high-quality, non-ambiguous questions, we enforce strict keyword constraints (Instruction 7) for each target category. For readability, we list representative keywords for the ”Geometry” and ”Material” categories; identical constraint logic is applied to the other categories (Color, Function, Taxonomy, Style).
Figure 10: Full Prompt for Shape Mating Rationale Generation. We utilize GPT-4o-mini to convert structured error tags (e.g., cut_mismatch ) into natural language explanations.
Figure 11: Prompt Templates for Change Captioning. We utilize a set of diverse templates for the Delta Captioning task. For the Verification task, we explicitly randomize the input order of the anchor and candidate objects and adjust the question wording (”first” vs ”second” object) accordingly to prevent positional bias.
| Template Variations |
| • Which pairs can mate? select one that applies. Options: (1,2), (1,3), (2,3), None Explain why each chosen pair can mate and why others cannot. |
| • Identify which pairs can mate. Choose the applicable option. Options: (1,2), (1,3), (2,3), None Provide explanations for why each selected pair can mate and why the others cannot. |
| • Determine which pairs are able to mate. select one that applies. Options: (1,2), (1,3), (2,3), None Explain the reasoning for each pair that can mate and why the remaining pairs cannot. |
| • Find the pairs that can mate together. Select the applicable option. Options: (1,2), (1,3), (2,3), None Describe why each chosen pair can mate and explain why the other pairs cannot. |
| • Which pairs can successfully mate? select one that applies. Options: (1,2), (1,3), (2,3), None Explain why each selected pair can mate and provide reasons why the others cannot. |
| • Determine all pairs that are capable of mating. Select the applicable options. Options: (1,2), (1,3), (2,3), None Provide explanations for why each selected pair can mate and why others cannot. |
Table 16: Instruction Templates for Shape Mating. We utilize 15 variations of the prompt to train the model, ensuring robustness to phrasing while maintaining a consistent task definition (Selection + Explanation).
| Task Type | Template Variations |
| Verification | • (Condition: Anchor is First object): Does the second object satisfy all of the following requirements compared to the first object? Requirements: - [Instruction] |
| • (Condition: Anchor is Second object): Does the first object satisfy all of the following requirements compared to the second object? Requirements: - [Instruction] | |
| Delta Captioning | • How would you transform the first object so that it matches the second object? |
| • Describe the edits needed to convert the first object into the second. | |
| • What modifications should be applied to the first object to obtain the second? | |
| • List the geometric adjustments required to turn the first object into the second. |
Table 17: Instruction Templates for Change Captioning. We use distinct template sets for the Verification and Delta Captioning tasks. For Verification, we explicitly randomize the object order (Anchor first vs. second) to prevent positional bias.
Figure 12: Distribution of instruction and response lengths in MO3D dataset.
Figure 13: Distribution of instruction and response lengths in Shape Mating.
Figure 14: Distribution of instruction and response lengths in Change Captioning.
Figure 15: System Prompts for Evaluation. We use specific prompts for different metric types. For MO3D (Top), the evaluator is explicitly instructed to prioritize visual evidence from multi-view renderings over text labels to handle valid open-ended descriptions.
Figure 16: Qualitative Comparison on MO3D (Positional Task).
Figure 17: Qualitative Comparison on MO3D (Positional Task).
Figure 18: Qualitative Comparison on MO3D (Comparison Task).
Figure 19: Qualitative Comparison on MO3D (Holistic Task).
Figure 20: Qualitative Comparison on MO3D (Holistic Task). (Note: This is a failure case where our model incorrectly generalizes the cylindrical shape to all objects.)
Figure 21: Qualitative Comparison on Shape Mating (Selection Task).
Figure 22: Qualitative Comparison on Shape Mating (Selection Task).
Figure 23: Qualitative Comparison on Change Captioning (Verify Task).
Figure 24: Qualitative Comparison on Change Captioning (Delta Captioning Task).
More in this topic
3D VLMs
Three-dimensional vision-language models connect language with spatially grounded visual data for 3D understanding, reasoning, and action.
See all 204 papers