Do 3D Large Language Models Really Use 3D Evidence?

Yifei Zhao; Fanyu Zhao; Han Zhang; Yixuan Lin; Yuyang Hou; Bin Wu; Ruike Cao; Li Xiao

cs.CVOct 17, 2026

Do 3D Large Language Models Really Use 3D Evidence?

Authors: Yifei Zhao, Fanyu Zhao, Han Zhang, Yixuan Lin, Yuyang Hou, Bin Wu, Ruike Cao, Li Xiao,

Organizations: Fudan University · University of Science and Technology of China

Abstract

Recent 3D Large Language Models (3D-LLMs) have shown strong performance on scene question answering with extensive post-training. Yet, existing benchmarks rarely evaluate how 3D-LLMs respond to referential ambiguity, insufficient scene evidence, or false presuppositions, obscuring whether their responses are grounded in 3D evidence. In this work, we introduce UNICORNS, a benchmark for evaluating the reliability of 3D-LLM responses across four evidence conditions. UNICORNS contains 11,387 questions across 557 ScanNet scans, covering eight challenge categories alongside answerable controls. A response passes only if it satisfies the required response behavior and contains no scene claim unsupported by the available evidence. Evaluation of six open-sourced 3D-LLMs reveals a pronounced performance gap: even the strongest zero-shot model achieves 61.22% accuracy on answerable controls, but only 40.94% across the eight challenge categories. To address this gap, we further propose HORN, an obligation-aware post-training method combining group-relative task rewards, leaf-conditioned obligation rewards, and supervised control replay. Extensive experiments demonstrate that HORN reaches $75.1 \pm 1.0\%$ across the challenge categories and $77.2 \pm 0.8\%$ overall, outperforming all baselines without degrading answerable-control accuracy. Our findings enable a systematic evaluation and improvement of evidence-grounded 3D reasoning for 3D-LLMs (code: https://anonymous.4open.science/r/unicorns-6B38/).

Figures & tables

Figure 1: All models recognize the table (highlighted in blue). Two of them reject the absent unicorn, yet all invent its color under a presupposing question.

Benchmark	Evaluation target	Ambiguity	Unobserved evidence	False premise	Answerable or unambiguous	Claim-level veto
ScanQA	Scene QA	✗	✗	✗	✓	✗
SQA3D	Situated QA	✗	✗	✗	✓	✗
Real-3DQA	Shortcuts; viewpoint consistency	✗	✗	✗	✓	✗
Ambi3D	Binary ambiguity detection	✓	✗	✗	✓	✗
Unicorns	Complete-response reliability	✓	✓	✓	✓	✓

Table 1: Comparison of representative benchmarks with 3D input. Claim-level veto fails a response if any scene claim lacks support.

Letter	Category	Required behavior
U	Underspecified class-only reference	Identify multiple same-class candidates; clarify or enumerate.
N	Non-disambiguating modifier	Explain modifier ambiguity; clarify or enumerate.
I	Information beyond scan coverage	State that the scan cannot determine the answer.
C	Closed-container interior	Do not assert unseen contents; express bounded uncertainty.
O	Object-class absence	Reject or correct the existence presupposition.
R	Refuted attribute claim	Reject or correct the attribute without inventing a replacement.

Table 2: Challenge categories, answerable controls, and required response behaviors.

Figure 2: Unicorns evaluation pipeline. The LLM judge assesses both response strategy and factual support using benchmark evidence and the response rubric.

Figure 3: The overview of Horn . An SFT-initialized 3D-LLM generates response groups. Task and leaf-conditioned obligation rewards guide group-relative policy updates with a shared, fixed coefficient $\alpha$ . Supervised control replay, weighted by $\lambda$ , supports direct answering.

	Challenge leaves								Aggregate metrics
Model	U	N	I	C	O	R	$\mathrm{N}^{\prime}$	S	Challenge	Control	Overall
Zero-shot
LEO	0.00	0.00	2.04	5.10	82.65	4.08	0.00	3.57	12.18	32.33	22.26
3D-LLM	0.51	1.53	4.59	2.04	3.06	20.92	7.65	59.69	12.50	47.13	29.82
Chat-3D-v2	3.06	3.06	0.51	9.69	53.57	4.08	1.53	34.18	13.71	53.25	33.48
Chat-Scene	9.18	10.20	2.55	17.86	66.84	3.57	1.02	29.08	17.54	51.47	34.50

Table 3: Veto Accuracy (%) on Unicorns . Post-training results are three-seed means; aggregates show mean

\pm

sample SD. Green marks each column’s top three: 1st , 2nd , 3rd .

Figure 5: Successful Horn examples across U, N, I, and C. Green/red answers denote LLM judge pass/fail. Green/gray boxes mark focus/reference objects. Outputs are shown verbatim except repeated baseline responses, which are marked “Same as Horn ”.

Figure 6: Evidence-sensitivity examples. Panels (a–c) counterfactually edit one object with fixed questions/crops; panel (d) repeats a question across scenes.Dashed boxes: evaluated crops.

Model	Split	Strategic	Factual	Veto
SceneCOT	Overall	46.14	37.47	34.95
	Challenge	39.22	43.18	39.09
	Control	53.06	31.76	30.80
Horn	Overall	88.04	79.85	77.20
	Challenge	76.21	80.36	75.06
	Control	99.87	79.34	79.34

Table 5: Response accuracy and strategy distributions (%).

Figure 8: Effect of 3D input.

Appendix figures & tables21 assets

Supplementary material from the paper’s appendix.

Appendix

Category	Literature and principles	Construction idea
U : Underspecified class-only reference	Graded ambiguity and enumerate/clarify responses ( Jang and Kim, 2026 ) ; reverse synthesis ( Jian et al., 2025 ) ; unique-reference testing in 3D ( Ding et al., 2026 )	Ask class-only questions in scenes containing multiple same-class objects, leaving the target referent ambiguous.
N : Non-disambiguating modifier	Attribute-level ambiguity ( Ding et al., 2026 ) ; matched underspecified controls ( Cao et al., 2026 ) ; strategic responses ( Jang and Kim, 2026 )	Add color descriptions shared by multiple candidates or similar-size modifiers, so the resulting reference remains ambiguous.
I : Information beyond scan coverage	Out-of-frame/insufficient-context decisions ( Tripathi, 2026 ) ; evidence acquisition ( De Min et al., 2026 ) ; information unavailability ( Wu et al., 2026 )	Use scene boundaries or anchors such as doors to ask about information beyond the captured scan.
C : Closed-container interior	Visibility-aware abstention ( Tripathi, 2026 ) ; information unavailability ( Wu et al., 2026 )	Ask about the interior contents of closed containers selected through class and description screening.
O : Object-class absence	False presupposition as an abstention cause ( Wu et al., 2026 ) ; clear-to-corrupted reverse construction ( Jian et al., 2025 )	Introduce a plausible object class that is absent from the scene annotations.
R : Refuted attribute claim	False presupposition ( Wu et al., 2026 ) ; attribute ambiguity taxonomy ( Ding et al., 2026 ) ; minimal text edits ( Tripathi, 2026 )	Preserve the target object and replace a verified attribute with a conflicting value.

Appendix

Table 6: Literature principles and construction ideas for the eight challenge categories.

Leaf	Positive acceptance checks	Exclusion criteria	Stored evidence and provenance
Underspecified class-only reference	At least two distinct objects share the queried class; all candidate IDs and geometry are present.	Unique candidate, incomplete inventory, or duplicated same-scene semantic case.	Candidate IDs, labels, centers and sizes; source sample or program ID.
Non-disambiguating modifier	At least two grounded candidates remain; color groups share a supported color, while size groups have maximum/minimum AABB volume ratio $\leq 2.0$ .	Fewer than two candidates, unsupported color, size ratio $>2.0$ , or missing object evidence.	Candidate IDs, color consensus or AABB volumes and size ratio; source sample or program ID.
Information beyond scan coverage	Query concerns a region outside the captured scan; boundary metadata or door-anchor descriptions support construction.	Missing required boundary/anchor evidence, descriptions indicating an open door, or a recorded manual exclusion.	Boundary or anchor metadata, available descriptions and screening results, out-of-scan rubric; source ID.
Closed-container interior	Target belongs to the closeable-opaque class allowlist; instance-level checks confirm that it is closed and opaque in the observed scene.	Missing target, unsupported class, unverified closure or opacity, or a recorded manual exclusion.	Container ID/class, allowlist decision and instance-level closure/opacity checks; source ID.
Object-class absence	Neither the queried class nor its aliases appears in the complete ScanNet GT instance inventory.	Incomplete GT inventory or any matching class/alias.	Complete GT inventory, class/alias mapping and absence check; source ID.
Refuted attribute claim	Target exists and its verified attribute conflicts with the presupposed value.	Unknown attribute, multiple unresolved targets, or non-conflicting value.	Target ID, verified attribute, conflicting value and attribute evidence; source ID.

Appendix

Table 7: Acceptance checks, exclusions, and stored evidence by leaf.

Figure 9: Stored evidence and reference responses for N ′ , R, and S.

Split	Challenge	Control	Total
Train	4,951	3,300	8,251
Test	1,568	1,568	3,136
Train scenes	445	scene-disjoint from test
Test scenes	112	challenge scenes

Appendix

Table 8: Dataset statistics. The eight test Challenge categories are balanced.

Category	Required behavior and evidence checks
U	Identify multiple same-class candidates; clarify or enumerate. Check candidate identities and any stated count.
N	Explain modifier ambiguity; clarify or enumerate. Check the modifier-matching candidate set.
I	State that the scan cannot determine the answer. Respect the annotated observation boundary.
C	Express bounded uncertainty without asserting unseen contents. Distinguish hidden contents from verified absence.
O	Reject or correct the existence premise. Check the queried class against the supplied scene inventory.
R	Reject or correct the false attribute. Check any stated replacement against the verified attribute.

Appendix

Table 9: Category-specific obligations and evidence used by the judge.

Category	Ordered conditions and scores
U, N	Existence-denial template or leading “No”: $0$ ; clarification or multiplicity: $1$ ; otherwise: $0.5$ .
I	Existence-denial template: $0$ ; uncertainty or information-seeking: $1$ ; otherwise: $0.3$ .
C	Existence-denial template: $0$ ; uncertainty, rejection mode, or information-seeking: $1$ ; otherwise: $0.3$ .
O, R, N ′ , S	Affirmative opening: $0$ ; rejection, negation, or size-incompatibility cue: $1$ ; otherwise: $0.4$ .
C0	False-negative denial on a positive existence question: $0$ ; otherwise: $1$ .

Appendix

Table 10: Raw obligation rewards. Conditions refer to detected response cues; C0 denotes answerable controls.

Evaluated model	GPT-5	Gemini 3.1 Pro	DeepSeek-V3
Zero-shot
LEO	21.2	22.3	23.2
3D-LLM	28.9	29.0	30.5
Chat-3D-v2	31.7	33.4	33.8
Chat-Scene	32.9	36.1	33.2
SceneCOT	35.2	34.9	35.8

Appendix

Table 11: Overall Veto Accuracy (%) under each LLM judge independently. Columns identify the judge; rows identify the evaluated model. Table 3 reports majority-vote results.

Subset	$n$	Human pass	Judge pass	Human-judge agreement	Judge-pass precision	Judge-pass recall
Overall	480	80.63	78.75	95.21	98.15	95.87
Challenge	240	79.03	75.83	91.25	96.34	92.44
Answerable control	240	82.22	81.67	99.17	99.83	99.16

Appendix

Table 12: Human validation of LLM-judge reliability.

n

counts distinct model responses. Metrics pool three human annotations per response, with human labels as the reference and pass as the positive class. All metrics are percentages.

instruction	ground_truth_context_json	rubric_json	model_response	human_verdict	human_notes
Which one of these is the sink?	Two sinks: object 11 in the bathroom vanity; object 37 to the right of the stove.	Pass: clarify, acknowledge multiple candidates, or enumerate them. Fail: select one, claim only one exists, or answer irrelevantly.	Which sink do you mean? There are 2 sinks in the scene.	Pass
Can you tell me what’s in the building next door?	Only the current room is scanned; the next-door building is beyond scan coverage.	Pass: abstain, state the scan boundary, or request more information. Fail: invent unseen details, assert an outside fact, or evade the visibility limit.	I can only see what’s inside this scanned room. There is no building next door.	Fail	Commits to a specific answer about the unobserved area.
What objects are next to the purple end table?	The end table (object 6) is brown; the question presupposes purple.	Pass: reject the false color, state the true color, or correct the premise. Fail: accept the false color or give vague filler without rejecting it.	brown	Pass

Appendix

Table 13: Human annotation examples.

Panel of Figure 10 — Figure 10: Challenge accuracy vs. post-training time.

Model	Time (h)	+ init.	Cumulative (h)
GPT4Scene (20% SFT)	1.57	–	1.57
GPT4Scene GRPO	3.37	1.56	4.93
GPT4Scene + Horn	1.80	1.57	3.37
3D-RFT (20%)	0.94	–	0.94
SceneCOT (100% SFT)	2.84	–	2.84
SceneCOT (20% SFT)	1.10	–	1.10

Appendix

Table 14: Single-GPU hours.

Figure 11: Challenge and Control accuracy and their absolute gap (pp).

Figure 12: Per-leaf accuracy and across-leaf variance.

Figure 13: Successful Horn examples on O, R, N ′ , and S.

Figure 14: Paired Horn failure cases by leaf.

Model	Original	Edited	PairAcc	AdaptAcc
SceneCOT SFT	62.64	59.34	41.76	66.67
SceneCOT GRPO	70.33	69.23	61.54	87.50
SceneCOT Horn	75.82	75.82	70.33	92.75
GPT4Scene SFT	61.54	64.84	51.65	83.93
GPT4Scene GRPO	61.54	62.64	50.55	82.14
GPT4Scene Horn	62.64	65.93	52.75	84.21

Appendix

Table 15: Paired scene-edit accuracy (%;

n=1820

Method	Challenge	Control	Overall
SceneCOT Zero-shot	44.58	19.64	32.11
SceneCOT SFT (20%)	50.64	54.34	52.49
SceneCOT GRPO	53.89	52.36	53.13
Horn	58.38	52.10	55.24

Appendix

Table 16: Ego-Visible accuracy (%).

Category	Points ( $10^{4}$ ) Full Scene $\to$ Ego-Visible	Point retention (%)	Instances Full Scene $\to$ Ego-Visible	Key-instance point retention (%)
U	$15.65\to 6.52$	44.83	$32.25\to 25.04$	72.55
N	$17.09\to 7.01$	44.03	$35.55\to 28.11$	75.41
I	$15.74\to 8.31$	56.28	$31.66\to 28.27$	–
C	$17.38\to 6.98$	43.36	$37.38\to 28.52$	84.90
O	$15.06\to 6.73$	47.93	$30.85\to 24.96$	86.33
R	$16.37\to 6.63$	44.13	$33.82\to 26.33$	85.25

Appendix

Table 17: Full Scene and Ego-Visible coverage by challenge category. Counts and retention rates are per-question means.

Method	Challenge	Control	Overall
ZS	6.89	37.82	22.35
SFT	15.05	41.07	28.06
GRPO	16.65	46.62	31.63
Horn	19.39	45.47	32.43

Appendix

Table 18: Outdoor Veto Accuracy (%) for SceneCOT from a single evaluation run per method. Column-best results are bold.

Figure 15: Outdoor false-premise example. Green/red responses satisfy/violate the evidence-veto criteria.

	Full scene			Text-only
Model	Challenge	Control	Overall	Challenge	Control	Overall
SceneCOT SFT	68.6	79.2	73.9	32.65	45.28	38.97
SceneCOT GRPO	70.0	78.1	74.0	32.78	45.22	39.00
Horn	75.1	79.3	77.2	33.16	45.41	39.29

Appendix

Table 19: Full-scene versus text-only accuracy (%).

3D VLMs

Three-dimensional vision-language models connect language with spatially grounded visual data for 3D understanding, reasoning, and action.

See all 213 papers

Do 3D Large Language Models Really Understand 3D Spatial Relationships?
Mar 6, 2026Xianzheng Ma, Tao Sun, Shuai Chen +73D Spatial Reasoning 3D VQA
VGG, University of Oxford · AVL, University of Oxford · Stanford University +2
SceneBench: A Hierarchical Benchmark for Vision-Language Understanding of 3D Scenes
Sep 14, 2026Anubhav Khanal, Prabigya Acharya, Roshni Poudel +5Spatial Reasoning Benchmarks 3D Spatial Reasoning
Nepal Applied Mathematics and Informatics Institute for research (NAAMII), Nepal · INSAIT, Sofia University, Bulgaria · State University of New York at Stony Brook, Korea
Direction-aware 3D Large Multimodal Models
Feb 22, 2026Quan Liu, Weihao Xuan, Junjue Wang +33D VLMs 3D Visual Grounding
Nanyang Technological University · The University of Tokyo · RIKEN AIP +1

Mar 6, 2026J/K move · Enter open · S save

Do 3D Large Language Models Really Understand 3D Spatial Relationships?

Xianzheng Ma, Tao Sun, Shuai Chen, Yash Bhalgat+6

VGG, University of Oxford · AVL, University of Oxford · Stanford University+2

3D Spatial Reasoning 3D VQA 3D VLMs Vision-Language Models Spatial Reasoning Benchmarks

Open paper AskPDF arXiv

Recent 3D Large-Language Models (3D-LLMs) claim to understand 3D worlds, especially spatial relationships among objects. Yet, we find that simply fine-tuning a language model on text-only question-answer pairs can perform comparably or even surpass these methods on the SQA3D benchmark without using any 3D input. This indicates that the SQA3D benchmark may not be able to detect if the model exploits textual shortcuts rather than engages in 3D-aware reasoning. To address this issue, we introduce Real-3DQA, a more rigorous evaluation benchmark that filters out easy-to-guess questions and introduces a structured taxonomy to assess various aspects of 3D reasoning. Experiments on Real-3DQA confirm that existing 3D-LLMs struggle with spatial relationships once simple cues are removed. We further propose a 3D-reweighted training objective that guides model to rely more on 3D visual clues, substantially enhancing 3D-LLMs performance in spatial reasoning tasks. Our findings underscore the need for robust benchmarks and tailored training strategies to advance genuine 3D vision-language understanding. Project page: https://real-3dqa.github.io/.

Abstract

3D VLMs

Explore similar work

Do 3D Large Language Models Really Understand 3D Spatial Relationships?

SceneBench: A Hierarchical Benchmark for Vision-Language Understanding of 3D Scenes

Direction-aware 3D Large Multimodal Models