ReRef-3D: A Benchmark for Spatial Referring Expression-Guided 3D Scene Rearrangement

Mary Lynn Martin; Yifei Zhang; Martha Palmer; Maria Leonor Pacheco

cs.CLAug 17, 2026

ReRef-3D: A Benchmark for Spatial Referring Expression-Guided 3D Scene Rearrangement

Authors: Mary Lynn Martin, Yifei Zhang, Martha Palmer, Maria Leonor Pacheco

Organizations: University of Colorado Boulder

Abstract

We introduce ReRef-3D, a benchmark for language-guided placement in 3D scenes. It contains 33,826 instructions across 998 CLEVR-derived scenes, spanning 16 placement families and direct, one-hop, and two-hop references. Each instruction must be resolved into a valid new placement position. Given that an instruction defines a region of acceptable placements rather than one coordinate, our evaluation inserts a prediction into the scene, recomputes relations, and tests relation satisfaction and physical validity. Each instruction also includes a verified naturalized rewrite. After fine-tuning, LLaVA-3D, 3D-LLM, and PlaceIt3D produce valid placements for 68.3%, 31.6%, and 22.4% of instructions, respectively. Across models, relation satisfaction surpasses physical validity, relations such as nearest and between are the most difficult, and phrasing has minimal effect on performance.

Figures & tables

Figure 1: The marker labeled 1 indicates the paired target location for the instruction, “Place the gray ball next to the block that is to the left of the yellow block that is behind the cyan block.” Shown in 2D and 3D.

Figure 2: Placement verification used during dataset generation and evaluation. The red marker indicates a proposed target location in a top-down view of the scene. After the target is placed at this location, the scene graph is recomputed and checked against the instruction. Only relevant edges are shown; the full scene graph is maintained internally. Green edges indicate verified relations.

Figure 3: Placement accuracy (%) by relation family for matched runs using relaxed tolerance. Results shown here are for templated expressions. Results for naturalized expressions are located in Appendix D.2 .

Model	Expressions	Placement acc. $\uparrow$	Relation sat. $\uparrow$	Physical valid. $\uparrow$	Mean dist. $\downarrow$	[email protected] $\uparrow$
LLaVA-3D	Template	66.7	92.5	72.6	1.11	60.8
LLaVA-3D	Natural	68.3	92.5	74.0	1.08	62.4
3D-LLM	Template	32.3	44.9	63.2	3.20	17.5
3D-LLM	Natural	31.6	43.3	62.0	3.27	16.3
PlaceIt3D	Template	17.7	32.3	54.6	3.22	19.8
PlaceIt3D	Natural	22.4	33.6	67.7	3.08	21.9

Table 1: Matched train/evaluation results (relaxed tolerance). All rates are percentages; mean distance is in scene units and acts only as a localization diagnostic. Best per column in bold .

		Evaluated on
Model	Trained on	Template	Natural
LLaVA-3D	Template	66.7	65.3
LLaVA-3D	Natural	67.8	68.3
3D-LLM	Template	32.3	31.8
3D-LLM	Natural	31.5	31.6
PlaceIt3D	Template	17.7	17.7

Table 2: Placement accuracy (%) for every train

\times

evaluation language combination (relaxed tolerance). Diagonal cells are the matched conditions of Table 1 .

	LLaVA-3D		3D-LLM		PlaceIt3D
Relation group	T	N	T	N	T	N
Farthest	93.6	93.7	51.9	54.9	34.0	41.9
Directional	80.1	79.8	48.2	46.5	22.1	30.5
Between	57.6	62.2	12.2	1.8	10.8	9.5
Nearest	31.9	35.1	4.2	4.7	9.3	11.9
Other	65.8	68.0	29.1	28.2	13.0	15.9

Table 3: Placement accuracy by relation group, averaged over families in each group, for templated (T) and naturalized (N) matched runs. Farthest , Directional , Between , and Nearest span direct, one-hop, and two-hop variants. Other covers ordinal, table-region, multi-relation, and aligned placements. Per-family values for templated expressions appear in Figure 3 . Per-family values for naturalized expressions are in Appendix D.2 .

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

	LLaVA-3D	3D-LLM	PlaceIt3D (PlaceWizard)
Language backbone	Vicuna/LLaMA-7B	Flan-T5-XL ( $\sim$ 3B)	Flan-T5-XL ( $\sim$ 3B)
3D input	posed RGB-D frames lifted to 3D patch tokens	precomputed lifted 2D features $[N,1408]$ + voxel indices	point cloud with $\sim$ 1024 superpoints
Geometry source	rendered RGB + depth + camera pose/intrinsics	feature maps back-projected via depth and camera	occupancy grid $\rightarrow$ point cloud + clusters
Target conditioning	instruction text only	instruction text only	asset embedding + instruction
Output	text [x, y, z]	text [x, y, z]	placement mask, rotation bucket, anchor mask
Supervision	goal coordinate string	goal coordinate string	placement-region mask over ground points

Appendix

Table 4: Implementation summary. The two general models coerce placement into coordinate text; the placement-specific model predicts a region and never emits a coordinate.

	LLaVA-3D	3D-LLM	PlaceIt3D
Epochs	3	20	20
Effective batch	16	4	16
Peak LR	2e-4	1e-4	1e-4
Min LR	—	1e-5	1e-5
Schedule	cosine	cosine	cosine
Warmup	3%	1000 steps	1000 steps

Appendix

Table 5: Fine-tuning configuration. Each model uses one 80 GB H100. LLaVA-3D converges in far fewer epochs because only low-rank adapters are trained.

Evaluation	Agreement	$\kappa$
Human semantic validity	100.0%	–
Gemini-human semantic validity	91.5%	–
Human best selection	40.0%	0.19
Gemini-human consensus	62.5%	0.48

Appendix

Table 6: Human and Gemini evaluation results.

Model	Instructions	Strict	Relaxed
LLaVA-3D	Template	64.6	66.7
LLaVA-3D	Natural	65.8	68.3
3D-LLM	Template	30.5	32.3
3D-LLM	Natural	29.8	31.6
PlaceIt3D	Template	15.9	17.7
PlaceIt3D	Natural	20.3	22.4

Appendix

Table 7: Matched placement accuracy (%) under strict (

1.0\times

) and relaxed (

1.5\times

) relation tolerances.

Figure 4: Placement accuracy (%) by relation family for matched runs using relaxed tolerance. Results shown here are for natural expressions.

Model	Train	Eval	Strict	Relaxed	Relation	Phys. valid	Mean dist.
	Instructions		Placement acc.
LLaVA-3D	Template	Template	64.6	66.7	92.5	72.6	1.11
	Template	Natural	63.2	65.3	91.2	71.9	1.15
	Natural	Template	65.5	67.8	92.3	73.8	1.08
	Natural	Natural	65.8	68.3	92.5	74.0	1.08
3D-LLM	Template	Template	30.5	32.3	44.9	63.2	3.20

Appendix

Table 8: Complete validation results for all train

{\times}

evaluation language combinations. All rates are percentages; mean distance is in scene units. Relation satisfaction, physical validity, and distance use the relaxed tolerance; strict and relaxed placement accuracy are computed from the same predictions.

#	Instruction (family)	Goal	Pred.	R	P	Pred.	R	P	Pred.	R	P
			LLaVA-3D			3D-LLM			PlaceIt3D
1	Place the gray ball in front of the cyan ball and to the left of the purple cylinder (multi-relation)	$(0.0,-3.0)$	$(0.0,-3.0)$	✓	✓	$(-3.0,-3.0)$	$\times$	✓	$(2.8,-2.6)$	$\times$	✓
2	Position the gray ball as close as possible to the cyan block (nearest)	$(-0.5,-0.5)$	$(-0.1,0.1)$	✓	✓	$(-1.0,-3.0)$	$\times$	$\times$	$(2.8,-2.6)$	$\times$	✓
3	Set the cyan block between the green cylinder and the gray ball (between)	$(0.5,-1.2)$	$(0.5,-1.2)$	$\times$	✓	$(-0.5,-3.0)$	$\times$	$\times$	$(2.8,-1.5)$	$\times$	✓
4	Position the small gray matte ball behind the small blue matte cylinder (pairwise)	$(-2.5,0.9)$	$(-2.1,1.4)$	✓	$\times$	$(-2.9,-3.0)$	$\times$	✓	$(2.5,-2.6)$	$\times$	✓
5	Set the large purple matte ball behind the small gray shiny block (pairwise)	$(-2.5,-2.1)$	$(-2.9,0.9)$	✓	✓	$(-2.9,-3.0)$	✓	✓	$(-1.0,1.1)$	✓	✓

Appendix

Table 9: Representative validation items with predictions from the templated fine-tuned checkpoints. Coordinates are the horizontal

(x,y)

components; R marks relation satisfaction and P physical validity. Placement accuracy requires both.

	LLaVA-3D	3D-LLM	PlaceIt3D
Relation satisfied, but collision	25.8	12.6	14.6
Physically invalid	27.4	36.8	45.4
Near miss ( $d<0.5$ , relation unsatisfied)	2.3	0.2	0.5
Correct placement with $d>2$	13.3	18.7	7.9

Appendix

Table 10: Percentage of validation examples in each failure or edge-case category for the matched template runs under the relaxed tolerance.

	LLaVA-3D	3D-LLM	PlaceIt3D	Goal
$r_{x}$ with goal	0.86	0.50	0.13	—
$r_{y}$ with goal	0.88	0.31	0.15	—
SD $x$	2.22	2.38	1.90	2.01
SD $y$	2.12	1.74	1.85	1.95
Modal cell share (%)	9.9	34.7	16.5	—
Distinct cells	157	27	66	—

Appendix

Table 11: Spatial distribution statistics for predicted and annotated goal locations.

Explore similar work

R $^3$ L: Reasoning 3D Layouts from Relative Spatial Relations
May 7, 2026Zhifeng Gu, Yuqi Wang, Bing Wang3D Spatial Reasoning 3D Scene Generation
Spatial Intelligence Group, The Hong Kong Polytechnic University.
SSR3D-LLM: Structured Spatial Reasoning via Latent Steps for Fine-Grained Grounding in Unified 3D-LLMs
May 27, 2026Jiawei Li, Ziyi Liu, Weijie Shi +33D Spatial Reasoning Latent Visual Reasoning
The Hong Kong University of Science and Technology · Soochow University
Repurposing 3D Generative Model for Autoregressive Layout Generation
Apr 17, 2026Haoran Feng, Yifan Niu, Zehuan Huang +43D Scene Generation Autoregressive Generation
School of Software, Beihang University · 2Tsinghua University · University of Hong Kong +2

May 7, 2026, cs.CVJ/K move · Enter open · S save

R $^3$ L: Reasoning 3D Layouts from Relative Spatial Relations

Zhifeng Gu, Yuqi Wang, Bing Wang

Spatial Intelligence Group, The Hong Kong Polytechnic University.

3D Spatial Reasoning 3D Scene Generation 3D Scene Layout Generation Spatial Reasoning

Open paper AskPDF arXiv

Relative spatial relations provide a compact representation of spatial structure and are fundamental to relative spatial reasoning in 3D layout generation. Recent works leverage Multimodal Large Language Models (MLLMs) to infer such relations, but the inferred relations are often unreliable and are typically handled with post-hoc heuristics. In this paper, we propose R

^3

L, a general framework that improves the reliability and consistency of relative spatial reasoning for 3D layout generation. Our key motivation is that multi-hop reasoning requires repeated reference-frame transformations, which accumulate errors in inferred relations and lead to semantic and metric drift. To mitigate this, we propose invariant spatial decomposition to break coupled relation chains, and consistent spatial imagination to promote self-consistency through an imagine-and-revise loop. We further introduce supportive spatial optimization to ease pose optimization via global-to-local coordinate re-parameterization. Extensive experiments across diverse scene types and instructions demonstrate that R

^3

L produces more physically feasible and semantically consistent layouts. Notably, our analysis shows that resolving frame-induced inconsistencies is crucial for reliable multi-hop relative spatial reasoning. The code is available at https://github.com/Neal2020GitHub/R3L.

Abstract

Explore similar work

R3^33L: Reasoning 3D Layouts from Relative Spatial Relations

SSR3D-LLM: Structured Spatial Reasoning via Latent Steps for Fine-Grained Grounding in Unified 3D-LLMs

Repurposing 3D Generative Model for Autoregressive Layout Generation

R $^3$ L: Reasoning 3D Layouts from Relative Spatial Relations