Large Language Models (LLMs) and Vision-Language Models (VLMs) increasingly generate indoor scenes through intermediate structures such as layouts and scene graphs, yet evaluation still relies on LLM or VLM judges that score rendered views, making judgments sensitive to viewpoint, prompt phrasing, and hallucination. When the evaluator is unstable, it becomes difficult to determine whether a model has produced a spatially plausible scene or whether the output score reflects the choice of viewpoint, rendering, or prompt. We introduce SceneCritic, a symbolic evaluator for floor-plan-level layouts. SceneCritic's constraints are grounded in SceneOnto, a structured spatial ontology we construct by aggregating indoor scene priors from 3D-FRONT, ScanNet, and Visual Genome. SceneOnto traverses this ontology to jointly verify semantic, orientation, and geometric coherence across object relationships, providing object-level and relationship-level assessments that identify specific violations and successful placements. Furthermore, we pair SceneCritic with an iterative refinement test bed that probes how models build and revise spatial structure under different critic modalities: a rule-based critic using collision constraints as feedback, an LLM critic operating on the layout as text, and a VLM critic operating on rendered observations. Through extensive experiments, we show that (a) SceneCritic aligns substantially better with human judgments than VLM-based evaluators, (b) text-only LLMs can outperform VLMs on semantic layout quality, and (c) image-based VLM refinement is the most effective critic modality for semantic and orientation correction.
Figures & tables
Figure 1: VLM Evaluation vs. SceneCritic . Given a piano room with nine objects, the VLM assigns different scores to the same scene depending on which view is rendered, while SceneCritic evaluates the layout directly by traversing relational constraints from SceneOnto (e.g., incorrect table scales, chairs not facing the table they co-occur with). This produces stable, object-level assessments that identify specific violations alongside successful placements. SceneCritic scores also align more closely with human preferences.
Figure 2: Relational sub-graph from SceneOnto for three common room types. Nodes represent object categories, with size proportional to frequency. Edges encode conditional co-occurrence probabilities max(P(B|A)), with thicker edges indicating stronger associations. Dotted red lines connect shared object categories across room types. SceneCritic traverses this structure during evaluation to verify whether object co-occurrences in a generated layout are plausible for the specified room type.
Figure 3: Summary statistics from SceneOnto . (Left) Object dimension distributions, with bubble size encoding depth. (Center) Co-occurrence frequency vs. semantic association (nPMI): 73.6% of 489 object-pair edges show positive association. (Right) Object placement strategies, showing the fraction of instances with back-to-wall vs. faces-center orientation (e.g., objects like beds and coffee tables cluster toward back-to-wall, while wardrobes and mirrors favor faces-center).
Re-eval
Left
Right
Front
Method
Backbone
Sem.
Ori.
Ovlp
Sem.
Ori.
Ovlp
Sem.
Ori.
Ovlp
Sem.
Ori.
Ovlp
Holodeck
Qwen72B
0.11
6.58
3.48
55.06
88.27
1.89
30.25
75.43
1.44
47.68
37.33
0.00
LayoutGPT
Qwen72B
19.32
1.09
0.18
3.65
25.00
0.41
2.39
9.46
4.08
16.61
0.24
46.04
LayoutVLM
Qwen72B
0.49
4.06
0.38
0.44
0.61
1.31
0.05
6.07
2.09
1.90
0.04
0.37
Holodeck
Gemini
0.30
1.97
0.00
4.69
8.56
0.00
0.42
0.09
7.02
1.22
1.97
1.56
LayoutGPT
Gemini
12.46
1.90
1.92
0.12
35.64
11.36
0.58
5.25
10.30
1.66
0.02
2.43
Table 1: Per-view variance relative to the top-view baseline. Each entry is computed as 4(View−Top1)2 for Semantic (Sem), Orientation (Ori), and Overlap (Ovlp). The Re-eval column measures variance when the same top-view image is evaluated twice under identical conditions. The remaining columns measure variance between the top-view and alternative viewpoints (Left, Right, Front).
VLM-Evaluation
SceneCritic
Method
Backbone
Sem.
Ori.
Ovlp.
Avg.
Sem.
Ori.
Ovlp.
Avg.
ProxOvlp
TrueOvlp
Holodeck
Qwen72B
42.61
60.61
75.17
59.46
76.62
52.75
95.97
95.99
75.12
LayoutGPT
Qwen72B
51.09
56.94
69.02
59.02
75.03
16.83
73.54
73.34
55.10
LayoutVLM
Qwen72B
50.82
45.48
71.06
55.79
89.46
59.38
91.92
92.31
80.32
Holodeck
Gemini
46.88
51.33
80.91
59.71
74.37
45.05
97.13
97.21
72.20
Table 2: Comparison of Baseline methods across VLM-based evaluator and SceneCritic on Semantic (Sem), Orientation (Ori), Overlap (Ovlp) and Average (Avg) scores. In SceneCritic , Overlap is decomposed into ProxOvlp (Axis-Aligned Bounding Box overlap) and TrueOvlp (Oriented Bounding Box overlap).
Figure 4: Qualitative results of SceneCritic : SceneCritic assessment for extended scenes, stating semantic, orientation and overlap scores and identifying specific violations (e.g., desk not backed against wall, cabinets not backed against wall) alongside successful placements across scale, orientation, co-occurrence, proximity, and completeness.
Model
Category
Post-Training Details
Params
Qwen3-14B [ 39 ]
General RL
GRPO-style reinforcement learning
14B
Qwen3-235B [ 39 ]
General RL
GRPO-style reinforcement learning
235B
UI-Venus-Navi-72B [ 17 ]
General RL
GRPO-based reasoning optimization
72B
Gemini-2.5-flash [ 9 ]
RLAIF + RLHF
SFT + Reward Model + RL
N/A
Qwen2.5-VL-7B-MM-RLHF [ 13 ]
RLHF
PPO-style human feedback alignment
7B
Qwen2.5-72B-VL [ 2 ]
RLHF
SFT + DPO (preference optimization)
72B
Table 3: Models categorized by post-training strategy and model parameters.
Semantic Verifier Evaluation
Method
Sem.
Ori.
Overlap
Avg.
ProxOvlp
TrueOvlp
Heuristic
Gemini-2.5-flash
76.3
43.7
99.3
98.9
73.03
Qwen2.5-72B-VL
72.2
48.1
91.3
91.3
70.53
DeepSeek-3.2V
82.6
64.4
95.3
95.7
80.83
Table 4: Comparison across refinement variants using MLLM backbones with SceneCritic . Overlap is decomposed into ProxOvlp (Axis-Aligned Bounding Box overlap) and TrueOvlp (Oriented Bounding Box overlap).
Appendix figures & tables6 assets
Supplementary material from the paper’s appendix.
Appendix
cooccur_thresh
func_thresh
count
0.2
0.5
678
0.2
0.7
678
0.5
0.7
624
Appendix
Table 5: Distribution of threshold combinations by count for Cooccurrence.
Count
(soft_angle, hard_angle) combinations
# count
431
(150, 180), (150, 270)
2
385
(90, 150), (90, 180), (90, 270)
3
235
(75, 90), (75, 150), (75, 180), (75, 270)
4
213
(45, 90), (45, 150), (45, 180), (45, 270)
4
Appendix
Table 6: Distribution of angle threshold combinations by count.
Figure 5: Impact of hyperparameter choice on SceneCritic Evaluation: SceneCritic shows strong alignment with human judgment under the selected hyperparameter configuration. In contrast, alternative hyperparameter settings lead to systematic biases, causing the evaluator to behave either overly lenient or excessively strict.
Top-view (1)
Top-view (2)
Left view
Right view
Front view
Method
Sem.
Ori.
Ovlp.
Avg.
Sem.
Ori.
Ovlp.
Avg.
Sem.
Ori.
Ovlp.
Avg.
Sem.
Ori.
Ovlp.
Avg.
Sem.
Ori.
Ovlp.
Avg.
Baselines
Holodeck + Gemini
46.88
51.33
80.91
59.71
47.97
48.52
80.83
59.11
51.21
45.48
80.92
59.21
48.18
50.73
75.61
58.17
49.09
48.52
78.41
58.67
Holodeck + Qwen72B
42.61
60.61
75.17
59.46
41.94
55.48
71.44
56.29
57.45
41.82
72.42
57.23
53.61
43.24
72.77
56.54
56.42
48.39
75.17
59.99
LayoutGPT + Gemini
48.18
53.82
77.68
59.89
55.24
56.58
74.91
62.24
47.48
41.88
70.94
53.43
49.70
49.24
71.26
56.73
50.76
54.12
74.56
59.81
LayoutGPT + Qwen72B
51.09
56.94
69.02
59.02
59.88
59.03
68.18
62.36
47.27
46.94
70.30
54.84
54.18
50.79
73.06
59.34
59.24
55.97
55.45
56.89
Appendix
Table 7: Comparison across all pipeline variants evaluated across multiple rendered views. Best results per column within each view are in bold .
Method A
Method B
Method A Scores
Method B Scores
Semantic
Orientation
Overlap
IAA (%)
Fleiss’ κ
VLM Agree
SceneCritic Agree
SceneCritic
VLM
SceneCritic
VLM
A W
B W
Tot
A%
B%
A W
B W
Tot
A%
B%
A W
B W
Tot
A%
B%
Sem
Ori
Ovlp
Sem
Ori
Ovlp
Sem
Ori
Ovlp
Sem
Ori
Ovlp
heuristic
holodeck
75.6
60
71.3
25
9
7
16
56.2
43.7
12
4
16
75.0
25.0
11
5
16
68.8
31.2
62.5
75.0
68.8
-0.098
0.126
-0.134
Y
Y
N
Y
Y
Y
heuristic
layoutgpt
75.6
60
53.6
100
15
1
16
93.8
6.2
14
2
16
87.5
12.5
16
0
16
100.0
0.0
93.8
87.5
100.0
0.739
0.490
1.000
N
Y
-
Y
Y
Y
heuristic
layoutvlm
75.6
60
81.9
60
12
4
16
75.0
25.0
14
2
16
87.5
12.5
11
5
16
68.8
31.2
75.0
87.5
68.8
0.164
0.490
-0.134
-
Y
N
N
Y
Y
holodeck
layoutgpt
71.3
25
53.6
100
14
2
16
87.5
12.5
15
1
16
93.8
6.2
15
1
16
93.8
6.2
87.5
93.8
93.8
0.512
0.727
0.691
N
N
Y
Y
Y
Y
holodeck
layoutvlm
71.3
25
81.9
60
6
10
16
37.5
62.5
6
10
16
37.5
62.5
14
2
16
87.5
12.5
62.5
62.5
87.5
-0.045
-0.093
0.423
Y
Y
N
Y
Y
Y
Appendix
Table 8: Pairwise comparison — Difficulty: Easy. Per-row inter-annotator agreement (IAA) is computed as max(A Wins,B Wins)/Total . We additionally report row-wise Fleiss’ κ using global label marginals for each metric across all Easy comparisons. SceneCritic Agree = 94.44% (IAA); Aggregated IAA = 77.08% / 83.33% / 84.38% (Sem/Ori/Ovlp); Fleiss’ κ = 0.30 / 0.46 / 0.39; VLM Agree = 58.82%.
Spatial question answering is the dominant paradigm for evaluating spatial intelligence in Vision-Language Models (VLMs), but it leaves a complementary axis of spatial competence under-evaluated: holistic 3D layout inference, which predicts every visible object's pose and extent from a single image in a structured form. To this end, we introduce IDEAL-Bench, an evaluation suite that requires VLMs to predict structured 3D layouts on photorealistic indoor scenes across 10 room types, scored along five numerical dimensions and a perceptual render-and-compare protocol. By operating on semantically realistic scenes with full asset substitution under controlled lighting and viewpoint, IDEAL-Bench moves beyond CLEVR-style simple geometric primitives so that any image-space discrepancy reflects spatial reasoning alone. The benchmark is built on IDEAL-Scenes, a procedurally generated dataset of 1,000 re-renderable Blender environments with ground-truth layouts. Evaluating 15 prominent VLMs reveals three findings: the task remains substantially unsolved, with the strongest model reaching only 62.1/100 overall; all models exhibit a sharp asymmetry between object recognition and geometric regression, indicating that current VLMs are trained to describe scenes rather than to measure them; model rankings partially diverge from those on QA-based and primitive-reconstruction benchmarks: top-tier consensus holds, but mid-tier rankings shift substantially. Collectively, these findings establish IDEAL-Bench as a diagnostic suite, targeting the geometric and structural competencies that QA-based evaluation cannot surface, and paving the way towards more rigorous evaluation of spatial intelligence in next-generation VLMs. Together, these findings position IDEAL-Bench as a principled diagnostic for whether future VLMs achieve genuine spatial understanding rather than linguistic approximations of it.
Automatically generating interactive 3D indoor scenes from natural language is crucial for virtual reality, gaming, and embodied AI. However, existing LLM-based approaches often suffer from spatial errors and collisions, in part because common scene representations-raw coordinates or verbose code-are difficult for models to reason about 3D spatial relationships and physical constraints. We propose SpatialGrammar, a domain-specific language that represents gravity-aligned indoor layouts as BEV grid placements with deterministic compilation to valid 3D geometry, enabling verifiable constraint checking. Building on this representation, we develop (1) SG-Agent, a closed-loop system that uses compiler feedback to iteratively refine scenes and enforce collision constraints, and (2) SG-Mini, a 104M-parameter model trained entirely on compiler-validated synthetic data. Across 159 test scenes spanning five scenarios of different complexity, SG-Agent improves spatial fidelity and physical plausibility over prior methods, while SG-Mini performs competitively against larger LLM-based baselines on single-shot generation scenarios.
Song Tang, Kaiyong Zhao, Yuliang Li +5
The Hong Kong University of Science and Technology (Guangzhou) · XGRIDS · Harbin Institute of Technology (Shenzhen)
Scene synthesis and editing has emerged as a promising direction in computer graphics. Current trained approaches for 3D indoor scene generation either oversimplify object semantics through one-hot class encodings (e.g., 'chair' or 'table'), require masked diffusion for editing, ignore room boundaries, or rely on floor plan renderings that fail to capture complex layouts. LLM-based methods enable richer semantics via natural language, but lack editing functionality, are limited to rectangular layouts, or rely on weak spatial reasoning from implicit world models. We introduce ReSpace, a generative framework for autoregressive text-driven 3D indoor scene synthesis and editing. Our approach features a compact structured scene representation with explicit room boundaries that enables asset-agnostic deployment and frames scene manipulation as a next-token prediction task, supporting object addition, removal, and swapping via natural language. We employ supervised fine-tuning with a preference alignment stage to train a specialized language model for object addition that accounts for user instructions, spatial geometry, object semantics, and scene-level composition. We further introduce a voxelization-based evaluation metric capturing fine-grained geometric violations beyond 3D bounding boxes. Experiments surpass state-of-the-art on object addition and achieve superior human-perceived quality on the application of full scene synthesis, despite not being trained on it.