SceneCritic: A Symbolic Evaluator for 3D Indoor Scene Synthesis

Kathakoli Sengupta; Kai Ao; Paola Cascante-Bonilla

cs.CVApr 14, 2026

SceneCritic: A Symbolic Evaluator for 3D Indoor Scene Synthesis

Authors: Kathakoli Sengupta, Kai Ao, Paola Cascante-Bonilla

Organizations: Stony Brook University

Abstract

Large Language Models (LLMs) and Vision-Language Models (VLMs) increasingly generate indoor scenes through intermediate structures such as layouts and scene graphs, yet evaluation still relies on LLM or VLM judges that score rendered views, making judgments sensitive to viewpoint, prompt phrasing, and hallucination. When the evaluator is unstable, it becomes difficult to determine whether a model has produced a spatially plausible scene or whether the output score reflects the choice of viewpoint, rendering, or prompt. We introduce SceneCritic, a symbolic evaluator for floor-plan-level layouts. SceneCritic's constraints are grounded in SceneOnto, a structured spatial ontology we construct by aggregating indoor scene priors from 3D-FRONT, ScanNet, and Visual Genome. SceneOnto traverses this ontology to jointly verify semantic, orientation, and geometric coherence across object relationships, providing object-level and relationship-level assessments that identify specific violations and successful placements. Furthermore, we pair SceneCritic with an iterative refinement test bed that probes how models build and revise spatial structure under different critic modalities: a rule-based critic using collision constraints as feedback, an LLM critic operating on the layout as text, and a VLM critic operating on rendered observations. Through extensive experiments, we show that (a) SceneCritic aligns substantially better with human judgments than VLM-based evaluators, (b) text-only LLMs can outperform VLMs on semantic layout quality, and (c) image-based VLM refinement is the most effective critic modality for semantic and orientation correction.

Figures & tables

Figure 1: VLM Evaluation vs. SceneCritic . Given a piano room with nine objects, the VLM assigns different scores to the same scene depending on which view is rendered, while SceneCritic evaluates the layout directly by traversing relational constraints from SceneOnto (e.g., incorrect table scales, chairs not facing the table they co-occur with). This produces stable, object-level assessments that identify specific violations alongside successful placements. SceneCritic scores also align more closely with human preferences.

Figure 2: Relational sub-graph from SceneOnto for three common room types. Nodes represent object categories, with size proportional to frequency. Edges encode conditional co-occurrence probabilities max(P(B|A)), with thicker edges indicating stronger associations. Dotted red lines connect shared object categories across room types. SceneCritic traverses this structure during evaluation to verify whether object co-occurrences in a generated layout are plausible for the specified room type.

Figure 3: Summary statistics from SceneOnto . (Left) Object dimension distributions, with bubble size encoding depth. (Center) Co-occurrence frequency vs. semantic association (nPMI): 73.6% of 489 object-pair edges show positive association. (Right) Object placement strategies, showing the fraction of instances with back-to-wall vs. faces-center orientation (e.g., objects like beds and coffee tables cluster toward back-to-wall, while wardrobes and mirrors favor faces-center).

		Re-eval			Left			Right			Front
Method	Backbone	Sem.	Ori.	Ovlp	Sem.	Ori.	Ovlp	Sem.	Ori.	Ovlp	Sem.	Ori.	Ovlp
Holodeck	Qwen72B	$0.11$	$6.58$	$3.48$	55.06	88.27	$1.89$	30.25	75.43	$1.44$	47.68	37.33	$0.00$
LayoutGPT	Qwen72B	19.32	$1.09$	$0.18$	$3.65$	$25.00$	$0.41$	$2.39$	$9.46$	$4.08$	$16.61$	$0.24$	46.04
LayoutVLM	Qwen72B	$0.49$	$4.06$	$0.38$	$0.44$	$0.61$	$1.31$	$0.05$	$6.07$	$2.09$	$1.90$	$0.04$	$0.37$
Holodeck	Gemini	$0.30$	$1.97$	$0.00$	$4.69$	$8.56$	$0.00$	$0.42$	$0.09$	$7.02$	$1.22$	$1.97$	$1.56$
LayoutGPT	Gemini	12.46	$1.90$	$1.92$	$0.12$	$35.64$	11.36	$0.58$	$5.25$	$10.30$	$1.66$	$0.02$	$2.43$

Table 1: Per-view variance relative to the top-view baseline. Each entry is computed as

\frac{(\text{View}-\text{Top1})^{2}}{4}

for Semantic (Sem), Orientation (Ori), and Overlap (Ovlp). The Re-eval column measures variance when the same top-view image is evaluated twice under identical conditions. The remaining columns measure variance between the top-view and alternative viewpoints (Left, Right, Front).

		VLM-Evaluation				SceneCritic
Method	Backbone	Sem.	Ori.	Ovlp.	Avg.	Sem.	Ori.	Ovlp.		Avg.
								ProxOvlp	TrueOvlp
Holodeck	Qwen72B	$42.61$	60.61	75.17	59.46	$76.62$	$52.75$	95.97	95.99	$75.12$
LayoutGPT	Qwen72B	51.09	$56.94$	$69.02$	$59.02$	$75.03$	$16.83$	$73.54$	$73.34$	$55.10$
LayoutVLM	Qwen72B	$50.82$	$45.48$	$71.06$	$55.79$	89.46	59.38	$91.92$	$92.31$	80.32
Holodeck	Gemini	$46.88$	$51.33$	80.91	59.71	$74.37$	$45.05$	97.13	97.21	$72.20$

Table 2: Comparison of Baseline methods across VLM-based evaluator and SceneCritic on Semantic (Sem), Orientation (Ori), Overlap (Ovlp) and Average (Avg) scores. In SceneCritic , Overlap is decomposed into ProxOvlp (Axis-Aligned Bounding Box overlap) and TrueOvlp (Oriented Bounding Box overlap).

Figure 4: Qualitative results of SceneCritic : SceneCritic assessment for extended scenes, stating semantic, orientation and overlap scores and identifying specific violations (e.g., desk not backed against wall, cabinets not backed against wall) alongside successful placements across scale, orientation, co-occurrence, proximity, and completeness.

Model	Category	Post-Training Details	Params
Qwen3-14B [ 39 ]	General RL	GRPO-style reinforcement learning	14B
Qwen3-235B [ 39 ]	General RL	GRPO-style reinforcement learning	235B
UI-Venus-Navi-72B [ 17 ]	General RL	GRPO-based reasoning optimization	72B
Gemini-2.5-flash [ 9 ]	RLAIF + RLHF	SFT + Reward Model + RL	N/A
Qwen2.5-VL-7B-MM-RLHF [ 13 ]	RLHF	PPO-style human feedback alignment	7B
Qwen2.5-72B-VL [ 2 ]	RLHF	SFT + DPO (preference optimization)	72B

Table 3: Models categorized by post-training strategy and model parameters.

	Semantic Verifier Evaluation
Method	Sem.	Ori.	Overlap		Avg.
			ProxOvlp	TrueOvlp
Heuristic
Gemini-2.5-flash	76.3	43.7	99.3	98.9	73.03
Qwen2.5-72B-VL	72.2	48.1	91.3	91.3	70.53
DeepSeek-3.2V	82.6	64.4	95.3	95.7	80.83

Table 4: Comparison across refinement variants using MLLM backbones with SceneCritic . Overlap is decomposed into ProxOvlp (Axis-Aligned Bounding Box overlap) and TrueOvlp (Oriented Bounding Box overlap).

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

cooccur_thresh	func_thresh	count
0.2	0.5	678
0.2	0.7	678
0.5	0.7	624

Appendix

Table 5: Distribution of threshold combinations by count for Cooccurrence.

Count	(soft_angle, hard_angle) combinations	# count
431	(150, 180), (150, 270)	2
385	(90, 150), (90, 180), (90, 270)	3
235	(75, 90), (75, 150), (75, 180), (75, 270)	4
213	(45, 90), (45, 150), (45, 180), (45, 270)	4

Appendix

Table 6: Distribution of angle threshold combinations by count.

Figure 5: Impact of hyperparameter choice on SceneCritic Evaluation: SceneCritic shows strong alignment with human judgment under the selected hyperparameter configuration. In contrast, alternative hyperparameter settings lead to systematic biases, causing the evaluator to behave either overly lenient or excessively strict.

	Top-view (1)				Top-view (2)				Left view				Right view				Front view
Method	Sem.	Ori.	Ovlp.	Avg.	Sem.	Ori.	Ovlp.	Avg.	Sem.	Ori.	Ovlp.	Avg.	Sem.	Ori.	Ovlp.	Avg.	Sem.	Ori.	Ovlp.	Avg.
Baselines
Holodeck + Gemini	46.88	51.33	80.91	59.71	47.97	48.52	80.83	59.11	51.21	45.48	80.92	59.21	48.18	50.73	75.61	58.17	49.09	48.52	78.41	58.67
Holodeck + Qwen72B	42.61	60.61	75.17	59.46	41.94	55.48	71.44	56.29	57.45	41.82	72.42	57.23	53.61	43.24	72.77	56.54	56.42	48.39	75.17	59.99
LayoutGPT + Gemini	48.18	53.82	77.68	59.89	55.24	56.58	74.91	62.24	47.48	41.88	70.94	53.43	49.70	49.24	71.26	56.73	50.76	54.12	74.56	59.81
LayoutGPT + Qwen72B	51.09	56.94	69.02	59.02	59.88	59.03	68.18	62.36	47.27	46.94	70.30	54.84	54.18	50.79	73.06	59.34	59.24	55.97	55.45	56.89

Appendix

Table 7: Comparison across all pipeline variants evaluated across multiple rendered views. Best results per column within each view are in bold .

Method A	Method B	Method A Scores		Method B Scores		Semantic					Orientation					Overlap					IAA (%)			Fleiss’ $\kappa$			VLM Agree			SceneCritic Agree
Method A	Method B	SceneCritic	VLM	SceneCritic	VLM	A W	B W	Tot	A%	B%	A W	B W	Tot	A%	B%	A W	B W	Tot	A%	B%	Sem	Ori	Ovlp	Sem	Ori	Ovlp	Sem	Ori	Ovlp	Sem	Ori	Ovlp
heuristic	holodeck	75.6	60	71.3	25	9	7	16	56.2	43.7	12	4	16	75.0	25.0	11	5	16	68.8	31.2	62.5	75.0	68.8	-0.098	0.126	-0.134	Y	Y	N	Y	Y	Y
heuristic	layoutgpt	75.6	60	53.6	100	15	1	16	93.8	6.2	14	2	16	87.5	12.5	16	0	16	100.0	0.0	93.8	87.5	100.0	0.739	0.490	1.000	N	Y	-	Y	Y	Y
heuristic	layoutvlm	75.6	60	81.9	60	12	4	16	75.0	25.0	14	2	16	87.5	12.5	11	5	16	68.8	31.2	75.0	87.5	68.8	0.164	0.490	-0.134	-	Y	N	N	Y	Y
holodeck	layoutgpt	71.3	25	53.6	100	14	2	16	87.5	12.5	15	1	16	93.8	6.2	15	1	16	93.8	6.2	87.5	93.8	93.8	0.512	0.727	0.691	N	N	Y	Y	Y	Y
holodeck	layoutvlm	71.3	25	81.9	60	6	10	16	37.5	62.5	6	10	16	37.5	62.5	14	2	16	87.5	12.5	62.5	62.5	87.5	-0.045	-0.093	0.423	Y	Y	N	Y	Y	Y

Appendix

Table 8: Pairwise comparison — Difficulty: Easy. Per-row inter-annotator agreement (IAA) is computed as

\max(\text{A Wins},\text{B Wins})/\text{Total}

. We additionally report row-wise Fleiss’

\kappa

using global label marginals for each metric across all Easy comparisons. SceneCritic Agree = 94.44% (IAA); Aggregated IAA = 77.08% / 83.33% / 84.38% (Sem/Ori/Ovlp); Fleiss’

\kappa

= 0.30 / 0.46 / 0.39; VLM Agree = 58.82%.

Method A	Method B	Method A Scores		Method B Scores		Semantic					Orientation					Overlap					IAA (%)			Fleiss’ $\kappa$			VLM Agree			SceneCritic Agree
Method A	Method B	SceneCritic	VLM	SceneCritic	VLM	A W	B W	Tot	A%	B%	A W	B W	Tot	A%	B%	A W	B W	Tot	A%	B%	Sem	Ori	Ovlp	$\kappa_{S}$	$\kappa_{O}$	$\kappa_{L}$	Sem	Ori	Ovlp	Sem	Ori	Ovlp
heuristic	holodeck	79.39	75	79.5	95	13	3	16	81.2	18.8	13	3	16	81.2	18.8	14	2	16	87.5	12.5	81.2	81.2	87.5	0.426	0.383	0.563	N	N	N	N	N	Y
heuristic	layoutgpt	79.39	75	68.3	100	14	2	16	87.5	12.5	14	2	16	87.5	12.5	16	0	16	100.0	0.0	87.5	87.5	100.0	0.562	0.530	1.000	N	Y	N	Y	Y	Y
heuristic	layoutvlm	79.39	75	94.0	95	3	13	16	18.8	81.2	1	15	16	6.2	93.8	2	14	16	12.5	87.5	81.2	93.8	87.5	0.426	0.787	0.563	Y	Y	Y	Y	Y	N
holodeck	layoutgpt	79.5	95	68.3	100	16	0	16	100.0	0.0	14	2	16	87.5	12.5	15	1	16	93.8	6.2	100.0	87.5	93.8	1.000	0.530	0.770	N	Y	Y	Y	Y	Y
holodeck	layoutvlm	79.5	95	94.0	95	1	15	16	6.2	93.8	1	15	16	6.2	93.8	1	15	16	6.2	93.8	93.8	93.8	93.8	0.787	0.787	0.770	-	N	N	Y	Y	Y

Appendix

Table 9: Pairwise comparison — Difficulty: Complex. Per-row inter-evaluator agreement (IAA) is computed as

\max(\text{A Wins},\text{B Wins})/\text{Total}

. We additionally report row-wise Fleiss’

\kappa

using global label marginals. SceneCritic Agree = 83.33% (IAA); Aggregated IAA = 90.63% / 89.58% / 93.75% (Sem/Ori/Ovlp); Fleiss’

\kappa

= 0.64 / 0.59 / 0.72; VLM Agree = 47.06%.

3D Scene Generation

3D scene generation creates structured, explorable environments from text, images, maps, or other inputs.

See all 190 papers

IDEAL-Bench: Indoor Dataset and Evaluation suite for Analyzing 3D Layout reasoning
Jul 3, 2026Yuening Cai, Junwei Zhou, Youran Qu +1VLM Evaluation Spatial Reasoning Benchmarks
Department of Computer Science Dartmouth College
SpatialGrammar: A Domain-Specific Language for LLM-Based 3D Indoor Scene Generation
Apr 30, 2026Song Tang, Kaiyong Zhao, Yuliang Li +53D Scene Generation 3D Scene Representation
The Hong Kong University of Science and Technology (Guangzhou) · XGRIDS · Harbin Institute of Technology (Shenzhen)
ReSpace: Text-Driven Autoregressive 3D Indoor Scene Synthesis and Editing
Jun 3, 2025Martin JJ. Bucher, Iro Armeni3D Scene Editing 3D Scene Representation
Stanford University

Jul 3, 2026J/K move · Enter open · S save

IDEAL-Bench: Indoor Dataset and Evaluation suite for Analyzing 3D Layout reasoning

Yuening Cai, Junwei Zhou, Youran Qu, Yu-Wing Tai

Department of Computer Science Dartmouth College

VLM Evaluation Spatial Reasoning Benchmarks 3D Spatial Reasoning Visually Grounded Reasoning 3D Scene Understanding

Open paper AskPDF arXiv

Spatial question answering is the dominant paradigm for evaluating spatial intelligence in Vision-Language Models (VLMs), but it leaves a complementary axis of spatial competence under-evaluated: holistic 3D layout inference, which predicts every visible object's pose and extent from a single image in a structured form. To this end, we introduce IDEAL-Bench, an evaluation suite that requires VLMs to predict structured 3D layouts on photorealistic indoor scenes across 10 room types, scored along five numerical dimensions and a perceptual render-and-compare protocol. By operating on semantically realistic scenes with full asset substitution under controlled lighting and viewpoint, IDEAL-Bench moves beyond CLEVR-style simple geometric primitives so that any image-space discrepancy reflects spatial reasoning alone. The benchmark is built on IDEAL-Scenes, a procedurally generated dataset of 1,000 re-renderable Blender environments with ground-truth layouts. Evaluating 15 prominent VLMs reveals three findings: the task remains substantially unsolved, with the strongest model reaching only 62.1/100 overall; all models exhibit a sharp asymmetry between object recognition and geometric regression, indicating that current VLMs are trained to describe scenes rather than to measure them; model rankings partially diverge from those on QA-based and primitive-reconstruction benchmarks: top-tier consensus holds, but mid-tier rankings shift substantially. Collectively, these findings establish IDEAL-Bench as a diagnostic suite, targeting the geometric and structural competencies that QA-based evaluation cannot surface, and paving the way towards more rigorous evaluation of spatial intelligence in next-generation VLMs. Together, these findings position IDEAL-Bench as a principled diagnostic for whether future VLMs achieve genuine spatial understanding rather than linguistic approximations of it.

Abstract

3D Scene Generation

Explore similar work

IDEAL-Bench: Indoor Dataset and Evaluation suite for Analyzing 3D Layout reasoning

SpatialGrammar: A Domain-Specific Language for LLM-Based 3D Indoor Scene Generation

ReSpace: Text-Driven Autoregressive 3D Indoor Scene Synthesis and Editing