ScIntBench: Physical Interaction Plausibility in 3D Scene Generation

Wendi Ying; Daochang Liu; Leonid Sigal; Ajmal Saeed Mian

cs.CVOct 27, 2026

ScIntBench: Physical Interaction Plausibility in 3D Scene Generation

Authors: Wendi Ying, Daochang Liu, Leonid Sigal, Ajmal Saeed Mian

Organizations: The University of Western Australia · University of British Columbia · Canada CIFAR AI Chair

Abstract

Existing evaluations of text-conditioned 3D indoor scene generation largely emphasize semantic fidelity and physical validity at rest, leaving interaction-time physical reliability insufficiently assessed. To address this gap, we introduce ScIntBench, a unified benchmark centered on interaction readiness: it executes prescribed manipulation actions (e.g., pushing, opening, and surface probing) in physics simulation and measures task success, capability and progress, post-interaction integrity, and dynamic plausibility. Multi-view semantic and static physical evaluations complement these metrics and help explain interaction outcomes. We systematically evaluate 12 state-of-the-art scene generation systems across 10 room types. Our results reveal that strong semantic and static physical performance does not necessarily translate into interaction readiness: a system with near-perfect static plausibility completes only 11.54% of interactions, 7 of the 12 systems produce scenes that cannot support any interaction, and even the best system succeeds in only 36.92% of prescribed tasks. These findings highlight two fundamental challenges in generating interaction-ready scenes: providing the necessary physical interaction capabilities and ensuring reliable physical behavior under active manipulation.

Figures & tables

Evaluation / Benchmark	Semantic View Setting	Static Physics	Active Interaction Physics	Methods Evaluated
Evaluation / Benchmark	Semantic View Setting	Static Physics	Active Interaction Physics	TSG	LVG	SAG	GCA
SceneWeaver [ 34 ]	Top-down	✓	✗	✓	✓	✓	✗
SceneEval [ 25 ]	Top-down	✓	✗	✓	✓	✗	✗
SceneCritic [ 22 ]	Per-view analysis	✓	✗	✗	✓	✗	✗
LEGO-Eval [ 7 ]	Constraint-specific	✗	✗	✗	✓	✗	✗
VWE-Bench [ 13 ]	5 views	✓	✗	✗	✗	✓	✓

Table 1: Comparison of representative benchmarks and evaluation protocols for text-conditioned 3D scene generation. Semantic view setting : camera viewpoints used to render scenes for semantic evaluation. Static physics : geometric and physical plausibility checks. Active interaction physics : evaluates physical behavior during object manipulation. Methods Evaluated : TSG (task-specific generators), LVG (LLM/VLM-based generators), SAG (specialized agentic systems), GCA (general-purpose coding agents).

Task-specific trained scene generators
Method	Semantic Quality				Static Physical Plausibility				Interactive Physical Plausibility
Method	Real $\uparrow$	Func $\uparrow$	Lay $\uparrow$	Comp $\uparrow$	COL $\downarrow$	OB $\downarrow$	SUP $\uparrow$	STAB $\uparrow$	ISR $\uparrow$	ICP $\uparrow$	PIV $\downarrow$	DP $\uparrow$
ATISS	3.00 $\pm$ 0.35	1.13 $\pm$ 0.50	1.93 $\pm$ 0.39	1.80 $\pm$ 0.32	59.20 $\pm$ 9.43	27.04 $\pm$ 5.65	27.52 $\pm$ 6.17	N.S.A.	0.00 $\pm$ 0.00	0.72 $\pm$ 0.44	N.S.A.	N.S.A.
DiffuScene	3.87 $\pm$ 0.48	1.30 $\pm$ 0.49	2.53 $\pm$ 0.39	2.13 $\pm$ 0.32	41.58 $\pm$ 7.70	14.69 $\pm$ 4.87	65.14 $\pm$ 6.15	N.S.A.	0.00 $\pm$ 0.00	0.84 $\pm$ 0.44	N.S.A.	N.S.A.
InstructScene	3.33 $\pm$ 0.32	1.40 $\pm$ 0.57	2.20 $\pm$ 0.37	2.00 $\pm$ 0.33	35.14 $\pm$ 7.58	23.22 $\pm$ 7.55	62.30 $\pm$ 7.63	N.S.A.	0.00 $\pm$ 0.00	0.77 $\pm$ 0.49	N.S.A.	N.S.A.
LLM/VLM scene workflows

Table 2: Performance comparison of different scene generation methods on ScIntBench. Metrics: Real = realism, Func = functionality, Lay = layout, Comp = completeness, COL = collision rate, OB = out-of-bounds, SUP = support, STAB = stability, ISR = interaction success rate, ICP = interaction capability progress, PIV = post-interaction violation rate, DP = dynamic plausibility. COL, OB, SUP, STAB, ISR, PIV are percentages, while all other metrics are scored on a 0–10 scale. N.S.A. denotes not support interaction due to missing targets or native physical prerequisites. We use mean values

\pm

95% t-distribution CI to quantify uncertainty.

\uparrow

indicates higher is better, and

\downarrow

indicates lower is better. Bold = best, underline = second best.

Figure 2: Qualitative comparison of generated scenes on ScIntBench. We show scenes generated by DiffuScene [ 26 ] , LayoutVLM [ 23 ] , Holodeck [ 33 ] , SAGE [ 28 ] , SceneSmith [ 17 ] , Codex [ 15 ] , and Claude Code [ 1 ] .

Figure 3: Qualitative comparison of interaction task outcomes. We show example outcomes in scenes generated by SAGE [ 28 ] , SceneSmith [ 17 ] , Codex [ 15 ] , and Claude Code [ 1 ] .

Task-specific trained scene generators
Method	Runtime (min)			API Cost (USD)
Method	Mean	Min	Max	Mean	Min	Max
ATISS	0.19	0.15	0.27	0	0	0
DiffuScene	0.53	0.51	0.55	0	0	0
InstructScene	0.05	0.03	0.06	0	0	0
LLM/VLM scene workflows

Table 3: Computational requirements statistics over these methods. For Codex and Claude Code, API costs are estimated from recorded token usage using OpenAI’s pricing and Anthropic’s pricing, respectively.

Metric	Spearman $\rho$ $\uparrow$	Kendall $\tau$ $\uparrow$
ICP	0.5126	0.4654
DP	0.4169	0.4215

Table 4: The correlation between MLLM-based evaluations and human evaluations.

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Method	Category	Training / Adaptation	Backbone
ATISS [ 16 ]	Trained Generator	3D-FRONT	–
DiffuScene [ 26 ]	Trained Generator	3D-FRONT	–
InstructScene [ 10 ]	Trained Generator	3D-FRONT	–
LayoutGPT [ 4 ]	LLM/VLM Workflow	Pretrained	GPT-4
LayoutVLM [ 23 ]	LLM/VLM Workflow	Pretrained	GPT-4o
Holodeck [ 33 ]	LLM/VLM Workflow	Pretrained	GPT-4

Appendix

Table 5: Configuration of the evaluated methods. 3D-FRONT denotes task-specific scene generation models trained on the 3D-FRONT [ 5 ] dataset, while Pretrained denotes approaches primarily relying on pretrained foundation models.

Figure 4: Overview of the 10 room types included in ScIntBench.

Action	Action Family	Target Object	Success Criteria
Open mini-fridge door	open_hinged	Mini fridge	Opening $\geq 60^{\circ}$ , held for $\geq 1$ s
Open room door	open_hinged	Room door	Opening $\geq 60^{\circ}$ , held for $\geq 1$ s
Push double bed	push_rigid	Double bed	Displacement $0.13$ – $0.17$ m
Push mini fridge	push_rigid	Mini fridge	Displacement $0.13$ – $0.17$ m
Push table	push_rigid	Table	Displacement $0.13$ – $0.17$ m
Place and push probe	place_push_probe	Bed surface	Sliding $\geq 0.02$ m for $\geq 0.2$ s

Appendix

Table 6: Interaction tasks constructed for Scene 3 (Bedroom). The main success criteria are shown; additional contact, support, and physical validity conditions are specified in the frozen action contracts.

Figure 5: Examples of robot policy calibration in controlled scenes. We validate the Franka Panda execution policies for hinged opening (left) and rigid pushing (right) before evaluating generated scenes.

Reference set	Source	ISR (%) $\uparrow$	ICP $\uparrow$	PIV (%) $\downarrow$	DP $\uparrow$
BEHAVIOR-1K	Reference	100.0	10.00	0.00	7.57
	SAGE	14.3	6.00	0.53	6.00
	SceneSmith	28.6	6.00	4.11	5.20
	Holodeck	42.9	5.43	6.58	6.50
	Codex	57.1	7.33	0.00	7.00
	Claude Code	57.1	9.20	10.30	7.00

Appendix

Table 7: Comparison with interactive reference scenes. We select BEHAVIOR-1K [ 9 ] , ReplicaCAD [ 24 ] , and iGibson [ 8 ] .

\uparrow

indicates higher is better, and

\downarrow

indicates lower is better. Bold = best.

Figure 6: Overview of interaction task outcomes across twelve scene generation methods.

Figure 7: Additional qualitative comparison of generated scenes across twelve scene generation systems.

Figure 8: Human evaluation interface for active metrics validation. Participants inspect synchronized target and overview videos, the intended action, and supporting physical measurements. They independently assess interaction capability and progress (ICP) using a six-level rubric and dynamic plausibility (DP) on a 0–10 scale.

3D Scene Generation

3D scene generation creates structured, explorable environments from text, images, maps, or other inputs.

See all 190 papers

PhyScene3D: Physically Consistent Interactive 3D Tabletop Scene Generation
Jun 1, 2026Weixing Chen, Zhuoqian Feng, Yang Liu +63D Scene Generation Functional 3D Scene Generation
Sun Yat-sen University, China · Guangdong Key Laboratory of Big Data Analysis and Processing · Huawei +1
PhyMix: Towards Physically Consistent Single-Image 3D Indoor Scene Generation with Implicit--Explicit Optimization
Apr 11, 2026Dongli Wu, Jingyu Hu, Ka-Hei Hui +4Image-to-3D Scene Generation 3D Scene Generation
1Lingnan University · 2The Hong Kong University of Science and Technology (Guangzhou) · 3The Chinese University of Hong Kong +2
ESG: Generating Physically Consistent Dynamic 3D Scenes from Text Descriptions
Sep 14, 2026Xintong Fang, Zhiyuan Fang, Rengan Xie +63D SGG 3D Scene Generation
Zhejiang University, Hangzhou, China · State Key Laboratory of CAD&CG, Zhejiang University, Hangzhou, China · Korea Advanced Institute of Science and Technology (KAIST) +1

Jun 1, 2026J/K move · Enter open · S save

PhyScene3D: Physically Consistent Interactive 3D Tabletop Scene Generation

Weixing Chen, Zhuoqian Feng, Yang Liu, Yexin Zhang+5

Sun Yat-sen University, China · Guangdong Key Laboratory of Big Data Analysis and Processing · Huawei+1

3D Scene Generation Functional 3D Scene Generation 3D Scene Layout Generation

Open paper AskPDF arXiv

Generating physically consistent 3D tabletop scenes is a fundamental yet underexplored problem for interactive and generalist robotic learning. The challenge stems from dense object hierarchies and irregular affordances. Here, an interactive scene denotes a physically valid, collision-free environment directly loadable into physics simulators. Existing methods, ranging from decoupled symbolic solvers to end-to-end regression models, often suffer from error propagation or overfitting to noisy supervision containing widespread physical violations. To address these limitations, we introduce PhyScene3D, a framework that reformulates generation as a Human-Mimetic Constructive Process. The proposed Cognitive Topological Reasoning Chain (CTRC) factorizes scene synthesis into a sequential, anchor-conditioned process. It employs a 3D AABB-based placement scheme that imposes a strong structural inductive bias. To address imperfect supervision and physical infeasibility, we introduce Physics-Aware Denoising Alignment (PADA). It integrates a differentiable Signed Distance Field (SDF) with Test-Time Optimization (TTO) to project generated scenes onto a physics-feasible manifold while preserving semantic intent. Experiments demonstrate that PhyScene3D outperforms state-of-the-art approaches in both semantic accuracy and physical validity, achieving a 40% reduction in scene-wise collision rate relative to the human-annotated training data.

Abstract

3D Scene Generation

Explore similar work

PhyScene3D: Physically Consistent Interactive 3D Tabletop Scene Generation

PhyMix: Towards Physically Consistent Single-Image 3D Indoor Scene Generation with Implicit--Explicit Optimization

ESG: Generating Physically Consistent Dynamic 3D Scenes from Text Descriptions