Organizations: University of Washington · Tencent AI Lab · The Hong Kong University of Science and Technology · University of Science and Technology of China · Shanghai Jiao Tong University
Recent LLM-centric advances can generate attractive furnished rooms by selecting and arranging assets. Yet evaluation remains limited to specific perspectives or predefined questions, failing to reflect overall scene quality. To address this pressing need, we introduce GREP. Its core is a focus shift from defining specific questions to establishing a three-tier evaluation philosophy: basic physical validity checks whether a scene obeys physical constraints; visual coherence concerns compatibility in object scale, style, and composition; and, at the highest level, human-use logic requires object arrangements to align with everyday human habits. Guided by these principles, an evaluation agent autonomously and adaptively identifies issues from scene context, gathers evidence, and produces scores and reports. This supports scene-specific issue discovery and reliable judgments. Experiments show that (1) GREP achieves 87% agreement with human majority preferences, substantially outperforming baseline evaluators. (2) GREP reports improve scene quality through refinement, whereas score-only and no-feedback baselines show slight declines. (3) GREP suggests that as planning ability improves, recent models such as GPT-5.6 Sol can move from relying on pipelines and harnesses to challenging their guidance and constraints, but still lack the ability to resolve the resulting conflicts. Stronger models are therefore expected not only to plan well, but also to adapt to external guidance and constraints more flexibly. Project page: https://han-808.github.io/GREP/
Figures & tables
Figure 1: GREP is a scoring framework for generated scenes, grounded in three progressive principles : (A) physical validity, (B) visual coherence, and (C) human-use logic. In contrast, existing metrics rely on narrow criteria or hard metrics, failing to assess the scene quality accurately. Moreover, GREP lets the evaluation agent autonomously identify issues, gather evidence, and reach comprehensive judgments. Check and cross marks indicate whether each evaluator’s output matches the actual state of the scene, not whether the scene passes. In (A), a pure contact rule would read the mirror 0.0008 m from the wall as floating; both VLM-assisted evaluators clear it, whereas only GREP separates the intended sofa–rug contact from the table–rug penetration. † A holistic VLM judge given only the global view.
Figure 2: GREP evaluation framework. (1) Analyze the room globally to raise questions under the three principles. (2) Adaptively gather evidence to resolve remaining checks. (3) Aggregate scores and compile a diagnostic report.
GPT-6 Astra
GPT-5.6 Sol
Claude Opus 5
Claude Fable 5
Claude Sonnet 5
Kimi K3
GLM- 5.3
HY4 Preview
Open space
Overall
84.5
73.7
81.5
77.9
65.3
73.2
70.8
61.8
Physical
85.0
81.0
90.3
77.8
72.4
70.2
67.2
55.9
Visual
88.4
84.8
95.3
95.1
87.0
91.2
90.2
86.8
Human-use
83.4
66.4
72.4
74.8
56.0
72.0
69.7
61.4
With wall layouts
Table 1: Model-native results in open spaces (without walls) and floorplans with rectangular wall layouts. Bold marks the highest value in each row. HY4 Preview’s with-wall average covers nine of the ten floorplans (Appendix C.3 ).
None
LayoutGPT
LayoutVLM
DirectLayout
SceneWeaver
GPT-5.6 Sol backbone
Overall
78.3
74.3
73.6
68.5
62.1
Physical
83.4
81.1
81.3
81.3
77.5
Human-use
74.5
69.4
68.0
59.2
50.9
GPT-4.1 backbone
Overall
40.2
31.0
54.5
44.1
40.0
Table 2: Results of specialized pipelines across model backbones and asset placement methods. Visual coherence metrics are excluded under FrozenAssets * . “None” means raising the asset arrangement plans with the native planning abilities of models without requiring a specialized pipeline. Bold marks the highest value in each row.
GPT-5.6 Sol
Kimi K3
GLM-5.3
HY4 Preview
Default constraints
Overall
83.0
88.5
84.7
88.5
Physical
75.8
78.5
84.5
89.6
Visual
98.8
97.3
98.0
97.8
Human-use
85.2
94.0
82.3
86.0
20% more assets
Table 3: Results of harnessed agents under default requirements and a stricter condition requiring 20% more assets. Bold marks the highest value in each row.
Figure 3: Qualitative examples of Model-native planning and pipeline-dependent generation. (a) HY4 Preview uses walls as placement references, (b) yet its layout contains a bed-chair collision, shown in the magnified view. With the same dining assets and GPT-5.6 Sol backbone, (c) Model-native arranges chairs around the round table, while (d) LayoutGPT places them in two straight rows.
Figure 4: Qualitative examples under the agent harness. (a) HY4 Preview uses a more diverse set of assets. (b) GPT-5.6 Sol repeats several assets, including wall pictures, chairs, and sofa cushions. Numbered boxes mark the enlarged insets below each layout: (1) wall pictures, (2) chairs, and (3) plants in (a) or sofa cushions in (b).
Figure 5: Report-guided scene refinement with GPT-4.1. (a) Mean score across five scenes from the shared initial layout (R0) through three revisions (R1–R3), using the 0–1 pipeline scoring protocol in Appendix E.1 ; shaded bands show ±1 standard error across the five scenes. (b) The dining layout of S101 before and after report-guided revision: dashed circles mark overlapping chairs, which are separated in the revised layout. These views were rendered again from the saved layouts.
Figure 6: Agreement with human overall preferences across 15 pairs of six scenes. (a) Agreement with the human majority; the error bar shows the between-run standard deviation of the Holistic VLM, and the dashed line marks chance (50%). (b) Pair P03: all five annotators and GREP prefer A, all three Holistic VLM runs prefer B, and Physical-only assigns equal scores.
Appendix figures & tables8 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 7: Per-metric evaluation procedures. Criterion-specific checks organize evidence for adjudication; conclusive geometric predicates are resolved directly. Bounded acquisition supports check preparation and requests for more evidence. The procedure depends on the metric (Table 4 ). OOB denotes out-of-bounds.
Metric
Check formulation
Evidence scope
Resolution
Collision
Mesh and oriented bounding box (OBB) screening of object pairs
Object pair
Deterministic for certified separation or a bounded interface overlap, else evaluator
Out-of-bounds
Screening against the room envelope
Object and boundary
Deterministic for containment within tolerance, else evaluator
Support
Support-chain screening along gravity
Object and its supports
Deterministic for grounded contact, else evaluator
Scale
VLM screen of structured object information
Group, after the screen
Screen for clearly valid cases, else evaluator
Style
Global visual screen
Room, then group
Screen, or evaluator after local confirmation
Pairing
VLM screen of the object inventory
Room inventory, at most one global view
Screen for clearly valid cases, else evaluator
Appendix
Table 4: Per-metric evaluation procedures. A group is an object group from the room partition (Appendix A.2 ), and the evaluator is the metric’s VLM evaluator.
Metric
Severity and penalty
Scale
Noticeable: p0 ; gross: p1
Style
Noticeable: p0 ; gross: p1
Pairing
Invalid: p1
Function
Impaired: p0 ; blocked: p1
Placement
Atypical: p0 ; implausible: p1
Appendix
Table 5: Severity levels and their event penalties.
Metric
wm
Collision
0.15
Support
0.15
Out-of-bounds
0.08
Scale
0.03
Style
0.03
Pairing
0.04
Appendix
Table 6: Metric weights used to aggregate room scores.
Limit
Value
Evidence rounds per episode
3
Images per episode
8
Views per round
2
Camera-selection calls per episode
4
Camera actions per episode
3
Deterministic camera candidates per request
6
Appendix
Table 7: Evidence-acquisition limits. Episode limits apply to each scoped evaluation episode, and Function check preparation has a separate budget.
Generator
Task
Mean
SD
Min–Max
GPT-6 Astra
S112
97.7
2.0
94.4–99.3
Kimi K3
S118
95.8
5.2
86.6–98.7
GPT-5.6 Sol
S118
94.4
0.4
93.9–94.8
Claude Fable 5
S110
92.1
0.7
90.9–92.4
Claude Opus 5
S114
89.1
2.8
84.1–90.7
Claude Sonnet 5
S116
88.1
1.0
87.1–89.6
Appendix
Table 8: Repeatability of GREP scores. Each scene was evaluated five times with fixed inputs, configuration, and judge (GPT-5.6 Sol).
Field
Content
Metric
Function
Check type
Relative-use geometry
Objects
Piano bench, with the digital piano
Severity
Blocked
Reason
“The bench is positioned about 2.0 m from the piano rather than within seated operating reach, so it cannot function as seating for the piano in the authored arrangement.”
Measurements
Center distance 2.0 m; the seating side of the bench and the control side of the piano each at 90∘ to the direction of the other object
Appendix
Table 9: A defect record from the diagnostic report * on the R0 layout of S104, shortened to the fields listed. Records also carry the verdict, a defect category, the check scope, an attribution field, and check identifiers, omitted here.
Scene
Branch
R0
R1
R2
R3
Mean R1–R3
S101
No feedback
0.534
0.489
0.456 †
0.526
0.490
Scores
0.494
0.517
0.499
0.503
Report
0.435
0.526
0.670
0.544
S102
No feedback
0.608
0.542 †
0.458 †
0.403 †
0.468
Scores
0.468 †
0.493 †
0.458 †
0.473
Report
0.569
0.612
0.712
0.631
Appendix
Table 10: Pipeline scores of the refinement study (0–1). R0 is shared by the three branches of a scene. A dagger marks a revision whose layout is identical to that of the previous round.
Large Language Models (LLMs) and Vision-Language Models (VLMs) increasingly generate indoor scenes through intermediate structures such as layouts and scene graphs, yet evaluation still relies on LLM or VLM judges that score rendered views, making judgments sensitive to viewpoint, prompt phrasing, and hallucination. When the evaluator is unstable, it becomes difficult to determine whether a model has produced a spatially plausible scene or whether the output score reflects the choice of viewpoint, rendering, or prompt. We introduce SceneCritic, a symbolic evaluator for floor-plan-level layouts. SceneCritic's constraints are grounded in SceneOnto, a structured spatial ontology we construct by aggregating indoor scene priors from 3D-FRONT, ScanNet, and Visual Genome. SceneOnto traverses this ontology to jointly verify semantic, orientation, and geometric coherence across object relationships, providing object-level and relationship-level assessments that identify specific violations and successful placements. Furthermore, we pair SceneCritic with an iterative refinement test bed that probes how models build and revise spatial structure under different critic modalities: a rule-based critic using collision constraints as feedback, an LLM critic operating on the layout as text, and a VLM critic operating on rendered observations. Through extensive experiments, we show that (a) SceneCritic aligns substantially better with human judgments than VLM-based evaluators, (b) text-only LLMs can outperform VLMs on semantic layout quality, and (c) image-based VLM refinement is the most effective critic modality for semantic and orientation correction.
Kathakoli Sengupta, Kai Ao, Paola Cascante-Bonilla
Generating complete, coherent, and usable 3D indoor scenes from natural language requires strong design intelligence: the ability to design high-quality scenes and critique their quality. However, existing approaches either rely on general-purpose models lacking domain-specific design experience or learn from static scenes. In this work, we introduce Scene-Designer, an agentic framework that learns design intelligence through complementary design and critique experiences. We construct two datasets: Scene-OTA-35K, which provides Observation--Thought--Action supervision for learning scene design and efficient error correction, and Scene-OTC-15K, which provides Observation--Thought--Critic supervision for learning scene quality critique across physical validity, functional correctness, and layout coherence. Based on these experiences, we propose Scene-GRPO, a Critic-guided GRPO framework that improves design decisions and reduces unnecessary trial-and-error corrections. Extensive experiments demonstrate that Scene-Designer achieves superior performance in physical validity, functional correctness, and layout coherence, while the learned Scene-Critic significantly outperforms general-purpose models in scene critique. Moreover, test-time Critic guidance further improves scene generation, showing that the Critic can provide useful guidance beyond training. Additional analysis suggests a trend of mutual enhancement between scene design and critique abilities.
Xuanyu Liu, Hongyan Feng, Miao Pan +5
Zhejiang University · Universal Identification Technology (Hangzhou) Co.,Ltd.
Before letting an agent operate over real context, can you prove it used the right evidence? GroundEval turns that question into a deterministic test of what the agent searched, fetched, cited, and was permitted to access. In one case study, two frontier LLM judges scored a plausible agent response 0.85 and higher. But the trace told a different story: the agent had never retrieved the artifact its answer depended on, yielding a GroundEval score of 0.000. We introduce GroundEval, a judge-free framework for evaluating agents against grounded, time-bounded, and access-controlled evidence. GroundEval uses a domain configuration to generate questions, lets the agent choose how to answer, and then scores both the final answer and the recorded trajectory that produced it. The benchmark targets three failures that LLM-as-judge evaluation struggles to detect: whether an agent checked before claiming absence, reasoned only from evidence available to the actor at the relevant time, and used the correct causal mechanism rather than a plausible one. These correspond to three tracks: Silence, Perspective, and Counterfactual. GroundEval exposes when plausible answers rest on invalid evidence paths, and produces structured per-question diagnostics that pair tool activity with the agent's turn-level narration, making each score inspectable rather than merely reported. Our case studies suggest this failure mode is common rather than exceptional, one that final-answer and judge-based evaluation cannot detect by construction.