GREP: Grounding Generated Room Evaluation in Principles
Organizations: University of Washington · Tencent AI Lab · The Hong Kong University of Science and Technology · University of Science and Technology of China ·
Abstract
Recent LLM-centric advances can generate attractive furnished rooms by selecting and arranging assets. Yet evaluation remains limited to specific perspectives or predefined questions, failing to reflect overall scene quality. To address this pressing need, we introduce GREP. Its core is a focus shift from defining specific questions to establishing a three-tier evaluation philosophy: basic physical validity checks whether a scene obeys physical constraints; visual coherence concerns compatibility in object scale, style, and composition; and, at the highest level, human-use logic requires object arrangements to align with everyday human habits. Guided by these principles, an evaluation agent autonomously and adaptively identifies issues from scene context, gathers evidence, and produces scores and reports. This supports scene-specific issue discovery and reliable judgments. Experiments show that (1) GREP achieves 87% agreement with human majority preferences, substantially outperforming baseline evaluators. (2) GREP reports improve scene quality through refinement, whereas score-only and no-feedback baselines show slight declines. (3) GREP suggests that as planning ability improves, recent models such as GPT-5.6 Sol can move from relying on pipelines and harnesses to challenging their guidance and constraints, but still lack the ability to resolve the resulting conflicts. Stronger models are therefore expected not only to plan well, but also to adapt to external guidance and constraints more flexibly. Project page: https://han-808.github.io/GREP/
Figures & tables
| GPT-6 Astra | GPT-5.6 Sol | Claude Opus 5 | Claude Fable 5 | Claude Sonnet 5 | Kimi K3 | GLM- 5.3 | HY4 Preview | |
| Open space | ||||||||
| Overall | 84.5 | 73.7 | 81.5 | 77.9 | 65.3 | 73.2 | 70.8 | 61.8 |
| Physical | 85.0 | 81.0 | 90.3 | 77.8 | 72.4 | 70.2 | 67.2 | 55.9 |
| Visual | 88.4 | 84.8 | 95.3 | 95.1 | 87.0 | 91.2 | 90.2 | 86.8 |
| Human-use | 83.4 | 66.4 | 72.4 | 74.8 | 56.0 | 72.0 | 69.7 | 61.4 |
| With wall layouts | ||||||||
| None | LayoutGPT | LayoutVLM | DirectLayout | SceneWeaver | |
| GPT-5.6 Sol backbone | |||||
| Overall | 78.3 | 74.3 | 73.6 | 68.5 | 62.1 |
| Physical | 83.4 | 81.1 | 81.3 | 81.3 | 77.5 |
| Human-use | 74.5 | 69.4 | 68.0 | 59.2 | 50.9 |
| GPT-4.1 backbone | |||||
| Overall | 40.2 | 31.0 | 54.5 | 44.1 | 40.0 |
| GPT-5.6 Sol | Kimi K3 | GLM-5.3 | HY4 Preview | |
| Default constraints | ||||
| Overall | 83.0 | 88.5 | 84.7 | 88.5 |
| Physical | 75.8 | 78.5 | 84.5 | 89.6 |
| Visual | 98.8 | 97.3 | 98.0 | 97.8 |
| Human-use | 85.2 | 94.0 | 82.3 | 86.0 |
| 20% more assets | ||||
Appendix figures & tables8 assets
Supplementary material from the paper’s appendix.
Appendix
| Metric | Check formulation | Evidence scope | Resolution |
|---|---|---|---|
| Collision | Mesh and oriented bounding box (OBB) screening of object pairs | Object pair | Deterministic for certified separation or a bounded interface overlap, else evaluator |
| Out-of-bounds | Screening against the room envelope | Object and boundary | Deterministic for containment within tolerance, else evaluator |
| Support | Support-chain screening along gravity | Object and its supports | Deterministic for grounded contact, else evaluator |
| Scale | VLM screen of structured object information | Group, after the screen | Screen for clearly valid cases, else evaluator |
| Style | Global visual screen | Room, then group | Screen, or evaluator after local confirmation |
| Pairing | VLM screen of the object inventory | Room inventory, at most one global view | Screen for clearly valid cases, else evaluator |
| Metric | Severity and penalty |
|---|---|
| Scale | Noticeable: ; gross: |
| Style | Noticeable: ; gross: |
| Pairing | Invalid: |
| Function | Impaired: ; blocked: |
| Placement | Atypical: ; implausible: |
| Metric | |
|---|---|
| Collision | 0.15 |
| Support | 0.15 |
| Out-of-bounds | 0.08 |
| Scale | 0.03 |
| Style | 0.03 |
| Pairing | 0.04 |
| Limit | Value |
|---|---|
| Evidence rounds per episode | 3 |
| Images per episode | 8 |
| Views per round | 2 |
| Camera-selection calls per episode | 4 |
| Camera actions per episode | 3 |
| Deterministic camera candidates per request | 6 |
| Generator | Task | Mean | SD | Min–Max |
|---|---|---|---|---|
| GPT-6 Astra | S112 | 97.7 | 2.0 | 94.4–99.3 |
| Kimi K3 | S118 | 95.8 | 5.2 | 86.6–98.7 |
| GPT-5.6 Sol | S118 | 94.4 | 0.4 | 93.9–94.8 |
| Claude Fable 5 | S110 | 92.1 | 0.7 | 90.9–92.4 |
| Claude Opus 5 | S114 | 89.1 | 2.8 | 84.1–90.7 |
| Claude Sonnet 5 | S116 | 88.1 | 1.0 | 87.1–89.6 |
| Field | Content |
|---|---|
| Metric | Function |
| Check type | Relative-use geometry |
| Objects | Piano bench, with the digital piano |
| Severity | Blocked |
| Reason | “The bench is positioned about 2.0 m from the piano rather than within seated operating reach, so it cannot function as seating for the piano in the authored arrangement.” |
| Measurements | Center distance 2.0 m; the seating side of the bench and the control side of the piano each at to the direction of the other object |
| Scene | Branch | R0 | R1 | R2 | R3 | Mean R1–R3 |
|---|---|---|---|---|---|---|
| S101 | No feedback | 0.534 | 0.489 | 0.456 † | 0.526 | 0.490 |
| Scores | 0.494 | 0.517 | 0.499 | 0.503 | ||
| Report | 0.435 | 0.526 | 0.670 | 0.544 | ||
| S102 | No feedback | 0.608 | 0.542 † | 0.458 † | 0.403 † | 0.468 |
| Scores | 0.468 † | 0.493 † | 0.458 † | 0.473 | ||
| Report | 0.569 | 0.612 | 0.712 | 0.631 |
Automated Evaluation
Automated evaluation develops metrics, benchmarks, and AI judges to assess model outputs and system behavior at scale.
See all 419 papers