Editing a 3D indoor scene from natural language is conceptually straightforward but technically challenging. Existing open-vocabulary systems often regenerate large portions of a scene or rely on image-space edits that disrupt spatial structure, resulting in unintended global changes or physically inconsistent layouts. These limitations stem from treating editing primarily as a generative task. We take a different view. A user instruction defines a desired world state, and editing should be the minimal sequence of actions that makes this state true while preserving everything else. This perspective motivates Edit-As-Act, a framework that performs open-vocabulary scene editing as goal-regressive planning in 3D space. Given a source scene and free-form instruction, Edit-As-Act predicts symbolic goal predicates and plans in EditLang, a PDDL-inspired action language that we design with explicit preconditions and effects encoding support, contact, collision, and other geometric relations. A language-driven planner proposes actions, and a validator enforces goal-directedness, monotonicity, and physical feasibility, producing interpretable and physically coherent transformations. By separating reasoning from low-level generation, Edit-As-Act achieves instruction fidelity, semantic consistency, and physical plausibility - three criteria that existing paradigms cannot satisfy together. On E2A-Bench, our benchmark of 63 editing tasks across 9 indoor environments, Edit-As-Act significantly outperforms prior approaches across all edit types and scene categories.
Figures & tables
Instruction
Semantic
Physical
Fidelity
Consistency
Plausibility
LayoutGPT [ 8 ]
✓
✗
✗
AnyHome [ 12 ]
✗
✗
✓
ArtiScene [ 15 ]
✗
✓
✗
Edit-As-Act (Ours)
✓
✓
✓
Table 1: Comparison of three essential requirements for different scene editing paradigms. LayoutGPT represents direct 3D layout editing. AnyHome represents constraint-based optimization. ArtiScene represents image-driven editing followed by 3D lifting.
Figure 1: Overview of Edit-As-Act . Step 1 : an LLM converts a source scene S0 and instruction into symbolic goal predicates GT in EditLang. Step 2 : a planner–validator loop iteratively selects EditLang actions at that satisfy goals Gt and regresses remaining goals until all are grounded in S0 . Step 3 : the resulting action sequence is applied to S0 to obtain the edited scene ST .
Bathroom
Bedroom
Computer Room
Dining Room
Game Room
Methods
IF
SC
PP
IF
SC
PP
IF
SC
PP
IF
SC
PP
IF
SC
PP
LayoutGPT-E
50.1
62.5
78.3
35.7
30.2
53.8
53.8
32.1
84.7
50.5
31.4
66.2
52.1
48.9
83.5
AnyHome
69.7
56.7
81.7
64.0
62.6
82.7
59.0
56.3
90.7
57.1
66.9
76.6
58.3
67.1
91.4
ArtiScene-E
61.7
73.4
83.7
43.0
39.0
90.4
41.9
41.0
90.6
37.3
37.1
90.4
61.3
54.4
89.4
Edit-As-Act (ours)
58.7
88.9
89.1
45.7
73.1
91.9
73.6
88.0
94.1
89.7
95.3
92.7
58.7
79.9
91.1
Table 2: Quantitative comparison across nine scene categories. Edit-As-Act achieves the strongest and most consistent performance in instruction fidelity (IF), semantic consistency (SC), and physical plausibility (PP), demonstrating robust generalization across diverse spatial configurations.
Figure 2: Representative qualitative results. Baseline methods often introduce unintended global changes, fail to satisfy multi-step instructions, or generate incomplete edits. Edit-As-Act produces precise, instruction-aligned modifications that remain physically valid and preserve the overall scene identity.
Method
IF ↑
SC ↑
PP ↑
Latency (s)
Avg. Calls
Avg. Tok
GPT-5
49.6
52.3
73.3
18.9
1.0
3357
Gemini-3-Pro-preview
43.1
48.7
71.7
47.7
1.0
6959
Claude-4.5-opus
50.2
43.5
68.5
11.3
1.0
3514
SceneWeaver
68.7
78.3
82.1
102.5
7.3
18335
Edit-As-Act (ours)
69.1
86.6
91.7
87.2
5.9
19056
Table 3: Comparison with contemporary reasoning baselines.
IF
SC
PP
Edit-As-Act (ours)
69.1
86.6
91.7
w/o Validator
55.3
75.1
86.0
w/o Source-Awareness
58.2
75.1
89.2
w/o EditLang
55.4
73.6
88.3
Forward Planning
61.2
78.7
90.3
Coord. Prediction
52.8
68.1
85.5
Table 4: Ablation study showing the impact of each component.
Figure 3: Effect of removing EditLang, which provides explicit preconditions that allow the chair to be rotated around the table.
Figure 4: User study results. Ten participants rated edited scenes produced by Edit-As-Act, ArtiScene, and AnyHome on three criteria. Edit-As-Act obtains the highest perceived instruction fidelity, semantic consistency, and physical plausibility.
Appendix figures & tables25 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 5: LLMs demonstrate strong capabilities in interpreting existing 3D layouts (top) but remain unreliable when directly generating 3D layouts from instructions (bottom). This asymmetry motivates our goal-regressive formulation.
Figure 6: Overview of the object generation and stylization pipeline. We utilize Hyper3D Gen-2 [ 19 ] to synthesize high-fidelity 3D assets. The pipeline operates in two modes: (1) Text-to-3D for generating new objects from scratch, and (2) Point Cloud-Guided Stylization (highlighted in blue dashed boxes) where the input 3D object is converted into a point cloud to condition the generation, ensuring the geometric structure remains preserved while the texture is updated according to the text prompt.
FAIL “Action undoes a previously satisfied goal: on(lamp, table) . Do not move objects that are already correctly placed.”
Symbolic (Relevance)
Action: stylize(curtain, "blue") Context: Instruction is “Rotate the chair”. No goal relates to the curtain.
FAIL “Action does not satisfy any current goal. Focus only on the chair and its orientation.”
Appendix
Table 6: Validator Decision Examples. Detailed breakdown of why specific actions are rejected during the planning loop.
Figure 7: Validation of Instruction Fidelity (IF) Metrics. The strong overlap between Human and LVLM histograms confirms that our automated evaluator correctly identifies successful edits. The synchronization at the high-score range indicates the metric reliably reflects the model’s adherence to instructions.
Figure 8: Validation of Physical Plausibility (PP) Metrics. Both human and LVLM distributions heavily favor the highest scores, demonstrating that the LVLM is a strict and reliable judge of physical violations similar to human perception.
Figure 9: Validation of Semantic Consistency (SC) Metrics. Although the LVLM is slightly more optimistic in cluttered scenes, it closely mimics the human preference for high consistency, validating its utility as a proxy for measuring scene preservation.
Figure 10: Absolute Score Difference (IF). Disagreements are localized to specific ambiguous scenarios.
Figure 11: Absolute Score Difference (PP). High agreement (low error) is observed across most categories.
Figure 12: Absolute Score Difference (SC). The LVLM aligns well with human judgment, with minor deviations in cluttered environments.
ADD
REMOVE
TRANSLATE
ROTATE
Methods
IF
SC
PP
IF
SC
PP
IF
SC
PP
IF
SC
PP
LayoutGPT-E
40.5
42.1
82.4
60.3
71.5
88.1
52.6
50.3
80.7
41.4
32.5
78.1
AnyHome
47.2
65.0
82.1
61.2
58.7
79.8
66.0
76.6
85.7
52.7
48.4
85.0
ArtiScene-E
39.9
50.0
89.1
80.1
78.4
94.0
50.2
58.0
86.1
39.9
28.0
90.8
Edit-As-Act (ours)
82.7
90.2
88.3
73.9
80.1
95.9
97.1
95.7
89.6
53.1
86.3
95.0
Appendix
Table 7: Performance by editing operation type. Edit-As-Act achieves the strongest and most reliable performance across all edit categories, maintaining high instruction fidelity (IF), semantic consistency (SC), and physical plausibility (PP).
Figure 13: Qualitative Failure Examples. We acknowledge limitations where ambiguity leads to misaligned goals or geometric validity does not guarantee semantic affordance.
Setting
IF
SC
PP
w/ GPT-OSS-20b
62.5
72.8
77.8
Small Pred. Set
52.4
78.1
82.6
High Sensitivity
66.5
84.5
90.2
Low Sensitivity
68.9
85.0
88.0
Ours
69.1
86.6
91.7
Appendix
Table 8: Additional ablation on backbone robustness, predicate complexity, and validator parameter sensitivity.
Method
OOB Scene Ratio (%) ↓
Floating Object Rate (%) ↓
ArtiScene-E
88.89
92.06
AnyHome
7.94
57.14
Edit-As-Act (Ours)
6.16
14.21
Appendix
Table 9: Comparison of Geometry-Based Metrics. Lower is better. Edit-As-Act achieves the best physical validity.
Figure 14: Diversity of E2A-Bench Source Scenes. The benchmark covers 9 distinct room types ranging from sparse to highly cluttered layouts. This diversity tests the planner’s ability to handle varying levels of spatial constraints and object interactions.
Text guided 3D scene editing provides an intuitive interface for modifying reconstructed environments, but remains difficult because natural language design requests are often semantically underspecified and must be grounded in cluttered 3D scenes. Existing methods typically formulate the task as one-shot conditional generation from a single prompt, failing to resolve ambiguous user intents or achieve precise spatial grounding. Consequently, they suffer from severe object localization drift, tracking failure under occlusions, and the notorious multi-view "sticker effect." To overcome these limitations, we present DesignAgent3D, an interactive multimodal agentic framework that reformulates 3D scene editing as a designer-like Plan-Perceive-Act paradigm. The agent first plans by interacting with the user to clarify underspecified design goals, then perceives by grounding the intended edit to specific objects or regions in the 3D scene, and finally acts by applying controlled visual modifications while preserving scene consistency. The edits are further integrated into the underlying 3D representation, supporting persistent and multi-view consistent novel-view rendering. Extensive experiments across both NeRF and 3D Gaussian Splatting backbones demonstrate that DesignAgent3D significantly outperforms state-of-the-art baselines, delivering superior semantic intent alignment, impeccable spatial localization accuracy, and high-fidelity multi-view consistency.
Xiujin Liu, Tianyu Yang, Yilun Zhao +1
University of Notre Dame South Bend, Indiana, USA · Yale University New Haven, Connecticut, USA
Scene synthesis and editing has emerged as a promising direction in computer graphics. Current trained approaches for 3D indoor scene generation either oversimplify object semantics through one-hot class encodings (e.g., 'chair' or 'table'), require masked diffusion for editing, ignore room boundaries, or rely on floor plan renderings that fail to capture complex layouts. LLM-based methods enable richer semantics via natural language, but lack editing functionality, are limited to rectangular layouts, or rely on weak spatial reasoning from implicit world models. We introduce ReSpace, a generative framework for autoregressive text-driven 3D indoor scene synthesis and editing. Our approach features a compact structured scene representation with explicit room boundaries that enables asset-agnostic deployment and frames scene manipulation as a next-token prediction task, supporting object addition, removal, and swapping via natural language. We employ supervised fine-tuning with a preference alignment stage to train a specialized language model for object addition that accounts for user instructions, spatial geometry, object semantics, and scene-level composition. We further introduce a voxelization-based evaluation metric capturing fine-grained geometric violations beyond 3D bounding boxes. Experiments surpass state-of-the-art on object addition and achieve superior human-perceived quality on the application of full scene synthesis, despite not being trained on it.
Existing 3D scene editing methods typically rely on per-scene optimization over explicit 3D representations or cascaded edit-and-reconstruct pipelines, resulting in high test-time cost, limited 3D awareness, and structural inconsistencies. To couple appearance synthesis with geometry prediction, we adapt a pretrained unified RGB-geometry latent space to feed-forward scene editing. Given a source video and an edited reference image, JointEdit3D performs asymmetric latent inpainting: it observes only the edited RGB reference latent and jointly generates the remaining RGB latents and the entire geometry latent along the source trajectory. JointEdit3D introduces a dedicated SceneAnchor Branch to inject source-scene structure without forcing direct copying, and adopts edit/background-aware losses to balance edited-region fidelity with unedited-content preservation. To address the lack of paired resources for standardized 3D scene editing evaluation, we introduce SceneEdit3D-15K, a dataset with 15K paired editing samples and renderer-provided 3D annotations, together with SceneEdit3D-Bench, a curated 100-sample benchmark. Experiments show that JointEdit3D improves edited-region quality and 3D structural completeness over prior baselines while maintaining competitive background preservation.
Xinnan Zhu, Ruijie Xu, Jiayu Ying +4
East China Normal University · Shanghai Artificial Intelligence Laboratory · Fudan University +1