Edit-As-Act: Goal-Regressive Planning for Open-Vocabulary 3D Indoor Scene Editing

Seongrae Noh; SeungWon Seo; Gyeong-Moon Park; HyeongYeop Kang

cs.CVMar 18, 2026

Edit-As-Act: Goal-Regressive Planning for Open-Vocabulary 3D Indoor Scene Editing

Authors: Seongrae Noh, SeungWon Seo, Gyeong-Moon Park, HyeongYeop Kang

Organizations: Korea University

Abstract

Editing a 3D indoor scene from natural language is conceptually straightforward but technically challenging. Existing open-vocabulary systems often regenerate large portions of a scene or rely on image-space edits that disrupt spatial structure, resulting in unintended global changes or physically inconsistent layouts. These limitations stem from treating editing primarily as a generative task. We take a different view. A user instruction defines a desired world state, and editing should be the minimal sequence of actions that makes this state true while preserving everything else. This perspective motivates Edit-As-Act, a framework that performs open-vocabulary scene editing as goal-regressive planning in 3D space. Given a source scene and free-form instruction, Edit-As-Act predicts symbolic goal predicates and plans in EditLang, a PDDL-inspired action language that we design with explicit preconditions and effects encoding support, contact, collision, and other geometric relations. A language-driven planner proposes actions, and a validator enforces goal-directedness, monotonicity, and physical feasibility, producing interpretable and physically coherent transformations. By separating reasoning from low-level generation, Edit-As-Act achieves instruction fidelity, semantic consistency, and physical plausibility - three criteria that existing paradigms cannot satisfy together. On E2A-Bench, our benchmark of 63 editing tasks across 9 indoor environments, Edit-As-Act significantly outperforms prior approaches across all edit types and scene categories.

Figures & tables

	Instruction	Semantic	Physical
	Fidelity	Consistency	Plausibility
LayoutGPT [ 8 ]	✓	✗	✗
AnyHome [ 12 ]	✗	✗	✓
ArtiScene [ 15 ]	✗	✓	✗
Edit-As-Act (Ours)	✓	✓	✓

Table 1: Comparison of three essential requirements for different scene editing paradigms. LayoutGPT represents direct 3D layout editing. AnyHome represents constraint-based optimization. ArtiScene represents image-driven editing followed by 3D lifting.

Figure 1: Overview of Edit-As-Act . Step 1 : an LLM converts a source scene $S_{0}$ and instruction into symbolic goal predicates $G_{T}$ in EditLang. Step 2 : a planner–validator loop iteratively selects EditLang actions $a_{t}$ that satisfy goals $G_{t}$ and regresses remaining goals until all are grounded in $S_{0}$ . Step 3 : the resulting action sequence is applied to $S_{0}$ to obtain the edited scene $S_{T}$ .

	Bathroom			Bedroom			Computer Room			Dining Room			Game Room
Methods	IF	SC	PP	IF	SC	PP	IF	SC	PP	IF	SC	PP	IF	SC	PP
LayoutGPT-E	50.1	62.5	78.3	35.7	30.2	53.8	53.8	32.1	84.7	50.5	31.4	66.2	52.1	48.9	83.5
AnyHome	69.7	56.7	81.7	64.0	62.6	82.7	59.0	56.3	90.7	57.1	66.9	76.6	58.3	67.1	91.4
ArtiScene-E	61.7	73.4	83.7	43.0	39.0	90.4	41.9	41.0	90.6	37.3	37.1	90.4	61.3	54.4	89.4
Edit-As-Act (ours)	58.7	88.9	89.1	45.7	73.1	91.9	73.6	88.0	94.1	89.7	95.3	92.7	58.7	79.9	91.1

Table 2: Quantitative comparison across nine scene categories. Edit-As-Act achieves the strongest and most consistent performance in instruction fidelity (IF), semantic consistency (SC), and physical plausibility (PP), demonstrating robust generalization across diverse spatial configurations.

Figure 2: Representative qualitative results. Baseline methods often introduce unintended global changes, fail to satisfy multi-step instructions, or generate incomplete edits. Edit-As-Act produces precise, instruction-aligned modifications that remain physically valid and preserve the overall scene identity.

Method	IF $\uparrow$	SC $\uparrow$	PP $\uparrow$	Latency (s)	Avg. Calls	Avg. Tok
GPT-5	49.6	52.3	73.3	18.9	1.0	3357
Gemini-3-Pro-preview	43.1	48.7	71.7	47.7	1.0	6959
Claude-4.5-opus	50.2	43.5	68.5	11.3	1.0	3514
SceneWeaver	68.7	78.3	82.1	102.5	7.3	18335
Edit-As-Act (ours)	69.1	86.6	91.7	87.2	5.9	19056

Table 3: Comparison with contemporary reasoning baselines.

	IF	SC	PP
Edit-As-Act (ours)	69.1	86.6	91.7
w/o Validator	55.3	75.1	86.0
w/o Source-Awareness	58.2	75.1	89.2
w/o EditLang	55.4	73.6	88.3
Forward Planning	61.2	78.7	90.3
Coord. Prediction	52.8	68.1	85.5

Table 4: Ablation study showing the impact of each component.

Figure 3: Effect of removing EditLang, which provides explicit preconditions that allow the chair to be rotated around the table.

Figure 4: User study results. Ten participants rated edited scenes produced by Edit-As-Act, ArtiScene, and AnyHome on three criteria. Edit-As-Act obtains the highest perceived instruction fidelity, semantic consistency, and physical plausibility.

Appendix figures & tables25 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 5: LLMs demonstrate strong capabilities in interpreting existing 3D layouts (top) but remain unreliable when directly generating 3D layouts from instructions (bottom). This asymmetry motivates our goal-regressive formulation.

Figure 6: Overview of the object generation and stylization pipeline. We utilize Hyper3D Gen-2 [ 19 ] to synthesize high-fidelity 3D assets. The pipeline operates in two modes: (1) Text-to-3D for generating new objects from scratch, and (2) Point Cloud-Guided Stylization (highlighted in blue dashed boxes) where the input 3D object is converted into a point cloud to condition the generation, ensuring the geometric structure remains preserved while the texture is updated according to the text prompt.

Instruction	Goal Predicates
“Move the dining set to the window.”	near(table_1, window_1), grouped_with(chair_*, table_1)
“Make the chair match the sofa.”	matches_style(chair_1, sofa_1)
“Place the lamp to the left of the bed.”	left_of(lamp_1, bed_1, cam_frame)

Appendix

Table 5: Examples of Natural Language to EditLang Goal Translation.

Check Type	Proposed Action & Context	Validator Decision & Feedback
Physical (Collision)	Action: move_to(chair_1, [1.2, 0, 1.5]) Context: Target coordinates overlap with table_1 .	FAIL “Target position causes collision with table_1 . Please select a clear region or move the obstacle first.”
Symbolic (Monotonicity)	Action: move_to(lamp, floor) Context: Previous step satisfied on(lamp, table) .	FAIL “Action undoes a previously satisfied goal: on(lamp, table) . Do not move objects that are already correctly placed.”
Symbolic (Relevance)	Action: stylize(curtain, "blue") Context: Instruction is “Rotate the chair”. No goal relates to the curtain.	FAIL “Action does not satisfy any current goal. Focus only on the chair and its orientation.”

Appendix

Table 6: Validator Decision Examples. Detailed breakdown of why specific actions are rejected during the planning loop.

Figure 7: Validation of Instruction Fidelity (IF) Metrics. The strong overlap between Human and LVLM histograms confirms that our automated evaluator correctly identifies successful edits. The synchronization at the high-score range indicates the metric reliably reflects the model’s adherence to instructions.

Figure 8: Validation of Physical Plausibility (PP) Metrics. Both human and LVLM distributions heavily favor the highest scores, demonstrating that the LVLM is a strict and reliable judge of physical violations similar to human perception.

Figure 9: Validation of Semantic Consistency (SC) Metrics. Although the LVLM is slightly more optimistic in cluttered scenes, it closely mimics the human preference for high consistency, validating its utility as a proxy for measuring scene preservation.

Figure 10: Absolute Score Difference (IF). Disagreements are localized to specific ambiguous scenarios.

Figure 11: Absolute Score Difference (PP). High agreement (low error) is observed across most categories.

Figure 12: Absolute Score Difference (SC). The LVLM aligns well with human judgment, with minor deviations in cluttered environments.

	ADD			REMOVE			TRANSLATE			ROTATE
Methods	IF	SC	PP	IF	SC	PP	IF	SC	PP	IF	SC	PP
LayoutGPT-E	40.5	42.1	82.4	60.3	71.5	88.1	52.6	50.3	80.7	41.4	32.5	78.1
AnyHome	47.2	65.0	82.1	61.2	58.7	79.8	66.0	76.6	85.7	52.7	48.4	85.0
ArtiScene-E	39.9	50.0	89.1	80.1	78.4	94.0	50.2	58.0	86.1	39.9	28.0	90.8
Edit-As-Act (ours)	82.7	90.2	88.3	73.9	80.1	95.9	97.1	95.7	89.6	53.1	86.3	95.0

Appendix

Table 7: Performance by editing operation type. Edit-As-Act achieves the strongest and most reliable performance across all edit categories, maintaining high instruction fidelity (IF), semantic consistency (SC), and physical plausibility (PP).

Setting	IF	SC	PP
w/ GPT-OSS-20b	62.5	72.8	77.8
Small Pred. Set	52.4	78.1	82.6
High Sensitivity	66.5	84.5	90.2
Low Sensitivity	68.9	85.0	88.0
Ours	69.1	86.6	91.7

Appendix

Table 8: Additional ablation on backbone robustness, predicate complexity, and validator parameter sensitivity.

Method	OOB Scene Ratio (%) $\downarrow$	Floating Object Rate (%) $\downarrow$
ArtiScene-E	88.89	92.06
AnyHome	7.94	57.14
Edit-As-Act (Ours)	6.16	14.21

Appendix

Table 9: Comparison of Geometry-Based Metrics. Lower is better. Edit-As-Act achieves the best physical validity.

Figure 14: Diversity of E2A-Bench Source Scenes. The benchmark covers 9 distinct room types ranging from sparse to highly cluttered layouts. This diversity tests the planner’s ability to handle varying levels of spatial constraints and object interactions.

Figure 17:

Figure 18:

Figure 19:

Figure 20:

Figure 21:

Figure 22:

Figure 23:

Figure 24:

3D Scene Editing

3D scene editing studies how to make controlled changes to objects, appearance, layout, and motion while keeping a scene coherent across views.

See all 88 papers

DesignAgent3D: Interactive 3D Scene Editing via Designer-like Multimodal Reasoning
Aug 18, 2026Xiujin Liu, Tianyu Yang, Yilun Zhao +13D Scene Editing Text-Guided 3D Scene Editing
University of Notre Dame South Bend, Indiana, USA · Yale University New Haven, Connecticut, USA
ReSpace: Text-Driven Autoregressive 3D Indoor Scene Synthesis and Editing
Jun 3, 2025Martin JJ. Bucher, Iro Armeni3D Scene Editing 3D Scene Representation
Stanford University
JointEdit3D: Feed-Forward 3D Scene Editing in a Unified Latent Space
Jun 11, 2026Xinnan Zhu, Ruijie Xu, Jiayu Ying +43D Scene Editing
East China Normal University · Shanghai Artificial Intelligence Laboratory · Fudan University +1

Aug 18, 2026J/K move · Enter open · S save

DesignAgent3D: Interactive 3D Scene Editing via Designer-like Multimodal Reasoning

Xiujin Liu, Tianyu Yang, Yilun Zhao, Xiangliang Zhang

University of Notre Dame South Bend, Indiana, USA · Yale University New Haven, Connecticut, USA

3D Scene Editing Text-Guided 3D Scene Editing Multimodal Agents

Open paper AskPDF arXiv

Text guided 3D scene editing provides an intuitive interface for modifying reconstructed environments, but remains difficult because natural language design requests are often semantically underspecified and must be grounded in cluttered 3D scenes. Existing methods typically formulate the task as one-shot conditional generation from a single prompt, failing to resolve ambiguous user intents or achieve precise spatial grounding. Consequently, they suffer from severe object localization drift, tracking failure under occlusions, and the notorious multi-view "sticker effect." To overcome these limitations, we present DesignAgent3D, an interactive multimodal agentic framework that reformulates 3D scene editing as a designer-like Plan-Perceive-Act paradigm. The agent first plans by interacting with the user to clarify underspecified design goals, then perceives by grounding the intended edit to specific objects or regions in the 3D scene, and finally acts by applying controlled visual modifications while preserving scene consistency. The edits are further integrated into the underlying 3D representation, supporting persistent and multi-view consistent novel-view rendering. Extensive experiments across both NeRF and 3D Gaussian Splatting backbones demonstrate that DesignAgent3D significantly outperforms state-of-the-art baselines, delivering superior semantic intent alignment, impeccable spatial localization accuracy, and high-fidelity multi-view consistency.

Abstract

3D Scene Editing

Explore similar work

DesignAgent3D: Interactive 3D Scene Editing via Designer-like Multimodal Reasoning

ReSpace: Text-Driven Autoregressive 3D Indoor Scene Synthesis and Editing

JointEdit3D: Feed-Forward 3D Scene Editing in a Unified Latent Space