This paper introduces Embodied Scene Rearrangement Planning (ESRP), a novel task requiring embodied agents to rearrange furniture in 3D scenes to match a target configuration using only egocentric observations and a top-down target layout. Unlike prior rearrangement tasks, ESRP precludes global state access and introduces mutual object occlusions, reflecting the practical constraints of real-world robotic deployment. These factors make aligning partial egocentric observations with the global target layout particularly challenging for long-horizon planning. To facilitate research, we present ESRP-Bench, a comprehensive benchmark built on OmniGibson featuring over 5,400 scene pairs and 8,200 objects. We define three multi-level metrics to evaluate rearrangement quality and provide four baselines: a hierarchical task-and-motion planning method, a vision-language-model-based method, and two learning-based approaches (IL and RL). Experimental results demonstrate that current methods struggle to complete the task efficiently, highlighting ESRP as a challenging frontier for embodied agents in scene understanding and long-horizon task planning. This work serves as a stepping stone toward deploying intelligent agents in real-world scenarios. Project page: https://pie-lab.cn/ESRP/.
Figures & tables
Fig. 1 : Overview of the esrp task. Given a top-down target layout, an embodied agent needs to rearrange furniture in a 3 D scene from an initial configuration (red) to a target state (green). The agent operates under realistic constraints, relying only on egocentric observations without access to global state information or ground-truth localization.
Fig. 2 : Initial-target scene layout pairs. We group scene layouts into three difficulty levels (as divided by green lines), with the first row showing the initial layout and the second row displaying the corresponding target layout. The top left shows easy-level scenes, where only 1 furniture object needs to be rearranged; the top right presents medium-level scenes with 2∼3 objects to be rearranged; and the bottom shows hard-level scenes with 4∼6 objects to be rearranged. The dataset covers a wide range of room types, including bedrooms, study rooms, and living rooms, along with diverse furniture types, such as chairs, tables, and plants.
Fig. 3 : Dataset distribution. (a) The scene clutter level in the dataset is defined as the sum of L 2 distances between the initial and target positions of all rearrangeable objects. Formally, more challenging scenes have higher clutter levels. (b) Distribution of rearrangeable objects across different room types in the dataset. We report only the top six room categories in the dataset, excluding those that constitute only a small fraction. Notably, these top six categories account for over 80% of all rooms.
Fig. 4 : Distribution of room types. The distribution of room categories is visualized on a logarithmic scale to account for class imbalance. Our dataset encompasses a diverse array of room types, comprehensively covering common household environments.
Fig. 5 : Pipeline overview. ( Left ) ESRP-Bench consists of the dataset and the esrp environment. ( Middle ) Four baseline models are: (1) il agent uses expert trajectories from the dataset to train a policy that imitates expert behavior; (2) rl agent learns a policy by interacting with the esrp environment; (3) foundation-model-based VLM agent prompts a pretrained vlm in a ReAct loop without task-specific training; and (4) planning-based agent utilizes global information to perform hierarchical task and motion planning. ( Right ) The model architectures for rl are shown, consisting of a CNN for visual perception and an LSTM module for memory. The gray box represents the embedding of the grasping state indicator g .
Fig. 6 : Examples of successfully rearranged scenes. The agent successfully moves the objects to match the target layout. The red curve represents the agent’s trajectory, while the blue curve depicts the object’s trajectory.
Fig. 7 : Failure cases. As the number of rearrangeable objects increases, the task becomes more challenging due to its long-horizon complexity. In such cases, the agent rearranges the scene to approximate the target layout.
Baseline
SR ↑
OSR ↑
RDR ↓
ESRP-PPO
20.18%
20.35%
0.9845
ESRP-BC
19.09%
20.31%
1.0170
ESRP-VLM
17.64%
18.97%
0.9830
ESRP-PLAN
30.20%
36.59%
0.6916
TABLE I : Performance of the baseline methods on the test set.
Baseline
Easy
Medium
Hard
SR ↑
OSR ↑
RDR ↓
SR ↑
OSR ↑
RDR ↓
SR ↑
OSR ↑
RDR ↓
ESRP-PPO
33.33%
33.33%
0.9040
0.54%
0.54%
1.0645
0.00%
0.00%
1.0048
ESRP-BC
29.44%
29.44%
0.9631
0.91%
4.20%
1.1157
0.00%
4.44%
1.0704
ESRP-VLM
27.64%
27.64%
0.9523
0.00%
3.99%
1.0372
0.00%
0.00%
1.0378
ESRP-PLAN
35.73%
35.73%
0.7101
22.40%
39.71%
0.6425
0.00%
22.00%
0.8165
TABLE II : The performance of baseline models across scenes of varying difficulty levels.
Baseline
Stage 1 (%)
Stage 2 (%)
Stage 3 (%)
Stage 4 (%) ↑
ESRP-PPO
13.21
5.76
60.85
20.18
ESRP-BC
11.33
6.73
62.85
19.09
ESRP-VLM
11.64
4.18
66.55
17.64
TABLE III : Failure-mode decomposition. Each test rollout is assigned to exactly one of four mutually exclusive stages, defined by how far the agent progresses through the rearrangement pipeline. Stage 1: the agent never approaches a rearrangeable object ( navigation failure ). Stage 2: the agent reaches an object but never grasps it ( engagement failure ). Stage 3: the agent grasps an object but fails to place it at the goal ( placement failure ). Stage 4: complete success. ESRP-VLM results are based on a single run.
Real-world robotic tasks are long-horizon and often span multiple floors, demanding rich spatial reasoning. However, existing embodied benchmarks are largely confined to single-floor in-house environments, failing to reflect the complexity of real-world tasks. We introduce MANSION, the first language-driven framework for generating building-scale, multi-floor 3D environments. Being aware of vertical structural constraints, MANSION generates realistic, navigable whole-building structures with diverse, human-friendly scenes, enabling the development and evaluation of cross-floor long-horizon tasks. Building on this framework, we release MansionWorld, a dataset of over 1,000 diverse buildings ranging from hospitals to offices, alongside a Task-Semantic Scene Editing Agent that customizes these environments using open-vocabulary commands to meet specific user needs. Benchmarking reveals that state-of-the-art agents degrade sharply in our settings, establishing MANSION as a critical testbed for the next generation of spatial reasoning and planning.
Lirong Che, Shuo Wen, Shan Huang +5
Tsinghua University · AgiBot · McGill University, MILA - Quebec AI Institute
Recent advances in vision language models (VLMs) have shown strong potential for spatial reasoning and 3D scene layout generation from open-ended language instructions. However, generating layouts that are not only semantically coherent but also feasible for interaction by embodied agents remains challenging, particularly in physically constrained indoor environments. In this paper, RoboLayout is introduced as an extension of LayoutVLM that augments the original framework with agent-aware reasoning and improved optimization stability. RoboLayout integrates explicit reachability constraints into a differentiable layout optimization process, enabling the generation of layouts that are navigable and actionable by embodied agents. Importantly, the agent abstraction is not limited to a specific robot platform and can represent diverse entities with distinct physical capabilities, such as service robots, warehouse robots, humans of different age groups, or animals, allowing environment design to be tailored to the intended agent. In addition, a local refinement stage is proposed that selectively reoptimizes problematic object placements while keeping the remainder of the scene fixed, improving convergence efficiency without increasing global optimization iterations. Overall, RoboLayout preserves the strong semantic alignment and physical plausibility of LayoutVLM while enhancing applicability to agent-centric indoor scene generation, as demonstrated by experimental results across diverse scene configurations.
Embodied instruction following (EIF) requires agents to understand and execute complex natural language commands within interactive 3D environments. Despite recent advances, existing methods often fail in long-horizon planning and handling irreversible state changes, resulting in low task success rates. To address these challenges, we introduce RePlan-Bot, a novel EIF agent that performs multi-level, continuous replanning throughout task execution. RePlan-Bot integrates a high-level LLM-based auditor for dynamic sub-goal adjustments guided by environmental feedback, a commonsense-guided search mechanism based on a multi-layered instance map for precise and structured object localization, and a lightweight ViT-based corrector to preemptively fix risky low-level actions. Evaluated on the ALFRED benchmark, RePlan-Bot achieves state-of-the-art performance in both seen and unseen environments, demonstrating superior adaptability and reliability.