This paper introduces Embodied Scene Rearrangement Planning (ESRP), a novel task requiring embodied agents to rearrange furniture in 3D scenes to match a target configuration using only egocentric observations and a top-down target layout. Unlike prior rearrangement tasks, ESRP precludes global state access and introduces mutual object occlusions, reflecting the practical constraints of real-world robotic deployment. These factors make aligning partial egocentric observations with the global target layout particularly challenging for long-horizon planning. To facilitate research, we present ESRP-Bench, a comprehensive benchmark built on OmniGibson featuring over 5,400 scene pairs and 8,200 objects. We define three multi-level metrics to evaluate rearrangement quality and provide four baselines: a hierarchical task-and-motion planning method, a vision-language-model-based method, and two learning-based approaches (IL and RL). Experimental results demonstrate that current methods struggle to complete the task efficiently, highlighting ESRP as a challenging frontier for embodied agents in scene understanding and long-horizon task planning. This work serves as a stepping stone toward deploying intelligent agents in real-world scenarios. Project page: https://pie-lab.cn/ESRP/.
Figures & tables
Fig. 1 : Overview of the esrp task. Given a top-down target layout, an embodied agent needs to rearrange furniture in a 3 D scene from an initial configuration (red) to a target state (green). The agent operates under realistic constraints, relying only on egocentric observations without access to global state information or ground-truth localization.
Fig. 2 : Initial-target scene layout pairs. We group scene layouts into three difficulty levels (as divided by green lines), with the first row showing the initial layout and the second row displaying the corresponding target layout. The top left shows easy-level scenes, where only 1 furniture object needs to be rearranged; the top right presents medium-level scenes with 2∼3 objects to be rearranged; and the bottom shows hard-level scenes with 4∼6 objects to be rearranged. The dataset covers a wide range of room types, including bedrooms, study rooms, and living rooms, along with diverse furniture types, such as chairs, tables, and plants.
Fig. 3 : Dataset distribution. (a) The scene clutter level in the dataset is defined as the sum of L 2 distances between the initial and target positions of all rearrangeable objects. Formally, more challenging scenes have higher clutter levels. (b) Distribution of rearrangeable objects across different room types in the dataset. We report only the top six room categories in the dataset, excluding those that constitute only a small fraction. Notably, these top six categories account for over 80% of all rooms.
Fig. 4 : Distribution of room types. The distribution of room categories is visualized on a logarithmic scale to account for class imbalance. Our dataset encompasses a diverse array of room types, comprehensively covering common household environments.
Fig. 5 : Pipeline overview. ( Left ) ESRP-Bench consists of the dataset and the esrp environment. ( Middle ) Four baseline models are: (1) il agent uses expert trajectories from the dataset to train a policy that imitates expert behavior; (2) rl agent learns a policy by interacting with the esrp environment; (3) foundation-model-based VLM agent prompts a pretrained vlm in a ReAct loop without task-specific training; and (4) planning-based agent utilizes global information to perform hierarchical task and motion planning. ( Right ) The model architectures for rl are shown, consisting of a CNN for visual perception and an LSTM module for memory. The gray box represents the embedding of the grasping state indicator g .
Fig. 6 : Examples of successfully rearranged scenes. The agent successfully moves the objects to match the target layout. The red curve represents the agent’s trajectory, while the blue curve depicts the object’s trajectory.
Fig. 7 : Failure cases. As the number of rearrangeable objects increases, the task becomes more challenging due to its long-horizon complexity. In such cases, the agent rearranges the scene to approximate the target layout.
Baseline
SR ↑
OSR ↑
RDR ↓
ESRP-PPO
20.18%
20.35%
0.9845
ESRP-BC
19.09%
20.31%
1.0170
ESRP-VLM
17.64%
18.97%
0.9830
ESRP-PLAN
30.20%
36.59%
0.6916
TABLE I : Performance of the baseline methods on the test set.
Baseline
Easy
Medium
Hard
SR ↑
OSR ↑
RDR ↓
SR ↑
OSR ↑
RDR ↓
SR ↑
OSR ↑
RDR ↓
ESRP-PPO
33.33%
33.33%
0.9040
0.54%
0.54%
1.0645
0.00%
0.00%
1.0048
ESRP-BC
29.44%
29.44%
0.9631
0.91%
4.20%
1.1157
0.00%
4.44%
1.0704
ESRP-VLM
27.64%
27.64%
0.9523
0.00%
3.99%
1.0372
0.00%
0.00%
1.0378
ESRP-PLAN
35.73%
35.73%
0.7101
22.40%
39.71%
0.6425
0.00%
22.00%
0.8165
TABLE II : The performance of baseline models across scenes of varying difficulty levels.
Baseline
Stage 1 (%)
Stage 2 (%)
Stage 3 (%)
Stage 4 (%) ↑
ESRP-PPO
13.21
5.76
60.85
20.18
ESRP-BC
11.33
6.73
62.85
19.09
ESRP-VLM
11.64
4.18
66.55
17.64
TABLE III : Failure-mode decomposition. Each test rollout is assigned to exactly one of four mutually exclusive stages, defined by how far the agent progresses through the rearrangement pipeline. Stage 1: the agent never approaches a rearrangeable object ( navigation failure ). Stage 2: the agent reaches an object but never grasps it ( engagement failure ). Stage 3: the agent grasps an object but fails to place it at the goal ( placement failure ). Stage 4: complete success. ESRP-VLM results are based on a single run.