Learning Contextual Motion Constraints for Object Rearrangement Exploration
Authors: Lexing Zhang, Zeyu Zhang, Gang Zeng, Hangxin Liu
Organizations: State Key Laboratory of General Artificial Intelligence, School of Intelligence Science and Technology, Peking University. · State Key Laboratory of General Artificial Intelligence, Beijing Institute for General Artificial Intelligence (BIGAI).
The motion of an articulated part in cluttered surroundings is constrained not only by an object's intrinsic joint limits but also by extrinsic obstacles. However, existing approaches primarily consider object articulation properties alone. Here we present Motion Constraint Prediction Network (MoCo-Net), a two-stream neural network that jointly predicts the continuous motion ranges of articulated parts and obstacle heatmaps highlighting potential collision regions from partial-view point cloud observation. By capturing both internal and external motion constraints, MoCo-Net produces more accurate feasible motion range prediction and obstacles recognition, improves generalization for novel objects in new scenes, and enables robots to construct a scene configuration space for efficient exploration of rearrangement actions. Our experiments show that MoCo-Net's two-stream design enhances each other to deliver more accurate predictions, which subsequently form a compact configuration space for a customized planner to efficiently rearrange the scene.
Figures & tables
Fig. 1 : An illustration of the proposed MoCo-Net. Given a partial scene point cloud and a query point for the articulated part of interest, MoCo-Net predicts that part’s feasible motion range and the heatmap highlighting surrounding regions where it may collide. Based on these predictions, an exploration algorithm could further compute rearrangement actions to effectively manipulate the part.
Fig. 2 : Network architecture for feasible range and obstacle heatmap prediction. (a) From the scene, we obtain (b) a partial scene point cloud PS visualized in blue and a query point MT visualized in red to specify the movable part. Our system predicts (c) the feasible range RT and obstacle heatmap HT for the movable part T . The feasible range is visualized through yellow and green points, representing upper and lower bounds respectively. The obstacle heatmap is displayed using a color gradient, where warmer colors indicate higher collision likelihood. In (d), we provide a zoomed view of highlighted areas in the heatmap to illustrate potential collision area.
Fig. 3 : Configuration space exploration for opening the dishwasher’s bottom drawer by 0.2m . The blue volume represents all feasible configurations of the scene (drawer joint position, door joint angle, trash can position simplified as 1D displacement). The blue circle marks the initial configuration and the blue cross marks the current configuration. Orange lines indicate previously explored configurations, while red lines show newly predicted feasible ranges where collision-free movement is possible. The algorithm efficiently expands within this space to reach the target, shown as the pink plane.
Feasible Range Prediction
Obstacle Heatmap Prediction
Method
Range Accuracy ↑
Range Error ↓
Collision Detection Rate ↑
Collision Distance Error ↓
Layout Var.
Unseen Obj.
Layout Var.
Unseen Obj.
Layout Var.
Unseen Obj.
Layout Var.
Unseen Obj.
GT-Sim
0.433
0.445
0.305
0.300
0.511
0.490
0.210
0.196
Backbone-PointNet
0.460
0.402
0.158
0.181
0.685
0.495
0.106
0.129
Backbone-PointNet++
0.545
0.506
0.126
0.138
0.738
0.532
0.133
0.164
FR-Only
0.604
0.566
0.114
0.128
-
-
-
-
TABLE I: Quantitative comparison of feasible range and obstacle heatmap prediction. ↓ indicates lower is better and ↑ indicates higher is better. Layout Var. evaluates performance on scenes with seen objects but different layouts. Unseen Obj. evaluates generalization to objects not present in training. The best results are highlighted in bold.
Fig. 4 : Qualitative results of predicted joint motion ranges and potential collisions . (a) and (c) show the ground-truth feasible ranges and collision heatmaps. (b) presents accurate predictions, while (d) illustrates less accurate and even false cases.
Fig. 5 : Efficiency in computing rearrangement actions through the proposed configuration space exploration . Rearrangement sequences shown in red are generated by the baseline method FR+Greedy , which directly relocates the obstacle closest to the target part. In contrast, sequences shown in blue are produced by the proposed configuration space exploration algorithm, resulting in more efficient rearrangement actions.
Method
ctar ↑
aan ↓
acn ↓
DirectManip
0%
6.055
2.525
FR+Greedy
80.0%
6.185
0.235
FR+OH (Ours)
82.4%
5.472
0.201
TABLE II: Quantitative results of planning. Arrows indicate the direction of improvement. The best results are highlighted in bold.
Object rearrangement planning in cluttered environments is often non-monotone, requiring objects to be temporarily relocated before reaching their final destinations. Although search-based planners can solve such tasks, they incur substantial computational cost and require expensive replanning when the environment changes. We propose the Visual Adaptive Rearrangement Planner (VARP), which learns non-monotone rearrangement directly from visual state representations generated from expert demonstrations. VARP predicts pick-and-place actions from the current and goal visual state representations, and incorporates an Action Refinement Strategy (ARS) to improve action feasibility and a Progress Enforcement Strategy (PES) to mitigate progress stagnation. By continuously updating actions from the latest visual state representations, VARP adapts online to disturbances without replanning from scratch or performing online combinatorial search. In challenging rearrangement benchmarks, VARP achieves a 98.3% success rate with only 0.4 s planning time per task, 97.5% success under execution-time disturbances, and 94.5% under continuous severe disturbances, while generalizing effectively to unseen object counts. These results demonstrate that VARP provides a scalable and robust solution for non-monotone object rearrangement.
This paper introduces Embodied Scene Rearrangement Planning (ESRP), a novel task requiring embodied agents to rearrange furniture in 3D scenes to match a target configuration using only egocentric observations and a top-down target layout. Unlike prior rearrangement tasks, ESRP precludes global state access and introduces mutual object occlusions, reflecting the practical constraints of real-world robotic deployment. These factors make aligning partial egocentric observations with the global target layout particularly challenging for long-horizon planning. To facilitate research, we present ESRP-Bench, a comprehensive benchmark built on OmniGibson featuring over 5,400 scene pairs and 8,200 objects. We define three multi-level metrics to evaluate rearrangement quality and provide four baselines: a hierarchical task-and-motion planning method, a vision-language-model-based method, and two learning-based approaches (IL and RL). Experimental results demonstrate that current methods struggle to complete the task efficiently, highlighting ESRP as a challenging frontier for embodied agents in scene understanding and long-horizon task planning. This work serves as a stepping stone toward deploying intelligent agents in real-world scenarios. Project page: https://pie-lab.cn/ESRP/.
Canzhi Chen, Zan Wang, Siqi Zhu +3
School of Computer Science and Technology, Beijing Institute of Technology, Beijing 100081, China
Motion feasibility prediction plays a central role in robotics, particularly in task and motion planning and manipulation. A major bottleneck for this problem in cluttered environments is that infeasible planning attempts by Sampling-based motion planners (SBMPs) can incur substantial computational cost. Also existing approaches for infeasibility certification are limited to low-dimensional configuration spaces and often assume simplified geometric environments represented by primitive objects with known parameters. We study the complementary problem of learning motion feasibility prediction directly from raw RGB-D observations for a 7-DOF manipulator operating in realistic cluttered scenes. We introduce the first large-scale benchmark for this setting, comprising 2.7M grasp feasibility labels over 88 scanned objects and 190 cluttered tabletop scenes. We benchmark three representative classifier families spanning MLP- based, volumetric-CNN, and point-cloud-based Transformer architectures under matched training conditions. Our best model, GRASPFC-PTX (a point-cloud transformer), achieves an AUROC of 0.996 on Novel objects while providing predictions significantly faster than SBMPs.
Sajid Ansari, Arthi, Girish Varma +1
International Institute of Information Technology Hyderabad Hyderabad 500032, India.