Emerging Extrinsic Dexterity in Cluttered Scenes via Dynamics-aware Policy Learning

Yixin Zheng; Jiangran Lyu; Yifan Zhang; Jiayi Chen; Mi Yan; Yuntian Deng; Xuesong Shi; Xiaoguang Zhao

cs.ROMar 10, 2026

Emerging Extrinsic Dexterity in Cluttered Scenes via Dynamics-aware Policy Learning

Authors: Yixin Zheng, Jiangran Lyu, Yifan Zhang, Jiayi Chen, Mi Yan, Yuntian Deng, Xuesong Shi, Xiaoguang Zhao,

Organizations: Institute of Automation, Chinese Academy of Sciences · Beijing Academy of Artificial Intelligence · Galbot · Peking University ·

Abstract

Extrinsic dexterity leverages environmental contact to overcome the limitations of prehensile manipulation. However, achieving such dexterity in cluttered scenes remains challenging and underexplored, as it requires selectively exploiting contact among multiple interacting objects with inherently coupled dynamics. Existing approaches lack explicit modeling of such complex dynamics and therefore fall short in non-prehensile manipulation in cluttered environments, which in turn limits their practical applicability in real-world environments. In this paper, we introduce a Dynamics-Aware Policy Learning (DAPL) framework that can facilitate policy learning with a learned representation of contact-induced object dynamics in cluttered environments. This representation is learned through explicit world modeling and used to condition reinforcement learning, enabling extrinsic dexterity to emerge without hand-crafted contact heuristics or complex reward shaping. We evaluate our approach in both simulation and the real world. Our method outperforms prehensile manipulation, human teleoperation, and prior representation-based policies by over 25% in success rate on unseen simulated cluttered scenes with varying densities. The real-world success rate reaches around 50% across 10 cluttered scenes, while a practical grocery deployment further demonstrates robust sim-to-real transfer and applicability.

Figures & tables

Fig. 2: Overview of the proposed two-stage learning framework. Stage 1 (World Model Pretraining): The model takes a cropped point cloud augmented with physical attributes (mass and velocity) as input. A Transformer-based architecture encodes these inputs into dynamics features ( $f_{dy}$ ), which are used by an MLP decoder to predict future per-point positions and velocities conditioned on robot actions. Stage 2 (Policy Learning): The pre-trained dynamics representations ( $f_{d}y$ ) are fed into an Actor-Critic policy network alongside proprioceptive data and task goals to facilitate efficient policy learning within a physical simulator.

Fig. 3: Overview of our proposed Clutter6D Benchmark and real-world setup. (a) Representative examples of the Clutter6D Benchmark with three levels of scene density: Sparse (4 objects), Moderate (8 objects), and Dense (12 objects). (b) The real-world experimental setup consisting of a Franka Research 3 robot and three Intel RealSense cameras for point cloud acquisition, along with the complete set of objects used in our real-world experiments.

Methods	Action Type	Sparse		Moderate		Dense
Methods	Action Type	S.R. $\uparrow$	M.O. $\downarrow$	S.R. $\uparrow$	M.O. $\downarrow$	S.R. $\uparrow$	M.O. $\downarrow$
Teleoperation [ 38 ]	Mixed	50.0	3.13	40.0	7.49	20.0	21.34
GraspGen + CuRobo [ 28 , 34 ]	Prehensile	26.6	–	15.6	–	3.13	–
Point2Vec [ 22 ]	Non-prehensile	6.89	5.09	1.95	3.36	0.78	5.35
Concerto [ 42 ]	Non-prehensile	3.13	1.65	1.56	2.90	0.39	7.56
CORN [ 8 ]	Non-prehensile	46.63	3.15	45.83	5.51	22.22	17.43

TABLE I: Quantitative results measured by success rate in the simulation benchmark. Note that Mean Offset (M.O.) for GraspGen + CuRobo is omitted because it relies on the CuRobo motion planner, which strictly prioritizes collision-free trajectories. Our method significantly outperforms all baselines across all clutter densities, maintaining high success rates even in dense environments where baselines experience a sharp performance decline.

Fig. 4: Training efficiency and convergence comparison. Our method (blue) achieves significantly higher sample efficiency compared to geometry-based baselines, reaching about 70% success rate within the initial

10^{4}

iterations. This demonstrates that our dynamics-aware representation provides a robust physical prior, facilitating the rapid acquisition of complex interaction strategies without the need for exhaustive sampling of contact physics.

Fig. 5: Compared to static geometric representation baseline CORN, which either triggers excessive contact chains leading to significant environment disturbance (middle left) or becomes ”stuck” due to a lack of dynamics reasoning (middle right), our policy (bottom row) exhibits selective and intentional interactions, successfully reorients the target while preserving scene stability.

Pretrain Task	Granularity	Velocity	Phys.	P.E. $\downarrow$	S.R. $\uparrow$	M.O. $\downarrow$
Recons.	Point-level	✗	✗	-	11.75	1.31
Recons.	Point-level	✓	✓	-	29.63	2.63
World Model	Object-level	✗	✗	3.1	14.13	3.27
World Model	Object-level	✓	✓	3.2	16.88	3.84
World Model	Point-level	✗	✗	4.1	42.00	4.91
World Model	Point-level	✓	✗	5.1	58.25	4.86

TABLE II: Ablation Study on Pre-training Objectives and Input Modalities (Sparse Track). P.E.: Chamfer Distance prediction error (cm). Results highlight the indispensability of velocity and physical features, while further establishing that point-level dynamics modeling is a more effective pretext task than either object-level relative pose prediction or simple reconstruction.

Fig. 6: Visualization of our world model predictions.

Fig. 7: Success rate over curriculum iterations. Performance steadily improves from 61.3% to 71.8%, demonstrating that iterative model refinement provides more accurate dynamics conditioning for learning effective manipulation strategies.

Method	Unnec. Contacts	Object Disturbance				Dexterous Behaviors (%)
Method	Unnec. Contacts	Small Trans.	Small Rot.	Large Trans.	Large Rot.	Avoid	Traverse	Leverage	Simple
CORN	3.6	2.8	2.6	4.4	4.8	22	14	6	62
Ours	2.4	3.2	3.0	1.8	1.2	52	38	42	28

TABLE III: Behavior analysis over 50 sampled trajectories per method. Unnec. Contacts and Object Disturbance metrics are reported as average counts per trajectory. Small/Large Trans. count non-target objects with translations below/above 5 cm, while Small/Large Rot. count non-target objects with rotations below/above

45^{\circ}

. Avoid, Traverse, and Leverage correspond to the dexterous behavior types illustrated in Fig. . Multiple behavior types may occur in one trajectory; Simple denotes trajectories in which none of the three is observed.

Noise Source	0.5%		5%		25%		50%
Noise Source	S.R.	M.O.	S.R.	M.O.	S.R.	M.O.	S.R.	M.O.
Mass	66.9	3.1	67.3	3.4	66.0	4.1	62.4	5.9
Velocity	64.0	3.6	64.0	3.2	61.1	3.9	58.5	4.4
Goal Pose	66.3	3.1	65.0	3.3	48.1	3.5	15.1	4.0
All Inputs	65.7	3.7	66.6	4.9	45.1	6.0	5.9	8.4

TABLE IV: Sensitivity to noisy physical and goal inputs under controlled simulation perturbations. Noise levels are reported as percentages of the corresponding input mean. S.R. denotes success rate and M.O. denotes mean offset (cm).

Fig. 8: Adaptive behavior guided by physical priors. In identical scenes, our policy autonomously adapts its interaction trajectory based on whether the pie or the Pringles can (pink/cyan mask) is assigned a heavy mass, transitioning between using objects as stable anchors versus avoiding them to prevent destabilization

Methods	S1		S2		S3		S4
	S.R. $\uparrow$	M.T. $\downarrow$	S.R. $\uparrow$	M.T. $\downarrow$	S.R. $\uparrow$	M.T. $\downarrow$	S.R. $\uparrow$	M.T. $\downarrow$
Teleop. [ 25 ]	3/5	77	4/5	59	3/5	39	1/5	81
Ours	2/5	51	1/5	26	3/5	33	1/5	73
Methods	S5		S6		S7		S8
	S.R. $\uparrow$	M.T. $\downarrow$	S.R. $\uparrow$	M.T. $\downarrow$	S.R. $\uparrow$	M.T. $\downarrow$	S.R. $\uparrow$	M.T. $\downarrow$
Teleop. [ 25 ]	3/5	28	2/5	39	4/5	20	0/5	90

TABLE V: Quantitative results across 10 scenes. Mean (last column) denotes average performance across all scenes. SR

\uparrow

indicates success rate and MT

\downarrow

indicates mean execution time.

Fig. 9: Qualitative Results in the real world. The goal pose is shown transparently.

Fig. 10: Humanoid grocery retrieval using extrinsic dexterity. Our policy enables the Galbot G1 to manipulate objects in cluttered shelves by sliding and reorienting them into graspable configurations

State Component	Symbol	Dimension	Description
Object point cloud	$P_{o}$	$\mathbb{R}^{512\times 7}$	Surface-sampled object point cloud in the environment frame, with per-point mass and contact-conditioned velocity $(x,y,z,m,v_{x},v_{y},v_{z})$
Environment point cloud	$P_{e}$	$\mathbb{R}^{512\times 7}$	Surface-sampled obstacle point cloud in the environment frame, with per-point mass and velocity $(x,y,z,m,v_{x},v_{y},v_{z})$
End-effector point cloud	$P_{ee}$	$\mathbb{R}^{256\times 7}$	Surface-sampled end-effector point cloud in the environment frame, with per-point mass and velocity $(x,y,z,m,v_{x},v_{y},v_{z})$
Hand state	$s_{t}^{EE}$	$\mathbb{R}^{9}$	End-effector position and orientation in 6D rotation representation (3D position + 6D orientation)
Robot state	$s_{t}^{q}$	$\mathbb{R}^{14}$	Joint positions and velocities of the 7-DoF arm
Relative goal pose	$T_{g}$	$\mathbb{R}^{9}$	Target object pose expressed relative to the current object pose (3D translation + 6D orientation)

TABLE I: Summary of our MDP components.

\dagger

: simulation only.

Parameter	Value
Value loss coefficient	0.5
Use clipped value loss	True
Clip parameter ( $\epsilon$ )	0.3
Entropy coefficient	0.006
Learning epochs	8
Mini-batches	8

TABLE II: PPO Algorithm Hyperparameters

Reward term	Weight
$r_{\text{contact}}$	1.0
$r_{\text{goal}}$	5.0
$r_{\text{goal-fine}}$	16.0
$r_{\text{success}}$	2000.0
$\sigma_{\text{coarse}}$	0.6
$\sigma_{\text{fine}}$	0.3

TABLE III: Reward terms and weights used for RL training.

Fig. 1: Visualizations of representative scenarios within the Sparse clutter level.

Fig. 2: Visualizations of representative scenarios within the Moderate clutter level.

Fig. 3: Visualizations of representative scenarios within the Dense clutter level.

Difficulty	Large Obstacles	Small Obstacles	Target
Sparse	1	2	1
Moderate	3	4	1
Dense	5	6	1

TABLE IV: Difficulty levels and procedural object counts.

Fig. 4: Adaptive Interaction under Mass Perturbations. The policy reconfigures trajectories by treating heavy objects as stable anchors. Case A: Utilizing the heavy pie for support. Case B: Leveraging the heavy can as a mechanical pivot after mass swapping.

Fig. 5: Disturbance Avoidance: The policy bypasses lightweight objects to maintain scene stability.

Fig. 6: Obstacle Traversal: The policy manipulates the target to traverse physical barriers (e.g., sink dividers).

Fig. 7: Leveraging Contact: The policy utilizes stable environmental objects as anchors to facilitate reorientation.

Non-Prehensile Robotic Manipulation

See all 16 papers

RetrDex: Efficient Object Retrieval in Cluttered Scenes with a Dexterous Hand
Feb 25, 2025Fengshuo Bai, Yu Li, Jie Chu +5RL for Robotics Robotic Manipulation
Shanghai Jiao Tong University · PKU-PsiBot Joint Lab · Peking University +2
Concurrent Prehensile and Nonprehensile Manipulation: A Practical Approach to Multi-Stage Dexterous Tasks
Mar 12, 2026Hao Jiang, Yue Wu, Yue Wang +2Dexterous Manipulation Long-Horizon Robotic Manipulation
Thomas Lord Department of Computer Science at the University of Southern California, USA.
AdaClearGrasp: Learning Adaptive Clearing for Zero-Shot Robust Dexterous Grasping in Densely Cluttered Environments
Mar 11, 2026Zixuan Chen, Wenquan Zhang, Jing Fang +7Robotic Grasping Robotics
Nanjing University · Nanjing University of Posts and Telecommunications · National University of Singapore

Feb 25, 2025J/K move · Enter open · S save

RetrDex: Efficient Object Retrieval in Cluttered Scenes with a Dexterous Hand

Fengshuo Bai, Yu Li, Jie Chu, Tawei Chou+4

Shanghai Jiao Tong University · PKU-PsiBot Joint Lab · Peking University+2

RL for Robotics Robotic Manipulation

Open paper AskPDF arXiv

Retrieving objects buried beneath clutter is both challenging and time-consuming, as complex support relationships make manipulation particularly difficult. Existing methods either focus on support relations and rely on sequential grasping to remove occluding objects, or perform preparatory actions such as pushing to facilitate subsequent grasps. However, these approaches are often inefficient and treat physical interactions as isolated auxiliary steps. In this paper, we propose RetrDex, an efficient framework for dexterous arm-hand systems to learn object retrieval in cluttered scenes. Our approach leverages large-scale parallel reinforcement learning (RL) in diverse cluttered scenes and incorporates a spatially aware representation that encodes occlusion patterns and spatial relationships among the target, the dexterous hand, and surrounding clutter. This representation enables the policy to develop diverse manipulation skills (e.g., pushing, stirring, and poking) that actively clear occluders. We evaluate RetrDex on 16 household objects across varied clutter configurations, and obtain strong retrieval performance and efficiency on both seen and unseen targets. Furthermore, we demonstrate successful zero-shot transfer to a real-world dexterous multi-fingered robot system, validating the practical applicability of our method. Videos can be found on our project website: https://RetrDex.github.io.

Abstract

Non-Prehensile Robotic Manipulation

Explore similar work

RetrDex: Efficient Object Retrieval in Cluttered Scenes with a Dexterous Hand

Concurrent Prehensile and Nonprehensile Manipulation: A Practical Approach to Multi-Stage Dexterous Tasks

AdaClearGrasp: Learning Adaptive Clearing for Zero-Shot Robust Dexterous Grasping in Densely Cluttered Environments