In densely cluttered environments, physical interference, visual occlusions, and unstable contacts often cause direct dexterous grasping to fail, while aggressive singulation strategies may compromise safety. Enabling robots to adaptively decide whether to clear surrounding objects or directly grasp the target is therefore crucial for robust manipulation. We propose AdaClearGrasp, a closed-loop decision-execution framework for adaptive clearing and zero-shot dexterous grasping in densely cluttered environments. The framework formulates manipulation as a controllable high-level decision process that determines whether to directly grasp the target or first clear surrounding objects. A pretrained vision-language model (VLM) interprets visual observations and language task descriptions to reason about grasp interference and generate a high-level planning skeleton, which invokes structured atomic skills through a unified action interface. For dexterous grasping, we train a reinforcement learning policy with a relative hand-object distance representation, enabling zero-shot generalization across diverse object geometries and physical properties. During execution, visual feedback monitors outcomes and triggers replanning upon failures, forming a closed-loop correction mechanism. To evaluate language-conditioned dexterous grasping in clutter, we introduce Clutter-Bench, the first simulation benchmark with graded clutter complexity. It includes seven target objects across three clutter levels, yielding 210 task scenarios. We further perform sim-to-real experiments on three objects under three clutter levels (18 scenarios). Results demonstrate that AdaClearGrasp significantly improves grasp success rates in densely cluttered environments. For more videos and code, please visit our project website: https://chenzixuan99.github.io/adaclear-grasp.github.io/.
Figures & tables
Fig. 1: We propose AdaClearGrasp , a closed-loop system that leverages VLM-based reasoning to perform adaptive clearing for robust dexterous grasping of target objects in densely cluttered scenes.
Fig. 2: Overview of AdaClearGrasp. The system adopts a hierarchical architecture in which a VLM-based Semantic Planner (left) reasons about the scene and generates high-level action plans. These plans are translated into atomic skills via the Model Context Protocol (MCP) Service (middle). The Skill Library contains heuristic clearing primitives, recovery mechanisms, and a grasping skill (the pretrained geometry-aware RL policy GeoGrasp). The system operates in a closed loop (right), continuously monitoring execution feedback to trigger replanning or recovery when necessary.
Fig. 3: A visualization example of each target object in Clutter-Bench and its corresponding three difficulty levels.
Fig. 4: Real-world Implementation Details. (Upper) Real-world experimental setup with xArm7, XHand, and the external camera Gemini 336L. (Lower) Visualization of 6D pose estimation using FoundationPose, which provides input to the VLM planner.
Fig. 5: Real-world experiments on 18 diverse scenarios.
Target Object
Level-1
Level-2
Level-3
VLM Scaffolding
AdaClearGrasp
VLM Scaffolding
AdaClearGrasp
VLM Scaffolding
AdaClearGrasp
Cube
0.2
0.8
0.0
0.9
0.0
1.0
Can
0.1
0.9
0.0
1.0
0.0
0.9
Pear
0.0
1.0
0.0
1.0
0.0
0.7
Apple
0.1
1.0
0.0
0.8
0.0
0.9
Mug
0.0
0.8
0.0
0.8
0.0
0.9
TABLE I: Success Rates Comparison on Clutter-Bench . We evaluate the zero-shot performance of AdaClearGrasp against the VLM Scaffolding baseline across three increasing difficulty tiers (Level-1 to Level-3). The results are averaged over 10 independent trials for each target object and clutter configuration.
Fig. 6: Ablation study on adaptive clearing and closed-loop feedback.
Fig. 7: The grasp success rates of our GeoGrasp policy on 7 object categories (3 seen, 4 unseen) in clutter-free environments.
Target Object
Clutter Level (Obstacles)
Avg.
2
4
6
Apple
9/10
7/10
5/10
70%
Orange Cube
10/10
8/10
6/10
80%
Cup
8/10
6/10
4/10
60%
Total Success
27/30
21/30
15/30
70%
Success Rate
90%
70%
50%
TABLE II: Real-world Sim-to-Real Transfer Results.
Parameter
Value
Optimizer
Adam
Learning Rate
3×10−4
Num. Environments
400
Rollout Steps per Env
5
Batch Size
800
Epochs per Rollout
5
TABLE III: PPO Hyperparameters for GeoGrasp Training
Tool Name
Parameters
Description
push
side (left, right, center), dist
Approaches the nearest obstacle relative to the target and pushes it away to clear a path.
pull
side (left, right, center), dist
Hooks the nearest obstacle and pulls it closer to the robot base to remove occlusion.
pick
None
Invokes the GeoGrasp policy to attempt a dexterous grasp on the target object.
move_to
target (string)
Moves the end-effector to a pre-grasp hover position above the specified object.
lift
height (optional)
Lifts the end-effector vertically by a specified distance to avoid collisions during transport.
initarm
None
Resets the arm to a safe home configuration to recover from singularities or execution failures.
TABLE IV: Atomic Skills and their Parameters exposed via MCP
Fig. 8: The structured prompt template used for VLM planning in AdaClearGrasp. It integrates system instructions, task context, and dynamic feedback to guide the VLM’s reasoning.
Target Name
YCB Model ID
Typical Clutter Objects
Apple
013_apple
Orange, Foam Brick, Mug, Pear, Ball, Cube
Cube
077_rubiks_cube
Can, Foam Brick, Mug, Apple, Ball, Orange
Can
010_potted_meat_can
Foam Brick, Mug, Apple, Ball, Orange, Pear
Pear
016_pear
Can, Foam Brick, Mug, Apple, Ball, Cube
Mug
025_mug
Can, Foam Brick, Apple, Ball, Orange, Cube
Lego
073-e_lego_duplo
Fish Can, Foam Brick, Mug, Apple, Ball, Orange
TABLE V: Target Objects and Clutter Pools in Clutter-Bench .
Dexterous grasping in cluttered scenes remains a central challenge in robotic manipulation, as it requires high-dimensional hand coordination under complex object-environment constraints. To address this problem, we introduce one of the largest simulation benchmarks to date, comprising around 0.5B grasp annotations across eight dexterous hands and approximately 50K objects. Furthermore, we propose T(R,O) Grasp++, a scalable diffusion-based framework for cross-embodiment dexterous grasp synthesis in cluttered environments. Extensive experiments demonstrate that our model significantly outperforms existing baselines, achieving an average success rate of 88% across eight dexterous embodiments. Meanwhile, our pipeline achieves only 0.3s inference latency while improving inference sample efficiency by over 10 times.
Xin Fei, Zhixuan Xu, Huaicong Fang +5
Department of Computer Science, National University of Singapore · RoboScience · College of Control Science and Engineering, Zhejiang University, China +2
Retrieving objects buried beneath clutter is both challenging and time-consuming, as complex support relationships make manipulation particularly difficult. Existing methods either focus on support relations and rely on sequential grasping to remove occluding objects, or perform preparatory actions such as pushing to facilitate subsequent grasps. However, these approaches are often inefficient and treat physical interactions as isolated auxiliary steps. In this paper, we propose RetrDex, an efficient framework for dexterous arm-hand systems to learn object retrieval in cluttered scenes. Our approach leverages large-scale parallel reinforcement learning (RL) in diverse cluttered scenes and incorporates a spatially aware representation that encodes occlusion patterns and spatial relationships among the target, the dexterous hand, and surrounding clutter. This representation enables the policy to develop diverse manipulation skills (e.g., pushing, stirring, and poking) that actively clear occluders. We evaluate RetrDex on 16 household objects across varied clutter configurations, and obtain strong retrieval performance and efficiency on both seen and unseen targets. Furthermore, we demonstrate successful zero-shot transfer to a real-world dexterous multi-fingered robot system, validating the practical applicability of our method. Videos can be found on our project website: https://RetrDex.github.io.
Fengshuo Bai, Yu Li, Jie Chu +5
Shanghai Jiao Tong University · PKU-PsiBot Joint Lab · Peking University +2
Robots that follow natural-language instructions unlock a wide range of autonomous applications in households, warehouses, and labs. Real-world environments are diverse and often heavily cluttered, requiring robots to plan over multiple steps, act, and replan as they work toward a goal. We present CLARO (CLutter-Aware Robot Orchestrator), a VLM agent that orchestrates a set of perception and manipulation skills: open-vocabulary object detection, segmentation, 6-DoF grasping, and non-prehensile pushing to locate and retrieve a language-specified target in clutter. CLARO maintains a memory of its past actions to support long-horizon execution, and emits manipulation targets (grasp poses/3D waypoints) allowing transfer across robot arms without task-specific training. We evaluate CLARO against a range of methods on a dynamic simulation environment and standard image-based benchmarks, demonstrating strong generalization across object classes, lighting, and clutter levels. CLARO attains an average success rate of 89.1%, a relative improvement of over 16% on the strongest competing method in high cluttered scenes. We further analyze performance along grounding and grasping axes to characterize where each method succeeds and fails. Finally, we deploy CLARO on a robot arm across diverse cluttered scenes, achieving a success rate of 88.8%, and demonstrating strong sim2real generalizability.
Sungsu Kim, Vineet Bhat, Prashanth Krishnamurthy +2