In densely cluttered environments, physical interference, visual occlusions, and unstable contacts often cause direct dexterous grasping to fail, while aggressive singulation strategies may compromise safety. Enabling robots to adaptively decide whether to clear surrounding objects or directly grasp the target is therefore crucial for robust manipulation. We propose AdaClearGrasp, a closed-loop decision-execution framework for adaptive clearing and zero-shot dexterous grasping in densely cluttered environments. The framework formulates manipulation as a controllable high-level decision process that determines whether to directly grasp the target or first clear surrounding objects. A pretrained vision-language model (VLM) interprets visual observations and language task descriptions to reason about grasp interference and generate a high-level planning skeleton, which invokes structured atomic skills through a unified action interface. For dexterous grasping, we train a reinforcement learning policy with a relative hand-object distance representation, enabling zero-shot generalization across diverse object geometries and physical properties. During execution, visual feedback monitors outcomes and triggers replanning upon failures, forming a closed-loop correction mechanism. To evaluate language-conditioned dexterous grasping in clutter, we introduce Clutter-Bench, the first simulation benchmark with graded clutter complexity. It includes seven target objects across three clutter levels, yielding 210 task scenarios. We further perform sim-to-real experiments on three objects under three clutter levels (18 scenarios). Results demonstrate that AdaClearGrasp significantly improves grasp success rates in densely cluttered environments. For more videos and code, please visit our project website: https://chenzixuan99.github.io/adaclear-grasp.github.io/.
Figures & tables
Fig. 1: We propose AdaClearGrasp , a closed-loop system that leverages VLM-based reasoning to perform adaptive clearing for robust dexterous grasping of target objects in densely cluttered scenes.
Fig. 2: Overview of AdaClearGrasp. The system adopts a hierarchical architecture in which a VLM-based Semantic Planner (left) reasons about the scene and generates high-level action plans. These plans are translated into atomic skills via the Model Context Protocol (MCP) Service (middle). The Skill Library contains heuristic clearing primitives, recovery mechanisms, and a grasping skill (the pretrained geometry-aware RL policy GeoGrasp). The system operates in a closed loop (right), continuously monitoring execution feedback to trigger replanning or recovery when necessary.
Fig. 3: A visualization example of each target object in Clutter-Bench and its corresponding three difficulty levels.
Fig. 4: Real-world Implementation Details. (Upper) Real-world experimental setup with xArm7, XHand, and the external camera Gemini 336L. (Lower) Visualization of 6D pose estimation using FoundationPose, which provides input to the VLM planner.
Fig. 5: Real-world experiments on 18 diverse scenarios.
Target Object
Level-1
Level-2
Level-3
VLM Scaffolding
AdaClearGrasp
VLM Scaffolding
AdaClearGrasp
VLM Scaffolding
AdaClearGrasp
Cube
0.2
0.8
0.0
0.9
0.0
1.0
Can
0.1
0.9
0.0
1.0
0.0
0.9
Pear
0.0
1.0
0.0
1.0
0.0
0.7
Apple
0.1
1.0
0.0
0.8
0.0
0.9
Mug
0.0
0.8
0.0
0.8
0.0
0.9
TABLE I: Success Rates Comparison on Clutter-Bench . We evaluate the zero-shot performance of AdaClearGrasp against the VLM Scaffolding baseline across three increasing difficulty tiers (Level-1 to Level-3). The results are averaged over 10 independent trials for each target object and clutter configuration.
Fig. 6: Ablation study on adaptive clearing and closed-loop feedback.
Fig. 7: The grasp success rates of our GeoGrasp policy on 7 object categories (3 seen, 4 unseen) in clutter-free environments.
Target Object
Clutter Level (Obstacles)
Avg.
2
4
6
Apple
9/10
7/10
5/10
70%
Orange Cube
10/10
8/10
6/10
80%
Cup
8/10
6/10
4/10
60%
Total Success
27/30
21/30
15/30
70%
Success Rate
90%
70%
50%
TABLE II: Real-world Sim-to-Real Transfer Results.
Parameter
Value
Optimizer
Adam
Learning Rate
3×10−4
Num. Environments
400
Rollout Steps per Env
5
Batch Size
800
Epochs per Rollout
5
TABLE III: PPO Hyperparameters for GeoGrasp Training
Tool Name
Parameters
Description
push
side (left, right, center), dist
Approaches the nearest obstacle relative to the target and pushes it away to clear a path.
pull
side (left, right, center), dist
Hooks the nearest obstacle and pulls it closer to the robot base to remove occlusion.
pick
None
Invokes the GeoGrasp policy to attempt a dexterous grasp on the target object.
move_to
target (string)
Moves the end-effector to a pre-grasp hover position above the specified object.
lift
height (optional)
Lifts the end-effector vertically by a specified distance to avoid collisions during transport.
initarm
None
Resets the arm to a safe home configuration to recover from singularities or execution failures.
TABLE IV: Atomic Skills and their Parameters exposed via MCP
Fig. 8: The structured prompt template used for VLM planning in AdaClearGrasp. It integrates system instructions, task context, and dynamic feedback to guide the VLM’s reasoning.
Target Name
YCB Model ID
Typical Clutter Objects
Apple
013_apple
Orange, Foam Brick, Mug, Pear, Ball, Cube
Cube
077_rubiks_cube
Can, Foam Brick, Mug, Apple, Ball, Orange
Can
010_potted_meat_can
Foam Brick, Mug, Apple, Ball, Orange, Pear
Pear
016_pear
Can, Foam Brick, Mug, Apple, Ball, Cube
Mug
025_mug
Can, Foam Brick, Apple, Ball, Orange, Cube
Lego
073-e_lego_duplo
Fish Can, Foam Brick, Mug, Apple, Ball, Orange
TABLE V: Target Objects and Clutter Pools in Clutter-Bench .
Department of Computer Science, National University of Singapore · RoboScience · College of Control Science and Engineering, Zhejiang University, China +2