cs.ROMar 11, 2026

AdaClearGrasp: Learning Adaptive Clearing for Zero-Shot Robust Dexterous Grasping in Densely Cluttered Environments

Authors: Zixuan Chen, Wenquan Zhang, Jing Fang, Ruiming Zeng, Zhixuan Xu, Yiwen Hou, Xinke Wang, Jieqi Shi, 

Organizations: Nanjing University · Nanjing University of Posts and Telecommunications · National University of Singapore

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

In densely cluttered environments, physical interference, visual occlusions, and unstable contacts often cause direct dexterous grasping to fail, while aggressive singulation strategies may compromise safety. Enabling robots to adaptively decide whether to clear surrounding objects or directly grasp the target is therefore crucial for robust manipulation. We propose AdaClearGrasp, a closed-loop decision-execution framework for adaptive clearing and zero-shot dexterous grasping in densely cluttered environments. The framework formulates manipulation as a controllable high-level decision process that determines whether to directly grasp the target or first clear surrounding objects. A pretrained vision-language model (VLM) interprets visual observations and language task descriptions to reason about grasp interference and generate a high-level planning skeleton, which invokes structured atomic skills through a unified action interface. For dexterous grasping, we train a reinforcement learning policy with a relative hand-object distance representation, enabling zero-shot generalization across diverse object geometries and physical properties. During execution, visual feedback monitors outcomes and triggers replanning upon failures, forming a closed-loop correction mechanism. To evaluate language-conditioned dexterous grasping in clutter, we introduce Clutter-Bench, the first simulation benchmark with graded clutter complexity. It includes seven target objects across three clutter levels, yielding 210 task scenarios. We further perform sim-to-real experiments on three objects under three clutter levels (18 scenarios). Results demonstrate that AdaClearGrasp significantly improves grasp success rates in densely cluttered environments. For more videos and code, please visit our project website: https://chenzixuan99.github.io/adaclear-grasp.github.io/.

Figures & tables

Fig. 1: We propose AdaClearGrasp , a closed-loop system that leverages VLM-based reasoning to perform adaptive clearing for robust dexterous grasping of target objects in densely cluttered scenes.
Fig. 2: Overview of AdaClearGrasp. The system adopts a hierarchical architecture in which a VLM-based Semantic Planner (left) reasons about the scene and generates high-level action plans. These plans are translated into atomic skills via the Model Context Protocol (MCP) Service (middle). The Skill Library contains heuristic clearing primitives, recovery mechanisms, and a grasping skill (the pretrained geometry-aware RL policy GeoGrasp). The system operates in a closed loop (right), continuously monitoring execution feedback to trigger replanning or recovery when necessary.
Fig. 3: A visualization example of each target object in Clutter-Bench and its corresponding three difficulty levels.
Fig. 4: Real-world Implementation Details. (Upper) Real-world experimental setup with xArm7, XHand, and the external camera Gemini 336L. (Lower) Visualization of 6D pose estimation using FoundationPose, which provides input to the VLM planner.
Fig. 5: Real-world experiments on 18 diverse scenarios.
Target ObjectLevel-1Level-2Level-3
VLM ScaffoldingAdaClearGraspVLM ScaffoldingAdaClearGraspVLM ScaffoldingAdaClearGrasp
Cube0.20.80.00.90.01.0
Can0.10.90.01.00.00.9
Pear0.01.00.01.00.00.7
Apple0.11.00.00.80.00.9
Mug0.00.80.00.80.00.9
TABLE I: Success Rates Comparison on Clutter-Bench . We evaluate the zero-shot performance of AdaClearGrasp against the VLM Scaffolding baseline across three increasing difficulty tiers (Level-1 to Level-3). The results are averaged over 10 independent trials for each target object and clutter configuration.
Fig. 6: Ablation study on adaptive clearing and closed-loop feedback.
Fig. 7: The grasp success rates of our GeoGrasp policy on 7 object categories (3 seen, 4 unseen) in clutter-free environments.
Target ObjectClutter Level (Obstacles)Avg.
246
Apple9/107/105/1070%
Orange Cube10/108/106/1080%
Cup8/106/104/1060%
Total Success27/3021/3015/3070%
Success Rate90%70%50%
TABLE II: Real-world Sim-to-Real Transfer Results.
ParameterValue
OptimizerAdam
Learning Rate3×10−43\times 10^{-4}
Num. Environments400
Rollout Steps per Env5
Batch Size800
Epochs per Rollout5
TABLE III: PPO Hyperparameters for GeoGrasp Training
Tool NameParametersDescription
pushside (left, right, center), distApproaches the nearest obstacle relative to the target and pushes it away to clear a path.
pullside (left, right, center), distHooks the nearest obstacle and pulls it closer to the robot base to remove occlusion.
pickNoneInvokes the GeoGrasp policy to attempt a dexterous grasp on the target object.
move_totarget (string)Moves the end-effector to a pre-grasp hover position above the specified object.
liftheight (optional)Lifts the end-effector vertically by a specified distance to avoid collisions during transport.
initarmNoneResets the arm to a safe home configuration to recover from singularities or execution failures.
TABLE IV: Atomic Skills and their Parameters exposed via MCP
Fig. 8: The structured prompt template used for VLM planning in AdaClearGrasp. It integrates system instructions, task context, and dynamic feedback to guide the VLM’s reasoning.
Target NameYCB Model IDTypical Clutter Objects
Apple013_appleOrange, Foam Brick, Mug, Pear, Ball, Cube
Cube077_rubiks_cubeCan, Foam Brick, Mug, Apple, Ball, Orange
Can010_potted_meat_canFoam Brick, Mug, Apple, Ball, Orange, Pear
Pear016_pearCan, Foam Brick, Mug, Apple, Ball, Cube
Mug025_mugCan, Foam Brick, Apple, Ball, Orange, Cube
Lego073-e_lego_duploFish Can, Foam Brick, Mug, Apple, Ball, Orange
TABLE V: Target Objects and Clutter Pools in Clutter-Bench .
More in this topic

Robotic Grasping

Robotic grasping studies how robots perceive, plan, and execute stable, task-appropriate contacts with objects across varied shapes and conditions.

See all 297 papers

Explore similar work

CardsList
Feb 25, 2025

RetrDex: Efficient Object Retrieval in Cluttered Scenes with a Dexterous Hand

Retrieving objects buried beneath clutter is both challenging and time-consuming, as complex support relationships make manipulation particularly difficult. Existing methods either focus on support relations and rely on sequential grasping to remove occluding objects, or perform preparatory actions such as pushing to facilitate subsequent grasps. However, these approaches are often inefficient and treat physical interactions as isolated auxiliary steps. In this paper, we propose RetrDex, an efficient framework for dexterous arm-hand systems to learn object retrieval in cluttered scenes. Our approach leverages large-scale parallel reinforcement learning (RL) in diverse cluttered scenes and incorporates a spatially aware representation that encodes occlusion patterns and spatial relationships among the target, the dexterous hand, and surrounding clutter. This representation enables the policy to develop diverse manipulation skills (e.g., pushing, stirring, and poking) that actively clear occluders. We evaluate RetrDex on 16 household objects across varied clutter configurations, and obtain strong retrieval performance and efficiency on both seen and unseen targets. Furthermore, we demonstrate successful zero-shot transfer to a real-world dexterous multi-fingered robot system, validating the practical applicability of our method. Videos can be found on our project website: https://RetrDex.github.io.

Fengshuo Bai, Yu Li, Jie Chu +5

Shanghai Jiao Tong University · PKU-PsiBot Joint Lab · Peking University +2

RL for RoboticsRobotic Manipulation
AskPDF
Jun 30, 2026

Agentic RAG-VLM: Affordance-Aware Retrieval-Augmented Generation with Self-Reflective Planning for Robotic Grasping

Generalizable robotic grasping in cluttered environments is essential for deploying manipulators in unstructured human spaces, yet existing VLM-based methods rely on visual similarity for object matching, neglecting physical affordances such as handle graspability and material fragility, and operate open-loop without spatial reasoning or failure recovery, limiting their effectiveness when objects are densely packed or physically diverse. We present Agentic RAG-VLM, a unified framework that bridges VLM-based semantic understanding and physically grounded grasp execution by integrating retrieval-augmented generation (RAG) with vision-language models (VLMs) and agentic self-reflective planning. Agentic RAG-VLM introduces three tightly coupled components: (1) a Hierarchical Affordance-Aware RAG (HAA-RAG) that encodes four-dimensional affordance descriptors, including type, material, fragility, and graspable region, and retrieves strategies by functional affordance compatibility rather than visual appearance; (2) a Scene Graph Constraint Reasoner that constructs spatial relationship graphs from VLM perception and translates proximity, occlusion, and support constraints into concrete grasp parameter adjustments; and (3) an Agentic Self-Reflective Pipeline with a 14-type failure taxonomy and three-level adaptive retry for closed-loop grasp refinement. Evaluated on a 12-task benchmark spanning single-grasp, interactive, and long-horizon scenarios with 360 trials per configuration, Agentic RAG-VLM achieves 78.3 percent overall success, a 53.3 percentage-point absolute gain over VLM-only baselines, demonstrating that affordance-aware retrieval, scene graph reasoning, and agentic recovery are jointly essential for robust manipulation.

Tao Chen, Lizheng Liu, Jiaxu Wang +4

Fudan University, Shanghai, China · Kean University

Robotic GraspingLong-Horizon Robotic ManipulationGeneralization in Robotic ManipulationVLMs for Robotics
AskPDF
Sep 21, 2026

When Does Touch Matter? Charting the Vision-Interaction Gap in Cluttered Dexterous Grasping

Dexterous grasping in clutter poses a basic sensing question: when do tactile measurements and external wrench estimates improve on visual geometry? Occlusion and contact can obscure grasp quality, motivating a controlled evaluation of these interaction signals. We present a controlled real-world study over five tabletop scene conditions on a dexterous hand (LEAP Hand) with vision, per-finger and wrist wrench estimates, and distributed fingertip taxels. With demonstrations, visual observations, action space, and compliant control fixed, we compare diffusion policies trained by imitation learning on vision only, wrench, taxels, or all combined, plus representation and fusion baselines. The combined policy succeeds in 24/25 trials versus 14/25 for vision only, and 15/15 versus 6/15 across the three confined conditions. Ablations show that wrench and taxel feedback are complementary. Behavioral comparisons show that interaction feedback enables earlier rejection of inadequate contacts, regrasping before lift, and more stable grasps. To our knowledge, this is the first real-world study to combine and separately evaluate these interaction modalities for target-oriented dexterous grasping in clutter. These results chart a widening vision-interaction gap and position cluttered dexterous grasping as a benchmark for determining when the learned policy needs interaction sensing. Project website: https://interaction-dex-grasp.github.io/

Hao Jiang, Luis Dominguez, Daniel Seita

Hao Jiang and Daniel Seita are with the Thomas Lord Department of Computer Science, University of Southern California, USA. · Luis Dominguez is with the University of New Mexico, USA.

Dexterous ManipulationTactile Sensing for Robotic Manipulation
AskPDF