GraspBench: Execution-Aware Cross-Robot Evaluation of 6-DoF Grasp Generation in MuJoCo

Abdul Basit; Bibek Poudel; Muhammad Shafique

cs.RONov 5, 2026

GraspBench: Execution-Aware Cross-Robot Evaluation of 6-DoF Grasp Generation in MuJoCo

Authors: Abdul Basit, Bibek Poudel, Muhammad Shafique

Organizations: The authors are with the eBRAIN Lab, Division of Engineering, New York University Abu Dhabi, Abu Dhabi, UAE.

Abstract

Modern 6-DoF grasp generators are often evaluated using grasp scores, dataset-level metrics, or single-robot demonstrations, yet proposal quality alone does not determine whether a grasp is executable by a particular robot. We present GraspBench, a controlled MuJoCo protocol for evaluating complete grasp-generation pipelines after robot-specific inverse-kinematics filtering, motion planning, and fixed approach-close-lift execution. GraspBench evaluates six configured pipelines across four robot embodiments, five diagnostic objects, and a structured tabletop pose grid, yielding 540 scenarios per method and 3,240 method-scenario evaluations in total. Missing outputs are conservatively counted as failures, and success follows the binary lift criterion produced by each method's execution runner. GraspGen achieves the highest overall success rate (73.1%), followed by EconomicGrasp (63.5%), while robot- and object-conditioned analyses reveal substantial variation across embodiments and object geometries. GPG produces the largest native candidate sets but achieves 44.3% success, indicating that proposal volume alone does not guarantee robot-executable grasps. Cluster-aware paired intervals support the principal method differences while not estimating repeated-execution uncertainty. The results characterize complete configured pipelines on fixed scenarios rather than intrinsic model quality or stochastic repeatability, showing that grasping performance depends jointly on the generator, robot embodiment, scene, and execution protocol.

Figures & tables

Work	Role	Scale	Exec.	$\times$ -robot	Cond.	Fail.
Dex-Net [ 3 ]	synthetic policy/data	$\surd$	$\surd$	$\times$	$\triangle$	$\times$
GraspNet [ 1 ]	RGB-D benchmark	$\surd$	$\triangle$	$\times$	$\triangle$	$\times$
ACRONYM [ 2 ]	simulated grasp data	$\surd$	$\triangle$	$\times$	$\times$	$\times$
Robosuite [ 13 ]	manipulation sim	$\triangle$	$\surd$	$\triangle$	$\surd$	$\times$
RLBench [ 14 ]	task benchmark	$\surd$	$\surd$	$\triangle$	$\surd$	$\times$
GPD/GPG [ 4 ]	geom. generator	$\times$	$\triangle$	$\times$	$\triangle$	$\times$

TABLE I: Evaluation-scope comparison with representative grasping datasets, generators, and manipulation benchmarks.

Fig. 1: GraspBench pipeline: proposal, robot-specific feasibility filtering, OMPL planning, MuJoCo execution, and outcome logging. Missing outputs and scenarios with no feasible grasp count as failures; Contact-GN denotes Contact-GraspNet (CGN).

Fig. 2: Diagnostic GraspBench object set.

Fig. 3: Evaluated robot configurations; each includes its arm, gripper, camera, controller, frame adapter, and collision model. “Piper” denotes AgileX PiPER.

Fig. 4: Tabletop distance, lateral-position, and pitch grid for condition-specific analysis.

Category	Values	Count
Robots	AUBO i5, Franka, KUKA iiwa, PiPER	4
Methods	AG, CGN, Econ., GPG, GG, HGGD	6
Objects	airplane, glass, cube, cylinder, bottle	5
Distance	Near, Mid, Far	3
Lateral	Left, Center, Right	3
Pitch	$-45^{\circ}$ , $0^{\circ}$ , $45^{\circ}$	3

TABLE II: GraspBench scenario grid used in the main evaluation.

Method	Success n/540 (%)	95% int. (%)	Missing	Cand. mean/med.	IK max
GraspGen	395 (73.1)	[69.3, 76.7]	0	296/304	50
EconomicGrasp	343 (63.5)	[59.4, 67.5]	0	1024/1024	50
AnyGrasp	259 (48.0)	[43.8, 52.2]	1	391/365	80
HGGD	249 (46.1)	[41.9, 50.3]	5	6/6	20
GPG	239 (44.3)	[40.1, 48.5]	0	7414/8088	–
Contact-GraspNet	178 (33.0)	[29.1, 37.0]	31	23/23	–

TABLE III: IK-filtered results. Missing outputs fail; intervals are Wilson grid summaries; candidates are native mean/median.

Fig. 5: Execution results: (a) success with Wilson intervals; (b) clustered paired differences; (c) native candidates versus success (descriptive because budgets differ). Abbreviations follow Table II .

Fig. 6: Diagnostics by (a) robot, (b) object, (c) pitch, and (d) failure type: other observed (amber), no grasp (red), and missing output (purple). Abbreviations follow Table II .

Method	Returned	E2E SR	SR\|record
	records	(%)	(%)
GraspGen	540 (100.0)	73.1	73.1
EconomicGrasp	540 (100.0)	63.5	63.5
AnyGrasp	539 (99.8)	48.0	48.1
HGGD	535 (99.1)	46.1	46.5
GPG	540 (100.0)	44.3	44.3

TABLE IV: Returned-record availability and success (%). End-to-end SR uses all 540 intended scenarios.

Robotic Grasping

Robotic grasping studies how robots perceive, plan, and execute stable, task-appropriate contacts with objects across varied shapes and conditions.

See all 324 papers

First Plan Then Evaluate: Multi-Target Planning with Post-Planning Success Evaluation Improves Learning-Based Grasping Pipelines
Sep 8, 2025Martin Matak, Mohanraj Devendran Shanthi, Karl Van Wyk +1Robotic Grasping Robot Motion Planning
Kahlert School of Computing and the Robotics Center, University of Utah. · NVIDIA, USA.
Beyond Visual Grasping: Benchmarking Complex Grasping from Detection to Execution
Jul 15, 2026Hanyi Zhang, Khang Nguyen, Charith Munasinghe +10Robotic Grasping Robot Foundation Models
University of Liverpool, UK · Mohamed bin Zayed University of Artificial Intelligence, UAE · Zurich University of Applied Science, Switzerland +3
MVB-Grasp: Minimum-Volume-Box Filtering of Diffusion-based Grasps for Frontal Manipulation
May 10, 2026Bibek Poudel, Abdul Basit, Muhammad ShafiqueRobotic Grasping
eBRAIN Lab, Division of Engineering New York University (NYU) Abu Dhabi, Abu Dhabi, UAE

Sep 8, 2025J/K move · Enter open · S save

First Plan Then Evaluate: Multi-Target Planning with Post-Planning Success Evaluation Improves Learning-Based Grasping Pipelines

Martin Matak, Mohanraj Devendran Shanthi, Karl Van Wyk, Tucker Hermans

Kahlert School of Computing and the Robotics Center, University of Utah. · NVIDIA, USA.

Robotic Grasping Robot Motion Planning

Open paper AskPDF arXiv

Autonomous multi-finger grasping is a fundamental capability in robotic manipulation. Optimization-based approaches show strong performance, but tend to be sensitive to initialization and are potentially time-consuming. As an alternative, the generator-evaluator-planner framework has been proposed. A generator generates grasp candidates, an evaluator ranks the proposed grasps, and a motion planner plans a trajectory to the highest-ranked grasp. If the planner doesn't find a trajectory, a new trajectory optimization is started with the next-best grasp as the target and so on. However, executing lower-ranked grasps means a lower chance of grasp success, and multiple trajectory optimizations are time-consuming. Alternatively, relaxing the threshold for motion planning accuracy allows for easier computation of a successful trajectory but implies lower accuracy in estimating grasp success likelihood. It's a lose-lose proposition: either spend more time finding a successful trajectory or have a worse estimate of grasp success. We propose a framework that plans trajectories to a set of generated grasp targets, the evaluator estimates the grasp success likelihood at the terminal configuration of the planned trajectories, and the robot executes the trajectory most likely to succeed. Our experiments show our approach improves over the traditional generator-evaluator-planner framework across different objects, generators, and motion planners, and successfully generalizes to novel environments in the real world, including different shelves and table heights. Project Website: https://martinmatak.github.io/fpte/

Abstract

Robotic Grasping

Explore similar work

First Plan Then Evaluate: Multi-Target Planning with Post-Planning Success Evaluation Improves Learning-Based Grasping Pipelines

Beyond Visual Grasping: Benchmarking Complex Grasping from Detection to Execution

MVB-Grasp: Minimum-Volume-Box Filtering of Diffusion-based Grasps for Frontal Manipulation