Koala Gripper: Co-designing Robotic Grippers and Data-Capture Devices for Scaling Dexterous Manipulation Learning
Authors: Amar Hajj-Ahmad, Zubin Kremer Guha, Tim Fofonoff, Zhi Ern Teoh, Ciarán T. O'Neill, Ben Thacher, Igor Fala, Vidullan Surendran, Murphy Wonsick, Peter Whitney, David Watkins
Organizations: * These authors contributed equally to this work.
As the demand for larger manipulation datasets grows, handheld robotic gripper data collection and the associated gripper designs become more vital. Current data collection device designs trend towards matching the morphologies of existing robotic grippers, sacrificing ergonomics and manipulation performance. In this paper, we propose a co-design framework that guides the simultaneous development of both data collection and robotic execution devices by weaving both platform constraints into the design process. Through this workflow, we present the Koala Gripper system, a data capture device and robotic gripper platform that improves dexterity and grasp capability compared to parallel jaw grippers while preserving scalability and ease-of-use. The design introduces a novel force-optimized finger/trigger linkage mechanism with directional reflected mass characteristics, a unique monolithic dual-thumb, and user-centered ergonomic design. The design's actuated robotic fingers are backdrivable, with effective mass on the order of tens of grams. We show that these grippers are capable of secure grasps over a wide range of objects, forceful tool use, and precise singulation. We further validate the platform by deploying it with an end-to-end data collection and policy execution pipeline that highlights its capabilities through learning from demonstration. More information available at http://koalagripper.rai-inst.com
Figures & tables
Fig. 1: The co-designed Koala Data Capture Device (a) and Robot Device (b) demonstrate a bolt fastening task using a standard drill.
Fig. 2: Task requirements, along with constraints and design targets from the CD and RD, feed common features and elements. These are then propagated back to each device, creating iterative cycles throughout the co-design process.
Fig. 3: The space around a robotic gripper can be segmented into several key active workspace areas that define how the gripper interacts with the environment and the associated design implications.
Fig. 4: The finger linkage mechanism integrates the common finger linkage (i) with the RD actuation mechanism (ii) and the CD trigger mechanism (iii)
Fig. 5: The path traced by the proposed trigger mechanism closely mimics natural fingertip closing paths [ 29 ] , improving energy harvesting performance.
Fig. 6: Mechanical advantage of the finger (top), trigger (middle), and combined (bottom) mechanism as configured in the CD. Augmenting the two mechanisms creates consistent mechanical advantage over the full stroke.
Fig. 7: The full workspace of the Koala gripper spans from 0mm pinch grasps to power grasps covering a 7 mm – 100 mm diameter range. Pad geometry targets power grasps of a cylindrical 44 mm diameter standard handle [ 30 ] .
Fig. 8: Triggers for the thumb (1), index finger (2), and middle finger (3) map human actuation to the gripper fingers and thumb. Stabilizing ergonomic features include a dorsal cushion (A), a ventral cushion (B), and a thumb-trigger side load support (C).
Fig. 9: Three wide-angle cameras are positioned on the gripper to observe the gripper fingers and the scene from multiple angles with minimal occlusions.
Fig. 10: Belted ellipsoid (a) and path-aligned (b) representations of finger effective mass over its complete stroke.
Thickness
CD
RD
Difference
10 mm
25.88±0.64N
25.06±0.45N
−3.17%±4.25%
20 mm
23.84±0.73N
24.88±0.91N
4.36%±6.73%
30 mm
24.41±0.65N
26.39±0.88N
8.11%±6.00%
40 mm
24.34±0.77N
26.81±1.16N
10.15%±7.50%
TABLE I: CD vs RD fingertip grasp forces
Fig. 11: Koala grasps the following objects: (a) nail † , (b) key † , (c) pencil (pinch), (d) card, (e) plate † , (f) bowl † , (g) mug † , (h) jug † , (i) box, (j) 25 lb sandbag, (k) pencil (power), (l) coffee can † , (m) 2 L bottle, (n) apple † , (o) ball † , (p) spoon † , (q) hammer † , (r) spray bottle † , (s) drill † . ( † From YCB object set [ 31 ] )
We introduce Yielding Universal Bidigital Interface (YUBI), a finger-aligned gripper designed to enable intuitive, ergonomic, and scalable data collection for bimanual dexterous manipulation. While handheld data collection systems such as Universal Manipulation Interface (UMI) enable affordable data collection, their bulky pistol-grip designs can pose ergonomic and usability challenges for fine-grained, dexterous manipulation tasks. To address this, YUBI presents a distinct design principle: yielding, finger-driven actuation that directly maps human finger movements to gripper jaw motion. Using the YUBI devices, we set up a data collection system with integrated VR-based 6 DoF tracking of the gripper, ensuring high-fidelity trajectory data acquisition. We curate a UMI-based dataset of unprecedented scale: 8,434 hours across 1.20M episodes and 119 tasks. Experiments show that YUBI offers advantages over the UMI gripper in versatility for complex bimanual tasks, dexterity, and operational efficiency. A single policy trained on the YUBI dataset transfers across multiple bimanual robots (UR, Franka, and ELEY) simply by mounting the gripper on each platform, confirming that the collected data are directly executable as policy supervision. We release the gripper hardware, data-collection software, and dataset as one integrated stack, offering the open community a reproducible path to large-scale data acquisition for advancing robotic foundation models.
Takehiko Ohkawa, Jumpei Arima, Yuki Noguchi +16
1AI Robot Association (AIRoA) · 2Toyota Motor Corporation · 4The University of Tokyo +5
Dexterous manipulation is limited by both control and design, without consensus as to what makes manipulators best for performing dexterous tasks. This raises a fundamental challenge: how should we design and control robot manipulators that are optimized for dexterity? We present a co-design framework that learns task-specific hand morphology and complementary dexterous control policies. The framework supports 1) an expansive morphology search space including joint, finger, and palm generation, 2) scalable evaluation across the wide design space via morphology-conditioned cross-embodied control, and 3) real-world fabrication with accessible components. We evaluate the approach across multiple dexterous tasks, including in-hand rotation with simulation and real deployment. Our framework enables an end-to-end pipeline that can design, train, fabricate, and deploy a new robotic hand in under 24 hours. The full framework and generated robot hands are open-sourced and available on our website.
Kehlani Fay, Darin Anthony Djapri, Anya Zorin +6
University of California, San Diego · University of California, Santa Barbara
Learning robust dexterous grasping requires real-world data that records the physical outcomes of grasp attempts. Such data is hard to obtain at scale: teleoperation yields valid physical outcomes but is slow and operator-biased, while simulation-based generation is cheap and scalable but cannot certify contact validity. A natural solution is to generate candidate grasps and verify them on real hardware, but this scales only if the entire collection loop (perception, execution, labeling, and reset) runs without human intervention. We present AutoDex, an automated real-world data-collection system that closes this loop: for each candidate from a replaceable generator, it localizes the object under severe hand-object occlusion with dense 20-camera perception, executes collision-monitored robot motions, labels lift-and-hold success or failure, and actively resets the object between trials to expose additional candidates across stable poses. The result is a reusable database of physically labeled grasp trials that downstream systems can query by retrieval and feasibility filtering. Using AutoDex, we collect 3,593 grasp trials across Allegro and Inspire hands on 100 diverse objects, with synchronized multi-view observations and robot-state logs. For a matched 500-trajectory collection, AutoDex requires 10.3 h versus 49.4 h for teleoperation, yielding a 4.8x throughput improvement, and grasps retrieved from the AutoDex-validated database succeed 76% versus 34% for simulation-only validation. Code and data will be publicly released.