Evolving Dexterous Robots from Scratch

Zihan Guo; Shuzhe Zhang; Muhan Li; Peiyang Li; Sam Kriegman

Figure 8: One design. The root is the central body: the hub at its centre and five spokes around it. Three spokes are active and so built, each ending at the tip from which an appendage grows; two are inactive and not built. A web fills the space between neighbouring built spokes. Each appendage is a chain of segments, and each joint between two segments is marked by the hinge axis it takes, lateral or normal. Segments are drawn slightly apart so that every joint shows. The parameters of the root and of an appendage are shown one by one in Figs. 9 and 10 .

Parameter	Range	Bins	Parameter	Range	Bins
Root hub			Appendage joint
root hub aspect	0.8 to 3.0	12	appendage joint yaw bend	$-0.24$ to $0.24$ rad	32
root hub ring fraction	0.5 to 1.0	12	appendage joint hinge axis	lateral or normal	2
root hub radius	0.02 to 0.05 m	12	appendage joint pitch bend	$-0.26$ to $0.26$ rad	32
root hub infill	1.0 to 4.0	8	appendage joint length	0.03 to 0.06 m	32
Root spoke and web			Appendage segment

Appendix

Table 1: Design-space parameters. The root is described by 39 values and each segment of an appendage, with its joint, by 12. Rows follow the panels of Figs. 9 and 10 . Bins is the number of bins used to write the parameter as a token (Appx. B.3 ).

Figure 9: The root. Top: one root from the design space, seen from above (left), from the end (right) and from the side (bottom). The purple ellipse is the hub. An orange bar with an arrow marks the tip of each spoke that is built. A–K : one parameter each, cut and enlarged from the views above; the small circle on the views marks where each was cut. Root hub: A Root hub aspect , B Root hub ring fraction , C Root hub radius , D Root hub infill . Root spoke and web: E Root spoke azimuth , F Root spoke activation , G Root spoke elevation , H Root spoke radius , I Root spoke length , J Root web radius , K Root web reach .

Figure 10: The appendage. Top: one appendage of the same root, growing from its spoke tip at the left; each segment is drawn as a chain of beads around its outline. A–L : one parameter each, cut and enlarged from the appendage above; the small circle marks where each was cut. Appendage joint: A Appendage joint yaw bend , B Appendage joint hinge axis , C Appendage joint pitch bend , D Appendage joint length . Appendage segment: E Appendage segment asymmetry , F Appendage segment gain , G Appendage segment lobe , H Appendage segment phase , I Appendage segment width , J Appendage segment depth , K Appendage segment taper , L Appendage segment twist .

Figure 11: Sensitivity analysis of the design variables. We measured how much each variable of the design changes the body. Across 1,000 designs, each variable in turn was changed by a random amount (10% of its range on average) and the body was voxelized before and after the change. The change of the body is one minus the intersection over union (IoU).

Figure 12: Data augmentation. Two designs and the operations used to create their positive pairs. The original design is shown in gray behind each augmented copy.

Symbol	Meaning
$\mathbf{p}$ , $\mathbf{p}_{0}$	position of the object, and its start position
$\mathbf{p}^{\ast}$	point under the root at which the object is held
$r$	distance of the object from its start position along the table, $\|\mathbf{p}_{xy}-\mathbf{p}_{0,xy}\|$
$h$	height of the object above its start position
$\theta$	angle between the orientation of the object and the goal orientation
$\omega$	rotation rate of the object about the task axis

Appendix

Table 2: Symbols used in the reward tables.

Figure 13: The 16 training objects.

	Wooden Cube (134.5 g)		Rubber Cube (26.8 g)		Rubik’s Cube 3x (92.9 g)		Rubik’s Cube 4x (132.3 g)		Fabric Cube (16.1 g)
Design	$\omega$	$T$	$\omega$	$T$	$\omega$	$T$	$\omega$	$T$	$\omega$	$T$
LEAP (Multi-obj)	—		—		—		—		—
LEAP (Cube-only)	$\text{1.7}_{\scriptscriptstyle\pm\text{0.1}}$	$\text{20.0}_{\scriptscriptstyle\pm\text{7.0}}$	$\text{0.2}_{\scriptscriptstyle\pm\text{0.0}}$	$\text{30.0}_{\scriptscriptstyle\pm\text{0.0}}$	$\text{0.3}_{\scriptscriptstyle\pm\text{0.0}}$	$\text{30.0}_{\scriptscriptstyle\pm\text{0.0}}$	$\text{0.3}_{\scriptscriptstyle\pm\text{0.0}}$	$\text{30.0}_{\scriptscriptstyle\pm\text{0.0}}$	—
Ours (Tetradactyl)	$\text{{1.8}}_{\scriptscriptstyle\bm{\pm}\text{{0.1}}}$	$\text{{30.0}}_{\scriptscriptstyle\bm{\pm}\text{{0.0}}}$	$\text{1.9}_{\scriptscriptstyle\pm\text{0.0}}$	$\text{30.0}_{\scriptscriptstyle\pm\text{0.0}}$	$\text{1.7}_{\scriptscriptstyle\pm\text{0.1}}$	$\text{30.0}_{\scriptscriptstyle\pm\text{0.0}}$	$\text{1.8}_{\scriptscriptstyle\pm\text{0.3}}$	$\text{21.3}_{\scriptscriptstyle\pm\text{7.0}}$	$\text{{1.7}}_{\scriptscriptstyle\bm{\pm}\text{{0.1}}}$	$\text{{30.0}}_{\scriptscriptstyle\bm{\pm}\text{{0.0}}}$
Ours (Didactyl)	$\text{1.5}_{\scriptscriptstyle\pm\text{0.3}}$	$\text{30.0}_{\scriptscriptstyle\pm\text{0.0}}$	$\text{{2.4}}_{\scriptscriptstyle\bm{\pm}\text{{0.5}}}$	$\text{{30.0}}_{\scriptscriptstyle\bm{\pm}\text{{0.0}}}$	$\text{{2.0}}_{\scriptscriptstyle\bm{\pm}\text{{0.0}}}$	$\text{{30.0}}_{\scriptscriptstyle\bm{\pm}\text{{0.0}}}$	$\text{{1.9}}_{\scriptscriptstyle\bm{\pm}\text{{0.2}}}$	$\text{{30.0}}_{\scriptscriptstyle\bm{\pm}\text{{0.0}}}$	$\text{1.4}_{\scriptscriptstyle\pm\text{0.2}}$	$\text{30.0}_{\scriptscriptstyle\pm\text{0.0}}$
	Tomato Soup Can (353.2 g)		Moon Ball (30.9 g)		Apple (364.6 g)		Orange (212.6 g)		Avocado (223.8 g)

Appendix

Table 3: Real-world in-hand rotation performance across diverse unseen objects. We report the mean rotation speed (

\omega

, rad/s) and stable rotation time (

T

, s), with sample standard deviations shown as subscripts. Rotation speeds are evaluated in two trials. Stable rotation time is evaluated in three 30 s trials. For each object, both metrics are bolded for the design with the highest mean rotation speed, determined before rounding. An em dash indicates failure to sustain in-hand rotation.

Term	Equation	Weight
rotation toward goal	$1/(\|\theta\|+0.1)$	$1$
goal reached	$\|\theta\|\leq 0.2$	$250$
steady rotation	$0.25<\omega_{z}<1.5$	$1$
object displacement	$\|\mathbf{p}-\mathbf{p}_{0}\|$	$-10$
object drift	$\max(\|\mathbf{p}-\mathbf{p}_{0}\|-0.03,\,0)$	$-0.1$
action magnitude	$\|\mathbf{a}_{t}\|^{2}$	$-2\mathrm{e}{-4}$

Appendix

Table 4: Reward terms of the rotate task.

Term	Equation	Weight
approach	$\mathrm{mean}_{i}\,\exp(-d_{i}/0.02)$ , before first contact	$0.4$
contact	$\mathrm{mean}_{i}\,f_{i}$	$0.8$
multiple contacts	at least two appendages touch the object	$0.8$
lift	$\mathrm{clip}\big((h-0.022)/0.008,\,0,\,1\big)$	$8$
near the root	$\exp(-\|\mathbf{p}-\mathbf{p}^{\ast}\|/0.05)$	$2$
stable grasp	held, at least two contacts, and nearly at rest	$2$

Appendix

Table 5: Reward terms of the pick task.

Wooden Cube	Rubber Cube	Rubik’s Cube 3x	Rubik’s Cube 4x	Fabric Cube	Cube Toy	Lemonade Mix Box	Tennis Ball	Moon Ball	Orange	Avocado	Donut Toy
10/10	10/10	8/10	8/10	10/10	7/10	10/10	6/10	5/10	9/10	5/10	4/10
Water Can	Candle Jar	Pudding	Polygon Toy	Jam Jar	Energy Stick	Meat Can	Toy Bear	Disney Toy	Baby Bottle	Motor	Salt Container
10/10	6/10	10/10	8/10	6/10	10/10	10/10	10/10	10/10	10/10	10/10	0/10

Appendix

Table 6: Real-world performance on the pick task across diverse objects. Results indicate successful picks out of 10 trials per object.

Term	Equation	Weight
approach	$\mathrm{mean}_{i}\,\exp(-d_{i}/0.02)$ , before first contact	$0.4$
contact	$\mathrm{mean}_{i}\,f_{i}$	$0.8$
multiple contacts	at least two appendages touch the bar	$0.8$
stability †	$\mathrm{mean}_{i}\,f_{i}\cdot\exp(-s/0.0075)$ , if at least two contacts	$8$
stable grasp †	at least two contacts and $s\leq 0.0075$	$2$
joint torques	$\|\bm{\tau}\|^{2}/\tau_{\mathrm{ref}}$	$-0.5$

Appendix

Table 7: Reward terms of the hang task. Terms marked with

\dagger

are active only after release.

\tau_{\mathrm{ref}}

is a hyperparameter.

Term	Equation	Weight
new progress	$\min(\Delta g^{\star},\,0.1)$	$40$
new progress in contact	$\min(\Delta g^{\star},\,0.1)$	$40$
new progress, unclipped	$\Delta g^{\star}$	$80$
new progress, squared	$\Delta(g^{\star})^{2}$	$80$
net progress	$\mathrm{clip}(\Delta g,\,\pm 0.1)$	$10$
proximity	$\mathrm{clip}(\Delta p,\,\pm 0.25)$	$5$

Appendix

Table 8: Reward terms of the retrieve task.

Term	Equation	Weight
Flip stage
progress toward goal	$\max(-\Delta\theta,\,0)$	$10$
goal reached	$\theta\leq 0.1$	$250$
steady rolling	$0.25<\omega<1.5$	$1$
rolling backward	$\max(-\omega,\,0)$	$-1$
Return stage

Appendix

Table 9: Reward terms of the flip task.

Term	Equation	Weight
grasp progress	$G_{t}-G_{t-1}$ , before release	$0.5$
contact †	$\mathrm{mean}_{i}\,f_{i}$	$0.25$
multiple contacts †	at least two appendages touch the handle	$0.25$
hold ‡	$1$	$1$
no slip ‡	$1/(1+S)$	$8$
action smoothing	$\|\mathbf{a}_{t}-\mathbf{a}_{t-1}\|^{2}$	$-0.005$

Appendix

Table 10: Reward terms of the hold task. The grasp score is

G=0.2\,\mathrm{mean}_{i}\,e^{-d_{i}/0.02}+0.4\,\mathrm{mean}_{i}\,f_{i}+0.4

, and the slip cost is

S=0.25\,(s_{\mathrm{a}}/0.02)^{2}+0.25\,(s_{\mathrm{l}}/0.015)^{2}+0.5\,(s_{\mathrm{r}}/0.35)^{2}

, where

s_{\mathrm{a}}

,

s_{\mathrm{l}}

and

s_{\mathrm{r}}

are the slip along, across, and about the handle. Terms marked with

\dagger

are active only after release; those marked with

\ddagger

are, in addition, multiplied by

\mathrm{mean}_{i}\,f_{i}

when at least two appendages touch the handle, and by 0 otherwise.

Figure 14: The evolution of multitaskers. Robots were evolved to rotate, flip, pick, hold, and hang. Unified fitness (A) was the sum of task normalized scores. The task scores of the best design found so far (B-F) reveal trade-offs: gains in unified fitness did not always improve performance on every task. Selected designs (G-M) correspond to the labeled points in A.

Figure 15: Real-world in-hand rotation across diverse objects for didactyl design.

Figure 16: Real-world in-hand rotation across diverse objects for didactyl design.

Figure 17: Real-world in-hand rotation across diverse objects for tetradactyl design.

Figure 18: Real-world picking across diverse objects using the four-appendage design.

Figure 19: Real-world picking across diverse objects using the four-appendage design.

Figure 20: Real-world test objects. The everyday objects used in the real-world rotate and pick experiments: rigid and soft, smooth and textured, from a 16 g fabric cube to a 687 g salt container.

Figure 21: The five physical designs. From left to right: the retriever (purple), the didactyl rotator (grey), the picker (white), the 4-appendage rotator (grey) and the hanger (tan). Each is 3D-printed in PLA with one actuator per joint; parts are listed in Table 17 . Their real-world results are in Tables 3 and 6 and Figs. 4 – 5 .

Model	Validity			Smoothness
Model	App. acc.	Joint acc.	Valid	Recall@1	Edit $\rho$	Topology 10-NN acc.	Length 10-NN $r$	Volume 10-NN $r$
Ours (2M)	1.00	0.96	0.82	1.00	$0.97\pm 0.10$	0.92	0.97	0.91
Ours (200k)	1.00	0.84	0.65	1.00	$0.99\pm 0.05$	0.76	0.96	0.90
w/o $z\in\mathbb{S}^{31}$	1.00	0.80	0.64	1.00	$0.99\pm 0.05$	0.71	0.95	0.88
w/o $\mathcal{L}_{\mathrm{con}}$	1.00	0.95	0.67	0.03	$0.97\pm 0.09$	0.59	0.82	0.73
VAE	0.44	0.06	0.33	0.01	$0.93\pm 0.13$	0.02	0.79	0.79

Appendix

Table 11: Ablation of the latent genome. All validity metrics, Recall@1, and Edit

\rho

use the 2,000-design test set; all three 10-NN metrics use a separate set of 4,500 designs. The best and second best values in each column are shaded.

Figure 22: Genetic coherence. Ten thousand unique, pentadactyl morphologies were generated, copied, and the copy was rotated by a randomly selected angle. The resulting design pairs are visualized by UMAP ( Healy and McInnes, 2024 ) along 2D projections of four different latent embeddings ( A-D ). A gray line connects the original (filled circle) to its rotated (open circle) counterpart. Five design variants were considered for this analysis, those with a pair of three-jointed appendages and three two-jointed appendages (3,3,2,2,2), four three-jointed and one two-jointed (3,3,3,3,2), one four-jointed and four three-jointed (4,3,3,3,3), three four-jointed and two three-jointed (4,4,4,3,3), and five four-jointed appendages (4,4,4,4,4). In the VAE, rotated copies can appear far from their originals in the UMAP projection (D). With the contrastive loss, rotated copies remain close to their originals in the UMAP projection (A–B); without it, the two views separate (C). Removing the spherical projection,

z\in\mathbb{S}^{31}

, had little effect on this pairing (B).

Layer	Description
Voxel input	$128^{3}\times 2$
Stem	Conv, kernel size $3^{3}$ , stride 2, to $64^{3}$ , 32 channels, batch normalization, ReLU
Stage 1	2 $\times$ ResBlock, downsample to $32^{3}$ , 64 channels
Stage 2	2 $\times$ ResBlock, downsample to $16^{3}$ , 128 channels
Stage 3	2 $\times$ ResBlock, downsample to $8^{3}$ , 256 channels
Stage 4	2 $\times$ ResBlock, downsample to $4^{3}$ , 512 channels

Appendix

Table 12: Encoder architecture.

Layer	Description
Input	$z\in S^{31}$ , and the tokens predicted so far
Memory	Linear in: 32, out: $8\times 384$ (eight vectors)
Embedding	token embedding (137) + learned position embedding (368), 384 channels
Layers 1–4	Transformer decoder layer: causal self-attention, attention to the eight vectors, feed-forward (1536, GELU); 8 heads, layer normalization before each part
Head	Linear in: 384, out: 137
Output	logits of the next token: 137

Appendix

Table 13: Decoder architecture.

Parameter	Value
CMA-ES
Population size	32
Initial step size ( $\sigma_{0}$ )	0.25
Fitness of Rejected Designs
Failed to compile	$-1000$
Motors do not fit	$-950$

Appendix

Table 14: CMA-ES hyperparameters.

Parameter	Range
Observation
Joint position noise (per step)	0 to 0.02 rad
Joint position offset (per episode)	$\pm$ 0.005 rad
Action
Action noise (per step)	0.05 to 0.1
Action delay	0 to 2 control steps

Appendix

Table 15: Domain randomization used for fine-tuning.

Parameter	Value
Parallel environments	4096
PPO iterations	500 for rotate and flip, 200 otherwise
Steps per iteration	32
Policy and value network	GRU (256) + MLP (512, 256, 128), ELU
Learning rate	$5\times 10^{-4}$ ( $3\times 10^{-4}$ ), adaptive
Target KL divergence	0.02

Appendix

Table 16: Reinforcement learning hyperparameters.

Item	Cost (US)	Rotate	Rotate-4	Pick	Hang	Retrieve
XL330-M288-T Actuator	$27.49	15	15	10	10	9
FPX330-H101 Hinge Frame with Idler a	$2.82	15	15	10	10	9
FPX330-S102 Base Frame (Radial) a	$2.50	7	7	10	5	1
FPX330-S101 Side Frame (Lateral) a	$2.50	8	8	0	5	8
Robot Cable-X3P 180 mm b	incl.	15	15	10	10	9
U2D2 USB Converter	$36.92	1	1	1	1	1

Appendix

Table 17: Bill-of-materials (BOM) of the five physical designs. All designs share the same parts: the base and every segment are 3D-printed in PLA, and each actuated joint is one DYNAMIXEL XL330-M288-T that drives the child link through an FPX330-H101 hinge frame and is seated in the parent link on an FPX330-S102 (radial mount) or FPX330-S101 (lateral mount) frame. The first column gives the unit cost (ROBOTIS US list price, September 2026) and the remaining columns the quantity used by each design; PLA is counted by mass. Totals are rounded to the nearest dollar. a Sold in sets of four; unit cost is one quarter of the set price. b One cable ships with every actuator, so cables add no cost. c M3 heat-set insert cost is based on a 100-pack of Uxcell M3

\times

6 mm brass heat-set inserts (5 mm OD) at

12.69. PLA cost is estimated at

0.02/g, with mass estimated from the printed volume at the effective printed density.

Evolving Dexterous Robots from Scratch

Abstract

Figures & tables

Appendix

Robotic Manipulation

House of Dextra: Cross-embodied Co-design for Dexterous Hands

ECo-MoE: Embodiment-Conditioned Mixture of Experts Increases the Evolvability of Robots

Shaping the Evolutionary Dynamics of Robot Morphology via Adaptive Control Learning

Abstract

Robotic Manipulation

Explore similar work

House of Dextra: Cross-embodied Co-design for Dexterous Hands

ECo-MoE: Embodiment-Conditioned Mixture of Experts Increases the Evolvability of Robots

Shaping the Evolutionary Dynamics of Robot Morphology via Adaptive Control Learning