Organizations: School of Computer Science, Shanghai Jiao Tong University, Shanghai, 200240, China · Intelligent Game and Decision Laboratory, Beijing, 100071, China · School of Statistics, Renmin University of China, Beijing, 100872, China · School of Artificial Intelligence, Shanghai Jiao Tong University, Shanghai, 200030, China · Shanghai Innovation Institute, Shanghai, China · Center for Applied Statistics, Renmin University of China, Beijing, 100872, China
Recent advances in robotics highlight hierarchical configurations of robot morphology, where multiple levels of functional substructures synergize to facilitate intelligent behaviors. This hierarchical perspective, while particularly advantageous for voxel-based soft robots (VSRs) to ease design and control complexities, is hindered by its heavy reliance on domain expertise. In this work, we address the following question: can we derive such hierarchical design principles solely from existing successful designs? We answer affirmatively by presenting RoboLDA, a Bayesian probabilistic model that decomposes VSR morphology generation into a four-level hierarchy: "task-robot-organ-voxel", and is trained via variational inference. Through extensive experiments on simulated VSRs, we verify the presence of consistent, intuitive hierarchical patterns underlying high-performing VSR designs and showcase RoboLDA's proficiency to extract and leverage these hierarchical priors for zero-shot robot design in unseen tasks. The generated designs, even without further optimization, achieve on average 106.4% of the optimized performance produced by evolutionary algorithms. Additionally, the organ structures inferred by RoboLDA serve as valid functional substructures, significantly enhancing synergistic motion control when integrated with modular control policies. Our work pioneers hierarchical generative modeling of robot morphology, offering a promising pathway towards more interpretable and generalizable development of embodied agents.
Figures & tables
Figure 1: VSR hierarchy from tasks to voxels. Robot types specialized for different tasks exhibit distinct features, but also share some common functional substructures (colored dashed boxes). “V" and “H" represent “vertical" and “horizontal", respectively.
Figure 2: The architecture of RoboLDA, including a decoder (generative process) and an encoder (approximate posterior). The generative process consists of four levels namely task, robot, organ and voxel. The approximate posterior infers latent variables to support variational inference.
Figure 3: MetaMorph with organ synergy. The upper half shows the architecture of MetaMorph. The lower half demonstrates how organs are converted into attention masks and imposed onto the attention layer.
Figure 4: An example of transformation between voxel matrices and rigid robots. For the purpose of illustration, robot joints are torqued so that the directions of rotation become more apparent. Different limb lengths are marked as different colors, while different rotation directions as different shades. Apart from ten different limb types (0 ∼ 9), we use a distinct number (10) to denote the torso.
Figure 5: Illustration of the hierarchical generative process learned by RoboLDA. The distributions of robot type are illustrated as histograms, whereas for organ and voxel layers, each position is labeled with the most likely outcome.
Figure 6: Learning curves of RoboLDA when being trained on morphologies with different performance levels.
Table 1: Task combinations for zero-shot evaluation. Each task is annotated with its difficulty level (easy/medium/hard) and task category (loc: locomotion, man: manipulation, cli: climbing). Following the task-relation perspective of PreCo [ 63 ] , these settings broadly cover three transfer types: more challenging variants (e.g., Climber-v0 → Climber-v2), transfers of comparable difficulty (e.g., Walker-v0 ↔ BridgeWalker-v0; PlatformJumper-v0 → GapJumper-v0), and reverse scenarios (e.g., UpStepper-v0 ↔ DownStepper-v0).
Task
RoboLDA
LASeR
MorphVAE
c-DM
RoboGAN
BO/GA
CPPN-NEAT
POET
High-similarity transfers
Pusher-v0 (easy)
11.42
10.97
11.22
6.99
7.60
7.79
9.67
12.50
Carrier-v0 (easy)
8.47
7.72
8.61
5.70
4.70
5.98
6.58
8.19
Climber-v1 (medium)
5.78
0.72
3.25
0.26
0.21
0.54
0.25
-
Medium-similarity transfers
Walker-v0 (easy)
10.65
10.62
10.64
10.21
8.45
9.85
6.42
-
Table 2: Comparison of zero-shot fitness between RoboLDA and baseline algorithms, grouped by train–test task similarity measured by average edit distance (Table 1 ). As both GA and BO use random sampling for their first generation, their results are averaged. The task names are followed by their difficulty levels. Harder tasks require more complex morphologies and thus present bigger challenges for robot design.
Task
BO
GA
CPPN-NEAT
BridgeWalker-v0
11
18
8
Walker-v0
N/A
N/A
N/A
Pusher-v0
N/A
29
N/A
Carrier-v0
N/A
18
N/A
UpStepper-v0
N/A
28
N/A
DownStepper-v0
23
17
3
Table 3: The number of generations for baseline EAs to reach RoboLDA’s zero-shot fitness (using the average-then-threshold protocol; see Section 4.4). “N/A" indicates that the EA fails to do so before evolution terminates.
Figure 7: Relative performance of zero-shot RoboLDA compared against optimal results evolved by baseline EAs (BO, GA, and CPPN-NEAT), using the ratio-then-average protocol (see Section 4.4).
Figure 8: Comparative results of robotic control.
Appendix figures & tables23 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 9: Results of comparison study on robotic control (training robots).
Task
BW-v0
Walker-v0
Pusher-v0
Carrier-v0
US-v0
DS-v0
Climber-v1
Climber-v2
GJ-v0
Sensitivity regarding hyperparameter settings
Std
0.222
0.008
0.990
1.246
0.983
0.641
0.831
0.313
0.776
COV
0.034
< 0.001
0.082
0.141
0.236
0.072
0.142
0.159
0.178
Sensitivity regarding random initialization
Std
0.004
0.003
1.213
1.098
0.006
0.277
0.050
0.027
0.375
COV
< 0.001
< 0.001
0.106
0.130
< 0.001
0.054
0.009
0.015
0.102
Appendix
Table 4: Sensitivity analysis of zero-shot fitness. “Std" and “COV" represent standard deviation and coefficient of variation, respectively. BW, US, DS and GJ stand for BridgeWalker, UpStepper, DownStepper and GapJumper, respectively.
Figure 10: Case studies on coordination and dexterity.
Figure 11: Locomotion environments of rigid robots.
Table 6: Zero-shot transfer results on 10×10 Walker-v0.
Figure 12: Organ mask ablation on four tasks. Each row: one task; left: training split; right: held-out test split.
Method
Climber-v1 (high)
Walker-v0 (med.)
UpStepper-v0 (low)
RoboLDA
5.78
10.65
5.13
w/o position modeling
4.13
10.59
4.02
task-organ-voxel
5.55
10.62
4.16
task-robot-voxel
5.72
10.62
3.93
task-voxel
5.70
10.63
4.29
Appendix
Table 7: Hierarchy ablation on three representative transfer settings.
Task
RoboLDA
MorphVAE
LASeR
BO/GA
CPPN-NEAT
UpStepper-v0
4.62 ± 0.71
3.72 ± 1.81
1.55 ± 0.63
2.86 ± 0.82
4.03 ± 0.56
Climber-v1
5.92 ± 0.84
4.35 ± 0.71
0.89 ± 0.08
0.27 ± 0.04
1.88 ± 2.20
Climber-v2
1.78 ± 0.07
1.11 ± 0.43
1.43 ± 0.39
0.27 ± 0.04
0.97 ± 0.57
GapJumper-v0
5.78 ± 0.86
4.45 ± 1.10
4.29 ± 0.14
2.40 ± 0.67
4.47 ± 1.39
Appendix
Table 8: Zero-shot fitness (mean ± std) over 10 independent runs on four challenging transfer settings.
Figure 13: Walker-v0
Figure 14: Pusher-v0
Figure 15: Carrier-v0
Figure 16: BridgeWalker-v0
Figure 17: UpStepper-v0
Figure 18: DownStepper-v0
Figure 19: Climber-v0. The figure has been rotated 90 degrees clockwise to save space, where the vertical line on the left represents the ground. The same is true for Figures 20 and 21 .
Figure 20: Climber-v1
Figure 21: Climber-v2
Figure 22: PlatformJumper-v0
Figure 23: GapJumper-v0
Hyperparameter
Value
RoboLDA
robot size
5×5
number of robot types
6
number of organ types
6
number of organ components
10
hidden dims of MLPs
[128]*3
Appendix
Table 9: Hyperparameter settings.
Figure 24: Visualization of the best-performing rigid robot morphologies generated by different algorithms in each trial, as well as the ones optimized by the genetic algorithm. f denotes fitness of each robot, which is measured as the cumulative sum of rewards over a complete episode of 400 timesteps, when controlled by the optimized controller. Reward of each step is in turn assigned according to the velocity of the robot along the x -axis.
Figure 25: Additional visualizations of hierarchical structures learned by RoboLDA.
Voxel-based soft robots (VSRs) present a promising avenue for developing artificial organisms with lifelike intelligence. However, the vast design spaces and expensive evaluations substantially challenge their design optimization. Here we develop MISCO, a novel evolutionary framework empowered by deep generative models to optimize VSR designs with theoretical guarantees. MISCO integrates an estimation-of-distribution algorithm with a meticulously designed variational autoencoder featuring multi-task learning, position awareness, and inter-voxel signaling. These key components enhance the representational capacity of VSR morphologies and facilitate highly efficient sampling and optimization of morphological distributions. We provide theoretical guarantees for MISCO's asymptotic convergence to globally optimal designs, alongside a favorable convergence rate. Extensive simulated experiments further demonstrate MISCO's exceptional effectiveness in navigating vast design spaces, evolving high-performing VSRs for diverse tasks while flexibly balancing optimization efficiency and morphological diversity. Being validated both empirically and theoretically, MISCO represents a step change towards more scalable and reliable soft robot development.
Junru Song, Huan Xiao, Yang Yang +8
School of Computer Science, Shanghai Jiao Tong University, Shanghai 200240, China · School of Statistics, Renmin University of China, Beijing 100872, China · Intelligent Game and Decision Laboratory, Beijing 100071, China +1
Bridging high-level semantic understanding with low-level physical control remains a persistent challenge in embodied intelligence, stemming from the fundamental spatiotemporal scale mismatch between cognition and action. Existing generative VLA policies typically adopt a "Generation-from-Noise" paradigm, which disregards this disparity, leading to representation inefficiency and weak condition alignment during optimization. In this work, we propose ResVLA, an architecture that shifts the paradigm to "Refinement-from-Intent." Recognizing that robotic motion naturally decomposes into global intent and local dynamics, ResVLA utilizes spectral analysis to decouple control into a deterministic low-frequency anchor and a stochastic high-frequency residual. By anchoring the generative process on the predicted intent, our model focuses strictly on refining local dynamics via a residual diffusion bridge. Extensive simulation experiments show that ResVLA achieves competitive performance, strong robustness to language and robot embodiment perturbations, and faster convergence than standard generative baselines. ResVLA also demonstrates strong performance in real-world robot experiments.
Yiming Zhong, Yaoyu He, Zemin Yang +5
ShanghaiTech University, Shanghai, China · Morphic Robotics, Shenzhen, China · The Chinese University of Hong Kong, Hong Kong, China.
In this paper, we introduce a model of evolution and learning in robots that co-optimizes a distribution of latent design vectors (genotypes) and a mixture of control experts (neural modules), which are gated by the latent coordinates of each decoded design (phenotype). This provides a scalable alternative to co-design algorithms that either train an individual policy for every robot, which is inefficient, or a monolithic universal controller for all robots, which results in overly conservative structures and behaviors. Our approach lies somewhere between these two extremes, preserving ancestral knowledge in a unified yet modular framework in which different body plans activate and deactivate different combinations of learned sensorimotor circuits for goal-directed behavior. This allows one part of the controller to be overhauled to better suit new species of designs as they emerge without disrupting the hard-earned knowledge contained within other expert modules. It also allows pretrained expert policies to be directly plugged into the mixture, which can steer evolution into otherwise unexplored areas of latent space containing desired morphological traits. We refer to this process as "evo by demo" and explore how it may be used to guide freeform evolution toward canonical structures defined by the pretrained model. Videos and code can be found at: https://eco-moe.github.io.