Efficient Morphology-Control Co-Design via Stackelberg Proximal Policy Optimization
Authors: Yanning Dai, Yuhui Wang, Dylan R. Ashley, Jürgen Schmidhuber
Organizations: Center of Excellence for Generative AI, King Abdullah University of Science and Technology (KAUST), Thuwal, Saudi Arabia. · Dalle Molle Institute for Artificial Intelligence Research (IDSIA), Lugano, Switzerland. · Università della Svizzera italiana (USI), Lugano, Switzerland. · Scuola universitaria professionale della Svizzera italiana (SUPSI), Lugano, Switzerland.
Morphology-control co-design concerns the coupled optimization of an agent's body structure and control policy. This problem exhibits a bi-level structure, where the control dynamically adapts to the morphology to maximize performance. Existing methods typically neglect the control's adaptation dynamics by adopting a single-level formulation that treats the control policy as fixed when optimizing morphology. This can lead to inefficient optimization, as morphology updates may be misaligned with control adaptation. In this paper, we revisit the co-design problem from a game-theoretic perspective, modeling the intrinsic coupling between morphology and control as a novel variant of a Stackelberg game. We propose Stackelberg Proximal Policy Optimization (Stackelberg PPO), which explicitly incorporates the control's adaptation dynamics into morphology optimization. By modeling this intrinsic coupling, our method aligns morphology updates with control adaptation, thereby stabilizing training and improving learning efficiency. Experiments across diverse co-design tasks demonstrate that Stackelberg PPO outperforms standard PPO in both stability and final performance, opening the way for dramatically more efficient robotics designs.
Figures & tables
Figure 1: Showcasing how our Stackelberg PPO can autonomously design task-specific robots for the “Pusher” task, starting with a bare-bones structure and ultimately evolving into a sophisticated design with arm-like structures for pushing boxes and leg-like limbs for movement. This highlights the method’s ability to create adaptive and complex designs. In comparison, the traditional PPO method generates simpler structures that can’t support more complex behaviors. For more examples of evolved designs and animations, as well as open-source code, visit: https://yanningdai.github.io/stackelberg-ppo-co-design .
Figure 2: Illustration of the phase-separated Stackelberg Markov Game for morphology–control co-design. In the leader phase (blue part), the agent incrementally edits the morphology via discrete topology-altering actions, producing a terminal morphology sTL . In the follower phase (green part), the control policy is optimized based on this morphology.
Figure 3: Performance curve with respect to the number of follower steps during training. Shaded regions denote standard error across seven random seeds.
Figure 4: (a) Evolved morphologies. Ablation studies on (b) λ sweep from 0 to ∞ and (c) Fisher information matrix on/off, both evaluated on Stepper-Regular task.
Figure 5: (a) Reward curves and (b) KL-divergence traces for different clipping thresholds ϵ . (c) Performance comparison under varying leader horizons T . All evaluated on Stepper-Regular.
Appendix figures & tables25 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 6: Visualization of the nine benchmark environments used in our experiments. Crawler, Stepper-Regular, Stepper-Hard, and Pusher are 3D tasks; others are 2D (x-z or x-y plane). The environments differ substantially in required morphology depth, symmetry, and limb arrangement, enabling evaluation on the generality of morphology–control co-design across locomotion and manipulation tasks.
Hyperparameter
Value
Fisher Regularization Coefficient λ
5.0
Maximum Conjugate Gradient Steps
20
CG Relative Error Tolerance
10−3
Follower Sampling Steps per Episode
6
Gradient Normalization Ratio α
1.0
Structure Design Steps Tstru
5
Appendix
Table 1: Hyperparameters of Stackelberg PPO adopted in all experiments
Figure 7: Visualization of co-evolved body designs generated through our Stackelberg PPO.
Methods
Crawler
Cheetah
Swimmer
Stackelberg PPO (Ours)
11047.90±126.20
13514.94±653.62
1334.98±16.06
BodyGen ( Lu et al., 2025 )
9098.72±558.26
11575.87±640.65
1302.64±3.71
Transform2Act ( Yuan et al., 2022 )
3950.80±268.43
8297.90±825.02
737.90±21.04
NGE ( Wang et al., 2019 )
1482.45±524.97
2534.76±428.68
384.45±112.03
ESS ( Sims, 1994 )
631.67±122.41
671.67±134.65
190.62±37.84
Methods
Walker-Hard
Glider-Hard
TerrainCrosser
Appendix
Table 2: Performance comparison of Stackelberg PPO against baseline methods across morphology-control co-design environments. Results show mean episode rewards and standard deviations over seven random seeds.
Figure 8: Extended evaluation on Glider and Walker environments under different morphology complexity budgets.(a) Training curves for the regular, medium, and hard variants of each environment. (b) Final generated morphologies under each complexity tier.
Clipping Parameter
Performance
Likelihood Ratio Violations (%)
Average KL Divergence
ϵ=0.1
4934.52±646.40
14.82±0.55
0.0030±0.0006
ϵ=0.2
7215.20±449.02
13.39±0.83
0.0196±0.0025
ϵ=0.4
7907.01±208.02
9.92±0.94
0.0343±0.0153
ϵ=0.6
4778.18±407.84
9.10±1.30
0.0665±0.0188
ϵ=0.8
2656.92±503.93
7.02±0.81
0.1340±0.0388
No Clipping
1233.26±443.98
0
1.7726±0.1539
Appendix
Table 3: Sensitivity of Stackelberg PPO to the clipping threshold ϵ .
Environment
SID+PPO (full)
PPO-only (no SID)
SID-only (no clipping)
Stepper-Regular
7215.20±449.02
4685.94±845.23
1257.33±530.25
Crawler
11047.90±126.20
9098.72±558.26
35.77±12.25
Cheetah
13514.94±653.62
11575.87±640.65
472.89±77.40
Glider
12414.50±498.53
11049.95±468.44
566.81±89.96
Appendix
Table 4: Ablation studies on the components of our SID estimator and PPO clipping, evaluated under the same phase-separated, non-differentiable Stackelberg setting.
Figure 9: Comparison of morphology evolution between Stackelberg PPO (ours) and Standard PPO (BodyGen) across four environments. BodyGen tends to collapse early into low-complexity designs, while Stackelberg PPO continues exploring structurally richer morphologies, yielding more capable final structures.
Stackelberg PPO (Ours)
BodyGen (PPO)
Performance Change After 1 Epoch
+0.392±0.075%
+0.224±0.043%
Appendix
Table 5: Average performance change after one epoch of training from the same checkpoint model, averaged over 10 checkpoints and 7 seeds, evaluated on Stepper-Regular.
Stackelberg PPO (10 cores)
BodyGen (10 cores)
NGE (10 cores)
NGE (64 cores)
Wall-clock Time
33.07±0.49 h
29.43±0.97 h
45.16±3.72 h
13.52±1.52 h
Appendix
Table 8: Wall-clock training time across methods. NGE results are shown under both 10 CPU cores and 64 cores to illustrate parallelization effects.
Figure 10: Morphological evolution trajectories across eight environments. Each row represents a distinct robot embodiment, with columns showing progressive stages of morphological adaptation from 5% to 100% training progress.
Figure 11: Power constraint under different penalty coefficients. As the penalty increases, the co-designed morphologies transition toward shorter, thicker, and more symmetric limbs.
Penalty Coef.
Performance
Power Penalty
Stackelberg PPO (ours)
BodyGen
Stackelberg PPO (ours)
BodyGen
0.0001
11047.90±126.20
9098.72±558.26
5631.42±674.03
2745.84±284.55
0.001
10191.15±371.81
7501.61±671.33
5342.16±584.25
5948.16±254.84
0.01
9853.19±229.37
8304.00±497.56
1582.48±697.61
468.12±516.33
0.1
10585.25±146.80
8974.48±574.29
25.50±22.27
26.34±23.64
Appendix
Table 9: Power-constraint setting: performance and power penalties under different penalty coefficients.
Figure 12: Manufacturability constraint under different penalty coefficients. Higher penalties on structure complexity and material mass encourage designs with fewer body elements, reduced branching, and mass concentrated near the root, producing compact morphologies that are easier to fabricate.
Penalty Coef.
Performance
Morphology Complexity
Material Cost
Stackelberg PPO (ours)
BodyGen
Stackelberg PPO (ours)
BodyGen
Stackelberg PPO (ours)
BodyGen
0
11047.90±126.20
9098.72±558.26
16.40±2.45
13.67±2.08
1.71±0.23
1.57±0.32
1
7892.93±349.84
6531.37±437.26
13.67±2.03
9.67±1.61
1.59±0.18
1.32±0.16
10
6825.47±303.09
5372.10±364.79
8.25±0.91
7.50±1.24
0.94±0.08
0.93±0.10
Appendix
Table 10: Manufacturability constraint setting: performance, morphology complexity, and material cost under different penalty levels.
Figure 13: Torque limits constraint under different penalty coefficients. Tighter actuation limits lead to noticeably simpler and more compact structures, with shorter limbs and reduced distal branching.
Penalty Coef.
Performance
Limit Violation Penalty
Stackelberg PPO (ours)
BodyGen
Stackelberg PPO (ours)
BodyGen
0.01
7893.42±84.62
6311.75±98.31
20210.50±6503.22
11350.45±5338.31
0.1
3133.01±70.44
3121.80±54.03
1106.45±64.69
899.35±49.40
Appendix
Table 11: Torque limits constraint: performance and torque-violation penalties under different torque-penalty coefficients.
Figure 14: Morphology comparison trained with and without payload. Payload induces more symmetric and load-supporting structures.
Payload Weight
Stackelberg PPO (ours)
BodyGen
0.2 kg
8675.08±286.42
5186.31±659.87
0.4 kg
6966.34±473.43
5116.54±546.55
0.6 kg
7347.46±478.01
4523.89±536.99
Appendix
Table 12: Payload constraint: performance comparison under different payload weights.
Level
Stackelberg PPO (ours)
BodyGen
2.0 N
11557.31±124.68
6963.05±450.48
4.0 N
11290.13±164.54
4621.82±597.71
6.0 N
10875.23±250.97
2816.16±857.83
Appendix
Table 13: Robustness evaluation: performance under different levels of external disturbance forces.
Level
Stackelberg PPO (ours)
BodyGen
30%
11424.66±112.08
7326.04±421.73
50%
11333.43±141.42
6795.55±493.31
70%
10892.09±149.72
5062.85±579.66
Appendix
Table 14: Robustness evaluation: performance under different levels of terrain friction noise.
Figure 15: Visualization the unseen obstacle-navigation task environment.
Obstacle Type
Spacing
Performance
Stackelberg PPO (ours)
BodyGen
Sparse Obstacle
16 m ( ∼ 4× robot width)
1790.45±161.77
1061.55±228.40
Dense Obstacle
8 m ( ∼ 2× robot width)
1698.52±733.02
1007.21±157.42
Appendix
Table 15: Performance in the unseen obstacle-navigation task under two obstacle densities.
Figure 16: Cross-task reuse of morphology priors: Crawler prior (left) and the resulting Pusher morphology (right).
Condition
Performance
Steps to Threshold (2500 Reward)
Stackelberg PPO (ours)
BodyGen
Stackelberg PPO (ours)
BodyGen
With Prior
4822.59±114.32
4575.52±112.78
∼8M
∼9M
Without Prior
3462.77±368.09
2779.95±509.18
∼32M
∼44M
Appendix
Table 16: Performance and sample efficiency in the Pusher task with and without morphology priors.
Morphology--control co-design jointly optimizes an agent's body structure and control policy as an integrated embodied system. However, existing methods typically model morphology design and control with separate networks coupled only indirectly through a shared task objective, limiting explicit high-level coordination. Inspired by natural genes that coordinate biological development, we introduce Morphogene, a compact latent blueprint that bridges an agent's body and brain. Through AdaConcat, Morphogene jointly conditions morphology and control generation at the limb level, allowing its variations to induce coordinated changes in both components. Building on this representation, we propose GeCode, which formulates co-design as exploration in the compact Morphogene space. Each Morphogene anchors a local design region in which nearby body--brain designs are explored, while performance-guided updates move these anchors toward promising regions for more efficient exploration of the broader design space. This process combines local refinement with global exploration while preserving body--brain compatibility. Extensive experiments across diverse 2D and 3D co-design tasks demonstrate that GeCode consistently outperforms existing state-of-the-art methods, achieving substantially faster convergence and higher final performance.
Fu Feng, Ruixiao Shi, Yucheng Xie +2
School of Computer Science and Engineering, Southeast University, Nanjing, China · Key Laboratory of New Generation Artificial Intelligence Technology and Its Interdisciplinary Applications (Southeast University), Ministry of Education, China
The intelligent behavior of robots does not emerge solely from control systems, but from the tight coupling between body and brain, a principle known as embodied intelligence. Designing soft robots that leverage this interaction remains a significant challenge, particularly when morphology and control require simultaneous optimization. A significant obstacle in this co-design process is that morphological evolution can disrupt learned control strategies, making it difficult to reuse or adapt existing knowledge. We address this by develop a Graph Neural Network-based approach for the co-design of morphology and controller. Each robot is represented as a graph, with a graph attention network (GAT) encoding node features and a pooled representation passed through a multilayer perceptron (MLP) head to produce actuator commands or value estimates. During evolution, inheritance follows a topology-consistent mapping: shared GAT layers are reused, MLP hidden layers are transferred intact, matched actuator outputs are copied, and unmatched ones are randomly initialized and fine-tuned. This morphology-aware policy class lets the controller adapt when the body mutates. On the benchmark, our GAT-based approach achieves higher final fitness and stronger adaptability to morphological variations compared to traditional MLP-only co-design methods. These results indicate that graph-structured policies provide a more effective interface between evolving morphologies and control for embodied intelligence.
Robot co-design via bi-level optimization couples within-lifetime controller learning for fitness evaluation with cross-generational morphological evolution. Prior work has established that well-adapted morphology facilitates faster control learning, a property termed morphological intelligence. Yet how control learning reciprocally shapes morphological evolution remains unexplored. This paper examines both directions for a holistic account of brain-body interplay. We first show that morphological contributions to control learning decouple into two orthogonal dimensions. We formalize the convergence speed as morphological intelligence and identify the performance ceiling as a complementary quantity termed true potential. A concise functional relation is then established to jointly characterize both quantities from individual learning curves, which, when aggregated at the population level, capture evolutionary profiles. Through extensive experiments on simulated voxel-based soft robots, we reveal that premature fitness evaluation systematically underestimates true potential and biases selection towards fast learners. This restricts design space exploration, compromising both optimization efficiency and morphological diversity. Notably, the widely recognized morphological Baldwin effect emerges as an artifact of this bias rather than a general evolutionary tendency. We therefore propose AdaControl, which monitors disproportionate selection for morphological intelligence during evolution and allocates minimally sufficient control learning for unbiased fitness evaluation. With AdaControl, a simple genetic algorithm rivals state-of-the-art generative-model-based co-design methods in discovering diverse high-performing designs while cutting computation by up to 80% versus exhaustive control.
Junru Song, Yang Yang, Yaqing Xu +5
Shanghai Jiao Tong University, Shanghai 200240, China · Intelligent Game and Decision Laboratory, Beijing 100048, China · Renmin University of China, Beijing 100872, China +1