Gradient-Based Co-Design of Systems and Reinforcement Learning Policies with Differentiable Simulators
Authors: Akhil S. Anand, Xun Shen, Marc Peter Deisenroth, Sebastien Gros
Organizations: Department of Engineering Cybernetics, Norwegian University of Science and Technology (NTNU), Trondheim, Norway · Graduate School of Engineering, Tokyo University of Agriculture and Technology, Tokyo, Japan · UCL Centre for Artificial Intelligence, University College London, London, United Kingdom
Co-design is the problem of jointly optimizing the design of a system and the policy that operates it. Neither can be chosen independently: a system design is only as good as the policy trained for it, and the optimal policy changes with the design. When that policy is learned by reinforcement learning (RL), evaluating a candidate design requires a full policy training run, and the design is usually optimized by black-box search such as Bayesian optimization (BO) or evolution strategies (ES). These methods scale poorly with the design dimension and ignore the differentiable structure of modern simulators and, in particular, the sensitivity of the optimal policy with respect to the design. We instead derive a gradient-based co-design method, where the design gradient decomposes into a direct effect of changing the design at a fixed policy and an indirect effect through the design-dependent optimal policy. The indirect effect requires the sensitivity of the converged policy to the design, which we obtain by applying the implicit function theorem to the stationarity condition of the policy gradient. Based on this sensitivity, we develop a proof-of-concept co-design algorithm with stochastic value gradient (SVG)-style policy optimization, prove that it converges to an approximately stationary design, and demonstrate on a differentiable networked-system benchmark with high-dimensional design spaces that it consistently outperforms BO and ES.
Figures & tables
Figure 1: Gradient-based co-design. (a) The co-design loop: the policy optimization loop (environment ↔ policy) yields the converged policy ϕ⋆(θ) , evaluated under the design objective JD ; the design gradient, a direct effect plus an indirect policy-sensitivity effect through κθ=∇θϕ⋆(θ) , used by another rl agent to update the design parameters. (b) Moving along the optimal policy manifold ϕ⋆(θ) (slope κθ ) decomposes the total design gradient ∇θJD into a direct step at fixed policy ( ∂θJD ) and an indirect step from the induced policy shift ( ∇ϕJDκθ ).
Figure 2: A node in inventory–routing network
Figure 3: Comparison with bo and es (with λy=0 , where κθ=0 by Lemma 1 ). Learning curves for bo , es and the gradient-based co-design method starting from the same initial design, for different design parameter sizes, over a budget of 100 design evaluations. Curves show the best co-design performance found so far relative to the shared initial design, JD−JD(θ0) , with mean ± SE over 25 trials; the percentage gaps quoted in the text are of the absolute final JD (Appendix E.2 ).
Figure 4: Top: Effect of the policy-sensitivity term (misaligned configuration, λy=0.1 ). Learning curves for the gradient-based co-design with the sensitivity-enabled gradient and the direct-only ( κθ=0 ). The curves show the resulting co-design performance JD with mean ± SE across the 15 trials. Bottom: Refining the best es design (misaligned configuration, λy=0.1 ). Learning curves for the gradient-based co-design starting from the design found by a converged es search, with the sensitivity-enabled gradient and the direct-only. The curves show the JD improvement over the es design (the zero line), with mean ± SE across the 15 trials; the annotated values are the final gains as % of the mean es performance.
nθ=15
30
60
bo
2.7
2.8
3.2
es
2.2
2.2
2.3
direct-only
2.0
2.0
2.1
sensitivity-enabled, dense rank- k
8.6
11.6
24.7
sensitivity-enabled, matrix-free
6.4
6.4
6.7
Table 1: Wall-clock per design evaluation (seconds).
Appendix figures & tables3 assets
Supplementary material from the paper’s appendix.
Appendix
Quantity
Definition
Computed by
Direct effect
∂θJ^HD ( ϕ fixed)
reverse-mode ad in θ
Inner gradient u
∇ϕJ^HD
reverse-mode ad in ϕ
Residual F
∇ϕJHπ
reverse-mode ad in ϕ
Av
∇ϕϕ2JHπv
forward-over-reverse HVP ( jvp of F )
Bv
∇θϕ2JHπv
jvp of F in θ
B⊤w
(∇θϕ2JHπ)⊤w
vjp of F in θ
Appendix
Table 2: Computation of each quantity. ad : automatic differentiation of the rollout surrogate; HVP: Hessian–vector product. Only κθ needs a solver beyond ad .
Figure 5: Per-trial paired differences at the common stopping point (iteration 200): ΔJD (sensitivity-enabled − direct-only, k=50/100/200 ); dots are the 15 trials per N , black markers the mean ± SE.
iterations
sensitivity-enabled
direct-only
difference
N=5 , k=50
50
+148.3 ( 65% , 15/15)
+91.2 ( 40% )
+57.1±23.0 (13/15, p=0.026 )
N=10 , k=100
100
+422.9 ( 58% , 15/15)
+255.6 ( 35% )
+167.3±58.6 (15/15, p=0.013 )
N=20 , k=200
150
+1772.8 ( 99% , 15/15)
+1396.3 ( 78% )
+376.5±128.9 (13/15, p=0.011 )
Appendix
Table 3: Refinement from the best es design (misaligned configuration, k≈3% of dimϕ ), scored at the stopping point of Figure 4 . Gain = JD improvement over the starting es design for the sensitivity-enabled and the direct-only variation (in brackets, as % of the mean es performance, and wins over the 15 trials); contrast = sensitivity-enabled − direct-only, paired over the 15 trials.
Reinforcement learning studies how agents learn sequential decisions from rewards and interaction, balancing exploration, performance, safety, and adaptation.
Finding an optimal behaviour policy within a given environment is a widely studied problem in domains as diverse as games, robotics, energy infrastructure, communication networks, and multi-agent systems. Numerous benchmarks have been developed to support such research, but the vast majority assume that the agent's embodiment (design) is fixed, focusing instead on policy learning alone. Lifting this assumption gives rise to a broader class of problems in which optimizing embodiment and policy separately is highly suboptimal. An agent's embodiment strongly shapes which control policies can be discovered, while the optimal embodiment is in turn defined by the policies it admits. To help the research community study this class of problems explicitly and systematically, we introduce Co-Design Gym - a suite of benchmark environments for jointly optimizing embodiment and policy. Our environments span domains such as robotic manipulation and locomotion, multi-robot cooperation, deformable and soft dynamics, video games, electricity grids, wireless networks, F1 racing, multi-agent warehouses, and optimal control, offering 20 environment families (domains), with over 85 distinct co-design presets in total. We further contribute a systematic evaluation of representative co-design algorithms, characterizing the current state of the art. Together, these contributions lay the groundwork for cumulative, comparable progress in co-design.
Aviraj Newatia, Yordan Tsvetkov, Leonard Pleiss +2
University of Cambridge · Technical University of Munich · Carnegie Mellon University +1
Reinforcement learning (RL) is traditionally concerned with learning a control policy for a fixed environment. In many engineering systems, however, the environment itself is alterable: physical or operational parameters can be tuned to shape the transition dynamics and costs experienced by the agent. This motivates jointly optimizing both the policy and the environment design parameters. To this end, we establish an Environment Parameter Gradient Theorem -- a formal expression for the gradient of the value function with respect to environment parameters. The key theoretical device is a generalized action-value function Qπ,ξ(s,a,ζ), which comprises two copies of the environment parameters: ζ governs the cost and transition dynamics at the current state--action pair, while ξ governs the future rollouts. This decoupling yields a tractable closed-form gradient expression and is essential to the theorem's derivation. Building on this result, we develop a model-free algorithm that simultaneously learns the optimal policy and the environment parameters. We demonstrate the efficacy of our framework on a UAV network design problem, where the optimal UAV placement (environment parameters) and communication routes (governed by the policy) are learned jointly to minimize the total communication cost in the network.
Amber Srivastava
Faculty of Mechanical Engineering, Indian Institute of Technology Delhi, India
Replicating and surpassing the autonomy of natural organisms remains a long-standing goal in robotics. Yet most robotic systems have their structure, materials, and control designed separately, in sharp contrast to the co-evolution in nature. This separation often leads to suboptimal designs, and we still have a limited understanding of the individual and collective contributions of these design entities. In this work, we propose a gradient-based co-design framework that simultaneously optimizes the topology, material distribution, and control policy of a truss-lattice robot. The framework embeds mixed-type topological and material variables into a continuous design space and integrates a neural network controller within a differentiable simulator, capturing their interactions and enabling efficient gradient calculation via automatic differentiation. Furthermore, we develop a constrained optimization to navigate the highly non-convex design landscape and jointly optimize all design entities. Case studies demonstrate that the proposed framework consistently discovers diverse locomotion strategies that outperform baselines obtained through separated design. The framework is also flexible to accommodate different functional requirements and boundary conditions. Using this framework, we further extract design insights that reveal the individual and collective effects of different entities on robotic performance. The proposed framework provides a computational foundation for the autonomous co-design of robotic systems, capable of reconfiguration, locomotion, and other complex autonomous behaviors.
Qinsong Guo, Liwei Wang
Dept. of Mechanical Engineering Carnegie Mellon University Pittsburgh, PA 15213