Gradient-Based Co-Design of Systems and Reinforcement Learning Policies with Differentiable Simulators
Organizations: Department of Engineering Cybernetics, Norwegian University of Science and Technology (NTNU), Trondheim, Norway · Graduate School of Engineering, Tokyo University of Agriculture and Technology, Tokyo, Japan · UCL Centre for Artificial Intelligence, University College London, London, United Kingdom
Abstract
Co-design is the problem of jointly optimizing the design of a system and the policy that operates it. Neither can be chosen independently: a system design is only as good as the policy trained for it, and the optimal policy changes with the design. When that policy is learned by reinforcement learning (RL), evaluating a candidate design requires a full policy training run, and the design is usually optimized by black-box search such as Bayesian optimization (BO) or evolution strategies (ES). These methods scale poorly with the design dimension and ignore the differentiable structure of modern simulators and, in particular, the sensitivity of the optimal policy with respect to the design. We instead derive a gradient-based co-design method, where the design gradient decomposes into a direct effect of changing the design at a fixed policy and an indirect effect through the design-dependent optimal policy. The indirect effect requires the sensitivity of the converged policy to the design, which we obtain by applying the implicit function theorem to the stationarity condition of the policy gradient. Based on this sensitivity, we develop a proof-of-concept co-design algorithm with stochastic value gradient (SVG)-style policy optimization, prove that it converges to an approximately stationary design, and demonstrate on a differentiable networked-system benchmark with high-dimensional design spaces that it consistently outperforms BO and ES.
Figures & tables
| bo | 2.7 | 2.8 | 3.2 |
|---|---|---|---|
| es | 2.2 | 2.2 | 2.3 |
| direct-only | 2.0 | 2.0 | 2.1 |
| sensitivity-enabled, dense rank- | 8.6 | 11.6 | 24.7 |
| sensitivity-enabled, matrix-free | 6.4 | 6.4 | 6.7 |
Appendix figures & tables3 assets
Supplementary material from the paper’s appendix.
Appendix
| Quantity | Definition | Computed by |
|---|---|---|
| Direct effect | ( fixed) | reverse-mode ad in |
| Inner gradient | reverse-mode ad in | |
| Residual | reverse-mode ad in | |
| forward-over-reverse HVP ( jvp of ) | ||
| jvp of in | ||
| vjp of in |
| iterations | sensitivity-enabled | direct-only | difference | |
|---|---|---|---|---|
| , | 50 | ( , 15/15) | ( ) | (13/15, ) |
| , | 100 | ( , 15/15) | ( ) | (15/15, ) |
| , | 150 | ( , 15/15) | ( ) | (13/15, ) |
Reinforcement Learning
Reinforcement learning studies how agents learn sequential decisions from rewards and interaction, balancing exploration, performance, safety, and adaptation.
See all 1,855 papers