cs.LGNov 14, 2026

Gradient-Based Co-Design of Systems and Reinforcement Learning Policies with Differentiable Simulators

Authors: Akhil S. Anand, Xun Shen, Marc Peter Deisenroth, Sebastien Gros

Organizations: Department of Engineering Cybernetics, Norwegian University of Science and Technology (NTNU), Trondheim, Norway · Graduate School of Engineering, Tokyo University of Agriculture and Technology, Tokyo, Japan · UCL Centre for Artificial Intelligence, University College London, London, United Kingdom

Abstract

Co-design is the problem of jointly optimizing the design of a system and the policy that operates it. Neither can be chosen independently: a system design is only as good as the policy trained for it, and the optimal policy changes with the design. When that policy is learned by reinforcement learning (RL), evaluating a candidate design requires a full policy training run, and the design is usually optimized by black-box search such as Bayesian optimization (BO) or evolution strategies (ES). These methods scale poorly with the design dimension and ignore the differentiable structure of modern simulators and, in particular, the sensitivity of the optimal policy with respect to the design. We instead derive a gradient-based co-design method, where the design gradient decomposes into a direct effect of changing the design at a fixed policy and an indirect effect through the design-dependent optimal policy. The indirect effect requires the sensitivity of the converged policy to the design, which we obtain by applying the implicit function theorem to the stationarity condition of the policy gradient. Based on this sensitivity, we develop a proof-of-concept co-design algorithm with stochastic value gradient (SVG)-style policy optimization, prove that it converges to an approximately stationary design, and demonstrate on a differentiable networked-system benchmark with high-dimensional design spaces that it consistently outperforms BO and ES.

Figures & tables

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. Co-design Gym: A Unified Benchmark for Embodiment-Policy Co-optimization

    Oct 1, 2026Aviraj Newatia, Yordan Tsvetkov, Leonard Pleiss +2Robot Morphology-Control Co-DesignRL Benchmarks

  2. Environment Parameter Gradient Theorem for Policy-Environment Co-Design in Reinforcement Learning

    Jul 14, 2026Amber SrivastavaPolicy Learning

  3. COSMIC: Concurrent Optimization of Structure, Material, and Integrated Control for robotic systems

    May 12, 2026Qinsong Guo, Liwei WangTopology OptimizationRobot Morphology-Control Co-Design