Warm Starting Neural Dynamics Models From Few Jacobian Samples
Organizations: Independent Researcher · NUMA, KU Leuven, Belgium
Abstract
Model-based reinforcement learning (MBRL) leverages a model of the environment, used for planning and generating samples without interacting with the real world. Motivated by evidence that informed initialization significantly improves the efficiency of MBRL, we propose TDec-Init, an algorithm to warm-start neural dynamics models using transition Jacobians. Using a decoupling algorithm based on the decomposition of a tensor of stacked Jacobian samples, we recover weights and flexible nonlinearities, and use them to initialize a feedforward neural network. In the low-data regime, we show on several benchmark environments that our approach outperforms random initialization and value training, and either matches or outperforms Sobolev training, while requiring fewer training steps between episodes.
Figures & tables
| step (s) | |||||||
|---|---|---|---|---|---|---|---|
| Reacher | 4 | 2 | 100 | 0.05 | 4 | 20 | 1000 |
| Pendulum | 2 | 1 | 200 | 0.05 | 8 | 20 | 1000 |
| Quadrotor | 6 | 2 | 200 | 0.02 | 16 | 40 | 1000 |
| CartPole | 4 | 1 | 200 | 0.05 | 16 | 20 | 1000 |
| Acrobot | 4 | 1 | 500 | 0.20 | 12 | 20 | 1000 |
Model-Based RL
Model-based reinforcement learning uses environment models to plan actions and learn policies with fewer real-world interactions.
See all 273 papers