Memory-Efficient Parallel Policy Learning for Collapse-Reactive Retreat via Hybrid Terrain Modeling
Organizations: Fukushima Institute for Research, Education and Innovation, Namie, Fukushima 979-1521, Japan. · Nara Institute of Science and Technology, Ikoma, Nara 630-0192, Japan.
Abstract
At disaster sites and on unstable slopes, robot contact can collapse uninspected terrain, requiring retreat upon support loss. Massively parallel reinforcement learning of collapse-reactive retreat requires independent state--action experiences. However, on shared rigid-body collapsible terrain, one robot's collapse alters others' contacts without direct interaction. We call this indirect interference. Duplicating terrain objects prevents this but is memory-intensive, limiting parallel sampling. Our key insight is that policy learning does not necessarily require rigid-body collapse modeling, which is feasible but computationally expensive; collapse-induced contact responses can suffice. We therefore propose a memory-efficient parallel policy-learning framework for collapse-reactive retreat via hybrid terrain modeling. Hybrid terrain modeling combines rigid-body simulation of non-collapsible terrain with virtual contact forces for collapsible regions. A State-Multiplexed Contact Model shares terrain geometry and a contact-response law while maintaining environment-specific collapse states, enabling independent experiences without terrain objects for collapsible regions. The framework trains a proprioceptive policy for both normal locomotion and collapse-reactive retreat. It supports over 100 times as many parallel environments as fine-resolution rigid-block terrain within the same GPU memory limit. This scalability and improved sample efficiency enable over 50 times faster policy convergence with comparable retreat performance. Sim-to-sim and real-robot experiments demonstrate retreat under multiple collapse conditions. A supplementary video is available at: https://youtu.be/_RrhwjzEPE0
Figures & tables
| Property | Perceptive RL [ 9 ] | Vision & Probing [ 7 ] | SPS [ 10 ] | Blind RL [ 11 ] | Non-Rigid RL [ 12 ] | Deformable RL [ 13 ] | RTDD [ 8 ] | PrePARE [ 14 ] | ABS [ 15 ] | Ours |
|---|---|---|---|---|---|---|---|---|---|---|
| (a) Targets ground that collapses under load | ✗ | ✓ | ✓ | ✗ | ✓ | ✓ | ✓ | ✗ | ✗ | ✓ |
| (b) No visual or probing inspection before contact | ✗ | ✗ | ✗ | ✓ | ✓ | ✓ | ✓ | ✓ | ✗ | ✓ |
| (c) Proprioception only, no exteroceptive sensing | ✗ | ✗ | ✓ | ✓ | ✗ | ✓ | ✓ | ✓ | ✗ | ✓ |
| (d) Changes motion after support loss or slip | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ | ✓ | ✗ | ✓ |
| (e) Single policy without controller switching | ✓ | ✗ | ✗ | ✓ | ✓ | ✓ | ✗ | ✗ | ✗ | ✓ |
| (f) Validated on a real robot | ✓ | ✓ | ✓ | ✓ | ✗ | ✓ | ✓ | ✓ | ✓ | ✓ |
| Property | DEM [ 24 ] | MPM [ 25 ] | GranularGym [ 22 ] | PRPD [ 23 ] | Movable Tiles [ 12 ] | RFT [ 26 ] | Legged Gym [ 19 ] | Ours |
|---|---|---|---|---|---|---|---|---|
| (a) Terrain state changes with robot contact | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✗ | ✓ |
| (b) Change persists over subsequent contacts | ✓ | ✓ | ✓ | ✓ | ✗ | ✗ | ✗ | ✓ |
| (c) Separate terrain history per environment | ✗ | ✗ | ✓ | ✓ | ✗ | ✗ | ✗ | ✓ |
| (d) No terrain copies per environment | ✗ | ✗ | ✗ | ✗ | ✗ | ✓ | ✓ | ✓ |
| (e) Learning with parallel environments | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ | ✓ | ✓ |
| Observation | Dim. | Actor | Critic |
| Base angular velocity in the body frame | 3 | ✓ | ✓ |
| Projected gravity in the body frame | 3 | ✓ | ✓ |
| Velocity command (forward, lateral, and yaw rate) | 3 | ✓ | ✓ |
| Joint positions relative to the default posture (wheel entries set to zero) | 16 | ✓ | ✓ |
| Joint velocities | 16 | ✓ | ✓ |
| Previous action | 16 | ✓ | ✓ |
| Parameter | Value |
|---|---|
| Episode and commands | |
| Initial distance | m from the opening center |
| Initial approach angle | from the approach axis |
| Initial heading error | rad |
| Approach and retreat speeds | m/s |
| Heading-hold yaw-rate gain | 1.5 s -1 on the heading error |
| Parameter | Range |
| Physical properties (once per environment) | |
| Static friction of the robot bodies | |
| Dynamic friction of the robot bodies | |
| Restitution of the robot bodies | |
| Base-mass offset | kg |
| Mass scale of each other link | |
| Term (weight ) | Definition |
|---|---|
| Command tracking | |
| Forward-velocity tracking ( ) | |
| Yaw-rate tracking ( ) | |
| Forward progress ( ) | |
| Lateral drift ( ) | |
| Collapse response and retreat | |
Reinforcement Learning
Reinforcement learning studies how agents learn sequential decisions from rewards and interaction, balancing exploration, performance, safety, and adaptation.
See all 1,852 papers