PROOF: Perceptive Robustness via Out-of-Distribution Fallback for Legged Locomotion
Organizations: Southern University of Science and Technology · Pengcheng Lab · Tsinghua University · The Hong Kong University of Science and Technology (Guangzhou)
Abstract
Vision-based perceptive locomotion has enabled legged robots to achieve agile and terrain-aware locomotion across complex environments. However, the reliability of such systems heavily depends on visual observations, which are inherently susceptible to occlusion, reflective surfaces, and illumination variations, often resulting in catastrophic locomotion failures under degraded perception conditions. In this paper, we present PROOF (Perceptive Robustness via Out-of-Distribution Fallback for Legged Locomotion), a perceptive locomotion framework that enables robust locomotion under severe visual disturbances. Our method adopts a two-stage teacher-student learning paradigm that explicitly models perception reliability: the teacher learns both perception-driven and proprioceptive fallback behaviors, while the student predicts exteroceptive features with uncertainty and suppresses them under out-of-distribution inputs. We evaluate our method on a Unitree Go2 quadruped equipped with a RealSense D435 depth camera in real-world settings. Under normal visual conditions, our method enables the robot to successfully perform high-dynamic parkour maneuvers, including traversing large gaps, elevated platforms, and narrow beam terrains. Under severe visual degradation, such as heavy occlusion and strong illumination changes, the robot continues to maintain stable and robust locomotion performance.
Figures & tables
| Method | Rough Flat | Uphill | Downhill | Upstairs | Downstairs |
|---|---|---|---|---|---|
| Baseline 1: w/o teacher mask | 0.28 / 0 | 0.11 / 0 | 0.35 / 0 | 0.15 / 0 | 0.22 / 0 |
| Baseline 2: MSE latent distillation | 0.83 / 96 | 0.76 / 79 | 0.75 / 84 | 0.69 / 73 | 0.71 / 91 |
| PROOF | 0.86 / 100 | 0.79 / 99 | 0.78 / 100 | 0.72 / 95 | 0.93 / 100 |
Appendix figures & tables12 assets
Supplementary material from the paper’s appendix.
Appendix
| Obs Group | Dim. / Shape | Used by | Contents |
|---|---|---|---|
| Proprioception | Teacher actor, student | Base angular velocity, projected gravity, velocity command, joint positions, joint velocities, and previous actions. | |
| Elevation map | Teacher actor/critic | Yaw-aligned terrain scan over with spacing; ray origin is shifted forward and heights use a vertical offset. | |
| Depth image † | Student | Two delayed forward-camera depth frames after ray clipping, crop, artifact noise, blur, and normalization. | |
| Exteroceptive mask | Teacher actor, distillation loss | Binary gate target; masks the teacher’s exteroceptive latent. | |
| Privileged state | Teacher critic | Base linear/angular velocity, gravity, command, joint positions, velocities, efforts, and accelerations, actions, foot contact flags, and foot heights. |
| Reward terms | Weight | Function |
| Task rewards | ||
| Linear velocity xy tracking | , where on parkour terrains and otherwise. | |
| Angular velocity z tracking | . | |
| Regularization rewards | ||
| Vertical velocity | . | |
| Roll/pitch angular velocity | . | |
| Term | Type | Condition |
|---|---|---|
| Episode length limit | Truncation | Episode length reaches , i.e., policy steps at . |
| Bad base orientation | Failure | , where is the vertical component of projected gravity in the base frame. |
| Base contact | Failure | The net contact force on robot base link exceeds . |
| D.R. Items | Range / setting |
|---|---|
| Foot friction | Static and dynamic friction in ; restitution in . |
| Base mass | Additive mass offset in . |
| Other body masses | Scale by . |
| Base center of mass | Offset each axis by . |
| Actuator gains | Scale stiffness and damping by . |
| Joint friction | Additive friction in . |
| Item | Range / setting |
|---|---|
| Camera translation | Uniform perturbation in per axis. |
| Camera rotation | Roll/yaw in ; pitch in . |
| Ray start point offsets | Positional in ; Rotational in per ray. |
| Sparse artifacts | Random depth artifacts with a probability of 0.0002. |
| Frame delay | sampling in simulation with – frame delay, i.e., – . |
| Algorithm 1: Teacher Policy Training |
|---|
| Obs proprioception , elevation map , privileged state , mask |
| /* Rollout collection */ |
| 1. for each rollout step do |
| 2. Sample terrain, command, and domain-randomized dynamics; |
| 3. if terrain is non-parkour then sample the exteroceptive mask; else keep perception unmasked; |
| 4. Compute , , and ; |
| Algorithm 2: Student Distillation |
|---|
| Obs proprioception , depth , elevation map , mask ; frozen teacher |
| /* Initialization */ |
| 1. Copy the teacher proprioceptive encoder and actor network; |
| 2. Freeze the student proprioceptive encoder; |
| /* Rollout and online teacher supervision */ |
| 3. for each rollout step do |
| Module | Specification |
|---|---|
| Proprioceptive encoder | MLP ; outputs from . The first three latent dimensions are supervised to estimate base linear velocity. |
| Elevation-map encoder | MLP ; outputs exteroceptive latent from elevation map . |
| Mask interface | Actor input is ; removes elevation-map information and keeps it. |
| Actor Network | MLP ; Gaussian policy with initial std. . |
| Critic | Pure MLP ; uses privileged state and elevation map. |
| Module | Specification |
|---|---|
| Proprioceptive branch | Copied from the teacher and frozen during distillation; outputs . |
| Depth encoder | CNN channels , kernels, stride , LayerNorm, ReLU; projection MLP gives visual feature . |
| GRU estimator | One-layer GRU with hidden size predicts from . |
| Uncertainty head | MLP predicts log-variance for the latent NLL loss. |
| Gate head | MLP predicts a sigmoid score . The actor receives a binary hysteresis mask: a closed gate opens at and an open gate closes below . |
| Actor network | Initialized from the teacher; input is and output is a -D Gaussian action. |
| Item | Teacher PPO | Student distillation |
| Rollout | ||
| Parallel envs | ||
| Horizon | steps/env | steps/env |
| Optimization | ||
| Optimizer | AdamW | AdamW |
| Learning rate | with adaptive KL schedule | |
| Condition | AUROC | AUPR | Prec. | Rec. | FPR | FNR |
|---|---|---|---|---|---|---|
| Seen, non-parkour | 0.999 | 0.999 | 0.996 | 0.969 | 0.004 | 0.031 |
| Unseen, non-parkour | 0.995 | 0.995 | 0.997 | 0.760 | 0.003 | 0.240 |
| Clean parkour | – | – | – | – | 0.004 | – |
| Flat | Stairs | |||||
|---|---|---|---|---|---|---|
| Corruption | 0.0 | 0.5 | 1.0 | 0.0 | 0.5 | 1.0 |
| Physical occlusion | 1.0 | 1.0 | 1.0 | 1.0 | 1.0 | 1.0 |
| Overexposure (flashlight) | 1.0 | 1.0 | 1.0 | 1.0 | 1.0 | 0.8 |
| Reflection (waving mirror) | 1.0 | 1.0 | 1.0 | 1.0 | 0.8 | 1.0 |
| Motion blur | 1.0 | 1.0 | 1.0 | 1.0 | 0.6 | 0.4 |
| Depth dropout | 1.0 | 1.0 | 1.0 | 1.0 | 1.0 | 1.0 |
Vision-Based Robot Control
Vision-based robot control uses camera images to guide robots’ navigation and manipulation through closed-loop perception and action.
See all 240 papers