Mitigating Covariate Shift in Behavioral Cloning via Robust Stationary Distribution Correction
Organizations: KAIST · Korea University · Gauss Labs Inc. · UC Berkeley
Abstract
We consider offline imitation learning (IL), which aims to train an agent to imitate from the dataset of expert demonstrations without online interaction with the environment. Behavioral Cloning (BC) has been a simple yet effective approach to offline IL, but it is also well-known to be vulnerable to the covariate shift resulting from the mismatch between the state distributions induced by the learned policy and the expert policy. Moreover, as often occurs in practice, when expert datasets are collected from an arbitrary state distribution instead of a stationary one, these shifts become more pronounced, potentially leading to substantial failures in existing IL methods. Specifically, we focus on covariate shift resulting from arbitrary state data distributions, such as biased data collection or incomplete trajectories, rather than shifts induced by changes in dynamics or noisy expert actions. In this paper, to mitigate the effect of the covariate shifts in BC, we propose DrilDICE, which utilizes a distributionally robust BC objective by employing a stationary distribution correction ratio estimation (DICE) to derive a feasible solution. We evaluate the effectiveness of our method through an extensive set of experiments covering diverse covariate shift scenarios. The results demonstrate the efficacy of the proposed approach in improving the robustness against the shifts, outperforming existing offline IL methods in such scenarios.
Figures & tables
Appendix figures & tables8 assets
Supplementary material from the paper’s appendix.
Appendix
| Divergence | |||
|---|---|---|---|
| KL Divergence | |||
| -Divergence | |||
| Soft -Divergence | |||
| TV-Distance | - | - | |
| Soft TV-Distance |
| Rooms | Room 1 | Room 2 | Room 3 | Room 4 |
|---|---|---|---|---|
| 0.342 | 0.291 | 0.089 | 0.278 | |
| Actions | "UP" | "DOWN" | "LEFT" | "RIGHT" |
| 0.080 | 0.453 | 0.016 | 0.451 |
| Problem setting | hopper | walker2d | halfcheetah |
|---|---|---|---|
| Scenario 1 | 100 | 100 | 250 |
| Scenario 2 | 100 | 100 | 250 |
| Scenario 3 | [10,20,30,40,50] | [10,20,30,40,50] | [50,100,150,200,250] |
Imitation Learning
Imitation learning trains agents to perform tasks by learning from expert demonstrations, feedback, or observed behavior.
See all 287 papers