Imitating the Imperfect: Offline-to-Online Robust Imitation Learning from Heterogeneous Demonstrators
Organizations: Swiss Federal Institute of Technology in Lausanne (EPFL), Lausanne, Switzerland. · University of Bristol, Bristol, UK. · Araya Inc., Tokyo, Japan.
Abstract
Imitation learning (IL) typically relies on large-scale offline demonstrations collected from multiple human or algorithmic demonstrators. However, most existing approaches assume these demonstrators are homogeneous or near-optimal experts, a convenient but unrealistic assumption in real-world applications. In practice, demonstrations are often collected from heterogeneous demonstrators with unknown and varying levels of expertise, resulting in highly mixed-quality data. In this work, we study a challenging yet practical setting of imitation learning from such heterogeneous imperfect demonstrators. We propose Latent Expertise and Optimality Scoring based Imitation Learning (LEOS-IL), a principled framework that jointly learns (i) a state-action optimality scoring model and (ii) latent expertise levels for each demonstrator from unlabeled suboptimal offline demonstrations. The learned scoring model is then integrated into an online policy optimization procedure, where the agent is trained to maximize estimated optimality scores while its online rollouts are iteratively leveraged to refine the scoring model, thereby mitigating distributional shift during online adaptation. We further provide theoretical guarantees for optimality score identification and conditional optimal occupancy and policy recovery under explicit sufficient conditions. Experiments on continuous control benchmarks demonstrate that our approach consistently outperforms existing baselines across highly heterogeneous and low-quality demonstration regimes.
Figures & tables
| Method | BC | GAIL | RIL | IC-GAIL | UID-GAIL | RCE | WGAIL | ILEED | Ours |
| Non-dominant optimal demo | ✗ | ✗ | ✗ | ✓ | ✓ | ✗ | ✗ | ✓ | ✓ |
| Multi-demo support | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ | ✓ | ✓ | ✓ |
| Handles structured suboptimality | ✗ | ✗ | ✓ | ✓ | ✓ | ✗ | ✓ | ✗ | ✓ |
| No optimality labels for training | ✗ | ✗ | ✓ | ✗ | ✓ | ✗ | ✓ | ✓ | ✓ |
| Probabilistic demo scoring | ✗ | ✓ | ✓ | ✗ | ✓ | ✓ | ✗ | ✗ | ✓ |
| Interactive IL | ✗ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✗ | ✓ |
| Environment | Dataset | GAIL o.s. | BC | ILEED | GAIL | RIL | WGAIL | LEOS-IL (ours) |
| General expertise test | ||||||||
| Ant-v4 | 4016.3 2776.8 | 5660.4 163.6 | 1341.8 164.5 | 1428.2 80.3 | 1387.2 301.0 | 101.8 118.0 | 1744.0 919.1 | 6084.6 146.9 |
| HalfCheetah-v4 | 9988.9 5959.9 | 9278.2 496.5 | 2318.1 521.2 | 1526.2 1020.4 | 3035.8 1186.4 | 1631.6 959.7 | 4336.2 335.6 | 8751.2 1797.9 |
| Swimmer-v4 | 190.7 149.1 | 293.6 28.2 | 21.3 23.0 | 16.8 20.5 | 76.6 28.3 | 28.8 16.3 | 115.6 83.7 | 305.4 15.7 |
| Walker2d-v4 | 3096.7 2549.6 | 3703.0 1011.9 | 308.4 111.5 | 440.3 313.0 | 948.6 297.2 | 1187.2 130.1 | 665.4 330.2 | 3959.2 671.4 |
| Low expertise test | ||||||||
| Experiment | Ant-v4 | HalfCheetah-v4 | Hopper-v4 | Swimmer-v4 | Walker2d-v4 |
| General expertise test | |||||
| Low expertise test |
| Test | Stage | Ant-v4 | Impr. | HalfCheetah-v4 | Impr. | Swimmer-v4 | Impr. | Walker2d-v4 | Impr. |
| General | 1 | 99.91 / 99.89 | - | 100.00 / 100.00 | - | 100.00 / 100.00 | - | 99.92 / 99.90 | - |
| expertise | 2 | 99.89 / 99.89 | -0.02 / 0.00 | 100.00 / 100.00 | 0.00 / 0.00 | 99.94 / 99.94 | -0.06 / -0.06 | 99.94 / 99.94 | 0.02 / 0.04 |
| Low | 1 | 90.33 / 51.78 | - | 96.33 / 73.68 | - | 97.19 / 78.84 | - | 86.39 / 43.01 | - |
| expertise | 2 | 99.65 / 98.23 | 10.32 / 89.71 | 99.98 / 99.91 | 3.79 / 35.60 | 97.58 / 87.91 | 0.40 / 11.50 | 98.48 / 92.40 | 13.99 / 114.83 |
Appendix figures & tables19 assets
Supplementary material from the paper’s appendix.
Appendix
| Environment | Optimal policy | Suboptimal policy |
| Ant-v4 | ||
| HalfCheetah-v4 | ||
| Swimmer-v4 | ||
| Walker-v4 |
| Demonstrator | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 |
| General expertise test | ||||||||
| 2 demonstrators | 0.100 | 0.900 | – | – | – | – | – | – |
| 4 demonstrators | 0.100 | 0.366 | 0.633 | 0.900 | – | – | – | – |
| 6 demonstrators | 0.100 | 0.260 | 0.420 | 0.580 | 0.740 | 0.900 | – | – |
| 8 demonstrators | 0.100 | 0.214 | 0.329 | 0.443 | 0.557 | 0.671 | 0.786 | 0.900 |
| Low expertise test | ||||||||
| Demonstrator | 1 | 2 | 3 | 4 | 5 | 6 | Mean error std |
| General expertise test | |||||||
| Ant-v4 | 0.100 | 0.260 | 0.420 | 0.580 | 0.740 | 0.900 | |
| HalfCheetah-v4 | 0.100 | 0.260 | 0.420 | 0.580 | 0.740 | 0.900 | |
| Hopper-v4 | 0.101 | 0.259 | 0.418 | 0.575 | 0.734 | 0.892 | |
| Swimmer-v4 | 0.100 | 0.260 | 0.420 | 0.580 | 0.740 | 0.900 | |
| Walker2d-v4 | 0.100 | 0.260 | 0.420 | 0.581 | 0.741 | 0.900 | |
| Environment | Stage | TP | TN | FP | FN | Accuracy / Precision | Improvement (%) |
| General expertise test | |||||||
| Ant-v4 | 1 | 14988.6 3.9 | 14984.2 10.1 | 15.8 10.1 | 11.4 3.9 | 0.9991 / 0.9989 | - |
| 2 | 14984.0 5.5 | 14984.0 5.5 | 16.0 5.5 | 16.0 5.5 | 0.9989 / 0.9989 | -0.02 / 0.00 | |
| HalfCheetah-v4 | 1 | 15000.0 0.0 | 14999.4 0.8 | 0.6 0.8 | 0.0 0.0 | 1.0000 / 1.0000 | - |
| 2 | 15000.0 0.0 | 15000.0 0.0 | 0.0 0.0 | 0.0 0.0 | 1.0000 / 1.0000 | 0.00 / 0.00 | |
| Swimmer-v4 | 1 | 15000.0 0.0 | 14999.4 1.2 | 0.6 1.2 | 0.0 0.0 | 1.0000 / 1.0000 | - |
| Method | Expertise [0.1 – 0.9] | Expertise [0.05 – 0.15] |
| LEOS-IL (ours) | 6084.6 146.9 | 5932.0 335.2 |
| Scoring (no relabel) | 5593.0 564.2 | 3316.2 1207.8 |
| Scoring (no early stop) | 2129.8 2205.8 | 1018.2 250.0 |
| Top-k fraction | Expertise [0.1 – 0.9] | Expertise [0.05 – 0.15] |
| 0.05 | – | 5887.6 263.8 |
| 0.1 | 5403.8 378.7 | 5932.0 335.2 |
| 0.15 | – | 3663.4 1950.9 |
| 0.2 | – | 1756.6 902.6 |
| 0.3 | 5809.0 274.8 | – |
| 0.5 | 6084.6 146.9 | – |
| Demonstrator | Type | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | Mean error std |
| General expertise test | ||||||||||
| 2 demonstrators | Est. | 0.101 | 0.899 | - | - | - | - | - | - | |
| True | 0.100 | 0.900 | - | - | - | - | - | - | - | |
| 4 demonstrators | Est. | 0.100 | 0.366 | 0.633 | 0.899 | - | - | - | - | |
| True | 0.100 | 0.367 | 0.633 | 0.900 | - | - | - | - | - | |
| 6 demonstrators | Est. | 0.100 | 0.260 | 0.420 | 0.580 | 0.740 | 0.900 | - | - | |
| Number of Demonstrators | Expertise [0.1 – 0.9] | Expertise [0.05 – 0.15] |
| 2 | 5634.0 617.8 | 6050.6 120.4 |
| 4 | 5934.0 264.3 | 5637.2 772.5 |
| 6 | 6084.6 146.9 | 5932.0 335.2 |
| 8 | 5735.0 225.6 | 6146.6 107.2 |
| Test | Stage | Query 1 | Impr. | Query 3 | Impr. | Query 5 | Impr. | Query 7 | Impr. | Query 9 | Impr. |
| General | 1 | 99.93 / 99.97 | - | 99.92 / 99.94 | - | 99.91 / 99.89 | - | 99.92 / 99.96 | - | 99.94 / 99.95 | - |
| Expertise | 2 | 99.92 / 99.92 | -0.01 / -0.05 | 99.90 / 99.90 | -0.02 / -0.04 | 99.89 / 99.89 | -0.02 / 0.00 | 99.89 / 99.89 | -0.03 / -0.07 | 99.91 / 99.91 | -0.03 / -0.04 |
| Low | 1 | 96.26 / 75.31 | - | 97.33 / 80.17 | - | 90.33 / 51.78 | - | 97.02 / 78.81 | - | 97.14 / 79.26 | - |
| Expertise | 2 | 99.64 / 98.22 | 3.51 / 30.42 | 99.52 / 97.60 | 2.25 / 21.74 | 99.65 / 98.23 | 10.32 / 89.71 | 99.48 / 97.40 | 2.54 / 23.59 | 99.73 / 98.65 | 2.67 / 24.46 |
| Query times | Expertise [0.1 – 0.9] | Expertise [0.05 – 0.15] |
| 1 | 5433.6 365.6 | 5810.2 301.3 |
| 3 | 5655.2 307.3 | 6011.2 219.9 |
| 5 | 6084.6 146.9 | 5932.0 335.2 |
| 7 | 5981.4 264.6 | 5944.2 279.8 |
| 9 | 5881.6 265.3 | 6130.8 126.8 |
| General expertise level | Low expertise level | |||
| Accuracy | Precision | Accuracy | Precision | |
| 0.1 | 59.97 19.94 | 20.00 39.99 | 90.00 0.00 | 0.00 0.00 |
| 0.2 | 89.94 19.97 | 79.97 39.99 | 90.00 0.00 | 0.00 0.00 |
| 0.3 | 99.92 0.02 | 99.96 0.03 | - | - |
| 0.4 | 99.94 0.01 | 99.98 0.02 | 90.00 0.00 | 0.00 0.00 |
| 0.45 | - | - | 96.44 2.43 | 88.46 11.92 |
| value | Expertise [0.1 – 0.9] | Expertise [0.05 – 0.15] |
| 0.1 | 1149.4 2298.8 | – |
| 0.2 | 3700.6 2286.5 | – |
| 0.3 | 5241.6 957.9 | – |
| 0.4 | 5308.0 377.7 | – |
| 0.45 | – | 5509.0 493.4 |
| 0.5 | 6084.6 146.9 | 5932.0 335.2 |
| Env | General expertise (Mean ± Std) | Low expertise (Mean ± Std) |
| Ant-v4 | ||
| HalfCheetah-v4 | ||
| Swimmer-v4 | ||
| Walker2d-v4 |
| Environment | BC (best policy) | ILEED (best policy) | BC (last 5 policies mean) | ILEED (last 5 policies mean) | LEOS-IL (ours), (last 5 policies mean) |
| General expertise test | |||||
| Ant-v4 | 2088.5 42.0 | 2188.6 70.3 | 1341.8 164.5 | 1428.2 80.3 | 6084.6 146.9 |
| HalfCheetah-v4 | 3352.5 214.2 | 3545.7 173.0 | 2318.1 521.2 | 1526.2 1020.4 | 8751.2 1797.9 |
| Swimmer-v4 | 329.4 17.7 | 344.6 1.4 | 21.3 23.0 | 16.8 20.5 | 305.4 15.7 |
| Walker2d-v4 | 1472.7 127.9 | 1439.2 97.2 | 308.4 111.5 | 440.3 313.0 | 3959.2 671.4 |
| Low expertise test | |||||