RAIL: Representation-Aligned Imitation Learning for Student-Compatible Teacher Policies
Organizations: Örebro University
Abstract
Reinforcement learning (RL) from raw sensory inputs such as images or onboard sensors can be challenging due to high-dimensional observations and sparse rewards. A common strategy is to train a teacher policy with access to privileged state information and then distill it into a student policy that acts from raw inputs alone. However, when the teacher relies on information unavailable to the student, exact imitation may be impossible, creating an irreducible imitation gap. Existing approaches typically mitigate this mismatch through careful reward shaping or additional reinforcement learning on the student policy after the teacher has been trained. Instead, we introduce Representation-Aligned Imitation Learning (RAIL), which addresses the mismatch during teacher training itself. RAIL learns a latent representation shared across teacher and student observations using contrastive learning, and trains the teacher policy directly in this space. Because the teacher policy is constrained to operate on this shared space, it learns behaviors that are reproducible from student observations. This substantially reduces the imitation gap while preserving task performance. Across multiple environments, RAIL outperforms strong baselines without reward modification or post-hoc student fine-tuning, and surpasses direct reinforcement learning from raw observations. The learned representations further enable zero-shot transfer to new tasks through teacher-only training. Project site: https://merajmammadov.com/rail/.
Figures & tables
| Corners | TunnelVision | CollectHealth | QuarterCheetah | |
| Task | Navigate from center to the hidden target corner (green). | Reach the goal (green) while avoiding obstacles (red). | Collect all health boxes in the room while avoiding walls. | Run forward as fast as possible. |
| Teacher obs. | Position, previous action, and the target corner identity. | Position, previous action, and occupancy of all neighboring cells (blue). | Position, previous action, and relative coordinates of all remaining boxes. | Full proprioceptive state (17D joint positions and velocities). |
| Student obs. | Position and previous action only; target identity is hidden. | Position, previous action, and three neighboring cells facing its current orientation (orange). | Stack of 5 limited field-of-view first-person RGB observations. | Third-person RGB observation with partial occlusion (left half is removed). |
| Action space | Move to any neighboring cells, . | Move or rotate to compass directions, . | Move in 8 directions or rotate , . | Continuous torques, . |
| Imitation gap | Teacher can move directly to the target, student must visit all corners. | Teacher can move directly to goal, student must turn to discover obstacles. | Teacher can move without turning, student must turn to discover next target. | Teacher has full state; student must infer dynamics and state from partial visual observations. |
| Corners | TunnelVision | CollectHealth | QuarterCheetah | |||||||||
| Method | ||||||||||||
| BC | 0.76 | 0.88 | 0.72 | 2186.83 82.47 | -268.17 29.02 | 2455.00 | ||||||
| DAgger | 0.62 | 0.85 | 0.16 | 2186.83 82.47 | -484.25 23.14 | 2671.08 | ||||||
| COSIL | 0.72 | 1.00 | 0.92 | 2186.83 82.47 | -308.04 14.46 | 2494.87 | ||||||
| SITT | 0.56 | 0.21 | 0.44 | 962.67 256.17 | 721.12 239.63 | 241.55 | ||||||
| RAIL (ours) | 0.00 | 0.00 | 0.01 | 2627.67 33.83 | 2510.24 72.05 | 117.43 | ||||||
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
| CornersPlus | TunnelVisionPlus | CollectHealthPlus | ||||
|---|---|---|---|---|---|---|
| Method | Task 2 | Task 1 2 | Task 2 | Task 1 2 | Task 2 | Task 1 2 |
| RAIL (student) | ||||||
| Student RL | – | – | – | |||
| Corners | TunnelVision | CollectHealth | QuarterCheetah | |
| training steps | 1.0M | 1.5M | 0.4M | 2.5M |
| parallel environments | 12 | 12 | 12 | 4 |
| PPO | ||||
| optimizer | Adam | Adam | Adam | Adam |
| learning rate | ||||
| batch size | 256 | 512 | 256 | 64 |
| Batch size | Corners | TunnelVision | CollectHealth | QuarterCheetah |
|---|---|---|---|---|
| 128 | ||||
| 256 | ||||
| 512 | ||||
| 1024 |
| Corners ∗ | TunnelVision ∗ | CollectHealth ∗ | QuarterCheetah ∗ | |||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Method | ||||||||||||
| BC | 0.00 | 0.00 | 0.00 | 2186.83 82.47 | 2113.43 57.39 | 73.40 | ||||||
| DAgger | 0.00 | 0.00 | 0.00 | 2186.83 82.47 | 2099.67 62.27 | 87.16 | ||||||
| COSIL | 0.00 | 0.00 | 0.00 | 2186.83 82.47 | 2090.52 68.50 | 96.31 | ||||||
| SITT | 0.00 | 0.00 | 0.00 | 1865.42 86.72 | 1720.43 74.45 | 144.99 | ||||||
Imitation Learning
Imitation learning trains agents to perform tasks by learning from expert demonstrations, feedback, or observed behavior.
See all 287 papers