RIFT: Reward Improvement Field Transfer for Cross-Domain Offline RL
Organizations: Massachusetts Institute of Technology, Cambridge, MA, USA
Abstract
Learning from limited target data is a central challenge in cross-domain offline reinforcement learning. Abundant source data can help, but dynamics differences can make direct transition reuse unreliable. We propose Reward Improvement Field Transfer (RIFT), which instead learns how low- and high-reward source transitions differ. RIFT encodes source and target transitions in a shared latent space, pairs low- and high-reward source transitions using optimal transport, and fits an affine field to their displacements. It applies this field to low-reward target anchors and decodes the results to augment the target dataset. The approach builds on the hypothesis that related domains can share useful low-to-high reward structure despite different dynamics. We bound synthesis error under shared class geometry, identify conditions favoring target anchoring over source-high transplant, and derive a conditional connection to downstream return. Experiments on dense-reward locomotion tasks under gravity, morphology, and friction shifts show improved aggregate performance over existing baselines.
Figures & tables
| Type | Source | Target | IQL | DARA | BOSA | SRPO | IGDF | OTDF | STC | RIFT (ours) |
|---|---|---|---|---|---|---|---|---|---|---|
| Gravity | half-med | med | ||||||||
| half-med-r | med | |||||||||
| half-med-e | med | |||||||||
| hopp-med | med | |||||||||
| hopp-med-r | med | |||||||||
| hopp-med-e | med |
Appendix figures & tables13 assets
Supplementary material from the paper’s appendix.
Appendix
| IQL (reported by STC) | IQL (ours) | |||||
| Shift | Environment | RIFT | ||||
| Gravity | Ant | |||||
| HalfCheetah | ||||||
| Hopper | ||||||
| Walker2d | ||||||
| Morphology | Ant | |||||
| Method | Hyperparameter | Value |
| Shared IQL backbone | ||
| Actor network | ||
| Critic network | ||
| Learning rate | ||
| Optimizer | Adam | |
| Discount factor | ||
| Type | Source | Target | IQL | DARA | BOSA | SRPO | IGDF | OTDF | STC | RIFT (ours) |
|---|---|---|---|---|---|---|---|---|---|---|
| Gravity | half-med | med | ||||||||
| half-med | med-e | |||||||||
| half-med | expert | |||||||||
| hopp-med | med | |||||||||
| hopp-med | med-e | |||||||||
| hopp-med | expert |
| Reward quantile | |||||||
| Shift | Task | 0 (no augmentation) | 0.1 | 0.2 (ours) | 0.3 | 0.4 | 0.5 |
| Gravity | Ant | ||||||
| HalfCheetah | |||||||
| Hopper | |||||||
| Walker2d | |||||||
| Morphology | Ant | ||||||
| Target transitions | IQL ∗ | RIFT | Relative difference |
|---|---|---|---|
| 500 | 257.6 | 245.6 | |
| 1,000 | 288.9 | 293.6 | |
| 2,500 | 336.8 | 398.6 | |
| 5,000 | 359.5 | 451.3 |
| Shift | Task | IQL ∗ | RIFT |
|---|---|---|---|
| Gravity | Ant | ||
| HalfCheetah | |||
| Hopper | |||
| Walker2d | |||
| Morphology | Ant | ||
| HalfCheetah |
| Method | ||
|---|---|---|
| IQL | 234.0 | 312.4 |
| SMOTE | 262.1 | 337.2 |
| RIFT | 280.5 | 392.3 |
| Shift | Task | Held-out decoded successor error | Generated successor error | Generated reward MAE (class gaps) | Generated realized reward |
|---|---|---|---|---|---|
| Gravity | Ant | 0.25 | 0.57 | 0.18 | 0.87 |
| HalfCheetah | 0.05 | 0.11 | 0.12 | 0.99 | |
| Hopper | 0.33 | 1.46 | 0.05 | 1.06 | |
| Walker2d | 0.65 | 7.07 | 0.08 | 0.80 | |
| Morphology | Ant | 0.16 | 0.29 | 0.14 | 0.91 |
| HalfCheetah | 0.05 | 0.08 | 0.05 | 1.08 |
| Task | VAE | Source | Target |
|---|---|---|---|
| HalfCheetah | base | ||
| bal | |||
| bal + reg | |||
| Walker2d | base | ||
| bal | |||
| bal + reg |
| Source | Target | IQL | DARA | BOSA | SRPO | IGDF | OTDF | STC | RIFT (ours) |
|---|---|---|---|---|---|---|---|---|---|
| halfcheetah-med | med | ||||||||
| halfcheetah-med | med-e | ||||||||
| halfcheetah-med | expert | ||||||||
| hopper-med | med | ||||||||
| hopper-med | med-e | ||||||||
| hopper-med | expert |
| Source | Target | DmC | RIFT (ours) |
|---|---|---|---|
| half-med | med | ||
| half-med-r | med | ||
| half-med-e | med | ||
| hopp-med | med | ||
| hopp-med-r | med | ||
| hopp-med-e | med |
| Component | Specification |
|---|---|
| CPU | AMD EPYC 9474F / 9654 |
| Cores | up to 96 in parallel |
| Memory | 4 GB per run |
Offline RL
Offline reinforcement learning learns decision-making policies from fixed datasets without interacting with the environment during training.
See all 328 papers