Hindsight Relabeling is All You Need for Reach-Avoid Learning
Organizations: Massachusetts Institute of Technology · Harvard Medical School · Kempner Institute at Harvard University
Abstract
Offline goal-conditioned reinforcement learning methods have shown promise for reach-avoid tasks, where an agent must reach a target state while avoiding undesirable regions of the state space. Existing approaches typically encode avoid-region information into an augmented state space and cost function, which prevents flexible, dynamic specification of novel avoid-region information at evaluation time. They also rely heavily on meticulously designed reward and cost functions, limiting transferability to novel environments and specifications. We introduce RADT, a decision transformer approach for offline, reward-free, goal-conditioned, avoid-region-conditioned RL. RADT encodes goals and avoid-regions directly as prompt tokens, allowing any number of avoid-regions of arbitrary size to be specified at evaluation time. Using only suboptimal offline trajectories from a random policy, RADT can learn reach-avoid behavior in a completely data-driven manner using our novel avoid-region hindsight relabeling approach, without the need for reward/cost supervision. We benchmark RADT against existing offline goal-conditioned RL models across 17 tasks, environments, and experimental settings. RADT generalizes in a zero-shot manner to out-of-distribution avoid-region sizes and counts, outperforming baselines that require retraining. In one such zero-shot setting, RADT achieves 35.7% improvement in normalized cost over the best retrained baseline while maintaining high goal-reaching success. We also apply RADT to cell reprogramming in biology, demonstrating its versatility.
Figures & tables
| Box Width | AM-Lag | RbSL | WGCSL | |||||
| MNC | SR | MNC | SR | MNC | SR | MNC | SR | |
| 0.16 | ||||||||
| 0.18 | ||||||||
| 0.20 | ||||||||
| 0.22 | ||||||||
| 0.24 | ||||||||
| UMazeObstacle | ||||||||
| # Avoid | AM-Lag | RbSL | WGCSL | |||||
| MNC (1e-2) | SR | MNC (1e-2) | SR | MNC (1e-2) | SR | MNC (1e-2) | SR | |
| 1 | ||||||||
| 3 | ||||||||
| 4 | ||||||||
| 5 | ||||||||
Appendix figures & tables23 assets
Supplementary material from the paper’s appendix.
Appendix
| Hyperparameter | Description | Search space |
| n_head | Number of attention heads | tune.choice([1, 3, 4, 6, 12]) |
| n_layer | Number of self-attention layers | tune.choice([3, 6, 12]) |
| embed_dim | Size of latent embedding space (hidden dimension) | Fixed to be 64 * n_head |
| adelta | Attention boosting bias to the prompt | tune.choice([0, 1, 2, 3]) |
| Hyperparameter | Description | Search space |
| batch_size | Batch size | tune.choice([32, 64, 128, 256]) |
| learning_rate | Maximum learning rate | tune.choice([1e-4, 1e-5]) |
| scheduler | Learning rate scheduler | tune.choice([‘cosinewarmrestarts’, ‘lambdalr’]) |
| warmup_steps | Number of warmup steps | tune.choice([500, 1000]) |
| T_0 | T_0 parameter to CosineAnnealingWarmupRestarts | Fixed to be 1000 |
| T_mult | T_mult parameter to CosineAnnealingWarmupRestarts | Fixed to be 1 |
| Initial state | Most visited intermediate | Examples |
| 000000000000000 | 100000000001100 | - avoid prompt : (000000000000000, 100000000001100, FHF) + avoid prompt : (000000000000000, 000011010100000, 000010010100000, FHF) |
| 010111111010011 | 001111011011111 | - avoid prompt : (010111111010011, 001111011011111, FHF) (10111111010011, 010111111010011, 001111011011111, ...000111010011111, 001111011011111, 100000000001100, FHF) (010111111010011, 001111011011111, 000111010011111, 001111011011111, FHF) + avoid prompt : (010111111010011, 000000000000011, 010010010100000, ...010111111010011, 000010010100000, FHF) (010111111010011, 000010010100000, 010111111010011, 000010010000000, FHF) |
| 000010010100000 | 000010010100000 | - avoid prompt : (000010010100000, 000010010100000, 000010010100000, ...000010010100000, 000010010100000, FHF) + avoid prompt : (000010010100000, 000010010100000, 000010010100000, FHF) |
| +SI | -SI, full data | -SI, successes only | |
| MNC | |||
| SR |
| Environment | +AS,GS | -AS,GS | ||
| MNC | SR | MNC | SR | |
| FetchReachObstacle-BoxWidth0.16 | ||||
| UMazeObstacle-NumAvoid3 | ||||
| UMazeObstacle-NumAvoid5 | ||||
| UMazeObstacle-NumAvoid7 | ||||
| Box width | +AR token | -whole AR tokens | -AR size component | |||
| MNC | SR | MNC | SR | MNC | SR | |
| 0.16 | ||||||
| 0.18 | ||||||
| 0.2 | ||||||
| 0.22 | ||||||
| 0.24 | ||||||
| # of avoid-regions | AM-Lag | |||
| MNC (1e-2) | SR | MNC (1e-2) | SR | |
| 10 | ||||
| 15 | ||||
| 20 | ||||
| Expert % | 0 | 10 | 20 | 30 | ||||
| Box Width | MNC | SR | MNC | SR | MNC | SR | MNC | SR |
| 0.16 | ||||||||
| 0.18 | ||||||||
| 0.20 | ||||||||
| 0.22 | ||||||||
| 0.24 | ||||||||
| Expert % | 0 | 10 | 20 | 30 | ||||
| Box Width | MNC | SR | MNC | SR | MNC | SR | MNC | SR |
| 0.16 | ||||||||
| 0.20 | ||||||||
| 0.24 | ||||||||
| # Avoid | AM-Lag | |||||
| MNC (1e-2) | SR | MNC (1e-2) | SR | MNC (1e-2) | SR | |
| 3 | ||||||
| 5 | ||||||
| 7 | ||||||
| Box Width | + avoid-region relabeling | - avoid-region relabeling | ||
| MNC | SR | MNC | SR | |
| 0.16 | ||||
| 0.18 | ||||
| 0.2 | ||||
| 0.22 | ||||
| 0.24 | ||||
| RADT | AM-Lag | RbSL | |
| MNC (1e-2) | |||
| SR |
| RADT | AM-Lag | RbSL | WGCSL | |
| MNC (1e-2) | ||||
| SR |
| Scenario + setup | discretized AM-Lag | dicretized RbSL | ||||
| % visitation | mean length | SR | % visitation | mean length | SR | |
| Scenario 1 -avoid | ||||||
| Scenario 1 +avoid | ||||||
| Scenario 2 -avoid | ||||||
| Scenario 2 +avoid | ||||||
| Scenario 3 -avoid | ||||||
| # Avoid | AM-Lag | RbSL | |
| 1 | |||
| 3 | |||
| 5 | |||
| 7 |
Goal-Conditioned RL
Goal-conditioned reinforcement learning trains agents to reach specified states or outcomes, enabling one policy to solve many tasks.
See all 98 papers