Alignment Flywheel-Guided Suppression Patching for Interactionless Inverse Reinforcement Learning
Organizations: IDLab, Ghent University – imec, Ghent, Belgium
Abstract
Learned reward models allow reinforcement-learning agents to use demonstrations, implicit feedback, safety constraints, social norms, or other sources that do not provide a complete hand-designed scalar reward. However, a learned reward model can assign high reward to unwanted behavior. A policy can exploit these errors during training. Later requirements can also identify additional regions where the deployed reward should change. We introduce flywheel-guided patching, a mechanism for making such localized corrections without retraining the learned evaluator. An Alignment Flywheel identifies overgeneralized or constraint-violating regions and compiles verified findings into suppression patches. We instantiate the mechanism for Interactionless Inverse Reinforcement Learning using MoE-GUIDE, placing patches in the model's latent space and retrieving only a small local subset during online reward evaluation. In the main MuJoCo benchmark, we evaluate patching without expert-state reset warm starts. We also test whether operator-specified norms absent from the demonstrations can be compiled into the same reward interface and compare this with enforcing the corresponding rules online. The results show that localized corrective information can modify an existing learned reward without providing a complete replacement reward. Compilation can reuse judgments that are expensive or unavailable during control. Cheap deterministic rules may be simpler to check directly at each step. Code for MoE-GUIDE, Alignment Flywheel, and flywheel patching is at https://github.com/decide-ugent/Alignment-Flywheel.
Figures & tables
| Controlled quadruped – throughput at matched collisions | ||
| Collisions per agent | ||
| Compiled patch | ||
| Runtime, formal rule | ||
| Runtime, graded rule | ||
| MuJoCo sparse Ant | ||
| Method | Return | Any violation |
| Method | Swimmer | Hopper | Walker2d ‡ | HalfCheetah ‡ | Ant ‡ |
| Expert | 360 | 3,761 | 4,150 | 7,582 | 4,443 |
| SAC | |||||
| ER+RND | |||||
| ER+Disagreement | |||||
| f-IRL | |||||
| MoE-GUIDE |
| Experiment | Finding |
| Latent vs. raw-state patches | Latent-space ball covers use fewer centers than raw-state covers at coverage levels of and above. |
| Hashed retrieval | Suppression is evaluated only for retrieved candidates. Kernel evaluation cost depends on the number of candidates returned. |
| Constraint-targeted Ant | Patches reduce violations under intrinsic-reward-only evaluation. Violation reductions are weaker when environment reward is included. |
| Runtime OOB enforcement | Direct runtime enforcement produces larger violation reductions in Ant, but can also reduce task return. |
Appendix figures & tables42 assets
Supplementary material from the paper’s appendix.
Appendix
| Rewarded | Too close | Off the | On the | |
| space | to wall | path | left side | |
| OOD only | ||||
| All norms |
| Method | Return | Any violation | Hip velocity |
| Unpatched MoE-GUIDE | |||
| Combined kernel, medium | |||
| Hip-velocity kernel, very tight | |||
| OOB hard combined safety | |||
| OOB smooth combined safety |
| Environment | Experts | Expert type | |||
| Swimmer-v2 | 8 | 3 | 1 | 5 | Perfect |
| Hopper-v2 | 11 | 4 | 2 | 5 | Perfect |
| Walker2d-v2 | 17 | 7 | 4 | 5 | Imperfect |
| HalfCheetah-v2 | 17 | 7 | 4 | 5 | Imperfect |
| Ant-v2 | 111 | 10 | 3 | 5 | Imperfect |
| Environment | Steepness | Steps | ||||
| Swimmer-v2 | 0.2 | 0.11 | 25 | 1 | 1.0 | 500K |
| Hopper-v2 | 0.15 | 0.13 | 25 | 2 | 1.0 | 1M |
| Walker2d-v2 | 0.6 | 0.07 | 100 | 2 | 0.9999995 | 1M |
| HalfCheetah-v2 | 1.0 | 0.5 | 100 | 1 | 0.999995 | 1M |
| Ant-v2 | 0.1 | 0.003 | 100 | 1 | 0.9999995 | 1M |
| Parameter | Value |
| Actor/Critic hidden layers | |
| Actor/Critic activation | ReLU |
| Learning rate (all networks) | |
| Discount factor | 0.99 |
| Polyak averaging | 0.995 |
| Automatic entropy tuning | Yes |
| Environment | Tightness | Patch method |
| Swimmer-v2 | Loose | Regression |
| Hopper-v2 | Tight | Regression |
| Walker2d-v2 | Tight | Regression |
| HalfCheetah-v2 | (original) | (original) |
| Ant-v2 | (original) | (original) |
| Level | ||||
| very_loose | 0.70 | 0.0 | 0.50 | 4.0 |
| loose | 0.50 | 0.0 | 0.40 | 3.5 |
| medium | 0.30 | 0.0 | 0.30 | 3.0 |
| tight | 0.15 | 0.0 | 0.20 | 2.5 |
| very_tight | 0.05 | 0.0 | 0.15 | 2.0 |
| adaptive | 0.05 | 0.0 | 0.15 | 4.0 |
| Parameter | Value |
| Number of tables | 4 |
| Bits per table | auto-scaled, range 14–24 |
| Bit cap | |
| Gaussian floor | |
| Gaussian cutoff radius | |
| RNG seed | 42 |
| Bits | Brute (ms) | LSH (ms) | Speedup | |
| 100 | 14 | 1.1 | 0.3 | 3.7 |
| 1,000 | 14 | 12.4 | 1.2 | 10.1 |
| 10,000 | 18 | 175.2 | 2.5 | 71.4 |
| 100,000 | 21 | 1,420 | 6.2 | 228 |
| 1,000,000 | 24 | 19,227 | 15.2 | 1,263 |
| Brute (ms) | LSH (ms) | Brute/ | LSH/ | ||
| 1 | 10,000 | 158 | 4.8 | 158 | 4.8 |
| 4 | 40,000 | 858 | 15.6 | 215 | 3.9 |
| 16 | 160,000 | 4,615 | 108 | 288 | 6.7 |
| 32 | 320,000 | 8,384 | 215 | 262 | 6.7 |
| 64 | 640,000 | 17,318 | 482 | 271 | 7.5 |
| Dense BF | Top-2 BF | Dense LSH | Top-2 LSH | |||
| 2 | 502 | 518 | 13.0 | 13.5 | 103% | 104% |
| 4 | 1,257 | 971 | 28.2 | 28.8 | 77% | 102% |
| 8 | 2,048 | 1,467 | 55.6 | 24.2 | 72% | 44% |
| 16 | 4,319 | 822 | 120 | 36.3 | 19% | 30% |
| 32 | 8,479 | 334 | 216 | 33.6 | 3.9% | 16% |
| 64 | 18,090 | 287 | 432 | 36.9 | 1.6% | 8.5% |
| Method | Ant-v2 (400K) | HalfCheetah-v2 (600K) | Hopper-v2 (600K) | Walker2d-v2 (600K) |
| MoE-GUIDE | ||||
| FP cap200 | ||||
| FP cap400 | ||||
| FP cap600 | ||||
| FP cap800 | ||||
| FP cap1000 |
| Method | Swimmer | Hopper | Walker2d | HalfCheetah | Ant |
| MoE-GUIDE | |||||
| Very Loose | |||||
| Loose | |||||
| Medium | |||||
| Tight | |||||
| Very Tight |
| Method | Swimmer | Hopper | Walker2d | HalfCheetah | Ant |
| MoE-GUIDE | |||||
| Regression | |||||
| DBSCAN |
| Method | Swimmer | Hopper | Walker2d ‡ | HalfCheetah ‡ | Ant ‡ |
| MoE-GUIDE | |||||
| FK ( ) | |||||
| FK ( ) | |||||
| FK ( ) |
| Tightness | Margin | Ant ‡ | HalfCheetah ‡ |
| Tight | |||
| Tight | |||
| Tight | |||
| Medium | |||
| Loose |
| Coverage | BN | Raw | Raw@BN- | Ratio | CoD |
| 50% | 37 | 26 | 996 | 0.7 | 26.9 |
| 75% | 192 | 322 | 1496 | 1.7 | 7.8 |
| 90% | 492 | 622 | 1796 | 1.3 | 3.7 |
| 95% | 592 | 722 | 1896 | 1.2 | 3.2 |
| Coverage | BN | Raw | Raw@BN- | Ratio | CoD |
| 50% | 35 | 30 | 885 | 0.9 | 25.3 |
| 75% | 128 | 160 | 1385 | 1.2 | 10.8 |
| 90% | 340 | 436 | 1685 | 1.3 | 5.0 |
| 95% | 440 | 536 | 1785 | 1.2 | 4.1 |
| Coverage | BN | Raw | Raw@BN- | Ratio | CoD |
| 50% | 47 | 39 | 988 | 0.8 | 21.0 |
| 75% | 129 | 157 | 1488 | 1.2 | 11.5 |
| 90% | 284 | 402 | 1788 | 1.4 | 6.3 |
| 95% | 384 | 502 | 1888 | 1.3 | 4.9 |
| Coverage | BN | Raw | Raw@BN- | Ratio | CoD |
| 50% | 57 | 44 | 967 | 0.8 | 17.0 |
| 75% | 127 | 156 | 1467 | 1.2 | 11.6 |
| 90% | 227 | 397 | 1767 | 1.7 | 7.8 |
| 95% | 313 | 497 | 1867 | 1.6 | 6.0 |
| Coverage | BN | Raw | Raw@BN- | Ratio | CoD |
| 50% | 48 | 34 | 981 | 0.7 | 20.4 |
| 75% | 150 | 188 | 1481 | 1.3 | 9.9 |
| 90% | 385 | 488 | 1781 | 1.3 | 4.6 |
| 95% | 485 | 588 | 1881 | 1.2 | 3.9 |
| Coverage | BN | Raw | Raw@BN- | Ratio | CoD |
| 50% | 39 | 29 | 998 | 0.7 | 25.6 |
| 75% | 159 | 184 | 1498 | 1.2 | 9.4 |
| 90% | 408 | 484 | 1798 | 1.2 | 4.4 |
| 95% | 508 | 584 | 1898 | 1.1 | 3.7 |
| Env | BN radius | Raw radius | Avg neigh (BN) | Avg neigh (Raw@BN- ) | ||
| Ant | 111 | 10 | 1.22 | 4.60 | 21.1 | 1.0 |
| HalfCheetah | 17 | 7 | 0.96 | 5.35 | 23.3 | 1.2 |
| Hopper | 11 | 4 | 0.45 | 3.91 | 15.4 | 1.0 |
| Swimmer | 8 | 3 | 0.25 | 2.56 | 11.8 | 1.0 |
| Walker2d | 17 | 7 | 1.17 | 5.15 | 16.0 | 1.0 |
| AntMaze | 29 | 8 | 1.15 | 7.55 | 29.2 | 1.0 |
| Loss | Avg neigh | Final | Final cov. | Train (s) | |||||
| 2 | 0.0999 | 13.5 | 51 | 125 | 241 | 336 | 416 | 99% | 124 |
| 3 | 0.0430 | 17.7 | 47 | 136 | 311 | 411 | 491 | 99% | 127 |
| 5 | 0.0080 | 19.5 | 42 | 159 | 408 | 508 | 588 | 99% | 126 |
| 7 | 0.0032 | 29.1 | 35 | 188 | 487 | 587 | 667 | 99% | 139 |
| 10 | 0.0026 | 25.5 | 37 | 191 | 489 | 589 | 669 | 99% | 136 |
| 15 | 0.0017 | 26.4 | 37 | 242 | 542 | 642 | 722 | 99% | 152 |
| Method | 0–100K | 100–200K | 200–400K | 400–600K |
| SAC | 0.804 | 0.806 | 0.808 | 0.801 |
| MoE-GUIDE | 0.734 | 0.690 | 0.776 | 0.794 |
| Fp tight | 0.734 | 0.645 | 0.655 | 0.684 |
| CP: Combined | 0.785 | 0.773 | 0.785 | 0.822 |
| CP: Velocity | 0.770 | 0.763 | 0.806 | 0.820 |
| CP: Tilt | 0.792 | 0.780 | 0.813 | 0.839 |
| Method | Speed | Tilt | Hip vel |
| SAC | 0.001 | 0.430 | 0.534 |
| MoE-GUIDE | 0.027 | 0.473 | 0.469 |
| Fp tight | 0.014 | 0.375 | 0.460 |
| CP: Combined | 0.033 | 0.457 | 0.529 |
| CP: Velocity | 0.019 | 0.479 | 0.503 |
| CP: Tilt | 0.027 | 0.519 | 0.503 |
| Method | (any) | (speed) | (tilt) | (hip) |
| SAC | ||||
| MoE-GUIDE | ||||
| FP tight | ||||
| CP: Combined | ||||
| CP: Tilt | ||||
| OOB Hard+Comb. |
| Violated ( ) | Safe ( ) | |||||||
| Kernel | Patches | |||||||
| CP Combined | 1,063 | 0.633 | 0.137 | 0.539 | 21.4% | 0.000 | ||
| CP Velocity | 416 | 0.633 | 0.011 | 0.624 | 3.9% | 0.000 | ||
| CP Tilt | 735 | 0.633 | 0.027 | 0.608 | 7.2% | 0.000 | ||
| CP Hip Vel | 699 | 0.633 | 0.121 | 0.557 | 15.4% | 0.000 | ||
| FP cap1000 | 922 | 0.633 | 0.000 | 0.633 | 0.0% | 0.000 | ||
| Method | ||||
| SAC | 0.811 | 0.811 | 0.803 | 0.791 |
| MoE-GUIDE | 0.687 | 0.748 | 0.788 | 0.809 |
| FP tight | 0.698 | 0.614 | 0.607 | 0.667 |
| CP: Combined | 0.783 | 0.773 | 0.779 | 0.769 |
| CP: Tilt | 0.796 | 0.771 | 0.805 | 0.835 |
| OOB Hard+Comb. | 0.194 | 0.312 | 0.586 | 0.638 |
| Method | Steps | Final | Final Viol. |
| OOB Hard (no safety) | 354K (5) | 2379 | — |
| OOB Hard+Combined | 447K (3) | 3590 | 0.677 |
| OOB Hard+Velocity | 308K (5) | 3675 | 0.773 |
| OOB Hard+Tilt | 386K (5) | 3253 | 0.655 |
| OOB Hard+Hip Vel | 590K (5) | 2860 | 0.820 |
| OOB Smooth (no safety) | 316K (5) | 2094 | — |
| Pipeline stage | Latency (ms) | vs. Patches | Realtime? |
| MoE + 800 patches (brute) | 0.36 | all freqs | |
| MoE + 800 patches (LSH) | 0.40 | all freqs | |
| MobileNetV2 classify | 19.8 | 50 Hz only | |
| ResNet-50 classify | 62.4 | 15 Hz only | |
| KeypointRCNN (R50-FPN) | 5,772 | 1 Hz | |
| Classify + Keypoint | 5,864 | 1 Hz |
| Safety signal | Example | Latency | Realtime? |
| Formal logic | Bound check, velocity cap | ms | |
| Light neural net | MobileNetV2 classify | ms | Marginal |
| Heavy neural net | Keypoint + detection | ms | |
| Multi-model consensus | Ensemble + voting | ms | |
| Simulation rollout | Forward dynamics check | ms | |
| Human review | Norm / ethical judgment | minutes–hours |
| Key | Description |
| moe_state_dict | MoE model weights |
| moe_config | {input_dim, bottleneck_dim, num_experts} |
| estimator_config | {l_min, l_max, steepness} |
| dim_stats | Per-dimension {min, max, mean, std} tensors |
| dim_groups | Position and velocity dimension groups |
| flywheel_patches | Per-expert patch centers, bandwidths, and strengths |
Reward Design for RL
Reward design for reinforcement learning studies how to specify, learn, and adapt signals that guide agents toward intended behavior.
See all 290 papers