cs.LGOct 31, 2026

Alignment Flywheel-Guided Suppression Patching for Interactionless Inverse Reinforcement Learning

Authors: Elias Malomgré, Pieter Simoens

Organizations: IDLab, Ghent University – imec, Ghent, Belgium

Abstract

Learned reward models allow reinforcement-learning agents to use demonstrations, implicit feedback, safety constraints, social norms, or other sources that do not provide a complete hand-designed scalar reward. However, a learned reward model can assign high reward to unwanted behavior. A policy can exploit these errors during training. Later requirements can also identify additional regions where the deployed reward should change. We introduce flywheel-guided patching, a mechanism for making such localized corrections without retraining the learned evaluator. An Alignment Flywheel identifies overgeneralized or constraint-violating regions and compiles verified findings into suppression patches. We instantiate the mechanism for Interactionless Inverse Reinforcement Learning using MoE-GUIDE, placing patches in the model's latent space and retrieving only a small local subset during online reward evaluation. In the main MuJoCo benchmark, we evaluate patching without expert-state reset warm starts. We also test whether operator-specified norms absent from the demonstrations can be compiled into the same reward interface and compare this with enforcing the corresponding rules online. The results show that localized corrective information can modify an existing learned reward without providing a complete replacement reward. Compilation can reuse judgments that are expensive or unavailable during control. Cheap deterministic rules may be simpler to check directly at each step. Code for MoE-GUIDE, Alignment Flywheel, and flywheel patching is at https://github.com/decide-ugent/Alignment-Flywheel.

Figures & tables

Appendix figures & tables42 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. Interactionless Inverse Reinforcement Learning: A Data-Centric Framework for Durable Alignment

    Feb 16, 2026Elias Malomgré, Pieter SimoensAI AlignmentReinforcement Learning

  2. Inverse RL Helps Align AI by Imitating Humans

    Jul 27, 2026Michał Wiliński, Liu Leqi, Chirag NagpalRL for Language ModelsLLM Alignment

  3. IR3^3: Contrastive Inverse Reinforcement Learning for Interpretable Detection and Mitigation of Reward Hacking

    Feb 23, 2026Mohammad Beigi, Ming Jin, Junshan Zhang +3Reward HackingReinforcement Learning