Learned reward models allow reinforcement-learning agents to use demonstrations, implicit feedback, safety constraints, social norms, or other sources that do not provide a complete hand-designed scalar reward. However, a learned reward model can assign high reward to unwanted behavior. A policy can exploit these errors during training. Later requirements can also identify additional regions where the deployed reward should change. We introduce flywheel-guided patching, a mechanism for making such localized corrections without retraining the learned evaluator. An Alignment Flywheel identifies overgeneralized or constraint-violating regions and compiles verified findings into suppression patches. We instantiate the mechanism for Interactionless Inverse Reinforcement Learning using MoE-GUIDE, placing patches in the model's latent space and retrieving only a small local subset during online reward evaluation. In the main MuJoCo benchmark, we evaluate patching without expert-state reset warm starts. We also test whether operator-specified norms absent from the demonstrations can be compiled into the same reward interface and compare this with enforcing the corresponding rules online. The results show that localized corrective information can modify an existing learned reward without providing a complete replacement reward. Compilation can reuse judgments that are expensive or unavailable during control. Cheap deterministic rules may be simpler to check directly at each step. Code for MoE-GUIDE, Alignment Flywheel, and flywheel patching is at https://github.com/decide-ugent/Alignment-Flywheel.
Figures & tables
Figure 1: Flywheel-guided patching. Demonstrations train a base IIRL reward artifact. The Alignment Flywheel uses external judgments to identify regions where reward should be reduced, either to address overgeneralization or to incorporate new requirements. Verified corrections are compiled into local suppression patches in latent space. These patches modify the reward mapping without retraining the learned evaluator.
Controlled quadruped – throughput at matched collisions
Collisions per agent
0.90
0.60
Compiled patch
3.83
3.62
Runtime, formal rule
3.03
2.87
Runtime, graded rule
3.63
3.29
MuJoCo sparse Ant
Method
Return
Any violation
Table 1: Compiled patches versus online enforcement of the same rule, in both settings. Upper: controlled quadruped, task throughput at matched compliance ( 8 agents, 8 seeds), read off each swept front by interpolation. Lower: MuJoCo sparse Ant at the last available checkpoint region, five seeds, expert-state reset curriculum retained. Higher return is better; lower violation is better.
Method
Swimmer
Hopper
Walker2d ‡
HalfCheetah ‡
Ant ‡
Expert
360
3,761
4,150
7,582
4,443
SAC
78±15
3459±127
4353±916
10125±657
2807±1149
ER+RND
98±29
3181±330
4450±778
10492±529
4018±771
ER+Disagreement
71±11
3240±261
4445±466
10359±282
3812±866
f-IRL
329±17
3464±91
4580±474
10996±496
3454±663
MoE-GUIDE
342±2
3540±86
4146±877
10646±657
3974±1099
Table 2: Mean undiscounted evaluation return ± standard deviation across five seeds, with 10 evaluation episodes averaged per seed. Swimmer is evaluated at 500K steps; all other environments at 1M steps. Bold marks the highest sample mean among the compared learning methods, excluding the expert reference. Methods differ in their use of demonstrations, exploration bonuses, and corrective checks. ‡ Imperfect expert demonstrations.
Figure 2: Learning curves accompanying the main results in Table 2
Experiment
Finding
Latent vs. raw-state patches
Latent-space ball covers use fewer centers than raw-state covers at coverage levels of 75% and above.
Hashed retrieval
Suppression is evaluated only for retrieved candidates. Kernel evaluation cost depends on the number of candidates returned.
Constraint-targeted Ant
Patches reduce violations under intrinsic-reward-only evaluation. Violation reductions are weaker when environment reward is included.
Runtime OOB enforcement
Direct runtime enforcement produces larger violation reductions in Ant, but can also reduce task return.
Table 3: Auxiliary findings in the tested settings. Detailed results are reported in Appendices 0.E – 0.H .
Appendix figures & tables42 assets
Supplementary material from the paper’s appendix.
Appendix
Rewarded
Too close
Off the
On the
space
to wall
path
left side
OOD only
100.0%
18.9%
4.1%
45.5%
All norms
73.5%
6.0%
0.0%
35.3%
Appendix
Table 4: What the artifact pays for, before and after compiling the four norms, over a grid of free space at the demonstrated speed. Percentages are over the states the model still rewards, so the columns measure what the reward function endorses rather than how much space it covers.
Figure 3: Compliance against task throughput for each norm, 8 agents, 8 seeds per point. Each point is one enforcement weight applied to identical compiled patches; larger markers are Pareto-optimal.
Figure 4: Unguided greedy-probe success against training episodes, 8 seeds, band =1 s.d. Dashed grey is the control: identical patches, enforcement off. Moderate enforcement of the agent-distance norm raises the ceiling; the three positional norms cost sample efficiency monotonically in weight.
Figure 5: Average forward trajectory per condition over the faded individual runs, with the distribution of lane position beside it. Only forward-running agents are drawn: “right” is relative to heading, so drawing both directions of two-way traffic places compliant traffic against both walls and cancels the effect visually. Both conditions compile identical patches and differ only in enforcement weight.
Figure 6: The average route coloured by mean distance to the nearest other agent. Red marks where agents habitually bunch up, which is where collisions come from; enforcement shifts the whole route away from the crowded end.
Method
Return
Any violation
Hip velocity
Unpatched MoE-GUIDE
2.89
88.4%
56.3%
Combined kernel, medium
5.22
80.2%
45.4%
Hip-velocity kernel, very tight
3.30
70.4%
41.4%
OOB hard + combined safety
0.20
1.9%
12.9%
OOB smooth + combined safety
1.72
1.6%
13.5%
Appendix
Table 5: Sparse Ant at the last available checkpoint region. Descriptive means over five-seed trajectories; no standard deviations or confidence intervals.
Environment
dobs
dbottleneck
Experts
Ntraj
Expert type
Swimmer-v2
8
3
1
5
Perfect
Hopper-v2
11
4
2
5
Perfect
Walker2d-v2
17
7
4
5
Imperfect
HalfCheetah-v2
17
7
4
5
Imperfect
Ant-v2
111
10
3
5
Imperfect
Appendix
Table 6: MoE reward model architecture per environment.
Environment
Lmax
Lmin
Steepness s
κ
κdecay
Steps
Swimmer-v2
0.2
0.11
25
1
1.0
500K
Hopper-v2
0.15
0.13
25
2
1.0
1M
Walker2d-v2
0.6
0.07
100
2
0.9999995
1M
HalfCheetah-v2
1.0
0.5
100
1
0.999995
1M
Ant-v2
0.1
0.003
100
1
0.9999995
1M
Appendix
Table 7: Intrinsic reward mapping parameters per environment. These are the best hyperparameters selected via the Lmax sweep.
Parameter
Value
Actor/Critic hidden layers
[256,256]
Actor/Critic activation
ReLU
Learning rate (all networks)
3×10−4
Discount factor γ
0.99
Polyak averaging τ
0.995
Automatic entropy tuning
Yes
Appendix
Table 8: SAC hyperparameters, shared across all environments.
Environment
Tightness
Patch method
Swimmer-v2
Loose
Regression
Hopper-v2
Tight
Regression
Walker2d-v2
Tight
Regression
HalfCheetah-v2
(original)
(original)
Ant-v2
(original)
(original)
Appendix
Table 9: Flywheel Patching variant used per environment in Table 2 .
Level
τfloor
Δmax
σmin
σmax
very_loose
0.70
0.0
0.50
4.0
loose
0.50
0.0
0.40
3.5
medium
0.30
0.0
0.30
3.0
tight
0.15
0.0
0.20
2.5
very_tight
0.05
0.0
0.15
2.0
adaptive
0.05
0.0
0.15
4.0
Appendix
Table 10: Strictness parameters for the Flywheel governance loop.
Parameter
Value
Number of tables T
4
Bits per table b
auto-scaled, range 14–24
Bit cap
B+2
Gaussian floor
10−3
Gaussian cutoff radius
3.72σ
RNG seed
42
Appendix
Table 11: LSH default hyperparameters.
K
Bits
Brute (ms)
LSH (ms)
Speedup
100
14
1.1
0.3
3.7 ×
1,000
14
12.4
1.2
10.1 ×
10,000
18
175.2
2.5
71.4 ×
100,000
21
1,420
6.2
228 ×
1,000,000
24
19,227
15.2
1,263 ×
Appendix
Table 12: Suppression time vs. patch count ( E=1 , B=10 , N=1000 ).
E
Ktotal
Brute (ms)
LSH (ms)
Brute/ E
LSH/ E
1
10,000
158
4.8
158
4.8
4
40,000
858
15.6
215
3.9
16
160,000
4,615
108
288
6.7
32
320,000
8,384
215
262
6.7
64
640,000
17,318
482
271
7.5
Appendix
Table 13: Suppression time vs. expert count ( Ke=10,000 , B=10 , N=1000 ).
E
Dense BF
Top-2 BF
Dense LSH
Top-2 LSH
BFBFK
LSHLSHK
2
502
518
13.0
13.5
103%
104%
4
1,257
971
28.2
28.8
77%
102%
8
2,048
1,467
55.6
24.2
72%
44%
16
4,319
822
120
36.3
19%
30%
32
8,479
334
216
33.6
3.9%
16%
64
18,090
287
432
36.9
1.6%
8.5%
Appendix
Table 14: Optional top- K scaling study: suppression cost vs. expert count ( Ktop=2 , Ke=10,000 , B=10 , N=1000 ). This is a large-scale extension and is not used in the reported MuJoCo experiments.
Method
Ant-v2 (400K)
HalfCheetah-v2 (600K)
Hopper-v2 (600K)
Walker2d-v2 (600K)
MoE-GUIDE
2692±792
9236±604
3369±116
3887±600
FP cap200
2639±799
9303±879
3371±221∗
3535±449∗
FP cap400
3433±783
9241±1003
3354±122
4367±316∗
FP cap600
2947±859
9870±375
3388±117
4270±663∗
FP cap800
2995±618
9637±164
2417±1255
3832±370∗
FP cap1000
2867±614
9744±253
2815±989
3587±967
Appendix
Table 15: FP patch cap sweep: mean eval return ± std. No comparison reaches significance (Welch’s t -test, all p>0.05 ). ∗ Fewer than 5 seeds due to log parser failures.
Figure 7: Learning curves for the FP patch budget sweep in Table 15 . Curves show mean evaluation return across seeds, with shaded bands for one standard deviation.
Method
Swimmer
Hopper
Walker2d
HalfCheetah
Ant
MoE-GUIDE
342±2
3540±86
4146±877
10646±657
3974±1099
Very Loose
339±3
3191±425
4174±457
10004±587
3858±366
Loose
341±3
3335±198
4259±405
10010±661
3858±366
Medium
339±3
3344±264
4280±374
9980±627
3941±527
Tight
339±3
3544±76
4535±722
10097±457
3827±681
Very Tight
340±2
3322±318
4172±354
10016±555
3875±403
Appendix
Table 16: FP constraint tightness ablation: final mean evaluation return ± std across 5 seeds. Swimmer at 500K steps; all others at 1M steps.
Figure 8: FP constraint tightness ablation: final mean evaluation return ± std across 5 seeds. Swimmer at 500K steps; all others at 1M steps.
Method
Swimmer
Hopper
Walker2d
HalfCheetah
Ant
MoE-GUIDE
342±2
3540±86
4146±877
10646±657
3974±1099
Regression
339±3
3544±76
4535±722
10097±457
3827±681
DBSCAN
250±4
3542±94
4186±293
9864±693
3917±498
Appendix
Table 17: Regression vs. DBSCAN patch identification: final mean evaluation return ± std across 5 seeds. Swimmer at 500K steps; all others at 1M steps.
Method
Swimmer
Hopper
Walker2d ‡
HalfCheetah ‡
Ant ‡
MoE-GUIDE
342±2
3540±86
4146±877
10646±657
3974±1099
FK ( m=0.0 )
36±17
3423±101
4424±536
11096±216
3647±926
FK ( m=0.5 )
117±5
3265±536
4260±284
10382±819
5086±217
FK ( m=1.0 )
191±4
3495±116
4268±344
10800±524
4443±370
Appendix
Table 18: OOB margin ablation: final mean evaluation return ± std across 5 seeds. Swimmer at 500K steps; all others at 1M steps. All variants use regression patch identification with cap 1,000. Each environment uses its best tightness level. Bold indicates best FK variant per environment.
Figure 9: OOB margin ablation across five MuJoCo environments. Bars show mean evaluation return ( ± std) for FK kernels with margin m∈{0,0.5,1.0} . Dashed line: MoE-GUIDE baseline; dotted line: expert demonstration return.
Figure 10: Effect of OOB margin on FK Patching in Swimmer (left) and Ant (right). Swimmer shows progressive recovery from collapse as margin increases. Ant peaks at m=0.5 , achieving the highest FK result across all ablations.
Tightness
Margin
Ant ‡
HalfCheetah ‡
Tight
m=0.0
3647±926
11096±216
Tight
m=0.5
5086±217
10382±819
Tight
m=1.0
4443±370
10800±524
Medium
m=1.0
4680±835
10542±612
Loose
m=1.0
4263±946
10790±606
Appendix
Table 29
Figure 11: Tightness–margin interaction for Ant-v2 (left) and HalfCheetah-v2 (right). Ant benefits from margin across all tightness levels. HalfCheetah performs best without margin at tight tightness.
Coverage
BN
Raw
Raw@BN- r
Ratio
CoD
50%
37
26
996
0.7 ×
26.9 ×
75%
192
322
1496
1.7 ×
7.8 ×
90%
492
622
1796
1.3 ×
3.7 ×
95%
592
722
1896
1.2 ×
3.2 ×
Appendix
Table 19: Ant ( D=111→B=10 , 3 experts): patches at each coverage level.
Coverage
BN
Raw
Raw@BN- r
Ratio
CoD
50%
35
30
885
0.9 ×
25.3 ×
75%
128
160
1385
1.2 ×
10.8 ×
90%
340
436
1685
1.3 ×
5.0 ×
95%
440
536
1785
1.2 ×
4.1 ×
Appendix
Table 20: HalfCheetah ( D=17→B=7 , 4 experts): patches at each coverage level.
Coverage
BN
Raw
Raw@BN- r
Ratio
CoD
50%
47
39
988
0.8 ×
21.0 ×
75%
129
157
1488
1.2 ×
11.5 ×
90%
284
402
1788
1.4 ×
6.3 ×
95%
384
502
1888
1.3 ×
4.9 ×
Appendix
Table 21: Hopper ( D=11→B=4 , 2 experts): patches at each coverage level.
Coverage
BN
Raw
Raw@BN- r
Ratio
CoD
50%
57
44
967
0.8 ×
17.0 ×
75%
127
156
1467
1.2 ×
11.6 ×
90%
227
397
1767
1.7 ×
7.8 ×
95%
313
497
1867
1.6 ×
6.0 ×
Appendix
Table 22: Swimmer ( D=8→B=3 , 1 expert): patches at each coverage level.
Coverage
BN
Raw
Raw@BN- r
Ratio
CoD
50%
48
34
981
0.7 ×
20.4 ×
75%
150
188
1481
1.3 ×
9.9 ×
90%
385
488
1781
1.3 ×
4.6 ×
95%
485
588
1881
1.2 ×
3.9 ×
Appendix
Table 23: Walker2d ( D=17→B=7 , 4 experts): patches at each coverage level.
Coverage
BN
Raw
Raw@BN- r
Ratio
CoD
50%
39
29
998
0.7 ×
25.6 ×
75%
159
184
1498
1.2 ×
9.4 ×
90%
408
484
1798
1.2 ×
4.4 ×
95%
508
584
1898
1.1 ×
3.7 ×
Appendix
Table 24: AntMaze ( D=29→B=8 , 3 experts): patches at each coverage level.
Env
D
B
BN radius
Raw radius
Avg neigh (BN)
Avg neigh (Raw@BN- r )
Ant
111
10
1.22
4.60
21.1
1.0
HalfCheetah
17
7
0.96
5.35
23.3
1.2
Hopper
11
4
0.45
3.91
15.4
1.0
Swimmer
8
3
0.25
2.56
11.8
1.0
Walker2d
17
7
1.17
5.15
16.0
1.0
AntMaze
29
8
1.15
7.55
29.2
1.0
Appendix
Table 25: Average neighbours within the ball radius.
B
Loss
Avg neigh
P50
P75
P90
P95
Final P
Final cov.
Train (s)
2
0.0999
13.5
51
125
241
336
416
99%
124
3
0.0430
17.7
47
136
311
411
491
99%
127
5
0.0080
19.5
42
159
408
508
588
99%
126
7
0.0032
29.1
35
188
487
587
667
99%
139
10
0.0026
25.5
37
191
489
589
669
99%
136
15
0.0017
26.4
37
242
542
642
722
99%
152
Appendix
Table 26: Full bottleneck dimension sweep for Ant ( D=111 , 3 experts, 2,000 epochs, 2,000 OOB test points).
Method
0–100K
100–200K
200–400K
400–600K
SAC
0.804
0.806
0.808
0.801
MoE-GUIDE
0.734
0.690
0.776
0.794
Fp tight
0.734
0.645
0.655
0.684
CP: Combined
0.785
0.773
0.785
0.822
CP: Velocity
0.770
0.763
0.806
0.820
CP: Tilt
0.792
0.780
0.813
0.839
Appendix
Table 27: Violation rate (any_rate) by training phase.
Method
Speed
Tilt
Hip vel
SAC
0.001
0.430
0.534
MoE-GUIDE
0.027
0.473
0.469
Fp tight
0.014
0.375
0.460
CP: Combined
0.033
0.457
0.529
CP: Velocity
0.019
0.479
0.503
CP: Tilt
0.027
0.519
0.503
Appendix
Table 28: Per-constraint violation rates, global average across all steps and seeds.
Method
r (any)
r (speed)
r (tilt)
r (hip)
SAC
−0.02
+0.33
−0.27
+0.31
MoE-GUIDE
+0.25
+0.70
+0.14
+0.25
FP tight
+0.08
+0.66
+0.02
+0.20
CP: Combined
+0.20
+0.74
+0.17
+0.16
CP: Tilt
+0.16
+0.68
+0.10
+0.25
OOB Hard+Comb.
+0.54
+0.81
+0.22
+0.52
Appendix
Table 29: Pearson r between evaluation reward and violation rate.
Violated ( n=22,810 )
Safe ( n=3,784 )
Kernel
Patches
rbase
s
rfinal
Δr
%>0.1
s
Δr
CP Combined
1,063
0.633
0.137
0.539
−0.094
21.4%
0.000
−0.000
CP Velocity
416
0.633
0.011
0.624
−0.009
3.9%
0.000
−0.000
CP Tilt
735
0.633
0.027
0.608
−0.025
7.2%
0.000
−0.000
CP Hip Vel
699
0.633
0.121
0.557
−0.076
15.4%
0.000
−0.000
FP cap1000
922
0.633
0.000
0.633
−0.000
0.0%
0.000
−0.000
Appendix
Table 30: Patch suppression on demonstration data. rbase is unpatched MoE reward, s is suppression magnitude, rfinal is reward after suppression, and Δr is change from unpatched.
Method
R∈[500,1K)
R∈[1K,1.5K)
R∈[2K,2.5K)
R∈[2.5K,3.5K)
SAC
0.811
0.811
0.803
0.791
MoE-GUIDE
0.687
0.748
0.788
0.809
FP tight
0.698
0.614
0.607
0.667
CP: Combined
0.783
0.773
0.779
0.769
CP: Tilt
0.796
0.771
0.805
0.835
OOB Hard+Comb.
0.194
0.312
0.586
0.638
Appendix
Table 31: Violation rate (any_rate) at matched reward levels.
Method
Steps R≥2K
Final R
Final Viol.
OOB Hard (no safety)
354K (5)
2379
—
OOB Hard+Combined
447K (3)
3590
0.677
OOB Hard+Velocity
308K (5)
3675
0.773
OOB Hard+Tilt
386K (5)
3253
0.655
OOB Hard+Hip Vel
590K (5)
2860
0.820
OOB Smooth (no safety)
316K (5)
2094
—
Appendix
Table 32: Full OOB results: final performance over the last 100K steps.
Pipeline stage
Latency (ms)
vs. Patches
Realtime?
MoE + 800 patches (brute)
0.36
1×
✓ all freqs
MoE + 800 patches (LSH)
0.40
1.1×
✓ all freqs
MobileNetV2 classify
19.8
55×
≤ 50 Hz only
ResNet-50 classify
62.4
173×
≤ 15 Hz only
KeypointRCNN (R50-FPN)
5,772
16,033×
< 1 Hz
Classify + Keypoint
5,864
16,289×
< 1 Hz
Appendix
Table 33: Per-frame latency: vision pipelines vs. patch evaluation (single CPU core, 640×480 image, median-of-5).
Safety signal
Example
Latency
Realtime?
Formal logic
Bound check, velocity cap
<0.01 ms
✓
Light neural net
MobileNetV2 classify
∼20 ms
Marginal
Heavy neural net
Keypoint + detection
∼6,000 ms
×
Multi-model consensus
Ensemble + voting
∼10,000 ms
×
Simulation rollout
Forward dynamics check
∼30,000 ms
×
Human review
Norm / ethical judgment
minutes–hours
×
Appendix
Table 34: Safety-signal latency spectrum: from formal logic to human review. Patches compile offline signals into runtime patch lookup.
Key
Description
moe_state_dict
MoE model weights
moe_config
{input_dim, bottleneck_dim, num_experts}
estimator_config
{l_min, l_max, steepness}
dim_stats
Per-dimension {min, max, mean, std} tensors
dim_groups
Position and velocity dimension groups
flywheel_patches
Per-expert patch centers, bandwidths, and strengths
AI alignment is growing in importance, yet many current approaches learn safety behavior by directly modifying policy parameters, entangling normative constraints with the underlying policy. This often yields opaque, difficult-to-edit alignment artifacts and reduces their reuse across models or deployments, a failure mode we term Alignment Waste. We propose Interactionless Inverse Reinforcement Learning, a framework for learning inspectable, editable, and reusable reward artifacts separately from policy optimization. We further introduce the Alignment Flywheel, a human-in-the-loop lifecycle for iteratively auditing, patching, and hardening these artifacts through automated evaluation and refinement. Together, these ideas recast alignment from a disposable training expense into a durable, verifiable engineering asset.
Language model alignment aims to make model behavior reflect desirable properties such as helpfulness, safety, and instruction following. Existing methods draw supervision from expert demonstrations, preference judgments, or verifiable outcomes. Demonstrations typically guide supervised fine-tuning (SFT), while preference judgments and verifiers provide rewards for reinforcement learning (RL). This raises a question: can the demonstrations alone be used to derive an explicit, interpretable reward useful in a post-training pipeline? Motivated by inverse reinforcement learning, we introduce Projected Alignment Reward Estimated from Demonstrations (PARED). PARED learns an explicit reward from expert demonstrations over a small set of response-level features, using a lightweight discriminator that separates demonstrations from the policy's own samples in this feature space. Experiments with inference-time reranking and adversarial on-policy RL show improvements both when starting from a base model without an SFT stage and when further optimizing an SFT-trained model. Additionally, we demonstrate that PARED can be used for contextual alignment, in which a single policy learns to adapt its responses to the preferences of different audiences.
Michał Wiliński, Liu Leqi, Chirag Nagpal
Carnegie Mellon University · The University of Texas at Austin · Independent Researcher
Reinforcement Learning from Human Feedback (RLHF) enables powerful LLM alignment but can introduce reward hacking - models exploit spurious correlations in proxy rewards without genuine alignment. Compounding this, the objectives internalized during RLHF remain opaque, making hacking behaviors difficult to detect or correct. We introduce IR3 (Interpretable Reward Reconstruction and Rectification), a framework that reverse-engineers, interprets, and surgically repairs the implicit objectives driving RLHF-tuned models. We propose Contrastive Inverse Reinforcement Learning (C-IRL), which reconstructs the implicit reward function by contrasting paired responses from post-alignment and baseline policies to explain behavioral shifts during RLHF. We then decompose the reconstructed reward via sparse autoencoders into interpretable features, enabling identification of hacking signatures through contribution analysis. Finally, we propose mitigation strategies - clean reward optimization, adversarial shaping, constrained optimization, and feature-guided distillation - that target problematic features while preserving beneficial alignment. Experiments across multiple reward model configurations show that IR3 achieves 0.89 correlation with ground-truth rewards, identifies hacking features with over 90% precision, and significantly reduces hacking behaviors while maintaining capabilities within 3% of the original model.