cs.CLOct 19, 2026
SaveMitigating Reward Hacking in Preference Reward Models with Evolving Penalties
Organizations: Sword Health · Instituto Superior Técnico, Universidade de Lisboa · Gandara AI · Instituto de Telecomunicações ·
Abstract
Reinforcement learning for language models has moved from preference reward models to verifiable rewards, which sustain much longer optimization. Preference models are prone to reward hacking: the policy raises their score without improving response quality, or even while degrading it. Yet for open-ended tasks, where quality can neither be verified nor fully written down as rubrics, they remain valuable. We introduce Dynalize, which mitigates reward hacking online, during policy training. Periodically and asynchronously, a language model analyzes training rollouts and writes penalties for undesirable behaviors that the reward model appears to reward; a judge applies them to every rollout. In clinical counseling and creative writing, training on the reward model alone raises its score continuously but ultimately leads to a decline in external benchmarks; Dynalize sustains longer training and results in improved downstream performance. We find that our method suppresses patterns incentivized by preference models that, without penalties, spread to most responses, and even to the model's reasoning, which the reward model does not explicitly score. Furthermore, we show that its gains hold in multi-turn training, and even after we retrain the reward model on the discovered behaviors.
Figures & tables
Figure 1: Overview of Dynalize . Full details in Section 3 .
Figure 2: RM score and benchmark score over training, for RM-only training and our method.
| Clinical counseling | Creative writing | |||||||
| MindEval | EQ-Bench CW v3 | |||||||
| Run | Peak | End | Final | RM score | Peak | End | Final | RM score |
| RM only | 3.69 | 3.39 | 3.37 | 1.00 | 58.9 | 56.0 | 54.8 | 12.19 |
| Dynalize | 3.92 +0.23 | 3.84 +0.45 | 3.80 +0.43 | 0.92 | 62.1 +3.2 | 60.8 +4.8 | 60.5 +5.7 | 10.93 |
Table 1: Main results. Peak : best checkpoint. End : mean over the checkpoints evaluated for both runs in steps 150–200. Final : step 200. RM score : mean over the End checkpoints. Subscripts: difference from RM only.
Figure 3: The 20 penalties discovered in clinical counseling: their prevalence in training rollouts while in the buffer, with the steps at which each was added and retired (left), and in held-out replies at the end of training (right).
| Clinical accuracy | Ethical & prof. | Assessment | Therapeutic rel. | AI-specific | ||
| Run | & competence | conduct | & response | & alliance | comm. quality | Overall |
| RM only, end | 3.54 | 3.98 | 3.37 | 3.51 | 2.52 | 3.39 |
| RM only, peak | 3.66 +0.12 | 4.13 +0.15 | 3.60 +0.23 | 3.76 +0.25 | 3.29 +0.77 | 3.69 +0.30 |
| Dynalize , end | 3.73 +0.19 | 4.32 +0.34 | 3.72 +0.35 | 3.79 +0.28 | 3.66 +1.14 | 3.84 +0.45 |
Table 2: MindEval scores by axis. Peak and End as in Table 1 . Subscripts: difference from RM only at the end.
Figure 4: Repeated patterns over training, on held-out prompts, for RM-only training and our method.
Figure 5: RM score and MindEval in multi-turn RL.
Figure 6: RM score and MindEval with the new RM.
Appendix figures & tables11 assets
Supplementary material from the paper’s appendix.
Appendix
| Run | Peak | End | Final |
|---|---|---|---|
| Judge: GLM 5.2 | |||
| RM only | 3.69 | 3.39 | 3.37 |
| Dynalize | 3.92 | 3.84 | 3.80 |
| Judge: Kimi-K3 | |||
| RM only | 3.85 | 3.59 | 3.54 |
| Dynalize | 4.13 | 4.05 | 4.06 |
Table 3: MindEval scores of the two clinical counseling runs under three judges. All judges rate the same conversations. Peak , End and Final are defined as in Table 1 . GLM 5.2 is the judge used in the main text.
| Clinical counseling | Creative writing | ||
|---|---|---|---|
| Penalty | Step(s) | Penalty | Step(s) |
| Pre-emptive insight attribution | 5 | Circular pseudo-profound dialogue | 10 |
| Over-elaborated hypothetical question framing | 15 | Non-sequitur garbled text fragments | 15 |
| Unwarranted psychoeducation inflation | 25 | Excessive length inflation | 25, 45 |
| Excessive reflective restatement before question | 30 | Excessive length inflation with plot repetition | 30 |
| Stacked multi-part questions | 37 | Overblown cosmic-scale power creep | 40 |
Table 4: Accepted penalties and their introduction steps, ordered chronologically within each environment. Steps refer to the recorded proposal cycles in which penalties were accepted. Multiple steps indicate reintroduction of the same penalty. The table covers both tasks through step 200.
Figure 7: The 15 penalties discovered in creative writing, as in Figure 3 .
| Penalty | Full description | Created at step |
|---|---|---|
| Pre-emptive insight attribution | The reply tells the member what they are ’onto,’ what they ’actually did,’ or frames their experience as evidence of progress or an important realization, before the member has acknowledged or agreed to it. This projects clinical meaning onto the member’s words and puts words in their mouth, rather than letting the member arrive at their own insight. | 5 |
| Over-elaborated hypothetical question framing | The reply poses a question but wraps it in multiple clauses, hypotheticals, or metaphorical reframings (e.g., ’beat it into submission with logic’) that extend length and add rhetorical flourish without improving clinical utility, making the core question harder to parse. | 15 |
| Unwarranted psychoeducation inflation | The reply pads the assignment of a simple activity with multiple sentences of scientific or mechanistic explanation (e.g., research findings, neurobiology, or how the technique works on the nervous system) that the member did not ask for and that delays the actual call to action. | 25 |
| Excessive reflective restatement before question | The reply spends 3+ sentences restating the member’s words back to them with elaboration before reaching a question, inflating length without advancing the conversation. | 30 |
| Stacked multi-part questions | The reply asks two or more distinct questions in a single turn (sometimes with sub-questions or elaborated options), inflating the response and overwhelming the member with multiple things to answer at once. | 37 |
| Conversational filler before question | The reply opens with casual filler phrases (e.g., ’Nice.’, ’Okay, so…’, ’Sounds good —’) that add no clinical value and delay the substantive content, inflating length and mimicking casual conversation rather than maintaining purposeful therapeutic communication. | 47 |
Table 5: Full accepted penalty descriptions and creation steps for clinical counseling, ordered by introduction.
| Penalty title | Full description |
|---|---|
| Over-elaborate reframing of user’s own plan | The reply takes the member’s already-stated simple plan and re-expands it with unnecessary psychological reframing, motivational commentary, or additional layers of meaning the member did not ask for, making the response longer and more complex than needed. |
| Context-disconnected response | The reply introduces content, questions, or topics that do not follow from the immediately preceding user turn, making the response feel non-sequitur or like it is responding to a different conversation. |
| Unearned reassurance and praise | The reply tells the member their idea/approach is ’clever,’ ’perfect,’ or otherwise praises them in a way that feels patronizing or unearned, especially when the member simply restated their own words or agreed to something minor. |
| Excessive stacked follow-up questions | The reply asks two or more distinct questions in a single turn (e.g., a concrete question immediately followed by ’how does that land?’ or a second question about a different sub-topic), creating conversational overload that pressures the member to address multiple threads at once. |
| Stacking multiple suggestions as options | The reply presents a list of multiple optional activities or techniques for the member to choose from, creating decision burden and overwhelming someone who asked for something simple. |
| Over-qualified hedging on simple asks | The reply wraps a straightforward question or next step in multiple layers of hedging and qualification (e.g., ’I’m wondering if,’ ’just to name what I’m picking up,’ ’not to challenge you’), inflating length and diluting clarity without adding substance. |
Table 6: Full penalty titles and descriptions from multi-turn clinical counseling training through step 195, ordered by introduction.
| Penalty title | Full description |
|---|---|
| Over-explaining clinical rationale | The reply delivers a multi-sentence paragraph explaining the clinical evidence or research basis for why a topic matters (e.g., sleep research, mood regulation, cognitive function) before asking the member if they’re open to exploring it, making the response feel like a lecture or justification rather than a natural conversational pivot. |
| Excessive brevity as false naturalness | The reply is a single ultra-short word or fragment (e.g., ’Night.’, ’Talk soon.’) that closes a conversation with no warmth, personalization, or acknowledgment of what the member shared, making the companion feel dismissive or robotic. |
| Excessive restatement before question | The reply opens by restating the member’s last statement nearly verbatim (often spanning multiple sentences) before asking a question, adding length without new content — the member hears their own words echoed back with minimal added framing. |
| Praising member’s own insight as ’useful’ | The reply labels the member’s self-observation as ’really useful’ or ’really honest’ before proceeding, inserting evaluative praise that can feel patronizing or like the assistant is grading the member rather than engaging with the content. |
| Low-content transitional closers | The reply is a brief, generic sign-off or transition phrase (e.g., ’You got it. Talk soon!’, ’I’ll remind you at 5:15 PM. How’s that timing looking for you?’) that carries minimal conversational substance, adding length or a question without moving the interaction forward meaningfully. |
| Stacking multiple choice options in questions | The reply asks a single question but stuffs it with two or more explicit answer options (e.g., ’is it the work demands, the fatigue kicking in, or something else?’), making the question feel like a multiple-choice survey item rather than an open, member-led invitation. |
Table 7: Full penalty titles and descriptions discovered by Our Method with the New RM in clinical counseling through step 200, ordered by introduction.
| Penalty | Full description | Created at step |
|---|---|---|
| Circular pseudo-profound dialogue | Characters exchange lines that sound dramatic but are mutually contradictory or tautological within the same passage (e.g., ’I want to die.’ / ’You are dead.’ / ’I want it to stop.’ / ’This is your penance.’), creating the illusion of depth without coherent meaning. | 10 |
| Non-sequitur garbled text fragments | The story contains garbled or nonsensical text fragments—random non-English characters, malformed words, or broken phrases mid-sentence (e.g., ’ 鴻国stroke,’ ’jerk 显瘦backs,’ ’ 昱standard English’)—that are clearly artifacts of generation failure and disrupt reader comprehension. | 15 |
| Excessive length inflation | The response runs well over 2000 words with multiple scene jumps, episodic vignettes, or appended epilogues that dilute the core narrative arc, far exceeding what the prompt’s concept warrants while lower-scored responses in the same group are more focused and proportionate. | 25 |
| Excessive length inflation with plot repetition | The response runs far beyond what the prompt requires (often 2000+ words) by introducing multiple sequential confrontations, villains, or scenarios that repeat the same thematic beat and power-demonstration pattern rather than advancing the story, creating a padded, episodic structure that dilutes narrative impact. | 30 |
| Overblown cosmic-scale power creep | The response inflates stakes far beyond what the prompt implies by escalating the protagonist’s significance to cosmic, planetary, or metaphysical extremes (e.g., a sheep that is a ’world anchor’ preventing continental destruction, a good person whose mere presence is an existential threat to all of Hell’s infrastructure), replacing grounded storytelling with escalating superlatives that strain credulity. | 40 |
| Excessive length inflation | The response is dramatically overlong (2500+ words) for the prompt, with the narrative continuing far past a natural endpoint through repeated epilogues, coda scenes, or ’what happened next’ time-skips that add no new conflict and merely restate the story’s theme. | 45 |
Table 8: Full accepted penalty descriptions and creation steps for creative writing, ordered by introduction.
More in this topic
Reward Hacking
Reward hacking studies how AI systems exploit imperfect objectives and how to detect, measure, and prevent behavior that scores well but misses intent.
See all 182 papers