cs.AINov 9, 2026

Shortcuts and Boundaries: Verifiable RL Misalignment and Contextual Safety Learning

Authors: Xinjie Shen

Organizations: Everywhere Safety

Abstract

Reinforcement learning (RL) can improve an agent's task capability while reinforcing violations of explicitly stated contextual requirements, even when reward accurately measures useful work. We study this reward-driven misalignment through verifiable reward--context conflicts, where compliant completion is feasible but executable violations earn more reward than every compliant strategy. Encoding these conflicts in task dynamics supports scalable environment construction and compliance feedback from execution histories. In resource and reporting workflows, task optimization increases real productivity through unauthorized resource use or raises reported scores while actual work declines. We observe both amplification of existing violations and the emergence of violations on tasks where none were observed before training. Controlled changes in the training distribution show that minority conflict exposure can still lead to widespread shortcut acquisition, reaching 97.9% certified profitable violations on an API task under a 25% conflict mixture after 128 updates. The same contextual checks supply a simple binary penalty that supports joint capability and compliance gains and partially mitigates acquired violations. Learned safety adapts to expanded authorization and transfers across mechanisms into workflows with evolving evidence and approval requirements. On 32 unseen changing-evidence tasks, resource-trained safety increases safe completion from 53.1% to 89.1% relative to task-only training, while both policies achieve full real work. These results connect a verifiable account of reward-driven misalignment with scalable task construction and safety learning that remains useful under changing execution requirements.

Figures & tables

Appendix figures & tables19 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. Safety Training Modulates Harmful Misalignment Under On-Policy RL, But Direction Depends on Environment Design

    Apr 14, 2026Leon Eshuijs, Shihan Wang, Antske FokkensReward HackingRL for Language Models

  2. Implicit Safety Alignment from Crowd Preferences

    May 20, 2026Qian Lin, Daniel S. BrownSafe RLPreference-Based RL

  3. Evaluation Metrics for Safe Reinforcement Learning

    Sep 14, 2026Lindsay Spoor, Aske Plaat, Thomas MoerlandRL BenchmarksConstrained RL