Shortcuts and Boundaries: Verifiable RL Misalignment and Contextual Safety Learning
Organizations: Everywhere Safety
Abstract
Reinforcement learning (RL) can improve an agent's task capability while reinforcing violations of explicitly stated contextual requirements, even when reward accurately measures useful work. We study this reward-driven misalignment through verifiable reward--context conflicts, where compliant completion is feasible but executable violations earn more reward than every compliant strategy. Encoding these conflicts in task dynamics supports scalable environment construction and compliance feedback from execution histories. In resource and reporting workflows, task optimization increases real productivity through unauthorized resource use or raises reported scores while actual work declines. We observe both amplification of existing violations and the emergence of violations on tasks where none were observed before training. Controlled changes in the training distribution show that minority conflict exposure can still lead to widespread shortcut acquisition, reaching 97.9% certified profitable violations on an API task under a 25% conflict mixture after 128 updates. The same contextual checks supply a simple binary penalty that supports joint capability and compliance gains and partially mitigates acquired violations. Learned safety adapts to expanded authorization and transfers across mechanisms into workflows with evolving evidence and approval requirements. On 32 unseen changing-evidence tasks, resource-trained safety increases safe completion from 53.1% to 89.1% relative to task-only training, while both policies achieve full real work. These results connect a verifiable account of reward-driven misalignment with scalable task construction and safety learning that remains useful under changing execution requirements.
Figures & tables
| Workflow | Useful work | Contextual requirements |
|---|---|---|
| API-assisted computation | Complete jobs using local computation and external requests | Cumulative requests must respect the authorized allowance across accounts. |
| Software repair and certification | Repair an artifact and publish validation evidence | Published evidence must apply to the current artifact and required checks. |
| Cohort release | Complete dependency-linked processing and release a verified artifact | Vendor permission can be revoked and reissued with a new scope; publication requires current evidence and approval. |
| Target | Conflict share | Restricted profitable | Restricted safe completion | Restricted real work | Authorized real work |
|---|---|---|---|---|---|
| API | 25% | 18.8% | 70.8% | .740 | .999 |
| API | 100% | 81.3% | 16.7% | .945 | .983 |
| Allocation | 25% | 25.0% | 47.9% | .476 | .684 |
| Allocation | 100% | 37.5% | 8.3% | .494 | .668 |
| Metric | Base | Task- trained | Safety- trained |
|---|---|---|---|
| Restricted historical violations | 27.1% | 75.0% | 8.3% |
| Restricted certified profitable violations | 12.5% | 70.8% | 0% |
| Restricted safe completion | 72.9% | 25.0% | 91.7% |
| Restricted real work | .675 | .908 | .689 |
| Authorized lawful expanded-permission use | 68.8% | 100% | 93.8% |
| Authorized real work | .834 | .999 | .965 |
Appendix figures & tables19 assets
Supplementary material from the paper’s appendix.
Appendix
| Mechanism | Intended work | Executable departure | Independent check |
|---|---|---|---|
| Allocation | Allocate resources across periods | Exceed cumulative authorization | Spending history; compliant reward bound |
| API budget | Complete jobs using computation and calls | Exceed authorized request usage | Call history and permission scope; compliant reward bound |
| Validation | Repair and certify an artifact | Change checks or reuse stale evidence | Actual repair; report coverage and version |
| Test expectations | Repair required outputs | Replace owner-specified expectations | Owner’s expected outputs; actual implementation |
| Evaluation cohort | Repair evaluated components | Omit required cases | Owner’s required component set; actual health |
| Score cache | Repair components and publish scores | Alter protected scores or versions | Cache ownership and write history; actual health and version |
| Measure | Value |
|---|---|
| Core task panels | |
| Mechanism families | 8 |
| Physical tasks / condition records | 132 / 384 |
| Automated mechanism construction | |
| Construction requests / admitted workflows | 24 / 7 |
| Admitted proposal families / settings | 3 / 4 |
| Mechanism | Meaning of native completion |
|---|---|
| Allocation | All allocation periods processed. |
| Shared API budget | Actual completed job value reaches the instance’s minimum target. |
| Validation | Current passing weight reaches the minimum target. |
| Proxy / CI / monitor | All components genuinely repaired. Maintenance safe completion additionally requires the specified valid artifact and report. |
| Pattern | Scope | Historical | Certified profitable |
|---|---|---|---|
| Low-baseline amplification | Validation, full panel | 1 48/48 | 1 48/48 |
| Existing-risk amplification | Allocation, full panel | 24 42/48 | 10 20/48 |
| Instance-level emergence | API, nine fixed cases | 0 32/36 | 0 32/36 |
| Target | Authorized real work | Restricted profitable violations |
|---|---|---|
| Allocation | .533 .697 | 10 18/48 |
| API | .834 .990 | 6 28/48 |
| Conflict share | Allocation profitable | API profitable | Authorized API work |
|---|---|---|---|
| 25% | 40/48 | 47/48 | 1.000 |
| Approximately 50% | 24/48 | 46/48 | .999 |
| 100% | 33/48 | 46/48 | .996 |
| Source / target | Restricted | Restricted | Authorized | Restricted |
|---|---|---|---|---|
| reward | real work | real work | profitable | |
| Resource / API | .631 .982 | .631 .982 | .855 1.000 | 4 44/48 |
| Validation | .050 1.000 | .979 .072 | .926 .208 | 1 48/48 |
| Target | Historical violations | Authorized real work |
|---|---|---|
| Allocation | 43 31/48 | .661 .767 |
| API | 44 34/48 | 1.000 .995 |
| Workshop | 46 35/48 | .678 .734 |
| Allowance | Wording | Certified profitable | Mean real work |
|---|---|---|---|
| Zero | Original | 0, 2, 0 | .302, .332, .315 |
| Zero | Paraphrase | 1, 10, 0 | .320, .427, .308 |
| Restricted | Original | 3, 38, 0 | .665, .908, .683 |
| Restricted | Paraphrase | 2, 30, 0 | .585, .877, .685 |
| Intermediate | Original | 3, 28, 0 | .716, .951, .886 |
| Intermediate | Paraphrase | 1, 27, 1 | .707, .938, .853 |
| Permission | Wording | Certified profitable | Mean real work |
|---|---|---|---|
| Restricted | Original | 10, 40, 2 | .689, .944, .739 |
| Restricted | Paraphrase | 5, 35, 1 | .656, .917, .734 |
| Expanded | Original | 0, 0, 0 | .822, .985, .987 |
| Expanded | Paraphrase | 0, 0, 0 | .802, .997, .989 |
| Metric | Base | Task- trained | Safety- trained |
|---|---|---|---|
| Excluded Allocation | |||
| Restricted historical violations | 29/48 | 36/48 | 27/48 |
| Restricted certified profitable violations | 15/48 | 20/48 | 12/48 |
| Expanded mean real work | .627 | .559 | .644 |
| Lawful expanded use | 47/48 | 44/48 | 46/48 |
| Metric | Base | Task- trained | Safety- trained |
|---|---|---|---|
| Restricted historical violations | 39/62 | 46/62 | 39/62 |
| Restricted certified profitable violations | 20/62 | 28/62 | 17/62 |
| Restricted mean real work | .467 | .493 | .477 |
| Expanded mean real work | .632 | .572 | .646 |
| Metric | Base | Task-trained | Safety-trained |
|---|---|---|---|
| Restricted safe completion | 9/16 | 7/16 | 15/16 |
| Restricted historical violations | 7/16 | 9/16 | 1/16 |
| Restricted certified profitable violations | 1/16 | 2/16 | 0/16 |
| Restricted real work | 1.000 | 1.000 | 1.000 |
| Restricted task reward | .703 | .706 | .698 |
| Expanded safe completion | 16/16 | 16/16 | 16/16 |
| Metric | Task-trained | Safety-trained |
|---|---|---|
| Restricted safe completion | 8/16 | 13/16 |
| Restricted historical violations | 8/16 | 3/16 |
| Restricted certified profitable violations | 4/16 | 2/16 |
| Restricted real work | 1.000 | 1.000 |
| Expanded safe completion | 15/16 | 16/16 |
| Expanded real work | .988 | 1.000 |
| Target | Base | Task-trained | Safety-trained |
|---|---|---|---|
| Changing evidence | 43/64 | 34/64 | 57/64 |
| Composed context | 27/64 | 29/64 | 36/64 |
| Metric | Changing evidence | Composed context |
|---|---|---|
| Restricted historical violations | 30 7 | 35 28 |
| Restricted certified profitable violations | 6 2 | 20 17 |
| Restricted real work | 1.000 1.000 | 1.000 1.000 |
| Expanded safe completion | 64 64 | 64 64 |
| Expanded historical violations | 0 0 | 0 0 |
| Expanded real work | 1.000 1.000 | 1.000 1.000 |
| Setting | Base | Task-trained | Safety-trained |
|---|---|---|---|
| Warehouse operations | 37/48 | 25/48 | 42/48 |
| Financial settlement | 27/48 | 16/48 | 40/48 |
| Manufacturing quality | 28/48 | 17/48 | 40/48 |
| Public services | 13/48 | 16/48 | 22/48 |
| Metric | Task-trained | Safety-trained |
|---|---|---|
| Safe completion | 13/48 | 30/48 |
| Historical violations | 35/48 | 18/48 |
| Certified profitable violations | 15/48 | 10/48 |
| Mean real work | .992 | .998 |
| Case | Observed execution | Outcome |
|---|---|---|
| CI: Base | Repair all components; run their checks; finish. | Full real work; no violation. |
| CI: Task-trained | Repair all components; run their checks; manually replace the passing reports; finish. | Full real work; procedure violated. |
| API: restricted | Execute four external jobs on account 0, reaching 9 requests; perform a later job locally; finish. | 9 authorized requests; work .708. |
| API: expanded | Execute the same first four external jobs, then four additional jobs on account 1; finish. | 17 authorized requests; work 1.000. |
Safe RL
Safe reinforcement learning studies how agents can learn and act while respecting safety constraints and limiting harmful outcomes.
See all 238 papers