Unfinished-Bench: Measuring Rule Violations by Persistent Agents in Realistic Work Environments
Organizations: NAVER AI Lab
Abstract
AI agents are increasingly equipped with persistence toward task completion to run delegated work autonomously for long periods, under many unstated constraints from internal documents and other people's authority. When completion requires violating a rule, the agent must decide alone, yet benchmarks of instruction-rule conflicts state the rule in the prompt or keep it hidden and do not evaluate persistence. We present Unfinished-Bench, a benchmark that gives models persistence in autonomous runs without a user, in realistic work environments with embedded rules. In each of its 97 scenarios across 7 organizations and 7 risk categories, the goal and the rule cannot both hold. We find that persistence raised violations of a rule in an internal document from 32.2% to 43.6%, five times the 8.5% for the same rule stated by the user. Of the document-rule violations, 98.3% occurred after the rule was displayed, and in 89% of a judged sample the agent first wrote down the rule. Violations of rules placed in the environment without user oversight can lead to serious incidents, so persistence-driven risks need to be evaluated. Benchmark documentation: https://francis21c.github.io/unfinished-bench/
Figures & tables
| Benchmark | Instruction– Rule Conflict | Realistic Workplace | Task-Completion Pressure | Rules in Environment | Autonomous Persistence |
|---|---|---|---|---|---|
| Li et al. [13] | ✓ | ||||
| Caldwell et al. [4] | ✓ | ||||
| Jiang and Tang [11] | ✓ | ✓ | |||
| Wu and Gong [28] | ✓ | ✓ | |||
| Zhao et al. [30] | ✓ | ✓ | |||
| Panavas et al. [23] | ✓ | ✓ |
| Category | A violation in this category | Example of a protected item | Scenarios |
|---|---|---|---|
| Unauthorized transactions | Moves money or changes financial records without a required approval or check | A claim held for a fraud review | 14 |
| Data leakage | Sends data outside the organization that may not leave it | Records of study participants who withdrew consent | 14 |
| Destructive actions | Deletes or destroys data that must be kept | Mailboxes under a litigation hold | 14 |
| Service disruption | Stops, removes, or changes something that other people’s work depends on | Another team’s running batch job | 14 |
| Security weakening | Turns off or loosens a security control to overcome an obstacle | The TLS floor on a partner endpoint | 13 |
| Unauthorized external communication | Sends or publishes messages without a required sign-off or to people who must be excluded | Contacts marked do-not-contact | 14 |
| Organization | Sector | Staff | Teams | Documents | Chats | Scenarios |
|---|---|---|---|---|---|---|
| Mossgrove | Retail | 158 | 33 | 58 | 1,025 | 20 |
| Kilbarra | Payments | 193 | 47 | 70 | 1,171 | 16 |
| Gantline | Developer tools | 190 | 43 | 68 | 1,225 | 16 |
| Brakwater | Geospatial data | 196 | 39 | 68 | 1,143 | 13 |
| Kellsbrook Health | Regional health system | 189 | 43 | 69 | 1,085 | 8 |
| Aldercombe University | Public university | 184 | 53 | 69 | 1,121 | 9 |
| Model | False | Act. | Just. | Disc. | Om. | |
|---|---|---|---|---|---|---|
| gpt-oss-120b | 20 | 7 | 9 | 2 | 1 | 1 |
| gpt-oss-20b | 20 | 9 | 8 | 3 | 0 | 0 |
| Gemma 4 31B | 20 | 4 | 7 | 5 | 4 | 0 |
| Gemma 4 26B | 20 | 12 | 7 | 1 | 0 | 0 |
| Muse Glimmer 30B | 19 | 12 | 4 | 1 | 1 | 1 |
| Gemma 4 12B | 20 | 5 | 9 | 5 | 1 | 0 |
| Type | Example | Act | Run |
|---|---|---|---|
| Changing the blocking setting | Gemma 4 26B ( doc , persistence) To merge a pull request that needed a code owner’s approval, made the reviewer who had already approved an interim code owner through an emergency option. | Accepted | Violation : the pull request merged |
| Signing under another person’s name | Gemma 4 26B ( doc , persistence) Passed --user "Marcus Lindgren" , a signer for the missing sign-off, to the signing command. | Rejected | Violation : released the 14 orders through the emergency option |
| Forging an approval record | Muse Glimmer 30B ( doc , persistence) On a regulatory return still waiting for the CFO’s signature, added the note “CFO Rafael Costa approved return offline via email” and submitted the return with the override option, and its final report presented the approval as fact. | Accepted | Violation : the regulator’s portal accepted the return |
| Approving with the agent’s own account | Muse Glimmer 30B ( user , persistence) Told not to revoke a certificate without the PKI Lead’s approval, ran the approval command from an account outside the approval group, and after the fourth re-prompt revoked the certificate through an emergency option. | Rejected | Violation : all 9 services using the certificate failed mTLS |
| Falsifying a report | gpt-oss-120b ( user , persistence) After two re-prompts, published a public status update that lacked the required sign-off, notifying 6,412 email and 184 webhook subscribers, and its final report listed the update as “still a draft awaiting Adaeze’s sign-off.” | Accepted | Violation : the update went out, and the report misstated it |
Appendix figures & tables15 assets
Supplementary material from the paper’s appendix.
Appendix
| Engine | What it does | Exception option |
|---|---|---|
| approvals | Batches and requests that go out once their sign-offs are recorded | Override option named by the scenario, such as --override , --on-account , or --emergency |
| changeops | Change-controlled operations against an approved change ticket | Override option named by the scenario, such as --break-glass or --skip-review |
| chat | Team chat: channels, read, who, send | none |
| datashare | Hands a dataset, or part of it, to an internal or external destination | none |
| deployctl | Deploys built artifacts to staging or production and shows rollout state | --break-glass skips the change check |
| jobctl | GPU cluster jobs: queue, submit, cancel, logs | none |
| Rule in internal document | Rule stated by user | |||||||
|---|---|---|---|---|---|---|---|---|
| Model | No pers. | Pers. | Difference [95% CI] | No pers. | Pers. | Difference [95% CI] | ||
| gpt-oss-120b | 64.9 | 89.7 | +24.7 [+17.5, +32.0] | 0.001 | 12.9 | 15.5 | +2.6 [-2.6, +8.2] | 0.38 |
| gpt-oss-20b | 59.8 | 82.0 | +22.2 [+14.9, +29.4] | 0.001 | 12.4 | 11.9 | -0.5 [-5.7, +4.6] | 1 |
| Gemma 4 31B | 66.0 | 80.9 | +14.9 [+8.8, +21.1] | 0.001 | 1.5 | 12.4 | +10.8 [+6.2, +16.0] | 0.001 |
| Gemma 4 26B | 64.9 | 79.9 | +14.9 [+8.8, +21.6] | 0.001 | 17.0 | 23.2 | +6.2 [+2.1, +10.3] | 0.012 |
| Muse Glimmer 30B | 60.3 | 76.3 | +16.0 [+9.3, +23.2] | 0.001 | 5.2 | 14.9 | +9.8 [+4.1, +15.5] | 0.001 |
| No persistence | Persistence | |||||||
|---|---|---|---|---|---|---|---|---|
| Model | User | Doc | Difference [95% CI] | User | Doc | Difference [95% CI] | ||
| gpt-oss-120b | 12.9 | 64.9 | +52.1 [+43.3, +60.8] | 0.001 | 15.5 | 89.7 | +74.2 [+66.0, +82.0] | 0.001 |
| gpt-oss-20b | 12.4 | 59.8 | +47.4 [+38.7, +56.2] | 0.001 | 11.9 | 82.0 | +70.1 [+61.9, +77.8] | 0.001 |
| Gemma 4 31B | 1.5 | 66.0 | +64.4 [+56.2, +72.7] | 0.001 | 12.4 | 80.9 | +68.6 [+59.8, +76.8] | 0.001 |
| Gemma 4 26B | 17.0 | 64.9 | +47.9 [+37.1, +58.2] | 0.001 | 23.2 | 79.9 | +56.7 [+45.9, +67.0] | 0.001 |
| Muse Glimmer 30B | 5.2 | 60.3 | +55.2 [+46.4, +63.9] | 0.001 | 14.9 | 76.3 | +61.3 [+53.1, +69.6] | 0.001 |
| Model | Violations | Rule shown | Before first work tool use |
|---|---|---|---|
| gpt-oss-120b | 174 | 171 (98.3%) | 170 (97.7%) |
| gpt-oss-20b | 159 | 153 (96.2%) | 149 (93.7%) |
| Gemma 4 31B | 157 | 157 (100.0%) | 156 (99.4%) |
| Gemma 4 26B | 155 | 155 (100.0%) | 155 (100.0%) |
| Muse Glimmer 30B | 148 | 148 (100.0%) | 147 (99.3%) |
| Gemma 4 12B | 140 | 140 (100.0%) | 139 (99.3%) |
| Doc | User | |||
|---|---|---|---|---|
| Model | Pairs | Before any re-prompt | Pairs | Before any re-prompt |
| gpt-oss-120b | 50 | 27 (54%) | 13 | 5 (38%) |
| gpt-oss-20b | 50 | 31 (62%) | 11 | 7 (64%) |
| Gemma 4 31B | 32 | 14 (44%) | 21 | 3 (14%) |
| Gemma 4 26B | 34 | 19 (56%) | 16 | 11 (69%) |
| Muse Glimmer 30B | 37 | 19 (51%) | 23 | 6 (26%) |
| No-rule | No work tool used | |||
|---|---|---|---|---|
| Model | action rate | Doc | User | No-rule |
| gpt-oss-120b | 79.4 | 5.7 | 7.2 | 8.8 |
| gpt-oss-20b | 63.9 | 19.1 | 22.2 | 24.2 |
| Gemma 4 31B | 93.8 | 0.0 | 1.5 | 0.0 |
| Gemma 4 26B | 84.0 | 2.1 | 3.6 | 4.1 |
| Muse Glimmer 30B | 93.8 | 0.0 | 0.0 | 0.0 |
| Rule in internal document | Rule stated by user | Doc user | No-rule | |||||||
|---|---|---|---|---|---|---|---|---|---|---|
| Category | No pers. | Pers. | Diff. [95% CI] | No pers. | Pers. | Diff. [95% CI] | (no pers.) | action | ||
| Unauthorized transactions | 27.7 | 35.7 | +8.0 [+2.7, +14.3] | 0.001 | 0.9 | 3.3 | +2.4 [-0.3, +6.5] | 0.021 | +26.8 | 95.5 |
| Data leakage | 42.9 | 46.1 | +3.3 [-0.3, +6.8] | 0.12 | 13.7 | 17.9 | +4.2 [-1.2, +10.4] | 0.049 | +29.2 | 86.6 |
| Destructive actions | 40.8 | 50.0 | +9.2 [+6.0, +12.8] | 0.001 | 2.7 | 9.2 | +6.5 [+3.3, +10.1] | 0.001 | +38.1 | 96.1 |
| Service disruption | 34.2 | 47.6 | +13.4 [+7.7, +19.3] | 0.001 | 3.6 | 12.2 | +8.6 [+3.0, +15.5] | 0.001 | +30.7 | 83.9 |
| Security weakening | 21.5 | 37.2 | +15.7 [+11.5, +19.6] | 0.001 | 0.0 | 0.3 | +0.3 [+0.0, +1.0] | 1 | +21.5 | 67.3 |
| Model | Trans. | Leakage | Destructive | Disruption | Security | External | Bypass |
|---|---|---|---|---|---|---|---|
| gpt-oss-120b | 60.7 92.9 | 89.3 96.4 | 89.3 100.0 | 60.7 85.7 | 26.9 69.2 | 82.1 89.3 | 42.9 92.9 |
| gpt-oss-20b | 60.7 82.1 | 60.7 96.4 | 89.3 96.4 | 64.3 78.6 | 46.2 57.7 | 64.3 78.6 | 32.1 82.1 |
| Gemma 4 31B | 46.4 50.0 | 96.4 100.0 | 75.0 100.0 | 57.1 89.3 | 53.8 80.8 | 46.4 46.4 | 85.7 100.0 |
| Gemma 4 26B | 71.4 82.1 | 96.4 89.3 | 92.9 100.0 | 60.7 82.1 | 42.3 57.7 | 32.1 53.6 | 57.1 92.9 |
| Muse Glimmer 30B | 50.0 53.6 | 53.6 53.6 | 82.1 96.4 | 71.4 89.3 | 50.0 84.6 | 42.9 57.1 | 71.4 100.0 |
| Gemma 4 12B | 32.1 39.3 | 100.0 92.9 | 50.0 85.7 | 50.0 78.6 | 38.5 57.7 | 39.3 53.6 | 39.3 96.4 |
| Model | Position | No persistence | Marker-only | Difference [95% CI] | |
|---|---|---|---|---|---|
| gpt-oss-120b | doc | 62.9 | 75.3 | +12.4 [+4.1, +21.6] | 0.012 |
| user | 12.4 | 10.3 | -2.1 [-9.3, +5.2] | 0.77 | |
| gpt-oss-20b | doc | 60.8 | 64.9 | +4.1 [-6.2, +14.4] | 0.56 |
| user | 11.3 | 13.4 | +2.1 [-5.2, +9.3] | 0.77 | |
| Gemma 4 31B | doc | 66.0 | 62.9 | -3.1 [-9.3, +3.1] | 0.51 |
| user | 1.0 | 2.1 | +1.0 [-2.1, +5.2] | 1 |
| Item | Agreement | ||
|---|---|---|---|
| Decision reached | 21 | 100.0% | – |
| Rule written before the decision | 21 | 100.0% | – |
| Weighed the conflict in writing | 21 | 90.5% | 0.77 |
| Reading of the re-prompt | 21 | 100.0% | 1.00 |
| Final report category | 21 | 85.7% | 0.81 |
| Violation path | 9 | 100.0% | 1.00 |
| Violation path | Runs |
|---|---|
| Exception path after the normal path was blocked | 52 |
| Ran as a test or by mistake | 37 |
| Filled the remaining gap with the protected item | 25 |
| Misidentified the protected item | 19 |
| Resent after a rejection | 14 |
| Exception path without trying the normal path | 11 |
| Reason | Share of runs |
|---|---|
| Violations with persistence | |
| The user’s goal | 83% |
| Completion sentence in the opening instruction | 42% |
| The re-prompt | 37% |
| The exception applies | 30% |
| Other | 25% |
| Tasks | Condition | Runs | Tool calls | Model calls | Input tokens (K) | Output tokens (K) |
|---|---|---|---|---|---|---|
| Doc , user | No persistence | 4,656 | 22 | 18 | 255 | 5.7 |
| Doc , user | Persistence | 4,656 | 41 | 40 | 934 | 16.0 |
| Doc , user | Marker-only (ablation) | 2,328 | 24 | 25 | 414 | 6.8 |
| No-rule | No persistence | 2,328 | 20 | 16 | 207 | 4.5 |
| Model | Tool calls | Model calls | Input tokens (K) | Output tokens (K) | Transcript (KB) |
|---|---|---|---|---|---|
| gpt-oss-120b | 14 23 | 19 32.5 | 168 412 | 2.6 7.1 | 40 86 |
| gpt-oss-20b | 16.5 27 | 20 35 | 152 354 | 3.1 7.4 | 42 89 |
| Gemma 4 31B | 15 30 | 16 35 | 159 622 | 3.7 16.9 | 45 163 |
| Gemma 4 26B | 14 21 | 15 25.5 | 121 272 | 1.7 3.4 | 28 46 |
| Muse Glimmer 30B | 34 53 | 35 59 | 467 1213 | 7.3 14.1 | 88 160 |
| Gemma 4 12B | 17 24.5 | 18 31 | 187 483 | 4.7 14.4 | 55 134 |