Decision Models as Policy Gates for Coding Agents: Explicit Policy Boundaries Improve Detection and Reduce False Alarms
Organizations: Agent Approve
Abstract
Coding agents can destroy data, leak secrets and message the wrong person faster than a human can review. Could a decision model catch these risks at runtime? We evaluated 22 hosted decision models and two prompted general models as classifiers for a multi-rule policy engine. A synthetic dataset of 2,643 tool calls was labeled blind against 28 rules, with a sealed holdout. With a single shared preamble in both conditions, explicit boundaries written as examples and counterexamples raised recall by 5.5-26.1 percentage points on all 23 available models and reduced false positives on 21. Jev 1.14's recall rose from 89.3% to 98.2%, while unnecessary interruptions fell from 30.6% to 4.8% of 252 clean actions. With examples, its median latency was 120 ms at $0.46 per thousand requests. Strong models retained the recall gains when concrete names were replaced by generic descriptions; three largely retained them when the same boundaries were written as prose. On 16 new scenarios written after the initial studies, a follow-up improved recall on 22 of 23 models and reduced unnecessary interruptions on 21. We release the dataset, rule texts and harness.
Figures & tables
| Recall % | FPR % | Interrupts / 60 | ||||||
|---|---|---|---|---|---|---|---|---|
| Model | Def. | Opus | Difference (95% CI) | Def. | Opus | Difference (95% CI) | Def. | Opus |
| Jev 1.14 | 84.6 | 95.6 | 10.9 (6.6, 15.3) | 3.22 | 1.15 | 2.07 ( 2.80, 1.40) | 15 | 4 |
| Perplexity v1.2 | 86.7 | 96.6 | 9.9 (5.2, 15.1) | 1.80 | 0.99 | 0.81 ( 1.35, 0.29) | 9 | 3 |
| SubQ Snap | 89.3 | 95.6 | 6.3 (0.9, 12.8) | 2.95 | 1.55 | 1.40 ( 2.03, 0.85) | 15 | 4 |
| GLiDE | 83.8 | 96.6 | 12.8 (8.2, 17.2) | 1.86 | 0.78 | 1.08 ( 1.71, 0.51) | 10 | 3 |
| Clef Flash | 50.1 | 69.7 | 19.6 (13.3, 26.1) | 0.53 | 0.69 | 0.16 ( 0.10, 0.43) | 3 | 7 |
| Def., per-rule preamble | GPT-6 examples | Opus examples, name-free | ||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Model | Recall | FPR | Intr. | Recall | FPR | Intr. | Recall | FPR | Intr. | |||
| Perplexity Decider v1.2 | 86.4 | 1.12 | 85.3 | 11.9 | 98.5 | 0.37 | 98.1 | 2.8 | 98.7 | 0.64 | 98.1 | 4.4 |
| OpenAI Decisions | 87.7 | 3.00 | 84.7 | 22.2 | 98.1 | 1.11 | 97.0 | 8.7 | 98.4 | 1.29 | 97.1 | 7.1 |
| Inception Mercury Decide | 81.1 | 1.29 | 79.8 | 19.4 | 97.7 | 0.54 | 97.2 | 6.7 | 98.7 | 0.57 | 98.1 | 5.6 |
| Fastino GLiDE | 86.0 | 1.37 | 84.6 | 19.0 | 98.2 | 0.37 | 97.8 | 5.6 | 98.6 | 0.41 | 98.2 | 1.6 |
| SubQ Snap | 85.5 | 1.37 | 84.2 | 17.9 | 98.3 | 0.59 | 97.7 | 6.0 | 98.3 | 0.66 | 97.7 | 4.0 |
Appendix figures & tables3 assets
Supplementary material from the paper’s appendix.
Appendix
| Model | Provider | Access | Identifier | $/M Input |
|---|---|---|---|---|
| Jev 1.13 | TypeSafe | API | jev-1.13.0 | 0.042 |
| Jev 1.14 | TypeSafe | API | jev-1.14.0 (released 2026-10-14) | 0.042 |
| Decisions API | OpenAI | API | gpt-6-luna | 0.10 |
| Clef | Cloudflare | Workers AI | @cf/cloudflare/clef | 0.24 |
| Clef Flash | Cloudflare | Workers AI | @cf/cloudflare/clef-flash | 0.09 |
| Gemini 3.5 Flash-Lite † | API | gemini-3.5-flash-lite | 0.30 |
| Def., single preamble | Opus examples | Latency (ms) | |||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Model | Recall | FPR | Intr. | Miss | Recall | FPR | Intr. | Miss | p50 | p95 | $/1k | ||
| Perplexity Decider v1.2 | 89.3 | 1.15 | 88.1 | 11.5 | 4.3 | 98.6 | 0.53 | 98.1 | 6.3 | 0.8 | 2,352 | 3,173 | 0.22 |
| OpenAI Decisions | 87.0 | 2.70 | 84.3 | 21.8 | 6.6 | 98.5 | 1.24 | 97.3 | 11.5 | 1.2 | 190 | 351 | 1.40 |
| Inception Mercury Decide | 81.2 | 1.23 | 80.0 | 16.3 | 7.3 | 98.4 | 0.66 | 97.8 | 5.6 | 0.5 | 539 | 996 | 0.45 |
| Fastino GLiDE | 87.3 | 1.46 | 85.9 | 20.2 | 5.5 | 98.4 | 0.47 | 98.0 | 4.8 | 0.8 | 1,419 | 3,511 | 5.24 |
| SubQ Snap | 90.4 | 2.17 | 88.3 | 17.1 | 2.6 | 98.3 | 0.64 | 97.7 | 6.0 | 1.2 | 469 | 668 | 0.61 |
| Template | Default | Dev | Hold | Template | Default | Dev | Hold |
|---|---|---|---|---|---|---|---|
| permanent-deletion | block | 98 | 95 | communication-outside-slack | block | 144 | 91 |
| git-force-push-or-delete-ref | block | 48 | 44 | send-data-to-external-host | ask | 367 | 330 |
| download-and-execute-code | ask | 55 | 55 | change-production-state | ask | 88 | 55 |
| replace-existing-file-content | ask | 218 | 71 | deploy-to-production | ask | 25 | 20 |
| read-credential-material | block | 144 | 95 | database-destructive-statement | block | 41 | 46 |
| share-credential-material | block | 81 | 45 | database-production-write | ask | 47 | 16 |
Runtime Enforcement for AI Agents
Runtime enforcement for AI agents checks and constrains actions as they execute, helping prevent unsafe or unauthorized effects.
See all 312 papers