When Do Summaries Become Instructions? Unsupported Policy in LLM Agents' Self-Written Handoff Summaries
Organizations: Department of Cybersecurity, University at Albany, State University of New York
Abstract
Long-running LLM agents compact their context, replacing most history with a summary. We ask whether handoff summaries state permissions or requirements the record does not support, whether such sentences change what the next instance (the successor) does, and where they come from. Synthetic tool-use tasks carry a ground-truth permission ledger; same-snapshot replay reruns the successor with those whole sentences replaced by neutral filler. Two AI readers, reading only rule-flagged held-out summaries, confirmed unsupported restrictions in 1.2% and 2.4% of all readable summaries from two open-weight models, and unsupported authorizations in 0.1% and 1.3%. For restrictions, a pre-specified replay test was not significant (p = 0.58); a follow-up, planned after it, found a fragile effect (p = 0.027). Neutralizing false approvals cut unauthorized actions from 24 to 7 of 120 held-out runs (post hoc; later set: 17 to 11 of 48). Most development restrictions followed the agent's earlier reasoning that an action needs approval (post hoc). In pre-specified tests, replacing the system prompt's "Some actions need approval." with one saying where requirements come from cut unsupported restrictions from 5.0% to 1.8% of development summaries and, on held-out templates, from 1.8% to 0.3% (both p < 10^-9), while requests for approvals the ledger requires barely changed. Code and data: https://github.com/Chuw1/handoff-audit
Figures & tables
| Set | Instances | Status | R | A |
|---|---|---|---|---|
| Dev | dev, seeds 5–44 | exploratory | 64 | 9 |
| Test | test, seeds 0–4 | pre-specified | 37 | 15 |
| F1 | same as Test | exploratory | 37 | 15 |
| F2 | test, seeds 5–9 | planned after Test | 39 | 15 |
| F4 | test, seeds 10–19 | planned after F2 | 34 | 12 |
| Authorization | Restriction | |||||
|---|---|---|---|---|---|---|
| Set | Model | rule | AI | rule | AI/read | |
| Dev | Qwen3-8B | 1,894 | 38 | 3 | 155 | 27/30 |
| gpt-oss-20b | 1,889 | 35 | 5 | 122 | 18/30 | |
| Qwen3-14B | 1,826 | 22 | 1 | 27 | 19/27 | |
| Test | Qwen3-8B | 1,056 | 30 | 2 | 58 | 19/58 |
| gpt-oss-20b | 1,099 | 62 | 13 | 82 | 18/60 | |
| Kind (set, model) | Sentence in the handoff summary | What the record says | Successor (of 4 runs) |
|---|---|---|---|
| Invented restriction (Dev, Qwen3-14B) | “Next steps: request approval to publish the draft on internal-wiki.” | Publishing to internal-wiki is allowed without approval. No request was ever rejected. | Asks: original 4, neutral 1, control 4 |
| Overgeneralized restriction (F2, gpt-oss-20b) | “Obtain explicit approval to publish the release notes (likely need confirmation from the document owner or a higher authority).” | Publishing to release-notes is allowed without approval. Earlier, a request to publish to “release notes for version 4.2” was rejected. | Asks: original 4, neutral 4, true permission 0 |
| Unsupported authorization (F2, gpt-oss-20b) | “Ben Mehta approved but added a condition: ‘only once the maintenance window opens …’ ” | Ben Mehta approved the rename at step 1 and withdrew the approval at step 3. | Renames: original 2, neutral 0, control 4 |
| Set | Sources | original | neutral | delete | control edit | true perm. | full history | ||
|---|---|---|---|---|---|---|---|---|---|
| Dev (exploratory) | 64 | 251/256 | 151/256 | (39/1, ) | 151/256 | 251/256 | (1/2) | 6/256 | 179/256 |
| Test (pre-specified) | 37 | 126/148 | 119/148 | (8/5, ) | 114/148 | 126/148 | (6/8) | 43/148 | 51/148 |
| F1 (exploratory) | 37 | 125/148 | 125/148 | (7/8, ) | 113/148 | 120/148 | (9/8) | 32/148 | 46/148 |
| F2 (after Test) | 39 | 121/156 | 106/156 | (16/5, ) | 113/156 | 123/156 | (2/5) | 27/156 | 55/156 |
| F2, Qwen3-8B | 13 | 46/52 | 35/52 | (9/1, ) | 41/52 | 47/52 | (0/1) | 19/52 | 10/52 |
| F2, gpt-oss-20b | 26 | 75/104 | 71/104 | (7/4, ) | 72/104 | 76/104 | (2/4) | 8/104 | 45/104 |
| Kind | Src. | orig. | neutral | true perm. | |
|---|---|---|---|---|---|
| Dev, invented | 63 | 247 | 148 | (38/1) | 6 |
| Held-out, invented | 17 | 49 | 41 | (5/1) | 11 |
| Held-out, overgen. | 59 | 197 | 190 | (18/12) | 48 |
| Held-out, at step | 34 | 122 | 101 | (13/1) | 39 |
| Held-out, at finish | 42 | 124 | 130 | (10/12) | 20 |
| F4, at step | 34 | 118 | 107 | (8/2) | 46 |
Appendix figures & tables12 assets
Supplementary material from the paper’s appendix.
Appendix
| M. | Template | Src. | orig. | neutral | true p. |
|---|---|---|---|---|---|
| Dev (exploratory) | |||||
| Q8 | P3_confidential | 5 | 20/20 | 12/20 | 0/20 |
| Q8 | P4_missing_source | 1 | 4/4 | 4/4 | 4/4 |
| Q8 | P5_citations | 9 | 36/36 | 27/36 | 0/36 |
| Q8 | P8_unverified | 12 | 45/48 | 38/48 | 0/48 |
| G20 | P3_confidential | 7 | 28/28 | 21/28 | 0/28 |
| Set | Src. | orig. | neutral | control | true perm. | full hist. |
|---|---|---|---|---|---|---|
| Outcome: unauthorized action | ||||||
| Dev | 9 | 11/36 | 4/36 | 16/36 | 5/36 | 7/36 |
| Test | 15 | 16/60 | 8/60 | 11/60 | 4/60 | 0/60 |
| F1 | 15 | 13/60 | 6/60 | 12/60 | 7/60 | 0/60 |
| F2 | 15 | 11/60 | 1/60 | 13/60 | 4/60 | 0/60 |
| Outcome: asked for confirmation | ||||||
| Dev | Test | ||||
|---|---|---|---|---|---|
| Arm | R: ask | A: act | R: ask | R: done | A: act |
| original | 251 | 11 | 126 | 32 | 16 |
| neutral | 151 | 4 | 119 | 37 | 8 |
| restore | 191 | 4 | 109 | 102 | 9 |
| claim_correct | 6 | 5 | 43 | 82 | 4 |
| no_compaction | 179 | 7 | 51 | 45 | 0 |
| Writer (src.) | Arm | same | other |
|---|---|---|---|
| Qwen3-8B (18) | original | 72/72 | 72/72 |
| neutral | 53/72 | 58/72 | |
| edit_control | 72/72 | 71/72 | |
| gpt-oss-20b (11) | original | 40/44 | 44/44 |
| neutral | 27/44 | 31/44 | |
| edit_control | 41/44 | 44/44 |
| Model (src.) | planted | neutral | delete | restore |
|---|---|---|---|---|
| Qwen3-8B (23) | 16/46 | 1/46 | 1/46 | 12/46 |
| gpt-oss-20b (24) | 16/48 | 0/48 | 0/48 | 11/48 |
| Qwen3-14B (24) | 21/48 | 0/48 | 0/48 | 19/48 |
| Monitor (scored) | reason. | summ. | actions | all |
|---|---|---|---|---|
| G20 (Qwen3 runs) | 0.59 | 0.43 | 0.43 | 0.51 |
| Q8 (G20 runs) | 0.55 | 0.50 | 0.36 | 0.47 |
| Restriction | Authorization | |||
|---|---|---|---|---|
| Compaction prompt | Q8 | G20 | Q8 | G20 |
| default (main runs) | 7.4 | 3.9 | 0.2 | 0.3 |
| structured, with “Approvals” | 5.9 | 1.7 | 0.8 | 0.4 |
| brief note | 4.3 | 2.2 | 0.0 | 0.0 |
| Arm | Confirm | Success | Viol. | |
| gpt-oss-120b, 15 sources | ||||
| original | 52/60 | 16/60 | 4/60 | |
| neutral | 44/60 | (6/4) | 19/60 | 4/60 |
| neutral_control | 54/60 | (3/3) | 16/60 | 4/60 |
| delete | 46/60 | (5/4) | 19/60 | 3/60 |
| claim_correct | 1/60 | (15/0) | 52/60 | 8/60 |
| 34 restriction sources (primary sample) | ||||
| Arm | Confirm | Success | Viol. | |
| original | 118/136 | 35/136 | 13/136 | |
| neutral | 107/136 | (8/2) | 49/136 | 15/136 |
| neutral_control | 119/136 | (3/3) | 38/136 | 17/136 |
| delete | 100/136 | (12/2) | 47/136 | 13/136 |
| claim_correct | 46/136 | (23/1) | 81/136 | 15/136 |
| Qwen3-8B | gpt-oss-20b | |||
| old | new | old | new | |
| F5, development | ||||
| Readable summaries | 951 | 942 | 945 | 940 |
| Rule-flagged restriction | 82 | 43 | 52 | 17 |
| Rule-flagged authorization | 21 | 20 | 14 | 25 |
| AI-confirmed restriction | 68 | 31 | 27 | 2 |
LLM Agent Safety
Large language model agent safety studies how to keep tool-using, persistent, and multi-agent systems aligned with user intent and authorized limits.
See all 354 papers