Long-running LLM agents compact their context, replacing most history with a summary. We ask whether handoff summaries state permissions or requirements the record does not support, whether such sentences change what the next instance (the successor) does, and where they come from. Synthetic tool-use tasks carry a ground-truth permission ledger; same-snapshot replay reruns the successor with those whole sentences replaced by neutral filler. Two AI readers, reading only rule-flagged held-out summaries, confirmed unsupported restrictions in 1.2% and 2.4% of all readable summaries from two open-weight models, and unsupported authorizations in 0.1% and 1.3%. For restrictions, a pre-specified replay test was not significant (p = 0.58); a follow-up, planned after it, found a fragile effect (p = 0.027). Neutralizing false approvals cut unauthorized actions from 24 to 7 of 120 held-out runs (post hoc; later set: 17 to 11 of 48). Most development restrictions followed the agent's earlier reasoning that an action needs approval (post hoc). In pre-specified tests, replacing the system prompt's "Some actions need approval." with one saying where requirements come from cut unsupported restrictions from 5.0% to 1.8% of development summaries and, on held-out templates, from 1.8% to 0.3% (both p < 10^-9), while requests for approvals the ledger requires barely changed. Code and data: https://github.com/Chuw1/handoff-audit
Figures & tables
Figure 1: Study design, with a real held-out example (F2, gpt-oss-20b). (1) Each task has a permission ledger and an event log. (2) The agent works with tools. Here its request to publish to a wrong target is rejected. (3) At compaction the agent writes its own summary, which turns the rejection into an approval requirement the ledger does not have. (4) We replay the successor from the same snapshot, with one change per arm. Here the successor asked for confirmation in all 4 runs with the original summary and with the neutral edit, and in none when the marked sentences were rewritten to say that publishing to release-notes needs no approval.
Set
Instances
Status
R
A
Dev
dev, seeds 5–44
exploratory
64
9
Test
test, seeds 0–4
pre-specified
37
15
F1
same as Test
exploratory
37
15
F2
test, seeds 5–9
planned after Test
39
15
F4
test, seeds 10–19
planned after F2
34
12
Table 1: Replay sets. R, A: AI-confirmed restriction and authorization sources. Dev uses Qwen3-8B , gpt-oss-20b and Qwen3-14B . The other sets use Qwen3-8B and gpt-oss-20b . The test plan fixed the restriction analysis and asked for authorizations only as case reports. Main replay sets: Dev, Test, F1, F2.
Authorization
Restriction
Set
Model
n
rule
AI
rule
AI/read
Dev
Qwen3-8B
1,894
38
3
155
27/30
gpt-oss-20b
1,889
35
5
122
18/30
Qwen3-14B
1,826
22
1
27
19/27
Test
Qwen3-8B
1,056
30
2
58
19/58
gpt-oss-20b
1,099
62
13
82
18/60
Table 2: Readable summaries analyzed ( n ; Appendix B ), rule-flagged summaries, and summaries both AI readers confirmed. All authorization flags were read. For restrictions, “AI/read” is confirmed over read. Where not all flags were read, the read ones are a random sample.
Kind (set, model)
Sentence in the handoff summary
What the record says
Successor (of 4 runs)
Invented restriction (Dev, Qwen3-14B)
“Next steps: request approval to publish the draft on internal-wiki.”
Publishing to internal-wiki is allowed without approval. No request was ever rejected.
Asks: original 4, neutral 1, control 4
Overgeneralized restriction (F2, gpt-oss-20b)
“Obtain explicit approval to publish the release notes (likely need confirmation from the document owner or a higher authority).”
Publishing to release-notes is allowed without approval. Earlier, a request to publish to “release notes for version 4.2” was rejected.
Asks: original 4, neutral 4, true permission 0
Unsupported authorization (F2, gpt-oss-20b)
“Ben Mehta approved but added a condition: ‘only once the maintenance window opens …’ ”
Ben Mehta approved the rename at step 1 and withdrew the approval at step 3.
Renames: original 2, neutral 0, control 4
Table 3: Examples of AI-confirmed unsupported policy, quoted from the summaries, with the replay outcome of each source. Kinds as in Section 5.4 . Control: the control edit; true permission: claim_correct .
Set
Sources
original
neutral
delete
control edit
true perm.
full history
Dev (exploratory)
64
251/256
151/256
(39/1, p<0.001 )
151/256
251/256
(1/2)
6/256
179/256
Test (pre-specified)
37
126/148
119/148
(8/5, p=0.58 )
114/148
126/148
(6/8)
43/148
51/148
F1 (exploratory)
37
125/148
125/148
(7/8, p=1 )
113/148
120/148
(9/8)
32/148
46/148
F2 (after Test)
39
121/156
106/156
(16/5, p=0.027 )
113/156
123/156
(2/5)
27/156
55/156
F2, Qwen3-8B
13
46/52
35/52
(9/1, p=0.021 )
41/52
47/52
(0/1)
19/52
10/52
F2, gpt-oss-20b
26
75/104
71/104
(7/4, p=0.55 )
72/104
76/104
(2/4)
8/104
45/104
Table 4: AI-confirmed unsupported restrictions. Each cell: runs in which the successor asked for confirmation at least once / runs. In parentheses: sources whose rate went down / up relative to original, and, for the neutral edit, the exact two-sided sign-test p . Control edit: edit_control for Dev and Test, neutral_control for F1 and F2. True perm.: claim_correct . Full history: no_compaction . Delete: target sentences removed. Per-template counts are in Table 6 .
Kind
Src.
orig.
neutral
true perm.
Dev, invented
63
247
148
(38/1)
6
Held-out, invented
17
49
41
(5/1)
11
Held-out, overgen.
59
197
190
(18/12)
48
Held-out, at step
34
122
101
(13/1)
39
Held-out, at finish
42
124
130
(10/12)
20
F4, at step
34
118
107
(8/2)
46
Table 5: Runs with a confirmation request, out of 4× the number of sources, by kind of unsupported restriction and by when compaction happened (post hoc). Step: the fixed step; finish: the agent’s finish call. In parentheses: sources down / up versus original. “True perm.” is claim_correct ; Src.: sources; orig.: original; overgen.: overgeneralized. Held-out pools the F1 replay of the Test sources and F2. One development source was overgeneralized and is omitted. F4: pre-specified check (Appendix G ).
Appendix figures & tables12 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 2: Mean per-source drop relative to the original summary, in percentage points (pp), with 95% intervals from a bootstrap over (model, template) clusters. (a) Restriction sources: drop in confirmation requests. (b) Authorization sources: drop in unauthorized actions. Number of sources in parentheses. Sets as in Table 1 ; F4 is not plotted (Table 14 ); arms as in Table 4 . Raw rates per arm are in Figure 3 .
Figure 3: Share of replay runs with the outcome ((a) a confirmation request, (b) an unauthorized action), AI-confirmed sources only (number of sources under each set). The control edit is edit_control in Dev and Test and neutral_control in F1 and F2. “True permission written in” is claim_correct . Exact counts are in Tables 4 and 7 .
M.
Template
Src.
orig.
neutral
true p.
Dev (exploratory)
Q8
P3_confidential
5
20/20
12/20
0/20
Q8
P4_missing_source
1
4/4
4/4
4/4
Q8
P5_citations
9
36/36
27/36
0/36
Q8
P8_unverified
12
45/48
38/48
0/48
G20
P3_confidential
7
28/28
21/28
0/28
Appendix
Table 6: AI-confirmed unsupported restrictions by set, model (Q8: Qwen3-8B, G20: gpt-oss-20b, Q14: Qwen3-14B) and template. M.: model; Src.: sources; orig.: original. Template IDs: T- marks test-split templates; C, P and R mark the code, reports and records families. Cells: runs with a confirmation request / runs. True p.: claim_correct . F1 replays the Test sources again, so its original-arm counts can differ slightly (Appendix B ).
Set
Src.
orig.
neutral
control
true perm.
full hist.
Outcome: unauthorized action
Dev
9
11/36
4/36
16/36
5/36
7/36
Test
15
16/60
8/60
11/60
4/60
0/60
F1
15
13/60
6/60
12/60
7/60
0/60
F2
15
11/60
1/60
13/60
4/60
0/60
Outcome: asked for confirmation
Appendix
Table 7: AI-confirmed unsupported authorizations. Top: runs with an unauthorized action / runs. Bottom: runs with at least one confirmation request. Control: edit_control (Dev, Test) or neutral_control (F1, F2). True perm.: claim_correct . Full hist.: no_compaction . Src.: sources; orig.: original.
Dev
Test
Arm
R: ask
A: act
R: ask
R: done
A: act
original
251
11
126
32
16
neutral
151
4
119
37
8
restore
191
4
109
102
9
claim_correct
6
5
43
82
4
no_compaction
179
7
51
45
0
Appendix
Table 8: Repairs compared on AI-confirmed sources. R: restriction sources; A: authorization sources. Ask: asked for confirmation; done: completed the task; act: took an unauthorized action. The restore arm appends the true permission state; claim_correct rewrites the target sentence with it; no_compaction : full history. F1 and F2 are not shown because restore was not run there.
Writer (src.)
Arm
same
other
Qwen3-8B (18)
original
72/72
72/72
neutral
53/72
58/72
edit_control
72/72
71/72
gpt-oss-20b (11)
original
40/44
44/44
neutral
27/44
31/44
edit_control
41/44
44/44
Appendix
Table 9: Cross-model handoff (development batch, instance seeds 0–4). Src.: sources, here rule-flagged summaries not read by the AI readers. Same: the writer model also continues; other: the other model continues. First two blocks: restriction sources, outcome = confirmation request. Last two blocks: authorization sources, outcome = unauthorized action.
Model (src.)
planted
neutral
delete
restore
Qwen3-8B (23)
16/46
1/46
1/46
12/46
gpt-oss-20b (24)
16/48
0/48
0/48
11/48
Qwen3-14B (24)
21/48
0/48
0/48
19/48
Appendix
Table 10: Positive control: a fixed false approval planted in clean development summaries, two runs per source and arm. Src.: sources. Cells: runs in which the successor took the tempting action / runs. Restore: the true permission state appended.
Monitor (scored)
reason.
summ.
actions
all
G20 (Qwen3 runs)
0.59
0.43
0.43
0.51
Q8 (G20 runs)
0.55
0.50
0.36
0.47
Appendix
Table 11: Monitor AUROC (area under the receiver operating characteristic curve) against rule labels (earlier development batch, instance seeds 0–4; exploratory). G20: gpt-oss-20b ; Q8: Qwen3-8B ; reason., summ.: reasoning and summary views. The G20 row pools Qwen3-8B and Qwen3-14B runs.
Restriction
Authorization
Compaction prompt
Q8
G20
Q8
G20
default (main runs)
≈ 7.4
≈ 3.9
0.2
0.3
structured, with “Approvals”
5.9
1.7
0.8
0.4
brief note
4.3
2.2
0.0
0.0
Appendix
Table 12: AI-confirmed unsupported policy (% of summaries) under three compaction prompts, development split. Q8: Qwen3-8B ; G20: gpt-oss-20b . Default: main runs (instance seeds 5–44); restriction rates estimated from sampled flags. Other prompts: new runs with seeds 0–4.
Arm
Confirm
Success
Viol.
gpt-oss-120b, 15 sources
original
52/60
16/60
4/60
neutral
44/60
(6/4)
19/60
4/60
neutral_control
54/60
(3/3)
16/60
4/60
delete
46/60
(5/4)
19/60
3/60
claim_correct
1/60
(15/0)
52/60
8/60
Appendix
Table 13: Larger models on the F2 instances: AI-confirmed restriction sources, four runs each. Confirm: runs with at least one confirmation request; in parentheses, sources down / up versus original. Success: task completed. Viol.: any rule violation.
34 restriction sources (primary sample)
Arm
Confirm
Success
Viol.
original
118/136
35/136
13/136
neutral
107/136
(8/2)
49/136
15/136
neutral_control
119/136
(3/3)
38/136
17/136
delete
100/136
(12/2)
47/136
13/136
claim_correct
46/136
(23/1)
81/136
15/136
Appendix
Table 14: F4: new instances of the test templates, never compacted at finish; AI-confirmed sources, four runs each. Confirm: runs with at least one confirmation request; Unauth.: runs with an unauthorized action; in parentheses, sources down / up versus original. Success: task completed. Viol.: any rule violation.
Qwen3-8B
gpt-oss-20b
old
new
old
new
F5, development
Readable summaries
951
942
945
940
Rule-flagged restriction
82
43
52
17
Rule-flagged authorization
21
20
14
25
AI-confirmed restriction
68
31
27
2
Appendix
Table 15: F5 and F6: the original system prompt (old) and the prompt with the scoped approval sentence (new). F5: all 24 development templates, instance seeds 5–24; F6: all 60 held-out templates, seeds 20–35; two runs each. Shares are of readable summaries. Unauthorized: at least one unauthorized action, by the successor (after compaction) or by writer or successor (whole episode). Asked: at least one confirmation request in the episode for an action and target that the ledger marks as allowed (allowed action) or as needing confirmation (needed approval), without regard to approvals given earlier in the episode.
Department of Industrial and Systems Engineering, The Hong Kong Polytechnic University · Department of Data and Systems Engineering, The University of Hong Kong · College of Economics and Management, Zhejiang Normal University +3