Organizations: Zhejiang University, China · Zhejiang Key Laboratory of Digital-Intelligence Service Technology · University of Oxford, UK · Xidian University, China
Much of the automation handed to language-model agents is procedural: each step needs little intelligence, but a long unattended run must follow a skill or SOP stated once at the start. We show that agents drift from such procedures over long runs, and that restating the instructions or resuming a stopped agent only moves the drift to another shortcut. We study an executable finite-state machine (FSM) that keeps the workflow state outside the model's context and enforces it at three points: tool calls, state transitions and the end of the agent's turn. In a pre-registered experiment on 200-account SOP-Bench batches, Claude Haiku 4.5 with the FSM decided on average 139 accounts correctly and by the procedure, against 48 for the strongest baseline (95% CI of the difference [46, 130]), and kept this rate to the end; across three models, the gain tracked how much the plain agent drifted. Because the runtime, not the model, supplies this long-horizon reliability, the FSM can let small models carry procedural work: Haiku 4.5 with it averaged more such decisions than plain Claude Sonnet 4.6 at 38% of the cost. Analyses on SWE-bench Verified, τ2-bench, and staged and cross-context workflows further suggest that reliable long-horizon automation can come from stronger execution control, not only stronger models.
Figures & tables
Figure 1: Four conditions on one 200-account batch.
Figure 2: Structure of the executable FSM , with the workflow of the main experiment.
Model
Condition
FC
FC per 50 accounts
Compliance
Decided
Correct
Cost ($)
1–50
51–100
101–150
151–200
Claude Haiku 4.5 (8 seeds)
Plain
9.4 ± 9.3
7.9
1.5
0.0
0.0
11.9%
143.8
100.0
0.96
Sys
17.1 ± 11.2
13.2
3.9
0.0
0.0
20.1%
120.1
86.1
1.07
Remind
14.6 ± 13.2
14.6
0.0
0.0
0.0
48.6 %
53.0
40.6
0.65
Plain-loop
18.2 ± 12.1
8.9
3.8
2.8
2.9
14.3%
200.0
124.1
1.71
Sys-loop
40.9 ± 47.0
11.4
7.0
10.9
11.6
28.9%
200.0
144.0
2.21
Table 1: Main results on 200-account unattended batches (mean over seeds ± s.d.).
Variant
Gates
Stop gate
FC
Decided
Compl.
Blind
Stops
Remind-loop
off
off
76.3
200.0
44.0%
0.0
7.3
Stop-only
off
on
5.7
146.7
3.7%
108.3
0.3
Gates-only
on
off
104.0
151.7
93.0%
0.0
44.7
FSM
on
on
128.0
200.0
93.3 %
0.0
76.3
Table 2: Ablation of the enforcement points.
Task set
Agent
Resolved
New tests
Time (s)
Cost ($)
Hard-100 (Sonnet 4.6)
Plain
62
24
–
–
FSM
69
39
–
–
Easy-100 (Sonnet 4.6)
Plain
86
57
310
1.30
FSM
87
64
381
1.73
Table 3: Analysis 1: SWE-bench Verified.
Domain
Tasks
Control
FSM
Discordant
p
(a) First frozen evaluation
Telecom
40
13
30
17 / 0
1.5×10−5
Airline
22
17
17
1 / 1
1.000
Banking
68
6
6
3 / 3
1.000
Retail
30
15
9
3 / 9
0.146
(b) Domain-specific revisions (descriptive)
Table 4: Analysis 2: τ2 -bench.
Protocol
Measure
Plain
FSM
AutoResearch (4 campaigns)
valid ledger rows
48/100
100/100
complete protocols
1/4
4/4
best accuracy (%)
99.33
99.33
Month-end close (6 periods per seed)
correct periods, 40 turns
8/12
11/12
correct periods, 24 turns
3/6
1/6
DuckDB optimizer (4 pairs)
mean official reward
0.19
0.75
Table 5: Analysis 3: staged protocols.
Appendix figures & tables6 assets
Supplementary material from the paper’s appendix.
Appendix
Prediction
Observed
Verdict
Claude Haiku 4.5, seeds 2–9
P1 Prompting is not enough: Plain, Sys, Remind each have fewer than 200 decisions or compliance below 80% in ≥ 6/8 seeds
8/8, 8/8, 8/8
holds
P2 Resumption only reroutes: Plain-loop, Sys-loop, Remind-loop each have ≥ 190 decisions and compliance below 80% in ≥ 6/8 seeds
8/8, 7/8, 7/8
holds
P3 FSM: FSM decides all 200 with compliance ≥ 95% in every seed; FSM − Remind-loop ≥ 30 on average and positive in ≥ 7/8; conservative contrast ≥ 20 and positive in ≥ 6/8
contrasts met ( + 90.9, 7/8; + 70.5, 7/8); compliance 80% in seeds 4 and 6
fails
P4 No harm: strict accuracy of FSM ≥ Remind-loop
146.5 vs. 138.9
holds
P5 Long horizon: FSM − Remind-loop in the last window ≥ first window
29.1 vs. 2.0
holds
Appendix
Table 6: Pre-registered predictions and outcomes.
Model
Comparison
Mean
95% CI
Seeds >0
Sign p
Wilcoxon p
Claude Haiku 4.5 (8 seeds)
FC: FSM − Remind-loop
+ 90.9
[46.2, 130.2]
7/8
0.070
0.023
FC: FSM − best -loop baseline
+ 70.5
[27.4, 114.5]
7/8
0.070
0.039
FC: FSM − Plain
+ 129.9
[104.1, 151.9]
8/8
0.008
0.008
Correct: FSM − Remind-loop
+ 7.6
[ − 16.1, 29.8]
5/8
0.727
0.484
Claude Sonnet 4.6 (3 seeds)
FC: FSM − Remind-loop
+ 55.0
[ − 15.0, 191.0]
1/3
1.000
1.000
FC: FSM − Plain
+ 79.3
[9.0, 184.0]
3/3
0.250
0.250
Appendix
Table 7: Paired comparisons over seeds.
Condition
Ends turn, accounts left
Decides without looking
Decides on partial data
Interleaved accounts
Submits for another account
First token- pressure remark
Plain
4/8 (0.5)
4/8 (49.1)
3/8 (17.5)
8/8 (65.8)
5/8 (87.8)
63
Sys
6/8 (0.8)
3/8 (32.8)
1/8 (2.5)
7/8 (63.6)
2/8 (32.4)
66.5
Remind
8/8 (1.0)
1/8 (20.2)
0/8
6/8 (15.9)
5/8 (28.2)
22
Plain-loop
6/8 (1.5)
5/8 (48.6)
0/8
8/8 (122.8)
4/8 (82.8)
81
Sys-loop
5/8 (3.0)
3/8 (38.2)
2/8 (3.9)
7/8 (100.0)
5/8 (88.4)
69.5
Remind-loop
8/8 (5.5)
3/8 (29.9)
5/8 (14.2)
7/8 (100.2)
7/8 (105.2)
26.5
Appendix
Table 8: How each condition left the procedure (Claude Haiku 4.5).
Task set
Model
Plain
FSM
Discordant
p
New tests
Empty
Time (s)
Cost ($)
Hard-100
Claude Sonnet 4.6
62
69
7 / 0
0.016
24 / 39
0 / 0
–
–
first 50
34
37
3 / 0
0.250
13 / 20
0 / 0
–
–
second 50
28
32
4 / 0
0.125
11 / 19
0 / 0
–
–
Easy-100
Claude Sonnet 4.6
86
87
5 / 4
1.000
57 / 64
2 / 1
310 / 381
1.30 / 1.73
Appendix
Table 9: Analysis 1 in detail (pairs of values are plain / FSM).
Valid ledger rows
Complete protocol
Best accuracy (%)
(a) AutoResearch
Plain
FSM
Plain
FSM
Plain
FSM
Campaign 1
7/25
25/25
no
yes
99.33
99.33
Campaign 2
8/25
25/25
no
yes
99.33
99.33
Campaign 3
8/25
25/25
no
yes
99.33
99.33
Campaign 4
25/25
25/25
yes
yes
99.33
99.33
Appendix
Table 10: Analysis 3 in detail.
Experiment
Workflow
States
YAML lines
Guards and other check code (lines)
Main experiment
ReferralAbusePerAccount
3
61
one guard script with three modes (156); stop-gate message wrapper (95); the YAML and the reminder text are generated from the SOP by a build script (119)
SWE-bench Verified
SWEDebug
4
78
seven inline shell checks (clean tree, notes file exists, no .py change before the fix state); no script
τ2 -bench Telecom
TauTelecomTroubleshootingV1
4
73
no hard checks, deny rules only; Python adapter: rules (535), runtime bridge (374), controller (377)
AutoResearch
AutoResearchSklearn
10
160
one stage-check script, nine calls (238)
Month-end close
MonthlyClose
7
108
one stage-check script, six calls (177)
DuckDB optimizer
DuckDBLongHorizonCausalV21
5
126
stage guard (341) and coherence check (62); context-boundary hook (1,214) and phase-action runner (262)
Appendix
Table 11: Size of the six workflow specifications and their check code.
Department of Industrial and Systems Engineering, The Hong Kong Polytechnic University · Department of Data and Systems Engineering, The University of Hong Kong · College of Economics and Management, Zhejiang Normal University +3