Organizations: Zhejiang University, China · Zhejiang Key Laboratory of Digital-Intelligence Service Technology · University of Oxford, UK · Xidian University, China
Much of the automation handed to language-model agents is procedural: each step needs little intelligence, but a long unattended run must follow a skill or SOP stated once at the start. We show that agents drift from such procedures over long runs, and that restating the instructions or resuming a stopped agent only moves the drift to another shortcut. We study an executable finite-state machine (FSM) that keeps the workflow state outside the model's context and enforces it at three points: tool calls, state transitions and the end of the agent's turn. In a pre-registered experiment on 200-account SOP-Bench batches, Claude Haiku 4.5 with the FSM decided on average 139 accounts correctly and by the procedure, against 48 for the strongest baseline (95% CI of the difference [46, 130]), and kept this rate to the end; across three models, the gain tracked how much the plain agent drifted. Because the runtime, not the model, supplies this long-horizon reliability, the FSM can let small models carry procedural work: Haiku 4.5 with it averaged more such decisions than plain Claude Sonnet 4.6 at 38% of the cost. Analyses on SWE-bench Verified, τ2-bench, and staged and cross-context workflows further suggest that reliable long-horizon automation can come from stronger execution control, not only stronger models.
Figures & tables
Figure 1: Four conditions on one 200-account batch.
Figure 2: Structure of the executable FSM , with the workflow of the main experiment.
Model
Condition
FC
FC per 50 accounts
Compliance
Decided
Correct
Cost ($)
1–50
51–100
101–150
151–200
Claude Haiku 4.5 (8 seeds)
Plain
9.4 ± 9.3
7.9
1.5
0.0
0.0
11.9%
143.8
100.0
0.96
Sys
17.1 ± 11.2
13.2
3.9
0.0
0.0
20.1%
120.1
86.1
1.07
Remind
14.6 ± 13.2
14.6
0.0
0.0
0.0
48.6 %
53.0
40.6
0.65
Plain-loop
18.2 ± 12.1
8.9
3.8
2.8
2.9
14.3%
200.0
124.1
1.71
Sys-loop
40.9 ± 47.0
11.4
7.0
10.9
11.6
28.9%
200.0
144.0
2.21
Table 1: Main results on 200-account unattended batches (mean over seeds ± s.d.).
Variant
Gates
Stop gate
FC
Decided
Compl.
Blind
Stops
Remind-loop
off
off
76.3
200.0
44.0%
0.0
7.3
Stop-only
off
on
5.7
146.7
3.7%
108.3
0.3
Gates-only
on
off
104.0
151.7
93.0%
0.0
44.7
FSM
on
on
128.0
200.0
93.3 %
0.0
76.3
Table 2: Ablation of the enforcement points.
Task set
Agent
Resolved
New tests
Time (s)
Cost ($)
Hard-100 (Sonnet 4.6)
Plain
62
24
–
–
FSM
69
39
–
–
Easy-100 (Sonnet 4.6)
Plain
86
57
310
1.30
FSM
87
64
381
1.73
Table 3: Analysis 1: SWE-bench Verified.
Domain
Tasks
Control
FSM
Discordant
p
(a) First frozen evaluation
Telecom
40
13
30
17 / 0
1.5×10−5
Airline
22
17
17
1 / 1
1.000
Banking
68
6
6
3 / 3
1.000
Retail
30
15
9
3 / 9
0.146
(b) Domain-specific revisions (descriptive)
Table 4: Analysis 2: τ2 -bench.
Protocol
Measure
Plain
FSM
AutoResearch (4 campaigns)
valid ledger rows
48/100
100/100
complete protocols
1/4
4/4
best accuracy (%)
99.33
99.33
Month-end close (6 periods per seed)
correct periods, 40 turns
8/12
11/12
correct periods, 24 turns
3/6
1/6
DuckDB optimizer (4 pairs)
mean official reward
0.19
0.75
Table 5: Analysis 3: staged protocols.
Appendix figures & tables6 assets
Supplementary material from the paper’s appendix.
Appendix
Prediction
Observed
Verdict
Claude Haiku 4.5, seeds 2–9
P1 Prompting is not enough: Plain, Sys, Remind each have fewer than 200 decisions or compliance below 80% in ≥ 6/8 seeds
8/8, 8/8, 8/8
holds
P2 Resumption only reroutes: Plain-loop, Sys-loop, Remind-loop each have ≥ 190 decisions and compliance below 80% in ≥ 6/8 seeds
8/8, 7/8, 7/8
holds
P3 FSM: FSM decides all 200 with compliance ≥ 95% in every seed; FSM − Remind-loop ≥ 30 on average and positive in ≥ 7/8; conservative contrast ≥ 20 and positive in ≥ 6/8
contrasts met ( + 90.9, 7/8; + 70.5, 7/8); compliance 80% in seeds 4 and 6
fails
P4 No harm: strict accuracy of FSM ≥ Remind-loop
146.5 vs. 138.9
holds
P5 Long horizon: FSM − Remind-loop in the last window ≥ first window
29.1 vs. 2.0
holds
Appendix
Table 6: Pre-registered predictions and outcomes.
Model
Comparison
Mean
95% CI
Seeds >0
Sign p
Wilcoxon p
Claude Haiku 4.5 (8 seeds)
FC: FSM − Remind-loop
+ 90.9
[46.2, 130.2]
7/8
0.070
0.023
FC: FSM − best -loop baseline
+ 70.5
[27.4, 114.5]
7/8
0.070
0.039
FC: FSM − Plain
+ 129.9
[104.1, 151.9]
8/8
0.008
0.008
Correct: FSM − Remind-loop
+ 7.6
[ − 16.1, 29.8]
5/8
0.727
0.484
Claude Sonnet 4.6 (3 seeds)
FC: FSM − Remind-loop
+ 55.0
[ − 15.0, 191.0]
1/3
1.000
1.000
FC: FSM − Plain
+ 79.3
[9.0, 184.0]
3/3
0.250
0.250
Appendix
Table 7: Paired comparisons over seeds.
Condition
Ends turn, accounts left
Decides without looking
Decides on partial data
Interleaved accounts
Submits for another account
First token- pressure remark
Plain
4/8 (0.5)
4/8 (49.1)
3/8 (17.5)
8/8 (65.8)
5/8 (87.8)
63
Sys
6/8 (0.8)
3/8 (32.8)
1/8 (2.5)
7/8 (63.6)
2/8 (32.4)
66.5
Remind
8/8 (1.0)
1/8 (20.2)
0/8
6/8 (15.9)
5/8 (28.2)
22
Plain-loop
6/8 (1.5)
5/8 (48.6)
0/8
8/8 (122.8)
4/8 (82.8)
81
Sys-loop
5/8 (3.0)
3/8 (38.2)
2/8 (3.9)
7/8 (100.0)
5/8 (88.4)
69.5
Remind-loop
8/8 (5.5)
3/8 (29.9)
5/8 (14.2)
7/8 (100.2)
7/8 (105.2)
26.5
Appendix
Table 8: How each condition left the procedure (Claude Haiku 4.5).
Task set
Model
Plain
FSM
Discordant
p
New tests
Empty
Time (s)
Cost ($)
Hard-100
Claude Sonnet 4.6
62
69
7 / 0
0.016
24 / 39
0 / 0
–
–
first 50
34
37
3 / 0
0.250
13 / 20
0 / 0
–
–
second 50
28
32
4 / 0
0.125
11 / 19
0 / 0
–
–
Easy-100
Claude Sonnet 4.6
86
87
5 / 4
1.000
57 / 64
2 / 1
310 / 381
1.30 / 1.73
Appendix
Table 9: Analysis 1 in detail (pairs of values are plain / FSM).
Valid ledger rows
Complete protocol
Best accuracy (%)
(a) AutoResearch
Plain
FSM
Plain
FSM
Plain
FSM
Campaign 1
7/25
25/25
no
yes
99.33
99.33
Campaign 2
8/25
25/25
no
yes
99.33
99.33
Campaign 3
8/25
25/25
no
yes
99.33
99.33
Campaign 4
25/25
25/25
yes
yes
99.33
99.33
Appendix
Table 10: Analysis 3 in detail.
Experiment
Workflow
States
YAML lines
Guards and other check code (lines)
Main experiment
ReferralAbusePerAccount
3
61
one guard script with three modes (156); stop-gate message wrapper (95); the YAML and the reminder text are generated from the SOP by a build script (119)
SWE-bench Verified
SWEDebug
4
78
seven inline shell checks (clean tree, notes file exists, no .py change before the fix state); no script
τ2 -bench Telecom
TauTelecomTroubleshootingV1
4
73
no hard checks, deny rules only; Python adapter: rules (535), runtime bridge (374), controller (377)
AutoResearch
AutoResearchSklearn
10
160
one stage-check script, nine calls (238)
Month-end close
MonthlyClose
7
108
one stage-check script, six calls (177)
DuckDB optimizer
DuckDBLongHorizonCausalV21
5
126
stage guard (341) and coherence check (62); context-boundary hook (1,214) and phase-action runner (262)
Appendix
Table 11: Size of the six workflow specifications and their check code.
Enterprise agents must follow long-horizon, conditional, safety-critical standard operating procedures (SOPs). We compile machine-readable SOP constraints into executable pseudo-code and run them with a program-guided (PG) stack machine that pages the active frame while an LLM performs semantic execution. A three-arm SOPBench study across six models separates representation from runtime: compiled text never significantly hurts and gains up to 16.0 points where official prose underperforms. Runtime guidance is capability-gated. Two strong models independently show positive seven-domain PG contrasts (58:19 and 75:31 discordant pairs), whereas weak models are harmed. A full-program cursor ablation (active frame first, complete program retained) recovers much of the strong-model refusal gain; selective visibility adds a smaller improvement. Paired probe and audit measurements track this divide to spontaneous state discipline rather than reconstruction ability. On Bank the three primary arms rise from 70.4 to 86.4 to 92.8, with 100% refusal correctness. Practical guidance: compile first; enable active-frame paging only after a model-level discipline check.
Chenglin Yu, Li Yin, Ying Yu +4
Department of Industrial and Systems Engineering, The Hong Kong Polytechnic University · Department of Data and Systems Engineering, The University of Hong Kong · College of Economics and Management, Zhejiang Normal University +3
As large language model (LLM) agents are applied to longer tasks, they increasingly modify workspace state across multiple rounds of iteration. However, agents typically observe only tool outputs and log fragments, while the actual state changes occur in the file system. Without explicit workspace boundaries, state-changing operations such as file writes and temporary artifact generation may scatter changes across paths. Over time, these weakly constrained changes accumulate, making states such as modified files difficult to track. This paper presents LemonHarness, an integrated execution framework for long-horizon agents. LemonHarness establishes an explicit execution boundary by constraining state-changing operations within a clearly defined workspace and bringing model invocation, tool execution, and rule knowledge within a single controlled boundary. State-changing operations, including file writes, dependency installation, and temporary artifact creation, are executed through structured tool interfaces, with execution feedback recorded as observations available to subsequent model decisions. The system also introduces a reusable rule knowledge base, which turns recurring execution rules and acceptance criteria into runtime knowledge. LemonHarness further adds a time-aware execution mechanism that exposes elapsed and remaining budget to the model, so it can rebalance exploration, implementation, and validation effort as time pressure shifts and avoid timeouts from long waits or excessive verification. On Terminal-Bench 2.0, LemonHarness_GPT-5.3-CodeX reached 84.49% accuracy over 445 trials; pairing the same framework with the stronger GPT-5.5 backbone raised the average accuracy to 86.52% across five jobs. The results suggest that a unified runtime boundary, callable rule knowledge, and time-aware execution can improve the stability of long-horizon agent execution.
Large language model (LLM) agents increasingly undertake long-horizon tasks that require sustained reasoning, tool use, and revision across many interdependent steps. However, existing agent harnesses maintain task execution, task state, and completion assessment within a growing context, making the state difficult to track and allowing incorrect self-assessments to propagate into later decisions. We reformulate long-horizon execution as a task-state management problem and propose LongHorizon-Harness, which maintains the task state explicitly outside execution and updates it only with facts independently verified from the environment. Its Manage-Execute-Audit(MEA) loop uses a manager to maintain the task state and determine the next subtask, a fresh-context executor to perform it, and a read-only auditor to verify the resulting environment state before the next round. A lightweight AgentAdapter supports interchangeable model and harness backends without modifying their native agent loops. LongHorizon-Harness improves Qwen3.7-Plus from 51.8% to 80.7% on WeaveBench, from 69.7% to 77.2% on Terminal-Bench2.1, and from 2.8% to 8.3% on OSWorld2.0. It also raises Claude Opus4.7 from 20.0% to 34.3% on an OSWorld2.0 subset, demonstrating consistent gains across models, harnesses, and interaction domains.