cs.AIAug 26, 2026
SaveLocalLSTC: A Long Short-Term Control Architecture for Locally Deployed GUI Agents
Organizations: School of Computer Science and Engineering The University of New South Wales Sydney, Australia
Abstract
Modern GUI-agent frameworks achieve strong desktop task performance with frontier API models, yet persistent control information often remains implicit in growing interaction trajectories. At each step, the planner reconstructs the active task stage, accumulated evidence, and runtime feedback before deciding the next action. This dependence becomes more pronounced under weaker local reasoning backbones. Across four representative state-of-the-art frameworks, replacing GPT-5 with Qwen3.5-9B reduces average OSWorld SR-100 from 60.9% to 35.2%. Trajectory annotation further identifies at least one control failure in 94.7% of failed trajectories. To address this problem, we introduce LocalLSTC, a training-free architecture that organizes control by temporal scope, maintaining persistent cross-step state to guide short-term execution commitments. The framework comprises two complementary mechanisms. Long-to-Short Planning forms each commitment from persistent state, while Short-to-Long Control integrates execution outcomes back into that state for progress assessment, recovery, and termination. With Qwen3.8-27B, LocalLSTC reaches 73.7% SR-100 on OSWorld and 68.5% on WindowsAgentArena, achieving the strongest local OSWorld result and a new state of the art on WindowsAgentArena. Ablations further support contributions from both mechanisms. Together, these results show that temporal organization of control information can reduce cross-step control failures and complement backbone scaling in locally deployed GUI agents.
Figures & tables
| Access | Method / Backbone | OSWorld | WindowsAgentArena | ||||
| SR-100 | SR-50 | SR-15 | SR-100 | SR-50 | SR-15 | ||
| API | o3 ( OpenAI, 2025b ) | 23.0 | 17.2 | 9.1 | - | - | - |
| Agent S / GPT-4o ( Agashe et al., 2025a ) | - | - | 20.6 | - | - | 18.2 | |
| Navi / GPT-4V-1106 ( Bonatti et al., 2025 ) | - | - | - | - | - | 19.5 | |
| computer-use-preview ( OpenAI, 2025a ) | 30.5 | 31.3 | 26.0 | - | - | - | |
| Claude 3.7 Sonnet ( Anthropic, 2025a ) | 35.6 | 35.8 | 27.1 | - | - | - | |
Table 1: Success rates on OSWorld and WindowsAgentArena under step budgets of 100, 50, and 15. LocalLSTC results report mean SD over two runs. Other methods use their reported results. Bold and underlining mark the highest and second-highest values, and “-” indicates an unreported result.
| Summary | Commitment | Execution Channels | Evidence Control | ||||||||||
| Domain | Tasks | SR (%) | steps | Subgoals / Task | Actions / Subgoal | Actions / step | GUI (%) | Bash (%) | API (%) | Context Refines | Pass (%) | Fail (%) | Bypass (%) |
| Chrome | 46 | 59.7 | 20.4 | 7.8 | 2.9 | 1.1 | 86.8 | 1.3 | 2.7 | 2.0 | 64.1 | 13.0 | 22.8 |
| GIMP | 26 | 78.8 | 14.3 | 5.1 | 3.9 | 1.4 | 86.3 | 7.4 | 0.0 | 1.1 | 75.0 | 5.8 | 19.2 |
| LibreOffice Calc | 47 | 28.7 | 30.1 | 11.3 | 4.2 | 1.6 | 75.5 | 5.0 | 13.3 | 3.2 | 42.6 | 20.2 | 37.2 |
| LibreOffice Impress | 47 | 33.4 | 24.7 | 8.1 | 4.7 | 1.6 | 81.0 | 1.6 | 9.5 | 2.2 | 41.5 | 19.1 | 39.4 |
| LibreOffice Writer | 23 | 39.1 | 21.7 | 8.0 | 4.2 | 1.6 | 83.4 | 3.8 | 4.7 | 2.1 | 45.7 | 19.6 | 34.8 |
Table 2: Domain-level runtime behavior of LocalLSTC with Qwen3.5-9B on OSWorld, averaged over two runs. Execution-channel columns report action shares, with the remainder reserved for control actions. Pass, Fail, and Bypass are mutually exclusive trajectory categories based on Final Verification history. Bypass takes precedence for consecutive-stall or maximum-step cutoffs without an explicit INFEASIBLE action. In all other cases, a trajectory is classified as Pass if it has no rejected Final Verification and as Fail if it has at least one, even when a later verification succeeds. An explicit INFEASIBLE action counts as an accepted Final Verification outcome, with earlier rejections still counted.
| Category | Method | SR-100 | Tokens (k) | Prompt (k) | Completion (k) | Steps | Time (s) |
| Full | LocalLSTC | 47.6 | 252.2 | 237.6 | 14.6 | 26.1 | 390.4 |
| Boundary | w/o L2S | 36.4 | 192.9 | 179.1 | 13.8 | 31.4 | 382.2 |
| w/o S2L | 41.5 | 246.5 | 229.1 | 17.4 | 27.1 | 472.7 | |
| Components | w/o persistent-subgoal interface | 38.0 | 276.6 | 257.3 | 19.3 | 29.6 | 554.4 |
| w/o candidate proposals | 40.7 | 273.0 | 259.1 | 13.9 | 31.2 | 412.5 | |
| w/o multi-action list | 43.5 | 260.7 | 243.0 | 17.8 | 28.8 | 445.5 |
Table 3: Grouped-mechanism and component ablations on OSWorld with Qwen3.5-9B. Resource metrics report per-task averages. Bold marks the highest SR or lowest resource value in each column.
| Stall / repetition | Recovery entry | Rejected termination | ||||
| Method | Rate (%) | Score (%) | Rate (%) | Score (%) | Rate (%) | Score (%) |
| Full | 4.5 | 34.3 | 60.0 | 34.0 | 32.1 | 38.9 |
| w/o L2S | 4.3 | 18.8 | 56.5 | 20.9 | 19.3 | 23.1 |
| w/o S2L | 4.6 | 11.8 | – | – | – | – |
Table 4: Runtime-event diagnostics for Full, w/o L2S , and w/o S2L on OSWorld. Rate reports the fraction of tasks that enter each event at least once, and Score reports mean evaluator score conditional on event entry. Bold marks the column maximum, and “–” indicates an event unavailable after removing the corresponding mechanism.
Appendix figures & tables7 assets
Supplementary material from the paper’s appendix.
Appendix
| Rule or hyperparameter | Value |
| Model-inference temperature | 0 |
| Model-inference top_p | 0.95 |
| Model-inference top_k | 20 |
| Model-inference sampling seed | 42 |
| Model-inference thinking token budget | 16384 |
| Maximum candidate proposals per inference step | 3 |
Table 5: Runtime parameters shared across LocalLSTC experiments.
| Label | A1 prev. (%) | A2 prev. (%) | Agreement (%) | F1 (%) | |
| Stage | 20.0 | 20.5 | 88.6 | 72.0 | 0.649 |
| Assess | 22.2 | 30.8 | 84.9 | 71.4 | 0.615 |
| Loop | 25.4 | 22.2 | 92.4 | 84.1 | 0.792 |
| Recover | 28.1 | 18.4 | 87.0 | 72.1 | 0.641 |
| Stop | 25.4 | 23.8 | 87.6 | 74.7 | 0.665 |
| Other | 9.7 | 10.8 | 90.3 | 52.6 | 0.472 |
Table 6: Inter-annotator agreement on 185 trajectories. A1 and A2 report label prevalence for the two independent human annotators. Agreement, F1, and Cohen’s are computed separately for each label.
| Annotator 1 | Annotator 2 | |||||||
| Label | P (%) | R (%) | F1 (%) | P (%) | R (%) | F1 (%) | ||
| Stage | 68.4 | 70.3 | 69.3 | 0.615 | 60.5 | 60.5 | 60.5 | 0.503 |
| Assess | 62.5 | 73.2 | 67.4 | 0.572 | 62.5 | 52.6 | 57.1 | 0.403 |
| Loop | 88.4 | 80.9 | 84.4 | 0.795 | 74.4 | 78.0 | 76.2 | 0.692 |
| Recover | 64.1 | 96.2 | 76.9 | 0.652 | 39.7 | 91.2 | 55.4 | 0.400 |
| Stop | 61.3 | 80.9 | 69.7 | 0.574 | 48.4 | 68.2 | 56.6 | 0.399 |
Table 7: Agreement between GPT-5.6-sol and the two independent human annotators on the same 185 trajectories. Precision, recall, F1, and Cohen’s are computed separately with each annotator as the reference.
| Method | Stage | Assess | Loop | Recover | Stop | None | Other | Any |
| HiSA | 36.3% | 19.2% | 14.1% | 28.2% | 51.2% | 31.2% | 12.7% | 56.1% |
| CoAct-1 | 10.3% | 27.6% | 27.4% | 38.2% | 47.2% | 28.5% | 3.8% | 67.8% |
| GTA1 | 11.7% | 23.0% | 26.3% | 31.4% | 49.1% | 32.2% | 5.4% | 62.3% |
| Agent S3 | 10.8% | 23.6% | 11.7% | 17.3% | 64.8% | 28.2% | 3.3% | 68.6% |
| LocalLSTC | 18.2% | 21.4% | 28.5% | 33.6% | 26.6% | 42.5% | 15.7% | 41.7% |
Table 8: Task-level incidence of control-failure labels on OSWorld under the fixed Qwen3.5-9B planner backbone. Each cell reports the percentage of all 369 tasks assigned the corresponding label.
| Summary | Commitment | Execution Channels | Evidence Control | ||||||||||
| Domain | Tasks | SR (%) | steps | Subgoals / Task | Actions / Subgoal | Actions / step | GUI (%) | Bash (%) | API (%) | Context Refines | Pass (%) | Fail (%) | Bypass (%) |
| Chrome | 17 | 34.7 | 24.1 | 8.1 | 3.3 | 1.1 | 84.5 | 0.9 | 5.3 | 2.0 | 35.3 | 23.5 | 41.2 |
| Clock | 4 | 37.5 | 34.5 | 10.5 | 5.5 | 1.7 | 96.0 | 0.0 | 0.0 | 3.4 | 50.0 | 0.0 | 50.0 |
| File Explorer | 19 | 63.2 | 18.5 | 7.8 | 3.0 | 1.3 | 73.4 | 14.4 | 0.0 | 1.6 | 50.0 | 28.9 | 21.1 |
| LibreOffice Calc | 24 | 8.3 | 34.7 | 13.8 | 4.5 | 1.8 | 69.3 | 1.1 | 23.3 | 3.9 | 12.5 | 16.7 | 70.8 |
| LibreOffice Writer | 19 | 7.9 | 24.4 | 11.2 | 3.1 | 1.4 | 79.4 | 0.2 | 7.3 | 2.4 | 18.4 | 7.9 | 73.7 |
Table 9: Domain-level runtime behavior of LocalLSTC with Qwen3.5-9B on WindowsAgentArena, averaged over two runs. Execution-channel columns report action shares, with the remainder reserved for control actions. Pass, Fail, and Bypass are mutually exclusive trajectory categories based on Final Verification history. Bypass takes precedence for consecutive-stall or maximum-step cutoffs without an explicit INFEASIBLE action. Otherwise, Pass indicates no rejected Final Verification, while Fail indicates at least one, even if a later verification succeeds. An explicit INFEASIBLE action counts as an accepted Final Verification outcome, with earlier rejections still counted.
| Configuration | Full | w/o L2S | w/o S2L |
| Planning context | Persistent , subgoal-aligned evidence, refined history, and recent abstracted steps | Task-conditioned evidence and history without a persistent subgoal in planner context | Planner retains a subgoal proposal, while evidence context is task-conditioned |
| Proposal and commitment | Up to three proposals compared within one inference, followed by an ordered action list | Single proposal with one action per planner step | Same as Full |
| Step Abstraction | Subgoal-conditioned evidence across execution channels | Retained with task-level conditioning | Retained with task-level conditioning |
| State update and routing | Evidence-conditioned continue , advance , error , stall , and finish routing | Task-level error, stall, and finish handling without subgoal-based stage transitions | Planner-intent state updates without S2L routing |
| Recovery feedback | State-conditioned error and stall feedback | Retained at the task level | Removed |
| Context Refinement | advance -triggered refinement with maximum-interval fallback | Periodic refinement without subgoal advance | Periodic refinement without state routing |
Table 10: Grouped L2S and S2L ablation configurations under the fixed Qwen3.5-9B backbone. Full denotes the complete LocalLSTC configuration.
| Ablation | Removed behavior | Retained or replacement behavior |
| Persistent-subgoal interface | The planner does not receive the previous subgoal and cannot use continue . | Each call generates a new subgoal label. Candidate proposals, multi-action commitments, all execution channels, evidence, routing, refinement, and Final Verification remain enabled. |
| Candidate proposals | The planner omits the candidate list, Best index, and selection rationale. | The planner directly returns one final JSON decision. Persistent subgoals, ordered action lists, all execution channels, and S2L remain enabled. |
| Multi-action list | The final decision cannot contain an ordered actions list. | The response uses one action per planner call. Candidate proposals, persistent subgoals, all execution channels, and S2L remain enabled. |
| Step Abstraction | The semantic Step Abstraction model call is removed. | Deterministic raw evidence records the action and observed change for GUI execution. Bash and API evidence additionally records status, exit code, and raw output. Routing, refinement, and Final Verification remain enabled. |
| State-conditioned routing | Assigned continue , advance , error , stall , and finish states are logged but not applied to runtime control. | Step Abstraction and planner subgoal updates remain. Context Refinement becomes periodic. Routing-dependent recovery, stall and loop enforcement, and Final Verification are bypassed. |
| Stall and loop handling | Repeated-action checks and action–result loop checks are disabled. Execution outcomes are not remapped to error or stall from tool failure or repeated no-change evidence. | Planner subgoal transitions, Step Abstraction, Context Refinement, and Final Verification remain. Runtime recovery feedback for these removed events is not added to planner context. |
Table 11: Component ablation settings under the fixed Qwen3.5-9B backbone.
More in this topic
GUI Agents
Graphical user interface (GUI) agents perceive and operate apps through screens, controls, and other software interfaces.
See all 178 papers