Absorbed in Inertia: Activation Analysis for Computer-Use Agents

Giulio Segalini; Zhi Wen Soi; Jérémie Decouchant; Lydia Chen

cs.AISep 29, 2026

Absorbed in Inertia: Activation Analysis for Computer-Use Agents

Authors: Giulio Segalini, Zhi Wen Soi, Jérémie Decouchant, Lydia Chen

Organizations: Université de Neuchâtel

Abstract

Computer-use agents have become increasingly capable of executing tasks on live desktops through natural-language instructions, based on trajectories of screenshots, actions, and reasoning. We discover that they can stealthily exhibit inertia, in which they repeat fruitless actions despite recognizing that these actions are ineffective. We hypothesize that inertia is reflected in the agent's internal state, i.e., the activation values of the agent's underlying model, and propose a protocol to measure the relationship between the two. Extensive analysis of high-dimensional activation states shows that inertia corresponds to an absorbing region of activation space, where activation values become stale across actions and even after attempts to steer them. We conjecture that drastically changing the agents' activations by re-initializing them is necessary to escape inertia. Specifically, we propose R $^3$ (Reset, Reroute, Restore), which temporarily resets the agent's context trajectory to escape the absorbing region and then restores the historical context to effectively complete the task. Our approach yields 17-55% lower measured inertia across models relative to unmodified agents. These results suggest that changing the context can interrupt recurrence more effectively than directly steering the resulting activations. Our code is available at https://anonymous.4open.science/r/vlm-agent-defense-D076

Figures & tables

Figure 1: Activation editing of an absorbing region. In inertia, the agent remains near a repeated pre-action state. Linear steering can preserve this pattern, while context reset recomputes the state from a fresh context and can lead to a different action. R 3 restores the earlier history after finding a screen-changing action. Geometry and arrows are schematic.

Figure 2: Recurrence concentrates in failed trajectories. Recurrence rate (%) by task outcome (left) and acknowledged share (%) of recurrences (right) per model. Data from both benchmarks are pooled.

Figure 3: Activation persistence during recurrence. Top: normalized values of 32 sampled pre-action activation coordinates across consecutive actions; the black bar denotes the screen–action recurrence run. Bottom: change between consecutive activation vectors, measured relative to the model’s typical activation magnitude (the median vector norm in its reference sample).

Figure 4: Activation readouts on held-out tasks. Mean AUC under nested task validation, with equal task weights. Error bars: 95% bootstrap intervals. Dashed line: chance.

Figure 5: Editing pre-action activations. Steering edits the activation, while context reset removes the interaction history before recomputing it. Both rows show measured changes in activations for the two policies. Activations matrices are extracted from real data.

Inertia (%, $\downarrow$ )
Method	UI-TARS	GUI-Owl	OpenCUA	EvoCUA
Failure notice	$-15.3\pm 8.2$	$+20.5\pm 26.8$	$\underline{-33.1}\pm 31.5$	$-14.4\pm 41.9$
Steering	$-7.0\pm 8.7$	$+44.0\pm 31.9$	$+36.7\pm 38.7$	$-33.8\pm 31.7$
Permanent Reset	$\underline{-27.3}\pm 7.8$	$\underline{-6.5}\pm 22.8$	$-9.5\pm 37.0$	$\mathbf{-64.9}\pm 30.2$
R 3 (ours)	$\mathbf{-33.9}\pm 9.2$	$\mathbf{-16.5}\pm 24.1$	$\mathbf{-39.2}\pm 30.1$	$\underline{-54.9}\pm 33.8$
Screen–action recurrence (%, $\downarrow$ )

Table 1: Intervention effects on OpenComputer. Paired changes from the unmodified agent in inertia, screen–action recurrence, and task score, with task-level standard errors after

\pm

. Inertia and screen–action recurrence are reported as relative changes in their rates per executed action. Bold and underline mark the best and second-best point estimates per model and metric.

	UI-TARS	GUI-Owl	OpenCUA	EvoCUA
Unmodified	79.8	81.0	87.5	84.8
Composite	68.5	83.3	85.7	84.8
Recurrence-only	71.0	82.1	83.9	84.8
Selective reset	70.2	60.7	62.5	73.9
Reset	0.8	2.4	3.6	0.0

Table 2: Same action again (%). Composite and recurrence-only steering have matched norms.

Figure 6: Activation neighborhoods after the first trigger. Share of the 25 nearest unmodified-agent activations associated with reuse of a previously unchanged-screen action. After reset, this share drops sharply at the trigger.

Figure 7: Screen changes and returns on OpenComputer. Percentages of all triggers, averaged over runs. A return repeats the original screen–action pair after a screen change and leaves the screen unchanged, within recorded follow-up.

Appendix figures & tables26 assets

Supplementary material from the paper’s appendix.

Appendix

Model	Benchmark	All trajectories	Failed	Successful	Acknowledged actions
UI-TARS	OSWorld	45.9	54.3	19.0	88.1
UI-TARS	OpenComputer	83.5	88.9	14.3	89.1
GUI-Owl	OSWorld	29.5	37.0	13.9	36.2
GUI-Owl	OpenComputer	52.6	57.3	0.0	59.3
OpenCUA	OSWorld	27.3	35.4	10.3	56.0
OpenCUA	OpenComputer	42.3	50.0	0.0	51.8

Appendix

Table 3: Screen–action recurrence and acknowledgment (%) by model and benchmark. Recurrence uses eligible trajectories within each outcome group. Acknowledgment uses actions with screen–action recurrence.

	trajectories				actions
Model	exact	$g8$	$g16$	$g32$	exact	$g8$	$g16$	$g32$
UI-TARS-1.5-7B	84%	84%	84%	84%	3,832	3,832	3,832	3,832
GUI-Owl-1.5-8B-Think	53%	53%	54%	54%	2,158	2,191	2,216	2,223
OpenCUA-32B	42%	46%	48%	49%	732	1,007	1,099	1,177
EvoCUA-32B-S2	42%	43%	43%	43%	1,441	1,445	1,449	1,460

Appendix

Table 4: Sensitivity of screen–action recurrence to coordinate tolerance. Coordinates are quantized to a

g

-pixel grid on the OpenComputer trajectories in Table 3 . Detections are counted once per action. Quantized matches can target different interface elements.

Model	Exact	$g=8$	$g=16$	$g=32$
UI-TARS	$-16.9\ [-26.6,-6.1]$	$-16.3\ [-26.0,-5.6]$	$-15.9\ [-25.7,-5.0]$	$-15.4\ [-25.2,-4.7]$
GUI-Owl	$-38.0\ [-52.1,-20.8]$	$-37.0\ [-51.2,-19.7]$	$-36.9\ [-50.9,-19.8]$	$-35.8\ [-50.0,-19.0]$
OpenCUA	$-42.7\ [-61.6,-15.6]$	$-28.2\ [-49.7,+1.5]$	$-25.8\ [-47.2,+3.5]$	$-20.1\ [-43.1,+11.0]$
EvoCUA	$-33.8\ [-47.8,-15.5]$	$-33.1\ [-47.4,-15.1]$	$-32.7\ [-46.9,-14.7]$	$-32.9\ [-47.1,-15.2]$

Appendix

Table 5: Coordinate sensitivity of R 3 on OpenComputer. Cells give relative changes in screen–action recurrence per executed action (%) with 95% task-bootstrap intervals. Coordinates are quantized to a

g

-pixel grid. GUI-Owl excludes two paired trajectories of one task with no executed-action denominator.

		Blind pass		Corrected pass
Model	Benchmark	Judge 1	Judge 2	Judge 1	Judge 2
UI-TARS	OSWorld	12/12	12/12	12/12	12/12
UI-TARS	OpenComputer	11/12	11/12	11/12	11/12
GUI-Owl	OSWorld	12/12	10/12	12/12	10/12
GUI-Owl	OpenComputer	11/12	12/12	11/12	12/12
OpenCUA	OSWorld	10/12	10/12	10/12	10/12

Appendix

Table 6: Human agreement on the separately sampled balanced check. Each cell gives agreeing actions out of 12. Blind labels precede comparison with the judge key. Corrected labels are reported separately.

		judged rows		acknowledged
Model	Benchmark	all	dropped	judge 1	judge 2	agreement
UI-TARS-1.5-7B	OSWorld	1,597	0	88.1%	89.9%	97.9%
	OpenComputer	3,832	81	89.1%	90.8%	98.1%
GUI-Owl-1.5-8B-Think	OSWorld	505	1	36.2%	44.4%	91.7%
	OpenComputer	2,158	1	59.3%	59.8%	99.5%
OpenCUA-32B	OSWorld	1,323	0	56.0%	54.6%	97.0%

Appendix

Table 7: Failure-acknowledgment agreement. Judge 1 supplies the reported labels, which are compared with Judge 2’s labels. The note below the table specifies exclusions and the actions used to calculate agreement.

Model	Benchmark	Macro acknowledgment	Cohen’s $\kappa$	Specific agreement
UI-TARS	OSWorld	76.57 [69.36, 83.23]	0.892 [0.792, 0.951]	98.80 / 90.34
UI-TARS	OpenComputer	87.30 [81.54, 92.60]	0.898 [0.742, 0.984]	98.96 / 90.81
GUI-Owl	OSWorld	45.94 [35.50, 56.24]	0.828 [0.672, 0.941]	89.66 / 93.02
GUI-Owl	OpenComputer	60.32 [49.34, 70.86]	0.990 [0.980, 0.998]	99.61 / 99.43
OpenCUA	OSWorld	55.17 [44.67, 65.16]	0.939 [0.878, 0.980]	97.27 / 96.62
OpenCUA	OpenComputer	50.26 [35.67, 64.59]	0.937 [0.817, 1.000]	96.99 / 96.68

Appendix

Table 8: Task-macro acknowledgment (%) and judge agreement. Brackets give 95% task-bootstrap intervals. Specific agreement is reported separately for positive / negative labels (%).

Figure 8: A repeated three-action cycle under the same failure-acknowledging reasoning. UI-TARS on OpenComputer, actions 35–37, with the tone dialog as recorded at action 37.

Figure 9: Failure acknowledgment with a repeated action, one example per model. Left, the screen at the acknowledged action, with the executed click circled. Right, the exposed reasoning at that action and the executed action, which repeats an earlier execution from the same screen and leaves it unchanged.

Model	Boundary	Captured layers	Steering layers
UI-TARS	Action marker	4–25	14, 15, 16, 17, 18, 19, 20, 21, 22, 23
GUI-Owl	Closing </think>	4–33	23, 24, 26, 27, 28, 29, 30, 31, 32, 33
OpenCUA	Action marker	4–61	41, 42, 43, 44, 45, 46, 47, 48, 49, 50
EvoCUA	Opening <tool_call>	4–61	47, 49, 50, 52, 53, 54, 55, 56, 57, 58

Appendix

Table 9: Activation extraction sites, captured layer ranges, and steering layers. Layer indices are one-based. Selected layers can be non-contiguous because selection ranks validation AUC rather than layer depth.

Model	Positive class	Contributing fitting tasks	Positive actions	Negative actions
UI-TARS	Screen–action recurrence	75	1134	1730
UI-TARS	Inertia	68	1020	1553
GUI-Owl	Screen–action recurrence	55	397	1957
GUI-Owl	Inertia	36	168	1262
OpenCUA	Screen–action recurrence	57	1019	1127
OpenCUA	Inertia	43	624	877

Appendix

Table 10: Contributing fitting tasks and actions for the two contrasts on OSWorld. Both contrasts use the same registered task split. A fitting task contributes when it contains both the corresponding positive class and eligible actions without screen–action recurrence.

Model	Layer	Val. AUC	Split-half cosine	Shuffled-label null
UI-TARS	20	0.826	0.963 [0.942, 0.974]	-0.002 [-0.215, 0.398]
GUI-Owl	33	0.721	0.307 [0.121, 0.436]	0.003 [-0.173, 0.389]
OpenCUA	48	0.952	0.822 [0.579, 0.888]	-0.005 [-0.436, 0.685]
EvoCUA	56	0.930	0.817 [0.708, 0.885]	0.000 [-0.289, 0.363]

Appendix

Table 11: Validation AUC and primary-direction stability. Stability entries give the median [minimum, maximum]. The null summarizes median split-half cosines under within-task label permutations. Validation also selects the layer.

Figure 10: Validation actions projected onto the fitted direction

v^{(\ell)}

and orthogonal training PC1, in training standard deviations. AUC averages within-task comparisons on the data used to select layers. Density reflects action counts.

Figure 11: Validation AUC of the recurrence readout (left, OSWorld) and the inertia readout (right, both benchmarks). The dashed line marks chance.

	Original library		Same-index library
Model	Unmodified	R 3	Unmodified	R 3
UI-TARS	62.8	0.0	48.7	30.5
GUI-Owl	24.7	0.0	31.3	18.5
OpenCUA	58.0	2.7	26.5	17.7
EvoCUA	59.0	0.0	29.8	16.5

Appendix

Table 12: Activation-neighborhood sensitivity at the first trigger. Cells give the percentage of 25 nearest reference neighbors associated with reuse of an unchanged-screen action. The restricted library matches trajectory action index, not prompt length.

Model	$\alpha$	Task score	$\Delta$ score	$\Delta$ screen–action recurrence	Mean actions	Evaluable	Selected
UI-TARS	Unmodified	28.00	0.00	0.00	35.88	50
UI-TARS	$0.5$	22.00	-6.00	+0.14	33.40	50
UI-TARS	$-0.5$	19.25	-8.75	+0.50	34.66	50
UI-TARS	$-1$	26.00	-2.00	+0.98	33.40	50
UI-TARS	$-2$	26.00	-2.00	-0.26	37.38	50
UI-TARS	$-5$	24.00	-4.00	-3.08	31.48	50	$\checkmark$

Appendix

Table 13: Steering-strength selection on the OSWorld validation tasks. Task score and its paired change are in points out of 100. The recurrence change is the paired change in screen–action recurrences per trajectory on the tasks common to all strengths. Evaluable counts tasks with a recorded score.

Model	Policy	Pairs	Re-entry totals	$\Delta$ re-entries / task [95% CI]	$\Delta$ score [95% CI]
UI-TARS	Failure notice	97	2113 / 2432	$+3.29\ [-3.30,+9.71]$	$+1.00\ [-2.10,+3.80]$
UI-TARS	Permanent Reset ( $k=8$ )	97	2113 / 1426	$-7.08\ [-12.46,-1.81]$	$+0.40\ [-3.20,+3.60]$
UI-TARS	R 3 ( $k=8$ )	193	4795 / 2607	$-11.24\ [-15.78,-6.96]$	$+0.60\ [-2.50,+3.40]$
UI-TARS	Linear steering ( $\alpha=-5$ )	97	2682 / 2335	$-3.58\ [-10.51,+3.69]$	$+2.20\ [-1.90,+6.50]$
GUI-Owl	Failure notice	97	1516 / 1544	$+0.29\ [-6.88,+7.62]$	$+0.20\ [-4.50,+4.90]$
GUI-Owl	Permanent Reset ( $k=8$ )	97	1516 / 834	$-7.03\ [-13.61,-0.54]$	$+3.20\ [-1.20,+7.80]$

Appendix

Table 14: Paired policy outcomes on OpenComputer. Re-entry columns count screen–action re-entries, as defined in Appendix D.3 , for the unmodified agent / intervention. This counter differs from the full detector used in Table 1 . Changes are paired means with 95% task-bootstrap intervals. Each policy uses all valid matched trajectories at its specified settings, so seed coverage differs across rows. Search thresholds are the reported per-model thresholds. Steering uses the fixed strength in Appendix D.1 .

Model	Unmodified	R 3	Change [95% CI]	Stops / trajectories
UI-TARS	42.78	35.56	$-7.22\ [-11.85,-2.66]$	10 / 193
GUI-Owl	24.54	15.21	$-9.33\ [-14.23,-4.55]$	9 / 194
OpenCUA	20.41	11.70	$-8.71\ [-15.29,-2.63]$	1 / 97
EvoCUA	17.19	11.39	$-5.81\ [-9.75,-2.14]$	38 / 194

Appendix

Table 15: Screen–action recurrence per executed action on OpenComputer under R 3 . Rates are task-macro percentages. Changes give percentage points with 95% task-bootstrap intervals. Stops are explicit controller terminations after search exhaustion.

Model	Policy	Pairs	Inertia (%) [95% CI]	Screen–action recurrence (%) [95% CI]
UI-TARS	Failure notice	97	$-15.3\ [-29.0,+0.9]$	$-4.6\ [-19.3,+13.9]$
UI-TARS	Steering	97	$-7.0\ [-22.4,+11.6]$	$-5.6\ [-18.0,+8.8]$
UI-TARS	Permanent Reset	97	$-27.3\ [-39.2,-13.4]$	$-14.6\ [-27.9,+0.9]$
UI-TARS	R 3	193	$-33.9\ [-47.2,-17.2]$	$-16.9\ [-26.3,-6.6]$
GUI-Owl	Failure notice	97	$+20.5\ [-23.1,+102.1]$	$-5.0\ [-32.3,+33.2]$
GUI-Owl	Steering	97	$+44.0\ [-12.3,+150.9]$	$-0.2\ [-27.0,+34.9]$

Appendix

Table 16: Relative changes from the unmodified agent reported in Table 1 , with 95% task-bootstrap intervals. Both measures are rates per executed action on OpenComputer.

Model	$k$	Pairs	$\Delta$ re-entries [95% CI]	$\Delta$ score [95% CI]	Selected
UI-TARS	8	193	$-11.24\ [-15.78,-6.96]$	$+0.60\ [-2.50,+3.40]$	$\checkmark$
UI-TARS	4	97	$-7.53\ [-13.64,-1.44]$	$+0.20\ [-3.80,+4.10]$
GUI-Owl	8	97	$-8.79\ [-15.08,-2.76]$	$+1.80\ [-2.30,+6.30]$
GUI-Owl	4	194	$-9.97\ [-14.52,-5.60]$	$+1.20\ [-2.60,+4.80]$	$\checkmark$
OpenCUA	8	97	$-2.85\ [-6.37,+0.68]$	$+2.80\ [-2.70,+8.30]$
OpenCUA	4	97	$-3.72\ [-7.09,-0.54]$	$-2.50\ [-8.30,+3.20]$	$\checkmark$

Appendix

Table 17: R 3 results by threshold. Pairs count trajectories of the same task and seed under matching collection settings.

Model	Seed	Policy	Screen changes	Original recurrence resumes	Follow-up actions
UI-TARS	0	R 3	32/37 86.5 [73.5, 97.1]	5/32 15.6 [4.2, 28.1]	46 [2, 88]
UI-TARS	1	R 3	39/45 86.7 [76.2, 95.6]	10/39 25.6 [11.9, 43.2]	33 [0, 84]
UI-TARS	default	Linear steering ( $\alpha=-5$ )	18/36 50.0 [31.2, 68.4]	6/18 33.3 [12.5, 57.1]	44.5 [0, 76]
GUI-Owl	0	R 3	28/33 84.8 [71.9, 96.0]	4/28 14.3 [3.4, 28.0]	32.5 [2, 91]
GUI-Owl	1	R 3	27/31 87.1 [72.4, 97.1]	5/27 18.5 [5.6, 33.3]	48 [7, 92]
GUI-Owl	default	Linear steering ( $\alpha=-1$ )	14/39 35.9 [18.9, 52.5]	2/14 14.3 [0.0, 33.3]	67.5 [2, 94]

Appendix

Table 18: Executed screen changes and subsequent recurrence on OpenComputer. Cells give counts, percentages, and 95% task-bootstrap intervals. Return rates use only events with a screen change. Follow-up gives median [minimum, maximum] executed actions after screen change. Each policy is measured on its own trajectories.

Model	Seed	Policy	Screen change (%)	Return (%)	Median follow-up
UI-TARS	seed 0	R 3	24/28 (85.7 [71.4, 96.4])	4/24 (16.7 [4.0, 33.3])	53.5
UI-TARS	seed 1	R 3	29/33 (87.9 [75.8, 97.0])	9/29 (31.0 [15.2, 50.0])	41
UI-TARS	default	Linear steering ( $\alpha=-5$ )	15/28 (53.6 [35.7, 71.4])	5/15 (33.3 [9.1, 58.8])	45
GUI-Owl	seed 0	R 3	19/21 (90.5 [76.2, 100.0])	4/19 (21.1 [5.0, 41.2])	40
GUI-Owl	seed 1	R 3	18/22 (81.8 [63.6, 95.5])	3/18 (16.7 [0.0, 35.7])	51
GUI-Owl	default	Linear steering ( $\alpha=-1$ )	11/27 (40.7 [22.2, 59.3])	1/11 (9.1 [0.0, 30.0])	81

Appendix

Table 19: Screen change and return using only the first trigger in each trajectory. Rates and intervals follow Table 18 ; follow-up counts executed actions after screen change.

Model	Seconds / action	Hours / task
UI-TARS	7.1	0.17
GUI-Owl	14.3	0.25
OpenCUA	240.4	2.26
EvoCUA	71.8	1.27

Appendix

Table 20: Elapsed time for unmodified OpenComputer agents. Command blocks count as one action; terminal declarations are excluded from the action count.

Model	Policy	$\Delta$ total records [95% CI]	Interventions	Trajectories affected
UI-TARS	Permanent Reset ( $k=8$ )	$-2.09\ [-7.68,+3.24]$	44	31
UI-TARS	R 3 ( $k=8$ )	$-5.29\ [-9.37,-1.41]$	82	61
GUI-Owl	Permanent Reset ( $k=8$ )	$-4.08\ [-10.55,+2.29]$	23	16
GUI-Owl	R 3 ( $k=4$ )	$-8.84\ [-13.51,-4.17]$	64	43
OpenCUA	Permanent Reset ( $k=8$ )	$+5.00\ [+0.59,+9.54]$	19	11
OpenCUA	R 3 ( $k=4$ )	$+0.90\ [-2.73,+4.60]$	19	14

Appendix

Table 21: Reset-policy trajectory length and intervention frequency on OpenComputer. Length is the paired change in trajectory records per task, including terminal records, with a 95% interval. Intervention totals and affected trajectory counts include the seeds used in the paired comparison.

Model	Layer	PC1 variance (%)	PC2 variance (%)
UI-TARS	20	14.08	4.82
GUI-Owl	33	11.59	5.13
OpenCUA	48	13.17	8.75
EvoCUA	56	10.46	7.36

Appendix

Table 22: Explained variance of the two-dimensional reference projection used in Figure 12 .

Figure 12: Activation movement does not always change the action. Task-weighted PCA of OSWorld reference states (gray) and OpenComputer interventions (colored). Lines join the unmodified mean to composite steering, matched-norm recurrence-only steering, and reset. Bars show all proposal outcomes; percentages count proposal escapes, not executed progress.

Model	Repeated	Different	Parse failure
UI-TARS	$70.2_{[56.5,\,82.3]}$	$26.6_{[14.5,\,39.5]}$	$3.2_{[0.0,\,9.7]}$
GUI-Owl	$60.7_{[42.9,\,77.4]}$	$39.3_{[22.6,\,57.1]}$	$0.0_{[0.0,\,0.0]}$
OpenCUA	$62.5_{[35.7,\,85.7]}$	$37.5_{[14.3,\,64.3]}$	$0.0_{[0.0,\,0.0]}$
EvoCUA	$73.9_{[58.7,\,87.0]}$	$26.1_{[13.0,\,41.3]}$	$0.0_{[0.0,\,0.0]}$

Appendix

Table 23: Selective-reset replay outcomes (%). Subscripts show 95% task-bootstrap intervals. All draws enter the denominator.

Explore similar work

Jun 16, 2026cs.AI

PreAct: Computer-Using Agents that Get Faster on Repeated Tasks

Computer-using agents drive real software through the screen -- clicking and typing -- but they solve every task from scratch: asked to repeat a task, an agent re-reads the screen, re-reasons every tap, and pays the full cost again. We present PreAct, which lets such an agent get faster on tasks it has done before. The first time it succeeds, PreAct compiles the run into a small state-machine program-states that check the screen, transitions that act-and on later runs replays it directly instead of invoking the agent 8.5-13x faster, with no per-step language-model calls. Replay is not blind: at each step PreAct checks that the screen matches what the program expects before acting, and hands control back to the agent the moment something is off. PreAct applies the same discipline when deciding what to keep: a freshly compiled program enters the store only if, re-run from a clean state, an independent evaluator confirms it solved the task-catching programs that replay to their last step yet leave the task undone. Across a mobile, a desktop, and a web benchmark, this store-time check separates repeated runs that improve from ones that degrade as faulty programs accumulate, worth 1.75-2.6 tasks per benchmark, the same direction on all three; a fallback that explores afresh when no program fits brings PreAct level with a strong record-and-replay baseline. We also report what did not matter: prompt wording, runtime guardrails, and whether a language model or a plain embedding retriever selects which program to reuse.

Bojie Li

Pine AI

Computer-Use Agents Replay Smooth Execution Automata

Ask PDF

Apr 29, 2026cs.AI

Step-level Optimization for Efficient Computer-use Agents

Computer-use agents provide a promising path toward general software automation because they can interact directly with arbitrary graphical user interfaces instead of relying on brittle, application-specific integrations. Despite recent advances in benchmark performance, strong computer-use agents remain expensive and slow in practice, since most systems invoke large multimodal models at nearly every interaction step. We argue that this uniform allocation of compute is fundamentally inefficient for long-horizon GUI tasks. Such trajectories are highly heterogeneous: many steps are routine and can be handled reliably by smaller, cheaper policies, while errors tend to concentrate at a relatively small number of high-risk moments. Across computer-use benchmarks, these failures repeatedly take two forms: progress stalls, where the agent loops, repeats ineffective actions, or fails to make meaningful progress, and silent semantic drift, where the agent continues taking locally plausible actions after already deviating from the user's true goal. To address this inefficiency, we propose an event-driven, step-level cascade for computer-use agents that runs a small policy by default and escalates to a stronger model only when lightweight learned monitors detect elevated risk. Our framework combines two complementary signals: a Stuck Monitor that detects degraded progress from recent reasoning-action history and triggers recovery, and a Milestone Monitor that identifies semantically meaningful checkpoints where sparse verification is most informative for catching drift. This design turns always-on frontier-model inference into adaptive, on-demand compute allocation over the course of an evolving interaction. The framework is modular and deployment-oriented: it can be layered on top of existing computer-use agents without changing the underlying agent architecture or retraining the large model.

Jinbiao Wei, Kangqi Ni, Yilun Zhao +2

Y Yale NLP Lab · CUniversity of North Carolina at Chapel Hill

Computer-Use Agents Intermediate Checkpoints Depth History

Ask PDF

Feb 24, 2026cs.AI

ActionEngine: From Reactive to Programmatic Web Agents via State Machine Memory

Many web agents operate through a reactive execution loop: they observe the current interface, reason about the next action, execute it, and repeat. This design incurs latency and cost that grow with the number of actions, while requiring agents to repeatedly rediscover how the same web application works. We present ActionEngine, a novel architecture that replaces step-by-step reasoning with programmatic execution using reusable knowledge of the application. A Crawling Agent explores the application offline and constructs an updatable state-machine memory that represents its GUI states, the operations available in each state, and the transitions between states. Unlike trajectory memory, this representation stores how the application works rather than solutions to individual tasks. At runtime, an Execution Agent uses this memory to synthesize a complete executable program in a single planning step, which is then executed deterministically without further planning calls. When the interface changes or the memory is incomplete, a reactive fallback repairs the failed action and updates the memory for future tasks. On 655 tasks across four WebArena domains, ActionEngine achieves a 91.2% success rate, outperforming the strongest reactive baseline, Claude Code, by 8.5 percentage points while reducing average task latency by 3.2x and cost by 8x.

Hongbin Zhong, Fazle Faisal, Luis França +4

Georgia Tech USA · Microsoft Research USA

Web Agents Reactive Progressive Reasoning

Ask PDF