CausalCache: Conditional High-Fidelity Restoration for Long-Horizon GUI Agents

Jiaxuan Luo; Zhanfeng Liao; Jiayao Teng; Yuan Wang

cs.AIAug 23, 2026

CausalCache: Conditional High-Fidelity Restoration for Long-Horizon GUI Agents

Authors: Jiaxuan Luo, Zhanfeng Liao, Jiayao Teng, Yuan Wang

Organizations: Johns Hopkins University · Tsinghua University

Abstract

Long-horizon GUI agents can retain complete action histories as compact text, but only a few historical screenshots fit in active context. We formulate this as budgeted fidelity restoration: every event remains summarized, while a fixed budget $B$ determines which events regain their archived screenshots. Recent- $B$ assigns all visual slots to the latest events. CausalCache instead scores the complete history and swaps in an older event only when its predicted utility exceeds that of a recent event. A history-gated key/value adapter modifies only restored history-image tokens and is exactly bypassed when no history image is active, preserving current-screen processing. The adapter and selector are trained with matched-budget interventions on desktop trajectories and evaluated zero-shot on mobile. On OSWorld-Verified, activating historical screenshots improves success by about $13$ percentage points over summary-only memory. Under the official $15$ -step limit, CausalCache and Recent- $4$ are statistically indistinguishable; in a $30$ -step diagnostic, CausalCache achieves $46.7\%$ success versus $42.4\%$ ( $+4.3$ points). Zero-shot on $117$ MobileWorld tasks, CausalCache improves over Recent- $4$ from $30.2\%$ to $36.8\%$ . The gain is concentrated on a pre-defined cross-app memory-candidate split ( $30.6\%$ vs. $19.4\%$ , $+11.2$ points), while single-app controls show no detectable difference ( $43.6\%$ vs. $42.4\%$ ). These results show that selecting which past events regain pixels is more effective than spending a fixed visual budget entirely on recency.

Figures & tables

Figure 1: CausalCache on a real cross-app episode (schematic; drawn with

B{=}4

). The agent must compose an SMS containing the order number it saw earlier in a shopping app. Every event persists as a one-line text summary; a budget of

B

events may additionally carry their archived pixels. After the app switch the evidence frame

e_{7}

lies outside the recent window: Recent-

B

spends every slot on the latest events and types the wrong number, while CausalCache swaps one recent slot for

e_{7}

at the same budget and succeeds.

Figure 2: The two learned components of CausalCache . (a) The history-gated K/V interface ( HGKV ): a low-rank

\Delta K,\Delta V

branch on the last eight frozen self-attention layers is multiplied by a binary token gate

g

that is

1

only on restored history-image tokens, so current-screen, text, and response tokens pass through unchanged, and at

B{=}0

the gate vanishes and the policy is bitwise-identical to the frozen base. (b) Proposal-conditioned re-decision: pass 1 renders the default Recent-

B

allocation and yields a proposal

a_{1}

; a two-tower scorer conditioned on per-event features with

a_{1}

as witness and on budget context scores the complete summarized trace and returns the exact-

B

set

S^{*}

. If

S^{*}

keeps the recent tail,

a_{1}

is emitted with no extra compute; only when a recent event is displaced is the prompt re-rendered under

S^{*}

and the same frozen weights generate the final action

a_{2}

. Event cards follow Figure 1 : dashed gray

=

summary-only, blue

=

recent with pixels, orange

=

promoted distant event; orange-outlined boxes are the trained modules; snowflakes denote frozen components.

Benchmark / stratum	Recent- $B$	CausalCache	$\Delta$
MobileWorld full ( $117$ )	$30.2$	$36.8$	$+6.6^{*}$
memory-critical ( $62$ )	$19.4$	$30.6$	$+11.2^{*}$
single-app control ( $55$ )	$42.4$	$43.6$	$+1.2$
split $\times$ method interact.	—	—	$+10.1$
OSWorld-Verified ( $361$ )	$33.0$	$33.3$	$+0.3$
$30$ max steps	$42.4$	$46.7$	$+4.3^{*}$

Table 1: Primary closed-loop contrasts at the deployment budget

B{=}4

(task-paired bootstrap; MobileWorld three-run means; the OSWorld row uses the official

15

-step cap). ∗ the

95\%

confidence interval strictly excludes zero (full CI tables in the supplement; the interaction interval accompanies them). The gain concentrates exactly on the construction-defined memory-critical stratum, with no detectable effect on controls; on desktop the allocations tie at the official

15

-step cap and separate when it is doubled to

30

Method	OSWorld-Verified ( $361$ )	MobileWorld ( $117$ )	MW memory-critical ( $62$ )	MW control ( $55$ )
Frozen, summary-only ( $B{=}0$ )	$19.9$	$28.2$	$21.0$	$36.4$
Frozen $+$ Recent- $B$	$32.7$	$29.9$	$22.6$	$38.2$
HGKV $+$ Recent- $B$	$33.0$	$30.2$	$19.4$	$42.4$
Frozen $+$ selector	$32.7$	$33.9$	$28.0$	$40.6$
CausalCache ( HGKV $+$ selector)	$\mathbf{33.3}$	$\mathbf{36.8}$	$\mathbf{30.6}$	$\mathbf{43.6}$
frozen-taught selector variant	—	$33.3$	$25.8$	$41.8$

Table 2: Closed-loop success rates (%) at the primary deployment budget

B{=}4

; the offline budget sweep is in the supplement. OSWorld-Verified: official no-GDrive roster and official

15

-step budget, one run per arm (a

30

-step extended-horizon rerun of both closed-loop arms is reported in the text). MobileWorld: zero-shot means over three runs for all multi-run arms; the memory-critical column restricts to the

62

construction-defined cross-app tasks, where the allocation contrast is largest, and the control column to the

55

single-app tasks, where no restoration method should help. CausalCache deploys the HGKV adapter plus the budget-aware selector; Frozen

+

selector runs the same selector on the fully frozen policy, and the frozen-taught variant additionally trains the selector on frozen-policy-scored labels. High-fidelity arms beat summary-only memory on desktop but remain pairwise indistinguishable there. Full paired statistics and per-round results are in the supplementary document.

Adapter	DiD select	$\|A_{r}\|$	Wrong drift
HGKV	$\mathbf{+0.0224}$	$0.0088$	$0.0085$
Ungated KV	$+0.0177$	$0.0086$	$0.0071$

Table 3: Desktop DiD evaluation on 94 development groups (

B{=}1

). DiD select is the target-relevant conditional margin (

A_{s}-A_{r}

); both adapters hold recent-window (

|A_{r}|

) and wrong-history drift within the cap

0.02

(intervals and all budgets in the supplement). The HGKV zero-budget path is exactly the frozen policy.

Figure 3: The core contrast at the primary budget: on construction-defined memory-critical tasks CausalCache gains

+11.2

pp over the same-budget recent allocation, while on matched single-app controls no effect is detectable (

+1.2

pp); the split-by-method interaction is

+10.1

pp (paired intervals in the supplement).

Allocation rule	$B{=}2$	$B{=}4$
Learned scorer	$+.030$	$+.017$
Random exact- $B$	$+.006$	$+.008$
RGB similarity	$+.006$	$+.007$
Inverted score	$-.016$	$-.009$
No recency feats	$+.012$	$+.008$

Table 4: Chooser ladder: log-likelihood margin over Recent-

B

at matched budgets with fresh full-set policy scoring. Only the learned ranking and its inversion separate from zero (episode-clustered intervals in the supplement).

Figure 4: Closed-loop MobileWorld success along the budget axis (full roster).

B{=}0/4

: three-run means; other budgets single rounds (

B{=}1/2

references: frozen dose curve; HGKV neutral on matched recent). Pooled over the sweep:

+4.9

pp (task-paired bootstrap; interval and full per-budget table in the supplement).

Computer-Use Agents

Computer-use agents perceive and operate graphical interfaces and software tools to complete tasks across websites, apps, and desktops.

See all 161 papers

MementoGUI: Learning Agentic Multimodal Memory Control for Long-Horizon GUI Agents
May 18, 2026Ziyun Zeng, Hang Hua, Bocheng Zou +3Multimodal Memory GUI Agents
University of Rochester · MIT-IBM Watson AI Lab · University of Wisconsin-Madison
ECHOES: Seeing Beyond the Current Image via Recursive History Compression and Privileged On-Policy Distillation
Oct 14, 2026Tengda Guo, Jiawang Hu, Yifei Tao +4Long-Horizon Agent Tasks Memory-Augmented Agents
Tsinghua University · StepFun · Nanyang Technological University +2
ReVision: Scaling Computer-Use Agents via Temporal Visual Redundancy Reduction
May 11, 2026Amirhossein Abaskohi, Yuhang He, Peter West +3Computer-Use Agents Long-Horizon Agent Tasks
University of British Columbia · Microsoft Research

May 18, 2026J/K move · Enter open · S save

MementoGUI: Learning Agentic Multimodal Memory Control for Long-Horizon GUI Agents

Ziyun Zeng, Hang Hua, Bocheng Zou, Mu Cai+2

University of Rochester · MIT-IBM Watson AI Lab · University of Wisconsin-Madison

Multimodal Memory GUI Agents Long-Horizon Agent Evaluation Long-Horizon Agent Tasks Agent Memory Management

Open paper AskPDF arXiv

Recent GUI agents have made substantial progress in visual grounding and action prediction, yet they remain brittle in long-horizon tasks that require maintaining task state across many interface transitions. Existing agents typically rely on raw history replay or text-only memory, which either overwhelms the model with redundant screenshots or discards localized visual evidence needed for future decisions. To address these limitations, we introduce MementoGUI, a plug-in agentic memory framework that equips MLLM-based GUI agents with MementoCore, a learned controller for online memory selection, compression, and retrieval. Rather than treating interaction history as a fixed context, MementoGUI formulates long-horizon GUI control as an online memory-control problem: working memory selectively preserves task-relevant interface events with textual summaries and ROI-level visual evidence, while episodic memory retrieves reusable past trajectories through learned relevance selection. MementoCore modularizes memory control into specialized operators for step processing, memory compression, episodic writing, and episodic selection, enabling plug-in memory augmentation without finetuning the GUI agent backbone. We further develop a scalable data curation pipeline that converts computer-use trajectories into memory-controller training data, introduce MementoGUI-Bench for evaluating long-horizon decision-making in GUI agents, and design MLLM-based metrics for semantic action matching, task progress, and memory consistency. Experiments on GUI-Odyssey, MM-Mind2Web, and MementoGUI-Bench show that MementoGUI consistently improves GUI agents over no-history, history-replay, and text-only memory baselines, with larger MementoCore backbones further strengthening memory-augmented GUI control.

Abstract

Computer-Use Agents

Explore similar work

MementoGUI: Learning Agentic Multimodal Memory Control for Long-Horizon GUI Agents

ECHOES: Seeing Beyond the Current Image via Recursive History Compression and Privileged On-Policy Distillation

ReVision: Scaling Computer-Use Agents via Temporal Visual Redundancy Reduction