Mobile GUI agents are typically fine-tuned 7B-scale vision-language models (VLMs) that generate each action as text. However, these VLMs decode every action token by token, and even agents that score candidate actions keep a large language model or VLM in the loop, making them slow and memory-hungry. To address these issues, we propose GUI-JEPA, to our knowledge the first decoder-free GUI agent policy built on a Joint-Embedding Predictive Architecture (JEPA), which selects actions in the latent space of a JEPA encoder pretrained on unlabelled GUI screenshots. Its action- and region-conditioned latent predictor turns the screen latent into one latent per candidate action, letting lightweight heads score all candidates against the instruction in a single batched pass, without decoding a token. We further show that the scorer relies on this latent pathway and that the predictor is trained for the decision itself: a randomly initialised predictor does just as well as one pretrained on screen dynamics. On AndroidControl, GUI-JEPA is 2.8-4.0x faster on the same GPU than a Qwen2.5-VL-7B agent we fine-tune with the same inputs, while reaching 86-92% of its in-distribution accuracy with 6% of its parameters. Code is available at https://github.com/bcwang-sjtu/GUI-JEPA
Figures & tables
Figure 1: Same inputs, same GPU: GUI-JEPA reaches 86–92% of the 7B agent’s accuracy at a fraction of its cost. Bars are relative to our same-protocol Qwen2.5-VL-7B SFT agent, fine-tuned with LoRA (IDD AC-Low / AC-High accuracy; latency: medians over 100 shared real AC-High steps on one H20 at batch 1; peak memory under HF serving; Table 3 ).
Figure 2: Overview of GUI-JEPA. Top: on a real AndroidControl step, the frozen JEPA encoder embeds the screenshot once, the residual predictor turns it into one latent per candidate in a single batch, and the structural head and the text matcher score every candidate (bars: a trained base policy’s probabilities on this step; grids are schematic). Bottom: the training stages, and scoring in one pass versus decoding token by token (measured in Section 4.5 ). Circled numbers mark the stage that trains each module (1: JEPA pretraining, 2: policy training, 3: text rankers on the fixed policy); flame = trained, snowflake = frozen; green marks the recorded (correct) action.
System
Params
Input
Judge
Split
AC-Low
AC-High
EM scores reported by other papers (screenshot input)
Aguvis-7B a
7B
img
EM
unspec.
89.4
54.2
OS-Atlas-7B a
7B
img
EM
unspec.
67.3
56.5
Qwen3-VL-8B m
8B
img
EM ∗
all-split
77.7
60.8
Qwen2.5-VL-7B, released a
7B
img
EM
unspec.
85.0
62.9
MIRAGE-4B m
4B
img
EM ∗
all-split
77.6
64.4
Table 1: AndroidControl step accuracy under disclosed protocols. img = screenshot; a11y = accessibility tree, a privileged input. Judges: exact = ours ( Section 4.1 ), EM = exact match as reported ( ∗ matching rule not defined in the source), relaxed = the original AndroidControl protocol. all-split = MIRAGE’s score over all subsplits; unspec. = split not stated. Our 7B reference has one LoRA adapter per setting, same recipe (zero-shot: Table 14 ; latency: Table 3 ). Sources: p Li et al. (2024, Tab. 5) ; a via Zhang et al. (2025, Tab. 2) (its re-evaluations; “released”: the released checkpoint); m via Yang et al. (2026b, Tab. 1) ; q Bai et al. (2025, Tab. 9) . Other protocols: Tables 11 and 12 .
Stage
IDD
full
OOD
val.
Base policy
56.84
49.79
43.85
57.54
+ span ranker
59.69
52.44
46.34
60.43
+ click reranker
60.99
53.75
47.67
62.46
Δ total
+4.15
+3.96
+3.82
+4.92
Table 2: Gains from the two text rankers (AC-High step accuracy, %; all stages on the same base-policy checkpoint; AC-Low: Table 7 ). OOD = full test minus IDD; val. = the 137-episode validation split, on which the reranker’s variant, weight and gate were selected.
System
Serving
p50
p90
Mem.
× ours
7B SFT, merged
HF
506.7
588.9
15.9
6.8/9.8
7B SFT, merged
vLLM
206.7
247.9
≥ 16
2.8 / 4.0
GUI-JEPA , fp32
PyTorch
74.3
127.8
3.45
1
GUI-JEPA , bf16
PyTorch
51.5
79.7
2.48
1
Table 3: Same-hardware latency per real step (ms; peak GPU memory in GB; same 100 AC-High IDD steps, one H20, batch 1). 7B = Qwen2.5-VL-7B (zero-shot and unmerged-LoRA timings: Appendix I ); ≥ : vLLM pre-allocates at least the weights. × ours: the 7B agent’s p50 divided by our fp32 or bf16 p50.
Figure 3: Where a step’s time goes (segment means, 100 shared real steps, one H20, batch 1; medians: Table 3 ). Decoding takes half or more of the 7B step, and its prefill alone outlasts a whole GUI-JEPA step.
Figure 4: The scorer relies on the latent pathway, which needs no dynamics pretraining. (a) Evaluation-time lesions (IDD; OOD and retrained controls in Tables 8 and 5 ). (b) Pilot AC-Low (an early configuration without the text matcher): only the structural head uses the pathway (orange box: deployed design). (c) Change in IDD accuracy against the jointly fine-tuned, dynamics-pretrained predictor when the predictor is kept frozen or randomly initialised (identical recipe).
Figure 5: GUI-JEPA on three real AC-High IDD steps , one per action kind: (a) an element click, (b) a global action, (c) a typed argument that the span ranker copies from the goal. Each step takes one scoring pass over all its candidates ( pt : probability of the chosen action; boxes: ground truth in green, GUI-JEPA’s choice in purple).
Appendix figures & tables19 assets
Supplementary material from the paper’s appendix.
Appendix
AC-High
AC-Low
Split
Stage
seed 1
seed 2
seed 3
mean ± sd
seed 1
seed 2
seed 3
mean ± sd
IDD
Base policy
56.35
55.21
56.84
56.13 ± 0.84
82.71
81.88
83.72
82.77 ± 0.92
+ span ranker
59.49
58.24
59.69
59.14 ± 0.79
82.63
81.78
83.64
82.68 ± 0.93
+ click reranker
60.26
59.10
60.99
60.12 ± 0.95
83.18
82.04
84.06
83.09 ± 1.01
Full test
Base policy
49.05
48.51
49.79
49.12 ± 0.64
78.13
76.65
78.74
77.84 ± 1.07
+ span ranker
51.79
51.21
52.44
51.81 ± 0.62
77.43
75.99
78.06
77.16 ± 1.06
Appendix
Table 4: Per-seed step accuracy of the three policy seeds at every stage of the waterfall (exact match; bold: seed 3, reported in the main text). Stages share the base checkpoint of each seed; the span ranker is trained once and shared by the three seeds, the click reranker once per seed. The OOD union is derived per seed from the full-test and IDD step counts.
Table 5: Per-seed IDD accuracy of the mechanism controls (s1–s3: training seeds; the main text reports seed 3). Lesion rows apply each intervention to the trained base policies. Means and out-of-distribution cells: Table 8 .
Subsplit
Episodes / steps
Base
+ ranker
+ reranker
Δ
PaLM 2S
IDD
716 / 3,858
56.13 ± 0.84
59.14 ± 0.79
60.12 ± 0.95
+3.98
70.8
app_unseen
631 / 3,475
41.30 ± 0.27
43.39 ± 0.26
44.56 ± 0.83
+3.25
58.5
task_unseen
803 / 4,464
43.03 ± 0.70
45.79 ± 0.68
46.87 ± 1.00
+3.84
59.6
category_unseen
700 / 3,891
41.71 ± 0.45
44.06 ± 0.44
45.20 ± 0.83
+3.48
57.4
OOD union
827 / 4,586
43.21 ± 0.55
45.65 ± 0.60
46.74 ± 0.94
+3.52
–
Full test
1,543 / 8,444
49.12 ± 0.64
51.81 ± 0.62
52.85 ± 0.94
+3.73
–
Appendix
Table 6: Per-subsplit waterfall on AC-High, with the a11y-text PaLM 2S cells (closest published protocol). Exact-match step accuracy of the base policy, after the goal-source span ranker and after the type-gated click reranker (final system); mean ± sd over 3 training seeds on identical base checkpoints (per seed: Table 4 ). Δ is final minus base; Δ and all differences in the text use unrounded means. The OOD union is derived per seed from the full-test and IDD results; the validation split selected the reranker’s variant, β and gate. PaLM 2S: the LoRA-tuned a11y-text model of Li et al. (2024, Tab. 5) on the same subsplits under the original protocol. The OOD subsplits overlap ( Appendix B ). AC-Low: Table 7 .
Subsplit
Base
+ ranker
+ reranker
PaLM 2S
IDD
82.77
82.68
83.09 ± 1.01
83.2
app_unseen
72.99
71.75
72.37 ± 1.58
78.5
task_unseen
73.66
72.65
73.15 ± 1.46
77.3
category_unseen
72.84
71.75
72.30 ± 1.68
76.8
OOD union
73.69
72.51
73.04 ± 1.44
–
Full test
77.84
77.16
77.64 ± 1.23
–
Appendix
Table 7: Per-subsplit waterfall on AC-Low (same models and conventions as Table 6 ; sd for the final system, 0.8–1.7 at every stage). The PaLM 2S AC-Low setting also receives the episode goal.
Figure 6: The text rankers’ gains transfer across subsplits. AC-High step accuracy (3 training seeds, mean ± sd) of the base policy, after the span ranker and after the click reranker, on IDD, the three overlapping OOD subsplits and the full test set; increments are computed on identical base checkpoints. Numbers: Table 6 .
IDD
app_unseen
task_unseen
category_unseen
Condition
AC-Low
AC-High
Low / High
Low / High
Low / High
Base policy (as is)
82.8 ± 1.0
56.2 ± 0.8
72.9 / 40.8
73.6 / 42.6
72.7 / 41.3
no predictor ( z~tc:=zt )
61.0 ± 1.7 − 21.9
41.7 ± 1.1 − 14.6
53.8 / 32.6
53.7 / 33.1
52.8 / 32.2
within-type action shuffle
71.9 ± 0.8 − 10.9
45.8 ± 0.8 − 10.4
62.1 / 32.3
63.5 / 34.5
61.9 / 32.7
all-candidate action shuffle
56.1 ± 0.5 − 26.7
34.4 ± 0.9 − 21.8
49.9 / 26.7
50.3 / 27.6
49.3 / 26.8
swapped, non-fine-tuned predictor
66.5 ± 1.9 − 16.3
45.5 ± 0.5 − 10.7
–
–
–
Appendix
Table 8: Ablations of the latent-predictor pathway on the full-scale base policy (exact-match step accuracy; IDD mean ± sd, OOD cells means AC-Low / AC-High; subscripts: change against the as-is row (top) or the jointly fine-tuned base (middle)). Top: inference-time lesions on all steps of each split, text matcher untouched except in the last row; the as-is row is an independent evaluation of the same policies (on IDD within 0.2 of Table 4 ). Middle: training-time variants (base-policy OOD: the as-is row); the two retrained rows use the identical full-scale recipe (OOD on all steps); the parameter-matched row lists its three training runs on IDD, and § marks the mean of the two that train stably (the third collapses on clicks: 45.5–47.7 Low / 28.8–31.4 High out of distribution). Bottom: predictive objectives at 4K-episode scale (auxiliary losses with λ=0.1 ; multi-horizon InfoNCE 0.2), AC-High change against the matched 4K reference, paired per seed unless a mean is stated. Per-seed IDD values: Table 5 ; concat-MLP head (pilot scale): Table 10 .
IDD
app_unseen
task_unseen
Probe steps / episodes
1,579 / 300
116 / 24
1,671 / 300
cos(prediction, true next latent)
0.467
0.467
0.460
cos(copy zt , true next latent)
0.788
0.773
0.786
cos(predicted change, true change)
0.006
0.003
0.004
Right direction (%)
52.6
47.4
51.6
Appendix
Table 9: Next-latent probes of the deployed (jointly fine-tuned) predictor. “Copy” keeps the current latent zt as the prediction; direction is the share of steps on which the predicted latent change points the same way as the true change (chance 50%).
Variant
Setting
Reference
Variant
Reading
Scoring head
Concat-MLP head over [zt,z~tc,c] replacing the structural head (ROI-conditioned predictor)
pilot, 2 ep
40.59 / 19.62
34.58 / 26.88
−6.0 Low, +7.3 High; its scores do not depend on the predictor
Encoder and resolution
DINOv2-L at 518 px as main features
4K, 4 ep
77.32 / 47.12
59.88 / 37.35
encoder and resolution change together
Appendix
Table 10: Design alternatives not adopted (AC-Low / AC-High exact-match step accuracy). Each variant is compared with a reference model that differs from it only in the stated factor and shares its scale, epochs and evaluation split. Setting: pilot = early pilot configuration (no text matcher); 4K = 4,000-episode training subset; ep = policy training epochs; evaluation on IDD (3,858 steps). Predictive objectives for the predictor: Table 8 .
System (source)
Params
Input
Judge
Split
AC-Low
AC-High
HiconAgent-3B ( Zhou et al., 2026 , Tab. 1)
3B
img
OS-Atlas
unspec. (“OOD setting”)
–
52.40
UI-TARS-2B ( Qin et al., 2025 , Tab. 8)
2B
img
OS-Atlas
7,708 (inferred)
89.3
68.9
LAMO-3B ( Wang et al., 2026 , Tab. 2)
3B
img
own (type + value match)
unspec.
92.1
65.5
SecAgent-3B ( Xie et al., 2026 , Tab. 3)
3B
img
unspec.
unspec.
–
69.5
GUI-Libra-3B ( Yang et al., 2026a , Tab. 3)
3B
img
own
AC-v2 (398)
85.9
57.3
Qwen2.5-VL-3B, re-run by GUI-Libra ( Yang et al., 2026a , Tab. 3)
3B
img
own
AC-v2 (398)
71.1
36.4
Appendix
Table 11: Small-model AndroidControl results not in Table 1 (larger and zero-shot models under other protocols: Table 12 ). Every cell is copied from the cited table. Apart from LiMAC and its two baselines, discussed in the text, none is set against our numbers because input, judge or split differ (ours for reference: 77.64 / 52.85 Low / High on the full test and 83.09 / 60.12 on IDD as 3-seed means, 78.64 / 53.75 and 84.06 / 60.99 for seed 3, under the exact-match judge with a11y input). Judge: relaxed = original protocol; OS-Atlas = a click within 14% of the screen width counts and typed text at F1 > 0.5 ( Wu et al., 2025 , App. D) ; own = the paper’s own definition. Split: 7,708 = OS-Atlas test samples; r500 / r800 = a random 500- / 800-step test subset; r500-IDD = IDD part of Random-500; unspec. = not stated in the source; AC-v2 (398) = the 398-example cleaned subset of GUI-Libra (its Pass@1 columns). † Goal-conditioned per-step action accuracy (“overall accuracy”), the analogue of AC-High. Parameter counts are nominal.
System (source)
Params
Input
Judge
Split
AC-Low
AC-High
GPT-4o, zero-shot ( Wu et al., 2025 , Tab. 5)
n/a
img
OS-Atlas
7,708
28.4
21.2
OS-Atlas-7B, zero-shot ( Wu et al., 2025 , Tab. 5)
7B
img
OS-Atlas
7,708
50.9
29.8
M3A (GPT-4-Turbo, text-only), zero-shot ( Li et al., 2024 , Tab. 4)
n/a
a11y
relaxed
r500-IDD
55.0
42.1
Aguvis-7B, SFT ( Xu et al., 2025 , Tab. 3)
7B
img
relaxed (inferred)
r500
80.5
61.5
Aguvis-72B, SFT ( Xu et al., 2025 , Tab. 3)
72B
img
relaxed (inferred)
r500
84.4
66.4
OS-Atlas-7B, SFT ( Wu et al., 2025 , Tab. 5)
7B
img
OS-Atlas
7,708
85.22
71.17
Appendix
Table 12: Zero-shot and larger models under other protocols, or with AC-High only (conventions and judge / split definitions as in Table 11 ). Every cell is copied from the cited table; none is set against our numbers. MAI-UI reports exact match on an unstated split and no AC-Low.
Figure 7: Lenient frozen-encoder localisation probe across pretraining recipes. ScreenSpot-Pro hit@8 of a soft-argmax localisation probe on frozen patch tokens (our encoders: mean ± sd over 5 probe seeds); hit@8 accepts errors up to 57% of the image side. Points, left to right: a web anchor ( ≈ 77K web screenshots, 60 epochs); mobile-only (119K GUI-Odyssey screenshots, 2 epochs); desktop-only (70K VideoCUA screenshots, 2 epochs); the policy encoder (blue; the mixed 189K mobile and desktop recipe, 2 epochs); a replicate pretraining of the same recipe after 2 epochs and, continued, after 4 and 6 epochs; DINOv3-ViT-L with its 95% bootstrap interval over probe examples (band). The recipes also differ in data size and training length, and the probe measures coarse localisation only (see text).
Component
Params
Setting
Encoder pretraining (frozen afterwards)
JEPA encoder, ViT-L/16
303.3M
vit_large_patch16_224 , 224 px, 14×14=196 patch tokens; ImageNet initialisation; JEPA objective after I-JEPA with random-patch masking at ratio 0.5 (the student sees the screenshot with the masked patches zeroed and regresses the EMA target’s tokens at those positions, smooth-L1); 189,262 unlabelled screenshots (118,791 GUI-Odyssey mobile, 70,471 VideoCUA desktop); 2 epochs at batch 96, 3,942 optimiser steps (the policy uses the epoch-2 checkpoint); EMA target decay 0.996; peak learning rate 5×10−5 with warm-up; JEPA loss 1.28 → 0.68; target-token std 1.8–2.2 (min 1.82, final 2.17; no collapse); 2,739 s ≈ 0.76 GPU-hour on one H20
Policy (trained end to end with the listwise loss)
Action code ac
–
16-d: action-type one-hot plus normalised coordinates or scroll direction
Residual latent predictor
39.9M
ROI-pooled encoder feature uc (1,024-d) ⊕ac→ MLP → added to a learned query token; 3 transformer layers over the 196 patch tokens and the query, 2-D sine–cosine positional embeddings; residual output z~tc=zt+Δ(zt,ac,uc) ; initialised from a dynamics predictor trained on AndroidControl transitions, then fine-tuned with the policy loss
Structural head
2.1M
cosine, with a learned scale, between an MLP of [poolroi(c)(z~tc);mean(z~tc)] and an MLP of the conditioning vector, plus a learned per-type bias; its only candidate-dependent visual input is z~tc
Appendix
Table 13: Components and hyperparameters of GUI-JEPA. Parameter counts are measured. The encoder and the CLIP text encoder are frozen and their outputs cached before policy training; the span ranker and the click reranker are trained after the base policy with the base policy fixed.
Qwen2.5-VL-7B
Qwen2.5-VL-7B
GUI-JEPA
GUI-JEPA
zero-shot
LoRA SFT
(ours, seed 3)
(ours, 3 seeds)
Parameters
8.29B
8.29B
0.50B
0.50B
ms/step (H20, batch 1, 100 real steps)
505 (HF)
207 (vLLM) / 507 (HF)
74 (fp32) / 51.5 (bf16)
Overall step acc.
43.2
70.9
61.0
60.1 ± 1.0
share of SFT (High / params)
–
–
86.1% / 6.0%
84.8% / 6.0%
Type accuracy
72.1
87.0
80.6
78.8 ± 3.4
Appendix
Table 14: Same-protocol comparison with a 7B VLM agent on AC-High (IDD). All systems see the same screenshots, enumerated candidates and ground-truth history and are scored with the same step-accuracy rules. The click rows use the 2,379 click and long-press steps whose ground-truth element is enumerated: type recall is the share predicted as a click, grounding (all) the share answered correctly with type errors counted as misses, and grounding ∣ type the share correct among the steps predicted as a click. ms/step: median over the 100 real IDD steps timed for all systems ( Section 4.5 ). Ours: seed 3, reported in the main text, and the mean ± sd over three seeds.
Step acc.
Type acc.
Action type
n
7B
ours
7B
ours
Gap
All
3,858
70.9
61.0
87.0
80.6
9.9
click
2,419
72.0
62.8
91.3
86.3
5.8
scroll
505
71.7
59.8
83.8
64.4
1.6
input_text
282
67.0
52.5
92.2
87.6
1.1
open_app
253
81.0
68.0
87.7
92.9
0.9
Appendix
Table 15: Where the same-protocol 7B agent leads (AC-High, IDD; same inputs and judge). Step and type accuracy (%) of the model reported in the main text by ground-truth action type; Gap: points of the overall 9.9-point gap each type contributes ( n× step-accuracy difference / 3,858).
Figure 8: Same-hardware per-step latency (median, one H20, batch 1). Stars: the deployed GUI-JEPA pipeline on the 100 real AC-High IDD steps timed for both systems, placed at their median of 28 scored actions (18 a11y candidates plus global actions). Lines: the synthetic fixed- N micro-benchmark with random inputs and N scored actions ( Table 16 ), a scaling illustration only. Horizontal lines: the same-protocol Qwen2.5-VL-7B SFT agent (merged LoRA) on the same steps with vLLM and HF transformers. Measurement boundaries: see the common protocol in the text.
Configuration
Steps
p50 ms
p90 ms
mean ms
peak mem
vs 7B vLLM
vs 7B HF (merged)
Real steps, fp32 (precision of all accuracy tables)
Table 16: Real-step serving path of the deployed GUI-JEPA pipeline and the synthetic micro-benchmark (one NVIDIA H20, batch 1). Real-step rows: the deployed pipeline, timed end to end on real AC-High IDD test steps; “shared 100” are the steps on which the 7B agent was also timed (identical screenshots, candidates and history), a subset of the “all 200” steps timed for GUI-JEPA. Ratios use the shared 100 steps only, against the same-protocol 7B SFT agent (merged LoRA) timed end to end: 206.7 ms with vLLM ( ≥ 16 GB peak memory) and 506.7 ms with HF transformers (15.9 GB). Synthetic rows: random image tensors and candidate vectors at a fixed number N of scored actions, 100-iteration median, a scaling illustration only. Measurement boundaries: see the common protocol below.
Figure 9: Latency decomposition in the synthetic fixed- N micro-benchmark (one NVIDIA H20, batch 1, fp32 eager, N=57 scored actions, random image tensors and candidate vectors, 100-iteration median). The click-reranker segment is amortised over the action mix; a click step takes 98.4 ms. The benchmark excludes JPEG decoding, CPU ROI pooling and the span ranker and is a scaling illustration only; real-step medians are in Table 16 .
Figure 10: Complementary errors behind the cascade on two real AC-High IDD steps (GUI-JEPA with both rankers and the same-protocol 7B SFT agent, given the same candidates). (a) GUI-JEPA is right with a margin above the 50%-coverage threshold τ (the median margin), so the cascade keeps its answer, while the 7B agent presses the wrong key. (b) GUI-JEPA picks the wrong destination with a margin below τ , so the cascade escalates the step to the 7B agent, which is right. Boxes: ground truth (green), GUI-JEPA (purple), 7B (rose).
Figure 11: GUI-JEPA as the cheap first stage of a cascade (AC-High, IDD). Steps with a large top-two margin are answered by GUI-JEPA, the rest by the 7B SFT agent; labels: share of steps kept by GUI-JEPA. Latency: the mean GUI-JEPA step plus, on escalated steps, the mean vLLM 7B step (same 100 real steps).
Vision-language models (VLMs) have become a common foundation for autonomous mobile GUI agents, but most existing systems rely on the VLM for both planning and action grounding at nearly every interaction step, leading to substantial latency and model-serving cost. We introduce Jev-Mobile, which shifts this paradigm to low-frequency VLM planning and high-frequency lightweight execution: the VLM specifies local goals, the accessibility tree defines a structured executable action space, and Jev, a fast typed decision model, repeatedly selects actions within this space. This design allows multiple GUI actions to be executed under a single VLM decision, reducing expensive VLM inference while preserving adaptive interaction. On the full AndroidWorld task suite, Jev-Mobile achieves 79% task success, compared with 78% for SeeAct-V and 84% for a Step-wise VLM baseline. Among successful trajectories, it reduces mean end-to-end execution time by 32.7% and mean model API cost by 73.4% relative to Step-wise VLM. These results show that decoupling high-level VLM reasoning from low-level action execution can substantially improve mobile GUI agent efficiency while maintaining competitive task performance.
Running GUI agents on edge devices can keep sensitive screens and interaction histories local, but the computational cost of inference at every action step makes deployment challenging. Existing GUI agent systems either perform full vision-language model (VLM) inference at each action step or reuse coarse-grained knowledge matched to prior tasks. However, dynamic mobile environments and user tasks make it difficult to fully utilize prior task executions without additional fine-tuning or task-specific offline exploration. To address this challenge, we present PastForward, a system that accelerates GUI agents through validated, fine-grained reuse of computational experience accumulated during ordinary task execution. During decoding, PastForward retrieves prior output sequences as device-adaptive multi-token proposals and verifies them in a single VLM forward pass. Across action steps, it uses prior GUI transitions to begin next-step inference while the device executes the current action, retains the early computation only when the predicted screen matches the observed screen, and carries reusable KV states forward. We evaluate PastForward on AndroidWorld workloads derived from real mobile usage patterns using multiple VLM backbones across server and edge platforms. On device, PastForward achieves action-step latency speedups of 1.63-2.36× while maintaining task success rates.
Recent advances in mobile GUI agents have shown strong potential for automating mobile tasks, but most effective systems still depend on large vision-language models for screenshot understanding and long-horizon planning. Small GUI agents that can be deployed directly on mobile devices are more attractive for practical use, offering lower inference cost and better protection of sensitive on-device information. However, due to limited model capacity, such lightweight agents remain unreliable when planning and executing GUI tasks end-to-end from screenshots alone. We propose Knowledge-Oriented Behavior Exploration (UI-KOBE), a framework that improves lightweight mobile GUI agents with reusable app-specific graph knowledge. UI-KOBE first autonomously explores a mobile application and constructs an app knowledge graph, where nodes represent distinct UI states and edges represent executable transitions. At runtime, a lightweight GUI agent uses the graph as external guidance: given a user task and the current screenshot, it identifies the current graph node and selects among self-loop actions, neighboring transitions, task completion, or fallback free actions associated with that node. By supporting runtime decisions with app-specific graph guidance, UI-KOBE reduces the burden of end-to-end GUI planning and helps lightweight models perform mobile GUI tasks more effectively, offering a practical step toward efficient, interpretable, and privacy-conscious on-device GUI agents.
Yuxiang Chai, Han Xiao, Xinyu Fu +3
1CUHK MMLab · 2Huawei Research · 3Shenzhen Loop Area Institute +1