GUI-JEPA: Candidate-Conditioned JEPA Latents as a Fast Action Policy for Mobile GUI Agents

Bingchao Wang; Yongcheng Jing; Shunyu Liu; Chengyu Wang; Jun Huang; Jie Yang; Wei Liu; Dacheng Tao

cs.AIOct 27, 2026

GUI-JEPA: Candidate-Conditioned JEPA Latents as a Fast Action Policy for Mobile GUI Agents

Authors: Bingchao Wang, Yongcheng Jing, Shunyu Liu, Chengyu Wang, Jun Huang, Jie Yang, Wei Liu, Dacheng Tao

Organizations: Shanghai Jiao Tong University · Nanyang Technological University · Alibaba Cloud Computing

Abstract

Mobile GUI agents are typically fine-tuned 7B-scale vision-language models (VLMs) that generate each action as text. However, these VLMs decode every action token by token, and even agents that score candidate actions keep a large language model or VLM in the loop, making them slow and memory-hungry. To address these issues, we propose GUI-JEPA, to our knowledge the first decoder-free GUI agent policy built on a Joint-Embedding Predictive Architecture (JEPA), which selects actions in the latent space of a JEPA encoder pretrained on unlabelled GUI screenshots. Its action- and region-conditioned latent predictor turns the screen latent into one latent per candidate action, letting lightweight heads score all candidates against the instruction in a single batched pass, without decoding a token. We further show that the scorer relies on this latent pathway and that the predictor is trained for the decision itself: a randomly initialised predictor does just as well as one pretrained on screen dynamics. On AndroidControl, GUI-JEPA is 2.8-4.0x faster on the same GPU than a Qwen2.5-VL-7B agent we fine-tune with the same inputs, while reaching 86-92% of its in-distribution accuracy with 6% of its parameters. Code is available at https://github.com/bcwang-sjtu/GUI-JEPA

Figures & tables

Figure 1: Same inputs, same GPU: GUI-JEPA reaches 86–92% of the 7B agent’s accuracy at a fraction of its cost. Bars are relative to our same-protocol Qwen2.5-VL-7B SFT agent, fine-tuned with LoRA (IDD AC-Low / AC-High accuracy; latency: medians over 100 shared real AC-High steps on one H20 at batch 1; peak memory under HF serving; Table 3 ).

Figure 2: Overview of GUI-JEPA. Top: on a real AndroidControl step, the frozen JEPA encoder embeds the screenshot once, the residual predictor turns it into one latent per candidate in a single batch, and the structural head and the text matcher score every candidate (bars: a trained base policy’s probabilities on this step; grids are schematic). Bottom: the training stages, and scoring in one pass versus decoding token by token (measured in Section 4.5 ). Circled numbers mark the stage that trains each module (1: JEPA pretraining, 2: policy training, 3: text rankers on the fixed policy); flame = trained, snowflake = frozen; green marks the recorded (correct) action.

System	Params	Input	Judge	Split	AC-Low	AC-High
EM scores reported by other papers (screenshot input)
Aguvis-7B a	7B	img	EM	unspec.	89.4	54.2
OS-Atlas-7B a	7B	img	EM	unspec.	67.3	56.5
Qwen3-VL-8B m	8B	img	EM ∗	all-split	77.7	60.8
Qwen2.5-VL-7B, released a	7B	img	EM	unspec.	85.0	62.9
MIRAGE-4B m	4B	img	EM ∗	all-split	77.6	64.4

Table 1: AndroidControl step accuracy under disclosed protocols. img = screenshot; a11y = accessibility tree, a privileged input. Judges: exact = ours ( Section 4.1 ), EM = exact match as reported ( ∗ matching rule not defined in the source), relaxed = the original AndroidControl protocol. all-split = MIRAGE’s score over all subsplits; unspec. = split not stated. Our 7B reference has one LoRA adapter per setting, same recipe (zero-shot: Table 14 ; latency: Table 3 ). Sources: p Li et al. (2024, Tab. 5) ; a via Zhang et al. (2025, Tab. 2) (its re-evaluations; “released”: the released checkpoint); m via Yang et al. (2026b, Tab. 1) ; q Bai et al. (2025, Tab. 9) . Other protocols: Tables 11 and 12 .

Stage	IDD	full	OOD	val.
Base policy	56.84	49.79	43.85	57.54
+ span ranker	59.69	52.44	46.34	60.43
+ click reranker	60.99	53.75	47.67	62.46
$\Delta$ total	+4.15	+3.96	+3.82	+4.92

Table 2: Gains from the two text rankers (AC-High step accuracy, %; all stages on the same base-policy checkpoint; AC-Low: Table 7 ). OOD = full test minus IDD; val. = the 137-episode validation split, on which the reranker’s variant, weight and gate were selected.

System	Serving	p50	p90	Mem.	$\times$ ours
7B SFT, merged	HF	506.7	588.9	15.9	6.8/9.8
7B SFT, merged	vLLM	206.7	247.9	$\geq$ 16	2.8 / 4.0
GUI-JEPA , fp32	PyTorch	74.3	127.8	3.45	1
GUI-JEPA , bf16	PyTorch	51.5	79.7	2.48	1

Table 3: Same-hardware latency per real step (ms; peak GPU memory in GB; same 100 AC-High IDD steps, one H20, batch 1). 7B = Qwen2.5-VL-7B (zero-shot and unmerged-LoRA timings: Appendix I );

\geq

: vLLM pre-allocates at least the weights.

\times

ours: the 7B agent’s p50 divided by our fp32 or bf16 p50.

Figure 3: Where a step’s time goes (segment means, 100 shared real steps, one H20, batch 1; medians: Table 3 ). Decoding takes half or more of the 7B step, and its prefill alone outlasts a whole GUI-JEPA step.

Figure 4: The scorer relies on the latent pathway, which needs no dynamics pretraining. (a) Evaluation-time lesions (IDD; OOD and retrained controls in Tables 8 and 5 ). (b) Pilot AC-Low (an early configuration without the text matcher): only the structural head uses the pathway (orange box: deployed design). (c) Change in IDD accuracy against the jointly fine-tuned, dynamics-pretrained predictor when the predictor is kept frozen or randomly initialised (identical recipe).

Figure 5: GUI-JEPA on three real AC-High IDD steps , one per action kind: (a) an element click, (b) a global action, (c) a typed argument that the span ranker copies from the goal. Each step takes one scoring pass over all its candidates ( $p_{t}$ : probability of the chosen action; boxes: ground truth in green, GUI-JEPA’s choice in purple).

Appendix figures & tables19 assets

Supplementary material from the paper’s appendix.

Appendix

		AC-High				AC-Low
Split	Stage	seed 1	seed 2	seed 3	mean $\pm$ sd	seed 1	seed 2	seed 3	mean $\pm$ sd
IDD	Base policy	56.35	55.21	56.84	56.13 $\pm$ 0.84	82.71	81.88	83.72	82.77 $\pm$ 0.92
	+ span ranker	59.49	58.24	59.69	59.14 $\pm$ 0.79	82.63	81.78	83.64	82.68 $\pm$ 0.93
	+ click reranker	60.26	59.10	60.99	60.12 $\pm$ 0.95	83.18	82.04	84.06	83.09 $\pm$ 1.01
Full test	Base policy	49.05	48.51	49.79	49.12 $\pm$ 0.64	78.13	76.65	78.74	77.84 $\pm$ 1.07
	+ span ranker	51.79	51.21	52.44	51.81 $\pm$ 0.62	77.43	75.99	78.06	77.16 $\pm$ 1.06

Appendix

Table 4: Per-seed step accuracy of the three policy seeds at every stage of the waterfall (exact match; bold: seed 3, reported in the main text). Stages share the base checkpoint of each seed; the span ranker is trained once and shared by the three seeds, the click reranker once per seed. The OOD union is derived per seed from the full-test and IDD step counts.

	IDD AC-Low			IDD AC-High
Condition	s1	s2	s3	s1	s2	s3
Inference-time lesions (text matcher untouched unless stated)
Base policy (as is)	82.79	81.80	83.88	56.48	55.39	56.84
no predictor ( $\tilde{\mathbf{z}}_{t}^{c}:=\mathbf{z}_{t}$ )	61.79	62.05	59.02	41.16	40.93	42.87
within-type action shuffle	72.01	71.12	72.65	46.47	44.95	46.06
all-candidate action shuffle	55.83	55.81	56.64	35.17	34.63	33.41

Appendix

Table 5: Per-seed IDD accuracy of the mechanism controls (s1–s3: training seeds; the main text reports seed 3). Lesion rows apply each intervention to the trained base policies. Means and out-of-distribution cells: Table 8 .

Subsplit	Episodes / steps	Base	+ ranker	+ reranker	$\Delta$	PaLM 2S
IDD	716 / 3,858	56.13 $\pm$ 0.84	59.14 $\pm$ 0.79	60.12 $\pm$ 0.95	+3.98	70.8
app_unseen	631 / 3,475	41.30 $\pm$ 0.27	43.39 $\pm$ 0.26	44.56 $\pm$ 0.83	+3.25	58.5
task_unseen	803 / 4,464	43.03 $\pm$ 0.70	45.79 $\pm$ 0.68	46.87 $\pm$ 1.00	+3.84	59.6
category_unseen	700 / 3,891	41.71 $\pm$ 0.45	44.06 $\pm$ 0.44	45.20 $\pm$ 0.83	+3.48	57.4
OOD union	827 / 4,586	43.21 $\pm$ 0.55	45.65 $\pm$ 0.60	46.74 $\pm$ 0.94	+3.52	–
Full test	1,543 / 8,444	49.12 $\pm$ 0.64	51.81 $\pm$ 0.62	52.85 $\pm$ 0.94	+3.73	–

Appendix

Table 6: Per-subsplit waterfall on AC-High, with the a11y-text PaLM 2S cells (closest published protocol). Exact-match step accuracy of the base policy, after the goal-source span ranker and after the type-gated click reranker (final system); mean

\pm

sd over 3 training seeds on identical base checkpoints (per seed: Table 4 ).

\Delta

is final minus base;

\Delta

and all differences in the text use unrounded means. The OOD union is derived per seed from the full-test and IDD results; the validation split selected the reranker’s variant,

\beta

and gate. PaLM 2S: the LoRA-tuned a11y-text model of Li et al. (2024, Tab. 5) on the same subsplits under the original protocol. The OOD subsplits overlap ( Appendix B ). AC-Low: Table 7 .

Subsplit	Base	+ ranker	+ reranker	PaLM 2S
IDD	82.77	82.68	83.09 $\pm$ 1.01	83.2
app_unseen	72.99	71.75	72.37 $\pm$ 1.58	78.5
task_unseen	73.66	72.65	73.15 $\pm$ 1.46	77.3
category_unseen	72.84	71.75	72.30 $\pm$ 1.68	76.8
OOD union	73.69	72.51	73.04 $\pm$ 1.44	–
Full test	77.84	77.16	77.64 $\pm$ 1.23	–

Appendix

Table 7: Per-subsplit waterfall on AC-Low (same models and conventions as Table 6 ; sd for the final system, 0.8–1.7 at every stage). The PaLM 2S AC-Low setting also receives the episode goal.

Figure 6: The text rankers’ gains transfer across subsplits. AC-High step accuracy (3 training seeds, mean

\pm

sd) of the base policy, after the span ranker and after the click reranker, on IDD, the three overlapping OOD subsplits and the full test set; increments are computed on identical base checkpoints. Numbers: Table 6 .

	IDD		app_unseen	task_unseen	category_unseen
Condition	AC-Low	AC-High	Low / High	Low / High	Low / High
Base policy (as is)	82.8 $\pm$ 1.0	56.2 $\pm$ 0.8	72.9 / 40.8	73.6 / 42.6	72.7 / 41.3
no predictor ( $\tilde{\mathbf{z}}_{t}^{c}:=\mathbf{z}_{t}$ )	61.0 $\pm$ 1.7 $-$ 21.9	41.7 $\pm$ 1.1 $-$ 14.6	53.8 / 32.6	53.7 / 33.1	52.8 / 32.2
within-type action shuffle	71.9 $\pm$ 0.8 $-$ 10.9	45.8 $\pm$ 0.8 $-$ 10.4	62.1 / 32.3	63.5 / 34.5	61.9 / 32.7
all-candidate action shuffle	56.1 $\pm$ 0.5 $-$ 26.7	34.4 $\pm$ 0.9 $-$ 21.8	49.9 / 26.7	50.3 / 27.6	49.3 / 26.8
swapped, non-fine-tuned predictor	66.5 $\pm$ 1.9 $-$ 16.3	45.5 $\pm$ 0.5 $-$ 10.7	–	–	–

Appendix

Table 8: Ablations of the latent-predictor pathway on the full-scale base policy (exact-match step accuracy; IDD mean

\pm

sd, OOD cells means AC-Low / AC-High; subscripts: change against the as-is row (top) or the jointly fine-tuned base (middle)). Top: inference-time lesions on all steps of each split, text matcher untouched except in the last row; the as-is row is an independent evaluation of the same policies (on IDD within 0.2 of Table 4 ). Middle: training-time variants (base-policy OOD: the as-is row); the two retrained rows use the identical full-scale recipe (OOD on all steps); the parameter-matched row lists its three training runs on IDD, and § marks the mean of the two that train stably (the third collapses on clicks: 45.5–47.7 Low / 28.8–31.4 High out of distribution). Bottom: predictive objectives at 4K-episode scale (auxiliary losses with

\lambda{=}0.1

; multi-horizon InfoNCE 0.2), AC-High change against the matched 4K reference, paired per seed unless a mean is stated. Per-seed IDD values: Table 5 ; concat-MLP head (pilot scale): Table 10 .

	IDD	app_unseen	task_unseen
Probe steps / episodes	1,579 / 300	116 / 24	1,671 / 300
cos(prediction, true next latent)	0.467	0.467	0.460
cos(copy $\mathbf{z}_{t}$ , true next latent)	0.788	0.773	0.786
cos(predicted change, true change)	0.006	0.003	0.004
Right direction (%)	52.6	47.4	51.6

Appendix

Table 9: Next-latent probes of the deployed (jointly fine-tuned) predictor. “Copy” keeps the current latent

\mathbf{z}_{t}

as the prediction; direction is the share of steps on which the predicted latent change points the same way as the true change (chance 50%).

Variant	Setting	Reference	Variant	Reading
Scoring head
Concat-MLP head over $[\mathbf{z}_{t},\tilde{\mathbf{z}}_{t}^{c},c]$ replacing the structural head (ROI-conditioned predictor)	pilot, 2 ep	40.59 / 19.62	34.58 / 26.88	$-6.0$ Low, $+7.3$ High; its scores do not depend on the predictor
Encoder and resolution
DINOv2-L at 518 px as main features	4K, 4 ep	77.32 / 47.12	59.88 / 37.35	encoder and resolution change together

Appendix

Table 10: Design alternatives not adopted (AC-Low / AC-High exact-match step accuracy). Each variant is compared with a reference model that differs from it only in the stated factor and shares its scale, epochs and evaluation split. Setting: pilot = early pilot configuration (no text matcher); 4K = 4,000-episode training subset; ep = policy training epochs; evaluation on IDD (3,858 steps). Predictive objectives for the predictor: Table 8 .

System (source)	Params	Input	Judge	Split	AC-Low	AC-High
HiconAgent-3B ( Zhou et al., 2026 , Tab. 1)	3B	img	OS-Atlas	unspec. (“OOD setting”)	–	52.40
UI-TARS-2B ( Qin et al., 2025 , Tab. 8)	2B	img	OS-Atlas	7,708 (inferred)	89.3	68.9
LAMO-3B ( Wang et al., 2026 , Tab. 2)	3B	img	own (type + value match)	unspec.	92.1	65.5
SecAgent-3B ( Xie et al., 2026 , Tab. 3)	3B	img	unspec.	unspec.	–	69.5
GUI-Libra-3B ( Yang et al., 2026a , Tab. 3)	3B	img	own	AC-v2 (398)	85.9	57.3
Qwen2.5-VL-3B, re-run by GUI-Libra ( Yang et al., 2026a , Tab. 3)	3B	img	own	AC-v2 (398)	71.1	36.4

Appendix

Table 11: Small-model AndroidControl results not in Table 1 (larger and zero-shot models under other protocols: Table 12 ). Every cell is copied from the cited table. Apart from LiMAC and its two baselines, discussed in the text, none is set against our numbers because input, judge or split differ (ours for reference: 77.64 / 52.85 Low / High on the full test and 83.09 / 60.12 on IDD as 3-seed means, 78.64 / 53.75 and 84.06 / 60.99 for seed 3, under the exact-match judge with a11y input). Judge: relaxed = original protocol; OS-Atlas = a click within 14% of the screen width counts and typed text at F1

>

0.5 ( Wu et al., 2025 , App. D) ; own = the paper’s own definition. Split: 7,708 = OS-Atlas test samples; r500 / r800 = a random 500- / 800-step test subset; r500-IDD = IDD part of Random-500; unspec. = not stated in the source; AC-v2 (398) = the 398-example cleaned subset of GUI-Libra (its Pass@1 columns). † Goal-conditioned per-step action accuracy (“overall accuracy”), the analogue of AC-High. Parameter counts are nominal.

System (source)	Params	Input	Judge	Split	AC-Low	AC-High
GPT-4o, zero-shot ( Wu et al., 2025 , Tab. 5)	n/a	img	OS-Atlas	7,708	28.4	21.2
OS-Atlas-7B, zero-shot ( Wu et al., 2025 , Tab. 5)	7B	img	OS-Atlas	7,708	50.9	29.8
M3A (GPT-4-Turbo, text-only), zero-shot ( Li et al., 2024 , Tab. 4)	n/a	a11y	relaxed	r500-IDD	55.0	42.1
Aguvis-7B, SFT ( Xu et al., 2025 , Tab. 3)	7B	img	relaxed (inferred)	r500	80.5	61.5
Aguvis-72B, SFT ( Xu et al., 2025 , Tab. 3)	72B	img	relaxed (inferred)	r500	84.4	66.4
OS-Atlas-7B, SFT ( Wu et al., 2025 , Tab. 5)	7B	img	OS-Atlas	7,708	85.22	71.17

Appendix

Table 12: Zero-shot and larger models under other protocols, or with AC-High only (conventions and judge / split definitions as in Table 11 ). Every cell is copied from the cited table; none is set against our numbers. MAI-UI reports exact match on an unstated split and no AC-Low.

Figure 7: Lenient frozen-encoder localisation probe across pretraining recipes. ScreenSpot-Pro hit@8 of a soft-argmax localisation probe on frozen patch tokens (our encoders: mean

\pm

sd over 5 probe seeds); hit@8 accepts errors up to 57% of the image side. Points, left to right: a web anchor (

\approx

77K web screenshots, 60 epochs); mobile-only (119K GUI-Odyssey screenshots, 2 epochs); desktop-only (70K VideoCUA screenshots, 2 epochs); the policy encoder (blue; the mixed 189K mobile and desktop recipe, 2 epochs); a replicate pretraining of the same recipe after 2 epochs and, continued, after 4 and 6 epochs; DINOv3-ViT-L with its 95% bootstrap interval over probe examples (band). The recipes also differ in data size and training length, and the probe measures coarse localisation only (see text).

Component	Params	Setting
Encoder pretraining (frozen afterwards)
JEPA encoder, ViT-L/16	303.3M	vit_large_patch16_224 , 224 px, $14\times 14=196$ patch tokens; ImageNet initialisation; JEPA objective after I-JEPA with random-patch masking at ratio 0.5 (the student sees the screenshot with the masked patches zeroed and regresses the EMA target’s tokens at those positions, smooth-L1); 189,262 unlabelled screenshots (118,791 GUI-Odyssey mobile, 70,471 VideoCUA desktop); 2 epochs at batch 96, 3,942 optimiser steps (the policy uses the epoch-2 checkpoint); EMA target decay 0.996; peak learning rate $5\times 10^{-5}$ with warm-up; JEPA loss 1.28 $\to$ 0.68; target-token std 1.8–2.2 (min 1.82, final 2.17; no collapse); 2,739 s $\approx$ 0.76 GPU-hour on one H20
Policy (trained end to end with the listwise loss)
Action code $a_{c}$	–	16-d: action-type one-hot plus normalised coordinates or scroll direction
Residual latent predictor	39.9M	ROI-pooled encoder feature $\mathbf{u}_{c}$ (1,024-d) $\oplus$ $a_{c}$ $\to$ MLP $\to$ added to a learned query token; 3 transformer layers over the 196 patch tokens and the query, 2-D sine–cosine positional embeddings; residual output $\tilde{\mathbf{z}}_{t}^{c}=\mathbf{z}_{t}+\Delta(\mathbf{z}_{t},a_{c},\mathbf{u}_{c})$ ; initialised from a dynamics predictor trained on AndroidControl transitions, then fine-tuned with the policy loss
Structural head	2.1M	cosine, with a learned scale, between an MLP of $[\operatorname{pool}_{\operatorname{roi}(c)}(\tilde{\mathbf{z}}_{t}^{c});\,\mathrm{mean}(\tilde{\mathbf{z}}_{t}^{c})]$ and an MLP of the conditioning vector, plus a learned per-type bias; its only candidate-dependent visual input is $\tilde{\mathbf{z}}_{t}^{c}$

Appendix

Table 13: Components and hyperparameters of GUI-JEPA. Parameter counts are measured. The encoder and the CLIP text encoder are frozen and their outputs cached before policy training; the span ranker and the click reranker are trained after the base policy with the base policy fixed.

	Qwen2.5-VL-7B	Qwen2.5-VL-7B	GUI-JEPA	GUI-JEPA
	zero-shot	LoRA SFT	(ours, seed 3)	(ours, 3 seeds)
Parameters	8.29B	8.29B	0.50B	0.50B
ms/step (H20, batch 1, 100 real steps)	505 (HF)	207 (vLLM) / 507 (HF)	74 (fp32) / 51.5 (bf16)
Overall step acc.	43.2	70.9	61.0	60.1 $\pm$ 1.0
share of SFT (High / params)	–	–	86.1% / 6.0%	84.8% / 6.0%
Type accuracy	72.1	87.0	80.6	78.8 $\pm$ 3.4

Appendix

Table 14: Same-protocol comparison with a 7B VLM agent on AC-High (IDD). All systems see the same screenshots, enumerated candidates and ground-truth history and are scored with the same step-accuracy rules. The click rows use the 2,379 click and long-press steps whose ground-truth element is enumerated: type recall is the share predicted as a click, grounding (all) the share answered correctly with type errors counted as misses, and grounding

\mid

type the share correct among the steps predicted as a click. ms/step: median over the 100 real IDD steps timed for all systems ( Section 4.5 ). Ours: seed 3, reported in the main text, and the mean

\pm

sd over three seeds.

		Step acc.		Type acc.
Action type	$n$	7B	ours	7B	ours	Gap
All	3,858	70.9	61.0	87.0	80.6	9.9
click	2,419	72.0	62.8	91.3	86.3	5.8
scroll	505	71.7	59.8	83.8	64.4	1.6
input_text	282	67.0	52.5	92.2	87.6	1.1
open_app	253	81.0	68.0	87.7	92.9	0.9

Appendix

Table 15: Where the same-protocol 7B agent leads (AC-High, IDD; same inputs and judge). Step and type accuracy (%) of the model reported in the main text by ground-truth action type; Gap: points of the overall 9.9-point gap each type contributes (

n\times

step-accuracy difference

/

3,858).

Figure 8: Same-hardware per-step latency (median, one H20, batch 1). Stars: the deployed GUI-JEPA pipeline on the 100 real AC-High IDD steps timed for both systems, placed at their median of 28 scored actions (18 a11y candidates plus global actions). Lines: the synthetic fixed-

N

micro-benchmark with random inputs and

N

scored actions ( Table 16 ), a scaling illustration only. Horizontal lines: the same-protocol Qwen2.5-VL-7B SFT agent (merged LoRA) on the same steps with vLLM and HF transformers. Measurement boundaries: see the common protocol in the text.

Configuration	Steps	p50 ms	p90 ms	mean ms	peak mem	vs 7B vLLM	vs 7B HF (merged)
Real steps, fp32 (precision of all accuracy tables)	shared 100	74.3	127.8	86.9	3.45 GB	2.8 $\times$	6.8 $\times$
Real steps, bf16 autocast	shared 100	51.5	79.7	56.2	2.48 GB	4.0 $\times$	9.8 $\times$
Real steps, fp32	all 200	74.8	134.0	87.5	3.45 GB	–	–
Real steps, bf16 autocast	all 200	48.0	79.0	53.4	2.48 GB	–	–
Synthetic fixed $N$ , fp32, $N{=}5$ / 23 / 57 / 100	–	28.6 / 53.9 / 95.8 / 151.5	–	–	1.9–3.3 GB	–	–
Synthetic fixed $N$ , bf16, $N{=}5$ / 10 / 23 / 57 / 100	–	19.4 / 22.2 / 28.3 / 43.6 / 63.8	–	–	1.8–2.4 GB	–	–

Appendix

Table 16: Real-step serving path of the deployed GUI-JEPA pipeline and the synthetic micro-benchmark (one NVIDIA H20, batch 1). Real-step rows: the deployed pipeline, timed end to end on real AC-High IDD test steps; “shared 100” are the steps on which the 7B agent was also timed (identical screenshots, candidates and history), a subset of the “all 200” steps timed for GUI-JEPA. Ratios use the shared 100 steps only, against the same-protocol 7B SFT agent (merged LoRA) timed end to end: 206.7 ms with vLLM (

\geq

16 GB peak memory) and 506.7 ms with HF transformers (15.9 GB). Synthetic rows: random image tensors and candidate vectors at a fixed number

N

of scored actions, 100-iteration median, a scaling illustration only. Measurement boundaries: see the common protocol below.

Figure 9: Latency decomposition in the synthetic fixed-

N

micro-benchmark (one NVIDIA H20, batch 1, fp32 eager,

N{=}57

scored actions, random image tensors and candidate vectors, 100-iteration median). The click-reranker segment is amortised over the action mix; a click step takes 98.4 ms. The benchmark excludes JPEG decoding, CPU ROI pooling and the span ranker and is a scaling illustration only; real-step medians are in Table 16 .

Figure 10: Complementary errors behind the cascade on two real AC-High IDD steps (GUI-JEPA with both rankers and the same-protocol 7B SFT agent, given the same candidates). (a) GUI-JEPA is right with a margin above the 50%-coverage threshold $\tau$ (the median margin), so the cascade keeps its answer, while the 7B agent presses the wrong key. (b) GUI-JEPA picks the wrong destination with a margin below $\tau$ , so the cascade escalates the step to the 7B agent, which is right. Boxes: ground truth (green), GUI-JEPA (purple), 7B (rose).

Figure 11: GUI-JEPA as the cheap first stage of a cascade (AC-High, IDD). Steps with a large top-two margin are answered by GUI-JEPA, the rest by the 7B SFT agent; labels: share of steps kept by GUI-JEPA. Latency: the mean GUI-JEPA step plus, on escalated steps, the mean vLLM 7B step (same 100 real steps).

Mobile GUI Agents

Mobile GUI agents use visual and structured interface information to carry out tasks across smartphone apps.

See all 84 papers

Jev-Mobile: Jev as an Executor for Mobile GUI Agents
Sep 24, 2026Linghua ZhangEfficient VLM Inference Mobile GUI Agents
PastForward: Faster On-Device GUI Agents via Computational Experience Reuse
Sep 26, 2026Taehwan Park, Changmin Lee, Hayeon Lee +1Mobile GUI Agents On-Device Inference
Meta Superintelligence Labs
UI-KOBE: Knowledge-Oriented Behavior Exploration for Lightweight Graph-Guided GUI Agents
May 28, 2026Yuxiang Chai, Han Xiao, Xinyu Fu +3Mobile GUI Automation GUI Agents
1CUHK MMLab · 2Huawei Research · 3Shenzhen Loop Area Institute +1

Sep 24, 2026J/K move · Enter open · S save

Jev-Mobile: Jev as an Executor for Mobile GUI Agents

Linghua Zhang

Efficient VLM Inference Mobile GUI Agents

Open paper AskPDF arXiv

Vision-language models (VLMs) have become a common foundation for autonomous mobile GUI agents, but most existing systems rely on the VLM for both planning and action grounding at nearly every interaction step, leading to substantial latency and model-serving cost. We introduce Jev-Mobile, which shifts this paradigm to low-frequency VLM planning and high-frequency lightweight execution: the VLM specifies local goals, the accessibility tree defines a structured executable action space, and Jev, a fast typed decision model, repeatedly selects actions within this space. This design allows multiple GUI actions to be executed under a single VLM decision, reducing expensive VLM inference while preserving adaptive interaction. On the full AndroidWorld task suite, Jev-Mobile achieves 79% task success, compared with 78% for SeeAct-V and 84% for a Step-wise VLM baseline. Among successful trajectories, it reduces mean end-to-end execution time by 32.7% and mean model API cost by 73.4% relative to Step-wise VLM. These results show that decoupling high-level VLM reasoning from low-level action execution can substantially improve mobile GUI agent efficiency while maintaining competitive task performance.

Abstract

Mobile GUI Agents

Explore similar work

Jev-Mobile: Jev as an Executor for Mobile GUI Agents

PastForward: Faster On-Device GUI Agents via Computational Experience Reuse

UI-KOBE: Knowledge-Oriented Behavior Exploration for Lightweight Graph-Guided GUI Agents