SecAgent: Efficient Mobile GUI Agent with Semantic Context

Yiping Xie; Song Chen; Jingxuan Xing; Wei Jiang; Zekun Zhu; Yingyao Wang; Pi Bu; Jun Song

cs.CVMar 9, 2026

SecAgent: Efficient Mobile GUI Agent with Semantic Context

Authors: Yiping Xie, Song Chen, Jingxuan Xing, Wei Jiang, Zekun Zhu, Yingyao Wang, Pi Bu, Jun Song,

Organizations: Taobao & Tmall Group of Alibaba

Abstract

Mobile Graphical User Interface (GUI) agents powered by multimodal large language models have demonstrated promising capabilities in automating complex smartphone tasks. However, existing approaches face two critical limitations: the scarcity of high-quality multilingual datasets, particularly for non-English ecosystems, and inefficient history representation methods. To address these challenges, we present SecAgent, an efficient mobile GUI agent at 3B scale. We first construct a human-verified Chinese mobile GUI dataset with 18k grounding samples and 121k navigation steps across 44 applications, along with a Chinese navigation benchmark featuring multi-choice action annotations. Building upon this dataset, we propose a semantic context mechanism that distills history screenshots and actions into concise, natural language summaries, significantly reducing computational costs while preserving task-relevant information. Through supervised and reinforcement fine-tuning, SecAgent outperforms similar-scale baselines and achieves performance comparable to 7B-8B models on our and public navigation benchmarks. Our dataset is available at https://huggingface.co/datasets/alibabagroup/CMGUI.

Figures & tables

Figure 1: Performance comparison of various agent models on three representative navigation benchmarks. SecAgent-3B achieves superior performance to 3B models and comparable performance to 7B-8B models.

Name	Training Data						Name	Benchmark Data
Name	Traj.	Steps	Apps	CoT	Human-Verified	Open-Source	Name	Traj.	Steps	Apps	Multi-Choice
Mobile-Bench-v2 [ 47 ]	12,856	93,592	49	✗	✗	✓	Magic-RICH [ 37 ]	-	4,000	150	✗
AgentCPM-GUI [ 57 ]	55k	470k	30	✗	✓	✗	CAGUI [ 57 ]	600	4,516	30	✗
CMGUI (Ours)	29,711	121,265	44	✓	✓	✓	CMGUI-Bench (Ours)	390	2,574	44	✓

Table 1: Comparison between our CMGUI dataset and existing Chinese mobile navigation datasets and benchmarks. We consider the scale (the number of trajectories and step annotations), diversity (the number of apps), whether each step includes a thought annotation, whether the dataset is verified by humans, whether it has been open-sourced, and whether the benchmark has multi-choice annotations. Note that Magic-RICH only contains step-level data.

Figure 2: Collection pipeline of our CMGUI dataset. Grounding data: UI elements are selected from random walk episodes, with MLLMs generating instructions. Navigation data: episodes are collected using a hybrid human-agent strategy, then humans annotate action correctness and bounding boxes, while MLLMs annotate semantic context and thoughts.

Figure 3: a) Samples from the GUIOdyssey dataset [ 25 ] , where SAM2 [ 31 ] is used to segment UI elements to obtain bounding boxes, inevitably leading to noise and errors ( e.g . , the red box is incorrectly sized). b) Samples from our CMGUI data. We employ well-trained annotators to conduct box-by-box annotation and implement a double-check mechanism to ensure precise bounding boxes. c) For our CMGUI-Bench, we conduct step-by-step and box-by-box annotation and consider all reasonable actions at each step, producing multi-choice action annotation.

Figure 4: Comparison of different methods. Left. a) Baseline: take the instruction and the current screenshot as input and directly output the action. b) AgentCPM [ 57 ] and UI-R1 [ 27 ] expect the model to have reasoning capabilities and thus introduce thought before output action. c) UI-Venus [ 29 ] , GUI-R1 [ 28 ] , and OS-Atlas [ 45 ] add history actions to the input. d) OdysseyAgent [ 25 ] uses both history actions and screenshots in the input. To ensure optimal results, it uses a large number of historical screenshots, for example, $N=5$ . e) Our SecAgent records history information by maintaining a concise and natural language-based summary. Thanks to this, SecAgent can achieve good results using only a historical image and action. Right. A demonstration of SecAgent.

Figure 5: The training framework of SecAgent. We observe that the original grounding and navigation datasets exhibit inherent biases. To mitigate this issue, we do not directly utilize the grounding data for SFT training as in prior work [ 57 , 53 ] . Instead, we transform the grounding annotations into navigation-compatible format by treating the center point of each bounding box as the target click coordinate.

Model	Step Acc.	Task Acc.	Action Acc.
Model	Step Acc.	Task Acc.	Click Acc.	Type Acc.	Swipe Acc.	Terminate Acc.
Closed-source Models
GPT-4o [ 15 ]	48.4	0.0	43.5	68.3	63.7	62.1
Claude-3-Sonnet [ 2 ]	81.4	25.8	91.1	30.0	81.3	53.4
Open-source Models (7B and 8B)
Qwen2.5-VL-7B [ 3 ]	83.8	35.6	89.0	75.9	4.4	79.0

Table 2: Comparison of various models on the CMGUI benchmark.

Models	AndroidControl	GUIOdyssey
Closed-source Models
GPT-4o [ 15 ]	20.8 ∗	29.3
Claude-3-Sonnet [ 2 ]	12.5 ∗	46.5
Open-source Models (7B and 8B)
Qwen2.5-VL-7B [ 3 ]	62.9 ∗	59.6
UI-TARS-7B [ 29 ]	74.4 ∗	37.5

Table 3: Performance comparison on two widely used English benchmarks. We report the step accuracies under high-level instructions. Bold text indicates the best score among 3B and 4B models, and underlined text represents the best score among all baseline models. Superscripts ∗ , † and ‡ indicate results from AgentCPM-GUI [ 57 ] , UI-Venus [ 29 ] , GUI-R1 [ 28 ] , respectively.

Settings	SA	TA	ITC	TTFT $\downarrow$	TPS $\uparrow$
N=0	85.3	36.2	1537	0.07	145
N=1	94.8	72.8	2239	0.11	140
N=2	95.1	73.8	2760	0.14	134
N=5	95.5	74.1	3642	0.22	122
N=5 w/o SC	95.0	73.0	3507	0.22	124
N=1 w/o SC	90.6	56.4	2171	0.11	142

Table 4: Performance and efficiency comparison across different history window sizes (

N

), and the ablation of Semantic Context (SC) and thought. For efficiency evaluation, we report the Input Token Count (ITC), Time to First Token (TTFT), Tokens per Second (TPS), and training hours, following [ 25 ] . ITC includes both text and vision tokens.

Training Data	Training Method	In-Domain Apps				Out-of-Domain Apps
Training Data	Training Method	Taobao SA / TA	Kuaishou SA / TA	Fliggy SA / TA	JD SA / TA	Rednote SA / TA	Bilibili SA / TA	Baidu Maps SA / TA	Pinduoduo SA / TA
Partial	SFT	91.2 / 56.9	89.1 / 44.8	91.9 / 58.3	91.9 / 61.1	91.7 / 54.5	88.4 / 45.8	86.6 / 41.7	94.4 / 66.7
Partial	SFT + RFT	94.5 / 70.8	90.1 / 48.3	96.0 / 79.2	93.5 / 61.1	96.4 / 80.0	90.2 / 45.8	91.3 / 62.5	97.6 / 83.3
Full	SFT	94.5 / 70.8	91.2 / 58.6	96.1 / 79.2	92.9 / 61.1	97.3 / 87.3	96.3 / 79.2	96.9 / 79.2	97.7 / 83.3
Full	SFT + RFT	95.5 / 73.8	93.2 / 75.9	96.6 / 79.2	97.6 / 83.3	98.4 / 89.1	97.6 / 91.7	96.6 / 79.2	97.6 / 88.9

Table 5: Performance comparison on in-domain and out-of-domain apps. The ‘Partial’ and ‘Full’ training data refer to the entire CMGUI dataset, excluding and including the four out-of-domain apps, respectively. For presentation conciseness, the results of four representative in-domain apps are presented.

Training Data	Training Method	SA	TA
Navigation	SFT	93.7	65.8
Grounding + Navigation	SFT	94.4	69.2
Gr2Nav + Navigation	SFT	94.8	72.8
Gr2Nav + Navigation	RFT ∗	95.2	75.9
Gr2Nav + Navigation	SFT + RFT	96.4	80.0

Table 6: Comparison of different training data and training methods. ‘Grounding’ denotes the original grounding data, and ‘Gr2Nav’ denotes the transformed grounding data with navigation-compatible format. ∗ Direct RFT on the base model fails to converge; thus we apply a lightweight SFT with 1k navigation step data before RFT.

Hyperparameter	Value
Supervised Fine-tuning
Epoch	6
Batch Size	2
Learning Rate	2e-5
LoRA Rank	8
LoRA Alpha	16

Table 7: The hyperparameters used for training SecAgent.

Action	Description	Format
CLICK	Click the point on the screen with coordinate (x, y).	{"name": "mobile_use", "arguments": {"action": "click", "coordinate": [x, y]}}
SWIPE	Swipe from the starting point with coordinate (x, y) to the end point with coordinates2 (x2, y2).	{"name": "mobile_use", "arguments": {"action": "swipe", "coordinate": [x, y], "coordinate2": [x2, y2]}}
TYPE	Input the specified text into the activated input box.	{"name": "mobile_use", "arguments": {"action": "type", "text": "specified_text"}}
SYSTEM_BUTTON	Press the specified system button ( e.g . , “Back”, “Home”, “Menu”, “Enter”).	{"name": "mobile_use", "arguments": {"action": "system_button", "button": "specified_button"}}
WAIT	Wait specified seconds for the change to happen.	{"name": "mobile_use", "arguments": {"action": "wait", "time": "specified_seconds"}}
TERMINATE	Terminate the current task and report its completion status (“success” or “failure”).	{"name": "mobile_use", "arguments": {"action": "terminate", "status": "success"}}

Table 8: The action space of CMGUI.

#ID	App Name	#Traj.	#ID	App Name	#Traj.	#ID	App Name	#Traj.	#ID	App Name	#Traj.
1	Taobao	4,722	2	Kuaishou	1,829	3	Rednote	1,341	4	Baidu Maps	1,213
5	Fliggy	1,184	6	Bilibili	1,157	7	WeChat	1145	8	JD	1,023
9	Pinduoduo	861	10	Weather	817	11	Meituan	777	12	Douyin	757
13	Calendar	741	14	Poizon	714	15	Aliyun Drive	693	16	Amap	686
17	Ele.me	640	18	Quark Browser	589	19	Toutiao	580	20	Tencent Maps	576
21	Calculator	541	22	Baidu	519	23	Tencent Meeting	499	24	Browser	496

Table 9: Our constructed CMGUI consists of 44 mainstream Chinese apps. The number of trajectories for each app is shown in the table.

Figure 6: Examples of grounding data in CMGUI. For each screenshot, multiple grounding data can be synthesized and then converted into multiple single-step navigation data.

Figure 7: Example trajectories in CMGUI-Bench. The multi-choice annotations are explicitly drawn on the image.

Figure 8: The prompt used by SecAgent. We note that the Chinese text has been translated into English for presentation purposes.

Figure 9: The prompt used to generate grounding instructions. We note that the Chinese text has been translated into English for presentation purposes.

Mobile GUI Automation

Mobile GUI automation develops agents and testing systems that interpret smartphone interfaces and carry out or verify multi-step tasks.

See all 37 papers

GUI-CEval: A Hierarchical and Comprehensive Chinese Benchmark for Mobile GUI Agents
Mar 16, 2026Yang Li, Yuchen Liu, Haoyu Lu +8Computer-Use Agent Benchmarks GUI Agents
HyperAI Team, Xiaomi Corporation
Beyond the GUI Paradigm: Do Mobile Agents Need the Phone Screen?
Jun 16, 2026Li Gu, Zihuan Jiang, Linqiang Guo +6Mobile GUI Automation AI Coding Agents
1Mila – Québec AI Institute · 2Concordia University · University of Toronto +1
Xiaomi-GUI-0 Technical Report
Jun 30, 2026Wanxia Cao, Chengzhen Duan, Pei Fu +29Mobile GUI Automation RL for GUI Agents

Mar 16, 2026J/K move · Enter open · S save

GUI-CEval: A Hierarchical and Comprehensive Chinese Benchmark for Mobile GUI Agents

Yang Li, Yuchen Liu, Haoyu Lu, Zhiqiang Xia+7

HyperAI Team, Xiaomi Corporation

Computer-Use Agent Benchmarks GUI Agents AI Agent Benchmarks Mobile GUI Agents

Open paper AskPDF arXiv

Recent progress in Multimodal Large Language Models (MLLMs) has enabled mobile GUI agents capable of visual perception, cross-modal reasoning, and interactive control. However, existing benchmarks are largely English-centric and fail to capture the linguistic and interaction characteristics of the Chinese mobile ecosystem. They also focus on isolated skills such as GUI grounding or offline agent, lacking a unified and fine-grained framework to assess the full capability chain from perception to execution. To address this gap, we introduce GUI-CEval, the first comprehensive benchmark for Chinese mobile GUI agents, built entirely on physical device environments. GUI-CEval spans 201 mainstream apps across four device types and adopts a two-level structure that evaluates both atomic abilities and realistic application-level performance along five dimensions: perception, planning, reflection, execution, and evaluation. All data are collected and verified through multi-stage manual processes to ensure authenticity and reproducibility. Extensive experiments on 20 representative MLLMs and multi-agent systems show that while models such as Qwen2.5-VL and UI-TARS perform competitively, most MLLMs still exhibit clear weaknesses in reflective decision-making and post-action self-evaluation, limiting their reliability in real-world interactions. We hope GUI-CEval provides a comprehensive and interpretable benchmark to guide capability diagnosis and advance the development of Chinese mobile GUI agents.

Abstract

Mobile GUI Automation

Explore similar work

GUI-CEval: A Hierarchical and Comprehensive Chinese Benchmark for Mobile GUI Agents

Beyond the GUI Paradigm: Do Mobile Agents Need the Phone Screen?

Xiaomi-GUI-0 Technical Report