MintAct: A Unified Visual Agent for Digital Environments

Mingfei Gao; Rui Tian; Haiming Gang; Bohan Zhai; Le Zhang; Yuanzheng Gong; Di Feng; Ege Özsoy

cs.CVSep 18, 2026

MintAct: A Unified Visual Agent for Digital Environments

Authors: Mingfei Gao, Rui Tian, Haiming Gang, Bohan Zhai, Le Zhang, Yuanzheng Gong, Di Feng, Ege Özsoy,

Organizations: Apple

Abstract

We present MintAct, a family of vision-language models that unifies UI grounding, multi-step navigation across mobile, desktop, and web, and visual tool use, trained at 2B, 4B, and 8B scales. Through careful design of our environments, data, and training recipes, MintAct models match the performance of per-domain specialists across all of these capabilities. To enable this, we develop a scalable environment and reinforcement learning (RL) infrastructure. On the environment side, we host hundreds of concurrent instances across heterogeneous per-domain backends, serving both trajectory data collection and online RL. To enable efficient and scalable RL training, an asynchronous framework keeps explicit control over the cross-domain training distribution and remains stable under noisy environment feedback and off-policy drift. Experimental results show that MintAct achieves state-of-the-art performance (48.9 on OSWorld-Verified) across a wide range of benchmarks at comparable model sizes.

Figures & tables

Figure 1: Overview of the diverse visual agentic tasks supported by MintAct . MintAct unifies three capabilities including UI grounding, multi-step navigation (across mobile, desktop and web) and visual tool use.

Figure 2: Overview of our training stages. MintAct starts from a high-resolution single-step SFT for building strong grounding capability followed by a low-resolution multi-step SFT to enable navigation and visual tool use. Then, RL specialists are developed in each domain and distilled back into a single model through RFT. The final stage is joint RL training to further strengthen model performance.

Category	Mobile	Desktop	Web
Pointing	click	left_click	click
	–	right_click , middle_click	–
	–	double_click , triple_click	–
	long_press	–	–
	–	mouse_move	hover
	–	left_click_drag	–

Table 1: Overview of the UI action spaces for the mobile, desktop, and web domains. Actions are grouped by category. Cells list the concrete action token(s) defined in each domain, or – if unavailable. All actions operate on raw screenshots and are grounded in pixel coordinates

(x,y)

. † Web has no dedicated answer action. The final response is returned as part of stop .

Figure 3: Top: Overview of the proposed asynchronous RL infrastructure. Bottom: Pipeline of trajectory group generation.

Mechanism	Monitoring Signal	Action	Prevented Failure
Per-domain quota	domain of each group	admit or discard the group	mixture drift
Quota-normalized backpressure	cumulative admitted groups	pause a domain’s producers	wasted generation
Dual-clip surrogate	$\pi_{\theta}/\pi_{\mathrm{prox}}$	clip, then floor at $-c\hat{A}$	staleness divergence
Truncated IS weight	$\pi_{\mathrm{prox}}/\pi_{\mathrm{roll}}$	down-weight the token	engine-mismatch blow-up

Table 2: The four mechanisms for improving joint asynchronous RL training. The top two balance data from different domains. The bottom two control staleness and train/inference mismatch.

Model	Grounding				Mobile Navigation		Desktop Navigation	Web Navigation		Toolcall
Model	MMBench -GUI	ScreenSpot -V2	UI-Vision	OS- World-G	Android Control	Android World	OSWorld -Verified	Weblica	Om2W	MMTool Sandbox
Qwen3-VL-2B ( Bai et al., 2025 )	69.5	86.8	13.7	46.1	55.5	36.4	17.0	27.7	10.8	0.0
Jedi-3B ( Xie et al., 2026 )	–	88.6	18.7	50.9	–	–	24.0	–	–	–
ScaleCUA-3B ( Liu et al., 2026 )	–	–	–	55.7	–	23.3	12.4	–	–	–
Ferret-UI-Lite-3B ( Yang et al., 2025c )	–	91.6	–	55.3	68.9	28.0	19.8	–	–	–
MAI-UI-2B ( Zhou et al., 2025 )	–	–	–	–	–	49.1	–	–	–	–

Table 3: Full comparison across model scales. Our numbers are marked in bold or underline when they are the best or second-best number compared to baselines.

Model	Grounding				Mobile Navigation		Desktop Navigation	Web Navigation		Toolcall
Model	MMBench -GUI	ScreenSpot -V2	UI-Vision	OS- World-G	Android Control	Android World	OSWorld -Verified	Weblica	Om2W	MMTool Sandbox
Qwen3-VL-8B	80.7	92.3	25.3	61.9	65.2	47.6	33.9	55.5	26.5	3.1
Grounding-specialist	84.1	94.6	59.3	64.2	–	–	–	–	–	–
Mobile-specialist	–	–	–	–	69.8	62.3	–	–	–	–
Desktop-specialist	–	–	–	–	–	–	37.7	–	–	–
Web-specialist	–	–	–	–	–	–	–	57.9	27.0	–

Table 4: Ablation: single-domain SFT specialists vs the jointly-trained MintAct -SFT-8B. Each specialist is trained only on its own domain’s data. The joint model is competitive with every specialist, matching or exceeding them on mobile, desktop, and tool use, and staying within a similar range on grounding and web, while covering all domains in a single model. Best numbers are marked in bold .

Stage 1	Stage 2	Grounding				Mobile Navigation		Desktop Navigation	Web Navigation		Toolcall
Stage 1	Stage 2	MMBench -GUI	ScreenSpot -V2	UI-Vision	OS- World-G	Android Control	Android World	OSWorld -Verified	Weblica	Om2W	MMTool Sandbox
$\times$	$\times$	80.7	92.3	25.3	61.9	65.2	47.6	33.9	55.5	26.5	3.1
✓	$\times$	84.1	94.6	59.3	64.2	70.0	45.7	9.4	15.6	2.0	0.0
$\times$	✓	80.1	93.1	25.0	56.6	60.7	65.5	41.5	59.1	25.5	21.7
✓	✓	82.7	94.2	56.9	64.7	71.0	63.8	42.6	57.5	30.0	19.8

Table 5: Ablation: impact of the two SFT stages on MintAct -8B. Stage 1 is high-resolution single-step SFT (grounding-focused); Stage 2 is low-resolution multi-step SFT. The last row is the full MintAct -SFT-8B.

Model	Grounding				Mobile Navigation		Desktop Navigation	Web Navigation		Toolcall
Model	MMBench -GUI	ScreenSpot -V2	UI-Vision	OS- World-G	Android Control	Android World	OSWorld -Verified	Weblica	Om2W	MMTool Sandbox
Teacher models: per-domain RL specialists.
RL-desktop	–	–	–	–	–	–	49.2	–	–	–
RL-mobile	–	–	–	–	70.5	67.8	–	–	–	–
RL-web	–	–	–	–	–	–	–	77.1	39.8	–
RL-VTU	–	–	–	–	–	–	–	–	–	22.5

Table 6: Ablation: RFT with rejection sampling. Each per-domain RL specialist starts from MintAct -SFT-8B and is trained only on its own domain. MintAct -RFT distills them into a single model, matching each specialist on its own domain while covering all domains. MintAct -Final improves on top of MintAct -RFT via joint RL.

Model	Grounding				Ground. Avg.	Mobile Navigation		Desktop Navigation	Web Navigation		Nav. Avg.	Toolcall
Model	MMBench -GUI	ScreenSpot -V2	UI-Vision	OS- World-G	Ground. Avg.	Android Control	Android World	OSWorld -Verified	Weblica	Om2W	Nav. Avg.	MMTool Sandbox
Qwen3-VL-8B-Instruct ( Bai et al., 2025 )	80.7	92.3	25.3	61.9	65.1	65.2	47.6	33.9	55.5	26.5	45.7	3.1
MintAct -SFT-8B	82.7	94.2	56.9	64.7	74.6	71.0	63.8	42.6	57.5	30.0	53.0	19.8
RL-mobile	82.6	94.1	56.7	65.8	74.8	70.5	67.8	44.2	59.8	28.4	54.1	24.7
RL-desktop	82.5	94.5	55.9	66.0	74.7	70.7	64.7	49.2	59.1	27.7	54.3	23.3
RL-joint	83.5	94.6	56.6	64.9	74.9	70.7	66.7	48.3	60.8	32.8	55.9	19.8

Table 7: Ablation: single-domain RL specialists vs MintAct -RL-8B. Each specialist is RL-trained only on its own domain’s environments, starting from the same MintAct -SFT-8B initialization. The joint model is RL-trained across mobile and desktop domains. Grounding Avg. averages the four grounding benchmarks. Nav. Avg. averages the five navigation benchmarks (AndroidControl, AndroidWorld, OSWorld-Verified, Weblica, Om2W).

Model	Mobile Navigation			Desktop Navigation		Web Navigation		Toolcall
Model	Android World	iOSWorld	Synthetic Mobile	OSWorld -Verified	Synthetic Desktop	Weblica	Om2W	MMTool Sandbox
Qwen3-VL-8B-Instruct ( Bai et al., 2025 )	47.6	2.3	26.9	33.9	39.3	55.5	26.5	3.1
+ RL-synthetic	60.3	18.8	45.6	38.6	67.9	64.0	30.2	9.3
MintAct -SFT-8B	63.8	12.8	43.1	42.6	50.0	57.5	30.0	19.8
+ RL-synthetic	59.8	22.3	51.9	39.4	66.4	66.4	28.9	21.1
+ RL-joint	66.7	24.8	43.8	48.3	56.4	60.8	32.8	19.8

Table 8: Ablation: impact of synthetic environments on MintAct -8B. We compare RL on synthetic environments from different initializations (Qwen3-VL-8B vs. MintAct -SFT-8B). RL-joint indicates training jointly on real environments (AndroidWorld and OSWorld).

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Figure A: We first choose the capabilities to target, mining them from real interaction traces and clustering them into groups. For each capability, a coding agent forks one of our four base operating systems and adapts it, together with accompanying tasks. The generated environments then serve three purposes: distilling a teacher model through supervised learning, improving the agent through direct interaction with reinforcement learning, and benchmarking progress on a held-out subset of capabilities.

Category	Hyperparameter	High-res single-step SFT	Low-res multi-step SFT	RFT
Model	initialization	Qwen3-VL-Instruct	High-res SFT model	MintAct -SFT
Optimization	epochs	1	1	1
	learning rate	$1\times 10^{-5}$	$1\times 10^{-5}$	$3\times 10^{-6}$
	LR schedule	cosine ( $0.1$ warmup)	cosine ( $0.1$ warmup)	cosine ( $0.1$ warmup)
	optimizer	AdamW	AdamW	AdamW
	weight decay	0	0	0

Appendix

Table A : Supervised fine-tuning hyper parameters of MintAct across the three stages. All stages perform full fine-tuning with the vision encoder and multimodal projector frozen.

Category	Hyperparameter	Value
RL optimization	group size ( $N$ )	8
	mini-batch size	48
	training steps	250
	learning rate	4e-6
	KL coefficient ( $\beta$ )	0.0
	TIS threshold ( $C$ )	2.0

Appendix

Table B: Joint RL training hyperparameters of MintAct .

GUI Automation

GUI automation uses AI agents to understand and operate graphical interfaces across desktop, mobile, and web applications.

See all 39 papers

How Mobile World Model Guides GUI Agents?
May 11, 2026Weikai Xu, Kun Huang, Yunren Feng +10World Model Learning GUI Agents
1Nanyang Technological University · 2MiLM Plus, Xiaomi Inc. · 3Independent Researchers +3
MobileWorldBench: Towards Semantic World Modeling For Mobile Agents
Dec 16, 2025Shufan Li, Konstantinos Kallidromitis, Akash Gokul +3World Models Mobile GUI Agents
UCLA · Panasonic AI Research · Salesforce AI Research
MolmoAct2: Action Reasoning Models for Real-world Deployment
May 4, 2026Haoquan Fang, Jiafei Duan, Donovan Clay +26Embodied Reasoning Efficient VLA Models
1Allen Institute for AI · University of Washington · 3National University of Singapore +6

May 11, 2026J/K move · Enter open · S save

How Mobile World Model Guides GUI Agents?

Weikai Xu, Kun Huang, Yunren Feng, Jiaxing Li+9

1Nanyang Technological University · 2MiLM Plus, Xiaomi Inc. · 3Independent Researchers+3

World Model Learning GUI Agents Mobile GUI Agents World Model Evaluation Interactive World Models

Open paper AskPDF arXiv

Recent advances in vision-language models have enabled mobile GUI agents to perceive visual interfaces and execute user instructions, but reliable prediction of action consequences remains critical for long-horizon and high-risk interactions. Existing mobile world models provide either text-based or image-based future states, yet it remains unclear which representation is useful, whether generated rollouts can replace real environments, and how test-time guidance helps agents of different strengths. To answer the above questions, we filter and annotate mobile world-model data, then train world models across four modalities: delta text, full text, diffusion-based images, and renderable code. These models achieve SoTA performance on both MobileWorldBench and Code2WorldBench. Furthermore, by evaluating their downstream utility on AITZ, AndroidControl, and AndroidWorld, we obtain three findings. First, renderable code reconstruction achieves high in-distribution fidelity and provides effective multimodal supervision for data construction, while text-based feedback is more robust for online out-of-distribution (OOD) execution. Second, world-model-generated trajectories can provide transferable interaction experience in the training process and improve agents' end-to-end task performance, although these data do not preserve the original distribution. Last, for overconfident mobile agents with low action entropy, posterior self-reflection provides limited gains, suggesting that world models are more effective as prior perception or training supervision than as universal post-hoc verifiers.

Abstract

GUI Automation

Explore similar work

How Mobile World Model Guides GUI Agents?

MobileWorldBench: Towards Semantic World Modeling For Mobile Agents

MolmoAct2: Action Reasoning Models for Real-world Deployment