AndroidFlux: Evaluating and Reward Modeling Failure Recovery Capabilities in Mobile Use Multimodal Agents

Gyubeum Lim; Yemo Koo; Ghunwoo Jang; Zsolt Kira

cs.LGOct 25, 2026

AndroidFlux: Evaluating and Reward Modeling Failure Recovery Capabilities in Mobile Use Multimodal Agents

Authors: Gyubeum Lim, Yemo Koo, Ghunwoo Jang, Zsolt Kira

Organizations: Georgia Institute of Technology · Konkuk University

Abstract

Mobile-use multimodal agents are typically evaluated by task success from a clean initial state, which does not isolate their ability to recover from intermediate errors. Because mobile environments change dynamically, a general mobile-use agent must complete tasks robustly even from unexpected states and in unseen environments. This requires knowing when and where an agent makes errors, and whether it can recover from errors and subsequent states. To evaluate this capability, we introduce AndroidFlux, a failure-recovery benchmark built on AndroidWorld. AndroidFlux restores recorded trajectories to controlled checkpoints before, at, and after the first recoverable error, so that different agents continue from the same device state and history. On these checkpoints, a complementary reward-modeling benchmark evaluates recovery-action preferences using downstream rollout outcomes. We further release an offline reinforcement learning (RL) dataset that improves both error recovery and overall task success. Our evaluations show that recovery performance only partially correlates with clean-start success, and that explicit reasoning helps only models trained with GUI-domain specific RL.

Figures & tables

Figure 1: Overview of AndroidFlux. A source agent commits a first error at step

t

. The error recovery benchmark replays to a checkpoint around

t

and resumes a target agent from the restored state. The reward modeling benchmark asks whether a reward model can rank recovery actions.

Benchmark	Platform	Live	Eval.	Error src.	Src. agents	Full trans.	$t{+}1$ rec. action	RM eval	Starts at	Scale
			Error recovery
AndroidControl	Mobile	✗	✗	–	–	✗	–	✗	any step	8,444 steps
AndroidWorld	Mobile	✓	✗	–	–	✗	–	✗	step 0	116 tasks
AndroidLab	Mobile	✓	✗	–	–	✗	–	✗	step 0	138 tasks
MobileWorld	Mobile	✓	✗	–	–	✗	–	✗	step 0	201 tasks
OSWorld	Desktop	✓	✗	–	–	✗	–	✗	step 0	369 tasks

Table 1: Comparison with other GUI benchmarks. Full trans. marks whether all three phases around the first error are covered;

t{+}1

rec. action gives which recovery actions are supplied at the post-error state; RM eval marks whether the benchmark offers action preference evaluation for reward models.

		Human-verified								LLM-verified
Type	Model	Scratch	$t{-}1_{\text{rep}}$	$t_{\text{rep}}$	$t_{\text{snap}}$	$t_{\text{all}}$	$t{+}1^{+}$	$t{+}1^{-}$	$t{+}1_{\text{all}}$	Scratch	$t{-}1$	$t$	$t{+}1^{+}$	$t{+}1^{-}$
General	Qwen3-VL-8B-Instruct	41.25	33.85	24.62	46.67	28.75	36.54	26.92	31.73	49.04	36.86	29.49	48.56	43.27
	Qwen3-VL-32B-Instruct	36.25	36.92	26.15	46.67	30.00	38.46	30.77	34.62	45.19	42.63	33.97	50.96	48.56
	Qwen3.8-27B	45.00	30.77	27.69	53.33	32.50	40.38	30.77	35.58	50.00	47.12	37.82	57.21	50.00
	Gemma-4-31B-IT (M3A)	37.50	40.00	36.92	66.67	42.50	48.08	36.54	42.31	43.27	51.60	44.55	62.50	61.06
	Gemini Flash 3.0 †	40.00	29.23	24.62	40.00	27.50	42.31	34.62	38.46	46.15	41.35	39.74	49.52	48.56

Table 2: General-purpose VLMs, GUI experts, and agentic frameworks on error recovery on both human-verified set and LLM-verified set. The columns follow the recovery checkpoints defined in Table 15 . All agentic frameworks are driven by Gemini Flash 3.0 and all proprietary models † are evaluated with reasoning enabled and Qwen3-VL AndroidWorld prompt.

	Single agent					UI-Genie RM				Oracle
	MAI-UI	GUI-Owl	GUI-Owl	Qwen3-VL	Qwen3-VL	BT	BT	SFT-128K	SFT-128K	5 agents	5 agents
	8B	1.5-8B	7B	8B	2B	128K	256K	Hard	Soft		+ BT-256K
UI-Genie	0.736	0.744	0.705	0.747	0.679	0.904	0.915	0.681	0.878	0.928	0.989
AndroidFlux	0.600	0.590	0.580	0.520	0.520	0.490	0.520	0.160	0.450	0.820	0.870
$\llcorner$ Error	0.597	0.569	0.569	0.500	0.514	0.444	0.486	0.097	0.403	0.819	0.875
$\llcorner$ Clean	0.607	0.643	0.607	0.571	0.536	0.607	0.607	0.321	0.571	0.821	0.857

Table 5: Pairwise accuracy on UI-Genie (1,000 pairs) and AndroidFlux (100 pairs), split into error-path and clean-path pairs. Single agent ranks by likelihood, UI-Genie RM is trained on UI-Genie, and Oracle is correct if any listed scorer is. Details are in Appendix D .

Appendix figures & tables31 assets

Supplementary material from the paper’s appendix.

Appendix

Action type	Typed arguments	Effect
click	index or (x,y)	Tap a UI element or screen coordinate
double_tap	index or (x,y)	Double tap
long_press	index or (x,y)	Press and hold
scroll	direction	Scroll the focused container
swipe	direction	Swipe across the screen
input_text	text , index	Enter text into a field

Appendix

Table 7: The action space of AndroidWorld and AndroidFlux. Actions with

\dagger

terminate the episode.

Model	Type	Reported SR (%)	Emulator Correct	Emulator SR (%)	Docker Correct	Docker SR (%)
Qwen3-VL-2B-Instruct	General	36.4	38/116	32.76	33/116	28.45
Qwen3-VL-4B-Instruct	General	45.3	55/116	47.41	51/116	43.97
Qwen3-VL-8B-Instruct	General	47.6	56/116	48.28	63/116	54.31
Qwen3-VL-30B-A3B-Instruct	General	54.3	56/116	48.28	27/116	23.28
Qwen3-VL-32B-Instruct	General	57.3	N/R	N/R	59/116	50.86
Qwen3.6-27B	General	70.3	N/R	N/R	32/116	27.59

Appendix

Table 8: Completed AndroidWorld clean-start measurements (116 tasks).

		AndroidWorld (116)		Scratch (80)		$t_{\mathrm{all}}$ (80)
Model	Type	Correct	SR (%)	Correct	SR (%)	Correct	SR (%)
Qwen3-VL-2B-Instruct	General	33/116	28.45	13/80	16.25	12/80	15.00
Qwen3-VL-4B-Instruct	General	51/116	43.97	21/80	26.25	21/80	26.25
Qwen3-VL-8B-Instruct	General	63/116	54.31	33/80	41.25	23/80	28.75
Qwen3-VL-30B-A3B-Instruct	General	27/116	23.28	11/80	13.75	21/80	26.25
Qwen3-VL-32B-Instruct	General	59/116	50.86	29/80	36.25	24/80	30.00

Appendix

Table 9: AndroidWorld clean-start success and human-verified recovery at the first-error checkpoint. AndroidWorld is the Docker run over the 116 AndroidWorld tasks in Table 8 , given for reference. Scratch is the same 116-task run restricted to the 80 recovery tasks;

t_{\mathrm{all}}

replays the recorded prefix to the first-error checkpoint over the same 80 cases.

Source model	Replay ( $n{=}65$ )	Snapshot ( $n{=}15$ )	Total ( $n{=}80$ )
Gemini Flash (Qwen3-VL prompt)	17 (26.15%)	5 (33.33%)	22 (27.50%)
Qwen3-VL-2B	10 (15.38%)	4 (26.67%)	14 (17.50%)
GUI-Owl-32B	10 (15.38%)	1 (6.67%)	11 (13.75%)
Mobile-Agent-v3-32B	10 (15.38%)	1 (6.67%)	11 (13.75%)
MAI-UI-8B	9 (13.85%)	2 (13.33%)	11 (13.75%)
Qwen3-VL-4B	9 (13.85%)	2 (13.33%)	11 (13.75%)

Appendix

Table 10: Source-model distribution of the human-verified set. Replay cases are restored by replaying the recorded action prefix; snapshot cases resume from a saved emulator snapshot.

Condition	Offset $n$	Feasible	Clipped	Feasible share
$t{-}1$	1	65	0	100.0%
$t{-}3$	3	39	26	60.0%
$t{-}5$	5	21	44	32.3%

Appendix

Table 11: Construction of the pre-error conditions on the 65 replay-based, human-verified cases. A clipped case is evaluated from a clean start because the source trajectory does not contain the requested pre-error offset.

Recovery-action source	$+$	$-$	Total	Recovery-action source	$+$	$-$	Total
MobileRun (Gemini Flash 3.0)	9	1	10	GUI-Owl-1.5-32B	0	4	4
Gemini Flash 3.5 Computer Use	3	1	4	GUI-Owl-1.5-8B (FinalRun app info)	0	4	4
GUI-Owl-1.5-2B	3	1	4	Gemini Flash 3.0 Computer Use	0	4	4
Gemma-4-31B-IT (M3A)	3	1	4	MAI-UI-2B	1	2	3
GUI-Owl-1.5-8B	3	1	4	MAI-UI-8B	2	1	3
GUI-Owl-1.5-8B-Thinking	4	0	4	Qwen3-VL-32B-Instruct	1	2	3

Appendix

Table 12: Recovery-action source distribution in the matched

t{+}1^{+}

t{+}1^{-}

subset. Counts identify the models that authored the selected actions, not the models that generated the original error trajectories.

Source model	Cases	Source model	Cases
Gemini Flash (Qwen3-VL prompt)	72 (23.08%)	GUI-Owl-1.5-2B	5 (1.60%)
GUI-Owl-32B	49 (15.71%)	Mobile-Agent-v3-7B	4 (1.28%)
Gemma-4-31B-it-0716 (M3A)	49 (15.71%)	Qwen3-VL-4B	4 (1.28%)
Gemma-4-31B (M3A)	42 (13.46%)	Qwen3-VL-4B-thinking	3 (0.96%)
Qwen3-VL-2B	24 (7.69%)	GUI-Owl-1.5-32B	3 (0.96%)
GUI-Owl-7B	17 (5.45%)	GUI-Owl-1.5-4B	2 (0.64%)

Appendix

Table 13: Source-model distribution of the LLM-verified set (

n{=}312

over 104 tasks, 21 source models).

Model	Type	Scratch (312)	SR (%)	$t$ (312)	SR (%)	AndroidWorld (116)	s104
Qwen3-VL-2B-Instruct	General	69/312	22.12	77/312	24.68	33/116	23/104
Qwen3-VL-4B-Instruct	General	120/312	38.46	101/312	32.37	51/116	40/104
Qwen3-VL-8B-Instruct	General	153/312	49.04	92/312	29.49	63/116	51/104
Qwen3-VL-32B-Instruct	General	141/312	45.19	106/312	33.97	59/116	47/104
Gemma-4-31B-IT (M3A, think off)	General	135/312	43.27	139/312	44.55	56/116	45/104
Gemma-4-31B-IT (M3A, think on)	General	–	–	148/312	47.44	–	–

Appendix

Table 14: LLM-verified set (312 cases), full table. Scratch projects the model’s 116-task AndroidWorld clean-start run onto these 312 cases;

t

replays the recorded prefix to the first-error checkpoint. AndroidWorld and s104 are the same 116-task run and its projection onto the 104 distinct tasks this set covers, given for reference.

	Human-verified										LLM-verified
Model	Scratch	$t{-}5_{\text{rep}}$	$t{-}3_{\text{rep}}$	$t{-}1_{\text{rep}}$	$t_{\text{rep}}$	$t_{\text{snap}}$	$t_{\text{all}}$	$t{+}1^{+}$	$t{+}1^{-}$	$t{+}1_{\text{all}}$	Scratch	$t{-}1$	$t$	$t{+}1^{+}$	$t{+}1^{-}$
Qwen3-VL-2B † ( Bai et al., 2025 )	16.25	13.85	16.92	23.08	12.31	26.67	15.00	28.85	17.31	23.08	22.12	27.56	24.68	41.83	32.69
Qwen3-VL-4B † ( Bai et al., 2025 )	26.25	29.23	24.62	27.69	21.54	46.67	26.25	34.62	34.62	34.62	38.46	40.71	32.37	50.96	41.83
GUI-Owl-32B ( Xu et al., 2026 )	53.75	40.00	43.08	35.38	21.54	46.67	26.25	28.85	34.62	31.73	60.58	41.35	28.21	41.35	42.79
Mobile-Agent-v3-32B	57.50	44.62	49.23	47.69	41.54	46.67	42.50	44.23	48.08	46.15	61.54	51.28	46.15	67.31	55.29
MAI-UI-8B ( Zhou et al., 2025 )	37.50	40.00	36.92	40.00	30.77	53.33	35.00	38.46	38.46	38.46	46.15	41.35	31.73	50.96	48.56

Appendix

Table 15: Human-verified (HV) and LLM-verified (LV) sets on error recovery over the HV set source agents. Scratch runs from step 0 of all 80 HV tasks. HV has 65 replay-based (

t_{\text{rep}}

) and 15 snapshot-based (

t_{\text{snap}}

) cases. At

t{+}1

, + and - take the recovery action from the successful and failed recovery trajectory, and

t{+}1_{\text{all}}

aggregates both. † Evaluated with the Qwen3-VL AndroidWorld prompt.

Source agents
Agent $\backslash$ Traj. Source	Mobile-Agent-v3-32B	Gemini Flash 3.0	GUI-Owl-32B	MAI-UI-8B	Qwen3-VL-2B	Qwen3-VL-4B	Overall
Mobile-Agent-v3-32B	30.0%	82.4%	30.0%	33.3%	10.0%	33.3%	41.5%
Gemini Flash 3.0	10.0%	52.9%	10.0%	33.3%	0.0%	22.2%	24.6%
GUI-Owl-32B	20.0%	41.2%	10.0%	22.2%	0.0%	22.2%	21.5%
MAI-UI-8B	20.0%	52.9%	30.0%	33.3%	0.0%	33.3%	30.8%
Qwen3-VL-2B	0.0%	41.2%	10.0%	0.0%	0.0%	0.0%	12.3%

Appendix

Table 16: Error recovery evaluation matrix at condition

t

on the human-verified set. Values are recovery success rates (%). The upper block is the square source-to-source matrix. The lower block averages evaluated agents that are not trajectory sources, grouped by class as in Tables 2 and 3 .

Model	Run 1	Run 2	Run 3	Run 4	Mean SR (%)	Range (pp)
	Successful tasks (SR %)				Aggregate
Qwen3-VL-4B-Instruct	46 (39.66)	46 (39.66)	46 (39.66)	47 (40.52)	39.88	0.86
Qwen3-VL-8B-Instruct	66 (56.90)	61 (52.59)	60 (51.72)	63 (54.31)	53.88	5.17
MAI-UI-8B	60 (51.72)	60 (51.72)	60 (51.72)	60 (51.72)	51.72	0.00
GUI-Owl-7B	73 (62.93)	71 (61.21)	68 (58.62)	71 (61.21)	60.99	4.31
GUI-Owl-1.5-4B	74 (63.79)	74 (63.79)	74 (63.79)	74 (63.79)	63.79	0.00

Appendix

Table 17: Repeated Docker runs on AndroidWorld (

116

tasks). Cells report the number of successful tasks with the corresponding success rate in parentheses.

Model	R1	R2	R3	R4	Mean (%)	R1	R2	R3	R4	Mean (%)	Mean (%)	Range (pp)
	Action-replay ( $65$ )					Snapshot ( $15$ )					Overall ( $80$ )
Qwen3-VL-4B-Instruct	12	14	15	13	13.50 (20.77)	8	8	8	7	7.75 (51.67)	21.25 (26.56)	3.75
Qwen3-VL-8B-Instruct	13	15	14	18	15.00 (23.08)	6	7	3	6	5.50 (36.67)	20.50 (25.63)	8.75
MAI-UI-8B	16	15	18	17	16.50 (25.38)	7	7	8	7	7.25 (48.33)	23.75 (29.69)	5.00
GUI-Owl-7B	20	14	18	17	17.25 (26.54)	6	5	8	3	5.50 (36.67)	22.75 (28.44)	8.75
GUI-Owl-1.5-4B	19	20	20	20	19.75 (30.38)	8	6	5	7	6.50 (43.33)	24.25 (30.31)	2.50

Appendix

Table 18: Repeated Docker runs on the AndroidFlux human-verified set (

80

tasks:

65

action-replay,

15

snapshot). Runs report the number of tasks recovered after resuming at the first-error step

t

; mean columns report the mean count with the corresponding rate in parentheses.

Model	Type	$\leq 3$	$4$ – $6$	$\geq 7$	Latest $-$ earliest
Qwen3-VL-2B-Instruct	General	$+0.02$	$+14.62$	$+3.37$	$+3.35$
Qwen3-VL-4B-Instruct	General	$-13.38$	$+6.75$	$+5.97$	$+19.34$
Qwen3-VL-8B-Instruct	General	$-9.48$	$-6.88$	$+10.64$	$+20.13$
Qwen3-VL-32B-Instruct	General	$-7.34$	$-3.06$	$-16.65$	$-9.30$
Qwen3.8-27B (think on)	General	$-17.67$	$+5.81$	$+7.52$	$+25.19$
Gemini Flash 3.0 (think on)	General	$-5.62$	$+5.81$	$-2.86$	$+2.76$

Appendix

Table 19: Per-model

\Delta

(injection

-

rewinding, pp) by steps before the error, and the latest

-

earliest contrast. 17 of the 18 models are positive.

Check	Group	Episodes	Contrast	95% CI	$p$
Difficulty	Q1 (hardest)	99	$+12.69$	$[+3.2,+22.8]$	0.010
	Q2	68	$+7.86$	$[-9.5,+25.2]$	0.380
	Q3	69	$+14.84$	$[-4.1,+36.5]$	0.143
	Q4 (easiest)	76	$+20.39$	$[+4.1,+37.3]$	0.011
Trajectory length	$\leq 10$ steps	156	$+21.03$	$[+2.5,+41.4]$	0.023
	$11$ – $16$ steps	54	$+18.38$	$[+0.9,+36.2]$	0.038

Appendix

Table 20: Latest

-

earliest contrast (pp) within groups of episodes. All nine are positive.

Source model	Cases	Stagnation	Direction	Execution
All	80	18 (22.50%)	35 (43.75%)	27 (33.75%)
Gemini Flash (Qwen3VL Prompt)	22	4 (18.18%)	9 (40.91%)	9 (40.91%)
Qwen3-VL-2B	14	4 (28.57%)	7 (50.00%)	3 (21.43%)
GUI-Owl-32B	11	3 (27.27%)	6 (54.55%)	2 (18.18%)
Mobile-Agent-v3-32B	11	3 (27.27%)	5 (45.45%)	3 (27.27%)
MAI-UI-8B	11	3 (27.27%)	3 (27.27%)	5 (45.45%)

Appendix

Table 21: First-error class distribution of the human-verified set (

n{=}80

), overall and by source model. Percentages are within each row.

Source model	Cases	Stagnation	Direction	Execution
All	312	147 (47.11%)	111 (35.58%)	54 (17.31%)
Gemini Flash (Qwen3VL Prompt)	72	55 (76.39%)	8 (11.11%)	9 (12.50%)
GUI-Owl-32B	49	10 (20.41%)	25 (51.02%)	14 (28.57%)
Gemma-4-31B-it-0716 (M3A)	49	12 (24.49%)	26 (53.06%)	11 (22.45%)
Gemma-4-31B (M3A)	42	10 (23.81%)	19 (45.24%)	13 (30.95%)
Qwen3-VL-2B	24	18 (75.00%)	6 (25.00%)	0 (0.00%)

Appendix

Table 22: First-error class distribution of the LLM-verified set (

n{=}312

, 21 source models), overall and by source model. Classes are the detector’s pred_error_type folded to the three-way taxonomy (stagnation

\rightarrow

Stagnation; goal_regression, goal_irrelevant, goal_direction and system_error

\rightarrow

Direction; incomplete_execution

\rightarrow

Execution). Percentages are within each row.

Group	Field	Prov.	Content
step	memory_step_idx	Norm.	Position in the replay prefix, contiguous from $0$
	source_step_idx	Copied	Index in the original source trajectory
	goal	Copied	Task instruction $c$ in force at this step
screenshot	before_screenshot	Copied	Observation $o_{i}$ before the action
	after_screenshot	Copied	Observation $o_{i+1}$ after the action
action	parsed_action	Norm.	Canonical tool call, coordinates rescaled to $[0,1]$

Appendix

Table 23: The schema-neutral step record. Provenance: Copied verbatim from the source step, Norm. deterministically re-encoded, Synth. generated when the source emits nothing usable, Annot. from the detector–verifier labels. Only Synth. fields reach a recovering agent; copied text is for auditing.

Panel of Figure 6 — Figure 6: Prompt used to generate the generalized history and context. A batch of up to ten screenshots is interleaved with one text block per step, and the model returns a batch_summary describing every evaluated step as a screen / action / result triple. That summary is carried forward verbatim into the next batch’s PRIOR BATCH SUMMARY HISTORY section, so long trajectories are summarized incrementally rather than re-read in full.

Pipeline stage (aggregate)
	Step-level				Trajectory-level
	Precision	Recall	F1	Accuracy	Precision	Accuracy
Detector	87.2	51.6	64.9	94.9	89.8	70.7
+ Verifier	90.8	50.8	65.2	95.0	92.1	71.0
By source model (verifier stage)
Mobile-Agent-v3-32B	89.5	37.8	53.1	96.2	89.5	74.8

Appendix

Table 24: First-error detection performance. The upper panel aggregates both pipeline stages over six source models (

N=666

steps); the lower panel breaks the verifier stage down by source model.

Panel of Figure 8 — Figure 8: Prompt used by the first-stage detector to evaluate Android GUI trajectory steps (Part 2).

Panel of Figure 9 — Figure 9: Cluster verifier prompt for first-counted-error refinement (Part 1).

Panel of Figure 10 — Figure 10: Cluster verifier prompt for first-counted-error refinement (Part 2).

Table 25: Provenance of the 100 restored checkpoints. Shares in (b) are normalized over all 100 cases and over the 72 detected-error cases respectively.

Model	Cand.	Panel	Model	Cand.	Panel
Qwen3-VL-2B-Instruct	✓	✓	MAI-UI-2B	–	✓
Qwen3-VL-4B-Instruct	–	✓	MAI-UI-8B	✓	✓
Qwen3-VL-8B-Instruct	✓	✓	Gemini Flash 3.0 (M3A)	–	✓
GUI-Owl-7B	✓	✓	MobileRun (Gemini Flash 3.0)	–	✓
GUI-Owl-1.5-4B	–	✓	Gemini Flash 3.5 Computer Use	–	✓
GUI-Owl-1.5-8B	✓	✓

Appendix

Table 26: Model roster of the reward modeling benchmark. Cand. marks the five policies that propose candidate recovery actions at a checkpoint; Panel marks the twelve continuation agents that roll out from each executed candidate. MAI-UI panel entries are history-conditioned and Gemini Flash 3.5 Computer Use is run task-only.

Table 27: Composition of the 100 released preference pairs.

Objective and readout
Hyperparameter	BT (64K)	BT (128K)	SFT
Base checkpoint	Qwen3-VL-8B-Instruct
Training objective	Bradley–Terry pairwise loss		Next-token cross-entropy
Loss scope	Chosen and rejected scalar rewards		Final judgment turn only
Training data	64K pairs	128K pairs	128K rows (64K pairs)
Adaptation

Appendix

Table 28: Full training configuration for the BT (64K and 128K pairs) and generative SFT baselines.

Table 29: Composition of the 8,499-prompt RL corpus. Shares in (b) are normalized within each checkpoint position.

Figure 11: Successful recovery trajectories from the first error at $t$ (GPT-5.6 Terra, think-on(medium), LLM-verified set).

Figure 12: Failed recovery trajectories from the first error at $t$ (GPT-5.6 Terra, think-on(medium), LLM-verified set).

Multimodal Agent Evaluation

See all 22 papers

AndroidReality: How Far Are Mobile Agents from the Real World?
Aug 7, 2026Xiaoou Liu, Longchao Da, Hanyang Chen +2Agent Reliability Mobile Agent Benchmarks
Arizona State University · Independent Researcher
MobiFlow: Real-World Mobile Agent Benchmarking through Trajectory Fusion
Feb 28, 2026Yunfei Feng, Xi Zhao, Cheng Zhang +5Computer-Use Agent Benchmarks Mobile Agent Benchmarks
Institute of Parallel and Distributed Systems (IPADS), Shanghai Jiao Tong University · Department of Precision Instrument, Tsinghua University · National Innovation Institute of High-end Smart Appliances
Lost in Recovery: Demystifying Recovery Behavior of Computer-Use Agents via Goal-Oriented Action Replay
Oct 20, 2026Byeonghun Bang, Sangwoo Park, Jinheon Baek +3Computer-Use Agents Computer-Use Agent Evaluation
KAIST · ETRI

Aug 7, 2026J/K move · Enter open · S save

AndroidReality: How Far Are Mobile Agents from the Real World?

Xiaoou Liu, Longchao Da, Hanyang Chen, Yuan Ling+1

Arizona State University · Independent Researcher

Agent Reliability Mobile Agent Benchmarks GUI Agents AI Agent Evaluation Mobile GUI Agents

Open paper AskPDF arXiv

Mobile agents have achieved promising results on clean online benchmarks such as AndroidWorld, yet their performance often degrades sharply in real-world deployment due to environmental variations and imperfect interface conditions. In this work, we introduce AndroidReality, a perturbation-based framework for evaluating and improving the robustness of mobile agents. Through a Markov Decision Process (MDP) perspective, we organize real-world interface variability into a principled taxonomy of perturbations along three axes: state, transition, and action. Guided by this taxonomy, we build a perturbed mobile benchmark on top of AndroidWorld with realistic and controllable perturbation injections, enabling systematic robustness evaluation of mobile agents. Our evaluation reveals substantial robustness gaps and four recurring error categories, motivating a simple training-free Test-Time Introspective Recovery (TTIR) mechanism that mitigates these failures on both perturbed and clean settings. Together, these results position robustness as a missing dimension in mobile agent evaluation and establish benchmark perturbation as an effective tool for both stress testing and surfacing latent weaknesses of mobile agents.

Abstract

Multimodal Agent Evaluation

Explore similar work

AndroidReality: How Far Are Mobile Agents from the Real World?

MobiFlow: Real-World Mobile Agent Benchmarking through Trajectory Fusion

Lost in Recovery: Demystifying Recovery Behavior of Computer-Use Agents via Goal-Oriented Action Replay