MobileDev-Bench: A Benchmark for Issue Resolution in Mobile Application Development

Moshood A. Fakorede; Krishna Upadhyay; A. B. Siddique; Umar Farooq

cs.SEMar 26, 2026

MobileDev-Bench: A Benchmark for Issue Resolution in Mobile Application Development

Authors: Moshood A. Fakorede, Krishna Upadhyay, A. B. Siddique, Umar Farooq

Organizations: Louisiana State University · University of Kentucky

Abstract

Large language models (LLMs) have shown strong performance on automated software engineering tasks, yet existing benchmarks focus primarily on library-style repositories, leaving mobile application development largely unexplored despite its framework-specific build systems, heterogeneous artifact types, and coordinated multi-file fix requirements. We introduce MobileDev-Bench, a benchmark comprising 407 real-world issue-resolution tasks collected from 19 production mobile applications spanning Android Native (Java/Kotlin), React Native (TypeScript), and Flutter (Dart). Each task pairs a verified developer-reported issue with executable test patches, enabling fully automated validation of model-generated fixes within mobile build environments. The benchmark exhibits substantially greater patch complexity than prior benchmarks: fixes modify 12.9 files and 334.6 lines on average, and 41% of instances require coordinated changes across multiple artifact types, such as source, build configuration, and resource files. Evaluation of four frontier LLMs (Claude Sonnet 4.5, Qwen3-Coder, GPT-5.2, and Gemini 2.5 Flash) yields end-to-end resolution rates of only 3.23% - 4.23% under automated retrieval and at most 5.69% under oracle retrieval, well below resolution rates reported on existing benchmarks. We release MobileDev-Bench with task instances, an evaluation harness, and containerized environments to support reproducible research on AI-assisted mobile application development.

Figures & tables

Benchmarks	Domain	#Tasks	#Repos	Verified	Difficulty level	Multi-type artifact fixes	Patch # Files	Complex. # Lines
SWE-bench	Python libraries	2294	12	✗	✗	✗	1.7	32.8
SWE-bench Verified	Python libraries	500	12	✓	✗	✗	1.2	14.3
SWE-bench Multimodal	JavaScript libraries	617	17	✓	✗	✓	3.9	131.6
SWE-PolyBench	Multi-lang. libraries	2110	21	✗	✗	✗	2.6	51.2
SWE-bench Multilingual	Multi-lang. libraries	300	42	✗	✗	✗	1.7	47.8
OmniGIRL	Multi-lang. repos	959	15	✗	✗	✗	1.2	46.3

Table 1: Comparison of issue-resolution benchmarks. Unlike prior work that primarily evaluates source-code fixes in library-style repositories, MobileDev-Bench targets mobile apps, incorporates human verification and difficulty stratification, evaluates multi-artifact fixes (e.g., source and resources artifacts such as manifest files), and restores pre-fix buildability to enable execution-based evaluation.

Figure 1: Overview of MobileDev-Bench construction pipeline. The framework consists of five phases: (1) filtering for high-quality repositories, (2) mining issue-linked PRs, (3) reproducing mobile builds via Docker, (4) executing compilation-aware validation, and (5) manually reviewing data to yield 407 verified, difficulty-stratified benchmark instances.

	Codebase		Instances		Fix Patch (Avg.)			Tests (Avg.)
Repository	#Files	#LoC	#	#Tok	#L	#H	#F	F2P	N2P	P2P
Flutter (Dart)
talawa	3,264	236k	12	280	256.6	18.0	5.9	1.0	4.2	3.8
zulip-flutter	514	141k	51	325	254.9	18.4	7.9	1.2	53.7	49.9
Android Native (Java/Kotlin)
antennapod	1,143	115k	9	1,567	348.1	35.7	12.9	0.0	15.7	0.0

Table 2: Per-repository statistics for MobileDev-Bench. #Files and #LoC report codebase size at the base commit. # is the instance count. #Tok is the average issue description length in tokens. Fix patch columns (#L: changed lines, #H: hunks, #F: files) and test transition columns (F2P: Fail-to-Pass, N2P: None-to-Pass, P2P: Pass-to-Pass) report per-instance averages.

	Issue	Codebase		Fix Patch			Tests
	Tokens	#Files	#Lines (K)	#Lines	#Files	#Hunks	F2P	N2P	P2P
Mean	308	4,605	389	334.6	12.9	27.5	1.1	31.1	11.9
Median	173	6,342	365	93	5	12	0	11	0
Max	11,501	7,923	2,652	11,915	237	592	30	375	267

Table 3: Summary statistics for MobileDev-Bench.

Figure 2: Distribution of MobileDev-Bench tasks by difficulty, category, and artifact pattern. Tasks are balanced across difficulty tiers, with bug fixes as the largest category. A significant number of tasks (41.0%) require coordinated changes across multiple artifact types.

Figure 3: Distribution of patch complexity metrics (files modified, hunks, changed lines) across five benchmarks (SWE-PB: SWE-PolyBench; SWE-MM: SWE-bench Multimodal; Multi-SWE: Multi-SWE-bench; Rust-SWE: Rust-SWE-bench; Ours: MobileDev-Bench). Each data point is the per-repository average. Y-axes are capped at the 90th percentile for visual clarity.

		Automated Retrieval				Oracle Retrieval
Difficulty / artifact types	Tasks	Claude	Qwen	GPT	Gemini	Claude	Qwen	GPT	Gemini
Easy	138 (33.9%)	8.0	8.5	8.0	8.0	13.0	6.7	10.9	4.3
Medium	139 (34.2%)	4.4	1.5	1.5	1.5	2.9	0.7	2.2	1.4
Hard	130 (31.9%)	0.0	0.0	0.0	0.0	0.8	0.0	0.0	0.0
Single-type artifact	240 (59%)	6.7	5.0	5.4	5.0	9.6	4.2	7.1	3.3
Multi-type artifact	167 (41%)	0.6	0.6	0.0	0.6	0.0	0.0	0.6	0.0

Table 4: Resolution rates (%) by difficulty level and artifact diversity under automated and oracle retrieval. Results show that resolution rates decline as difficulty and artifact complexity increase.

Figure 4: Resolution rates by ground-truth file count and patch size. Single-file tasks achieve 13–16% (Automated) and 10–25% (Oracle); rates collapse for multi-file and large-patch tasks.

Figure 5: Recall and F1 scores decrease across all evaluated models as the number of modified files increases. Oracle retrieval consistently outperforms Automated retrieval across complexity levels.

Appendix figures & tables15 assets

Supplementary material from the paper’s appendix.

Appendix

Repository	Summary	License
palisadoesfoundation/talawa	Community organization management	GNU GPLv3
zulip/zulip-flutter	Zulip mobile apps for Android and iOS	Apache 2.0
antennapod/antennapod	Podcast manager for Android	GNU GPLv3
commons-app/apps-android-commons	Wikimedia Commons Android uploader	Apache 2.0
element-hq/element-x-android	Android Matrix messenger (Jetpack Compose)	GNU AGPLv3
futsch1/medtimer	Medication reminder app	MIT

Appendix

Table 5: Summary and licenses for all GitHub repositories represented in MobileDev-Bench.

	PRs Crawled	Filtered	Validated	Verified
palisadoesfoundation/talawa	1,743	212 $\downarrow$ 1531	13 $\downarrow$ 199	12 $\downarrow$ 1
zulip/zulip-flutter	1,051	96 $\downarrow$ 955	51 $\downarrow$ 45	51 .0
antennapod/antennapod	2,031	89 $\downarrow$ 1942	11 $\downarrow$ 78	9 $\downarrow$ 2
commons-app/apps-android-commons	1,477	200 $\downarrow$ 1277	17 $\downarrow$ 183	10 $\downarrow$ 7
element-hq/element-x-android	3,899	303 $\downarrow$ 3596	77 $\downarrow$ 226	68 $\downarrow$ 9
futsch1/medtimer	643	25 $\downarrow$ 618	12 $\downarrow$ 13	8 $\downarrow$ 4

Appendix

Table 6: Benchmark repository instance conversion pipeline showing progression from collected pull requests through attribute-based filtering, execution-based validation, and manual verification.

Figure 6: Distribution of manual annotation scores for (a) problem-statement clarity and (b) unit-test coverage adequacy, across all 541 annotated candidate instances. Green bars (scores 0–1) indicate instances accepted into the benchmark; hatched red bars (scores 2–3) indicate excluded instances. The majority of exclusions arise from vague issue descriptions (score 2/3 on dimension a) or tests that impose undue implementation constraints (score 2/3 on dimension b).

Figure 7: Task characteristics of the 407 verified benchmark instances. (a) Difficulty distribution stacked by issue type, using the four time-based difficulty labels assigned during annotation. (b) Overall issue-type breakdown. The benchmark spans the full difficulty range and covers all three major categories of mobile development activity.

Metric	Android Native	Flutter	React-Native
Instance distribution
Repositories	13	2	4
Instances	328	63	16
Fix patch statistics
Avg. # files	14.0	7.6	9.5
Avg. # hunks	28.8	18.3	36.2

Appendix

Table 7: Framework-wise instance distribution and code change characteristics in MobileDev-Bench

	Overall		% by Framework
Combination	#	%	Android Native	Flutter	React Native
Single Artifact Type
Src	238	58.5	55.8	74.6	50.0
Res	1	0.3	0.3	0.0	0.0
Build	1	0.2	0.3	0.0	0.0
Subtotal	240	59.0	56.4	74.6	50.0

Appendix

Table 8: Distribution of artifact type combinations in MobileDev-Bench. 41.0% of instances require coordinated changes across multiple artifact types.

Figure 8: Percentage of instances per framework whose fix patches involve each technical knowledge domain, inferred from import statements, annotations, and identifiers in the added lines of fix patches. An instance may span multiple domains.

		Resolved				Rate (%)
#Files	Tasks	C. Sonnet 4.5	Qwen3	GPT 5.2	G. 2.5	C. Sonnet 4.5	Qwen3	GPT 5.2	G. 2.5
Automated Retrieval
1 file	71	11	9	11	9	15.5	12.7	15.5	12.7
2–3 files	89	4	2	2	2	4.5	2.2	2.2	2.2
4–5 files	52	2	2	0	2	3.8	3.8	0.0	3.8
6–10 files	70	0	0	0	0	0.0	0.0	0.0	0.0

Appendix

Table 9: Resolution rates by ground truth file count under the Automated and Oracle retrieval settings. Rate is resolved / evaluated instances (instances without a submitted prediction excluded in Automated retrieval).

Figure 9: Resolution rates by number of files modified, for Automated (solid lines) and Oracle (dashed lines) retrieval settings. Single-file tasks achieve 12.7–15.5% under Automated and up to 25.4% under Oracle, dropping to near zero for 6+ files.

Figure 10: Resolution rates by patch size, for Automated (solid lines) and Oracle (dashed lines) retrieval. Tiny patches (

\leq

10 lines) achieve 14.3–16.1% under Automated and 10.7–30.4% under Oracle, dropping to near zero for patches

>

100 lines.

Figure 11: Patch size distribution for all instances versus resolved instances. Resolved instances skew heavily toward patches

\leq

50 lines (92.9% vs. 35.1%).

		Automated Retrieval		Oracle Retrieval
Category	Bench.	Resolved	Ratio	Resolved	Ratio
Artifact Type Diversity
Single-artifact	59.0%	96.4%	1.63x	97.1%	1.65x
Multi-artifact	41.0%	3.6%	0.09x	2.9%	0.07x
File Count Distribution
1 file	17.4%	75.0%	4.31x	76.5%	4.40x

Appendix

Table 10: Resolution rate breakdown by task characteristics (407 tasks). Resolved column shows the share of uniquely resolved instances (28 under Automated, 34 under Oracle) in each category vs. the overall benchmark share. Ratio = Resolved% / Benchmark%.

Model	#L	#A	#R	#H	#F
Automated Retrieval
Claude Sonnet 4.5	8.5	6.2	2.3	1.4	1.0
Gemini 2.5 Flash	8.7	5.9	2.8	1.4	1.0
GPT-5.2	10.2	7.3	2.9	1.8	1.1
Qwen3 Coder	12.7	9.0	3.7	1.6	1.1
Oracle Retrieval

Appendix

Table 11: Average fix patch statistics for model predictions under Automated and Oracle retrieval settings. #L: total changed lines (added + removed). #A: added lines. #R: removed lines. #H: hunks. #F: files modified. Averages are over non-empty submitted patches only.

		Claude Sonnet 4.5			Qwen3-Coder			GPT 5.2			Gemini 2.5 Flash
#Files	$n$	R	P	F1	R	P	F1	R	P	F1	R	P	F1
Automated
1 file	71	54.9	54.9	54.9	46.5	46.5	46.5	35.2	35.2	35.2	35.2	35.2	35.2
2–3 files	89	22.7	49.4	30.5	23.6	48.3	30.9	20.4	44.4	27.5	20.2	44.9	27.5
4–5 files	52	12.1	51.9	19.6	12.7	53.8	20.5	9.9	40.4	15.8	10.5	42.3	16.7
6–10 files	70	9.4	65.7	16.4	7.5	52.1	13.0	7.4	49.3	12.8	6.7	46.4	11.6

Appendix

Table 12: File-level retrieval metrics (Recall / Precision / F1, macro-averaged, %) by ground-truth file count under Automated and Oracle retrieval settings (407 tasks).

Figure 14: Retrieval metrics (recall, precision, F1) by ground-truth file count across all four models. Recall collapses from 35–55% on single-file tasks to 2–3% on 11+ file tasks, while precision remains comparatively stable (35–66%), widening the precision-recall gap with complexity. F1 degrades sharply, driven entirely by recall failure.

Software Engineering

Software engineering studies how to design, build, test, deploy, and maintain reliable software systems and the processes behind them.

See all 270 papers

MobileKernelBench: Can LLMs Write Efficient Kernels for Mobile Devices?
Mar 12, 2026Xingze Zou, Jing Wang, Yuhua Zheng +8LLM Code Generation Efficient Neural Network Inference
Zhejiang University · Westlake University · HKUST +2
Benchmarking LLM Judges for Mobile Agent Evaluation
Aug 11, 2026Ziqiang Wang, Li Gu, Zhixiang Chi +4LLM-as-a-Judge Mobile Agent Benchmarks
Mila – Québec AI Institute · Concordia University · University of Toronto +2
AndroidDaily: A Verifiable Benchmark for Mobile GUI Agents on Real-World Closed-Source Applications
May 26, 2026Yifan Sui, Xin Huang, Hongbing Li +14Automated Evaluation Computer-Use Agent Benchmarks
1Beijing University of Posts and Telecommunications · 2StepFun · 3Waseda University

Mar 12, 2026J/K move · Enter open · S save

MobileKernelBench: Can LLMs Write Efficient Kernels for Mobile Devices?

Xingze Zou, Jing Wang, Yuhua Zheng, Xueyi Chen+7

Zhejiang University · Westlake University · HKUST+2

LLM Code Generation Efficient Neural Network Inference

Open paper AskPDF arXiv

Large language models (LLMs) have demonstrated remarkable capabilities in code generation, yet their potential for generating kernels specifically for mobile devices remains largely unexplored. In this work, we extend the scope of automated kernel generation to the mobile domain to investigate the central question: Can LLMs write efficient kernels for mobile devices? To enable systematic investigation, we introduce MobileKernelBench, a comprehensive evaluation framework comprising a benchmark prioritizing operator diversity and cross-framework interoperability, coupled with an automated pipeline that bridges the host-device gap for on-device verification. Leveraging this framework, we conduct extensive evaluation on the CPU backend of Mobile Neural Network (MNN), revealing that current LLMs struggle with the engineering complexity and data scarcity inherent to mobile frameworks; standard models and even fine-tuned variants exhibit high compilation failure rates (over 54%) and negligible performance gains due to hallucinations and a lack of domain-specific grounding. To overcome these limitations, we propose the Mobile Kernel Agent (MoKA), a multi-agent system equipped with repository-aware reasoning and a plan-and-execute paradigm. Validated on MobileKernelBench, MoKA achieves state-of-the-art performance, boosting compilation success to 93.7% and enabling 27.4% of generated kernels to deliver measurable speedups over native libraries.

Abstract

Software Engineering

Explore similar work

MobileKernelBench: Can LLMs Write Efficient Kernels for Mobile Devices?

Benchmarking LLM Judges for Mobile Agent Evaluation

AndroidDaily: A Verifiable Benchmark for Mobile GUI Agents on Real-World Closed-Source Applications