CoME: Empowering Channel-of-Mobile-Experts with Informative Hybrid-Capabilities Reasoning
Authors: Yuxuan Liu, Weikai Xu, Kun Huang, Changyu Chen, Jiankun Zhao, Pengzhi Gao, Wei Liu, Jian Luan, Shuo Shang, Bo Du, Ji-Rong Wen, Rui Yan
Organizations: Gaoling School of Artificial Intelligence, Renmin University of China · MiLM Plus, Xiaomi Inc. · Nanyang Technological University · Wuhan University
Mobile Agents can autonomously execute user instructions, which requires hybrid-capabilities reasoning, including screen summary, subtask planning, action decision and action function. However, existing agents struggle to achieve both decoupled enhancement and balanced integration of these capabilities. To address these challenges, we propose Channel-of-Mobile-Experts (CoME), a novel agent architecture consisting of four distinct experts, each aligned with a specific reasoning stage, CoME activates the corresponding expert to generate output tokens in each reasoning stage via output-oriented activation. To empower CoME with hybrid-capabilities reasoning, we introduce a progressive training strategy: Expert-FT enables decoupling and enhancement of different experts' capability; Router-FT aligns expert activation with the different reasoning stage; CoT-FT facilitates seamless collaboration and balanced optimization across multiple capabilities. To mitigate error propagation in hybrid-capabilities reasoning, we propose InfoGain-Driven DPO (Info-DPO), which uses information gain to evaluate the contribution of each intermediate step, thereby guiding CoME toward more informative reasoning. Comprehensive experiments show that CoME outperforms dense mobile agents and MoE methods on both AITZ and AMEX datasets.
Figures & tables
Figure 1: Left is an example of hybrid-capabilities reasoning. Right is the difference between input-oriented activation in MoE and output-oriented activation in CoME.
Figure 2: CoME architecture.
Figure 3: An example of InfoGain reward.
Method
#Params
SCROLL
CLICK
TYPE
PRESS
STOP
Overall
type
match
type
match
type
match
Auto-GUI ∗ ( Zhang and Zhang, 2024 )
700M
61.40
74.56
32.20
87.80
81.40
57.70
74.40
82.98
47.69
Qwen2VL † ( Wang et al., 2024b )
2B
23.05
78.38
43.74
60.40
59.40
52.22
47.02
62.25
43.80
ShowUI ( Lin et al., 2024 )
2B
23.79
53.29
40.34
89.60
85.20
59.00
87.55
64.22
49.55
UITars ( Qin et al., 2025 )
2B
30.17
83.81
55.25
83.97
84.77
53.78
61.39
74.23
55.63
GUI-R1 ( Luo et al., 2025 )
3B
25.53
79.00
50.47
21.80
20.80
32.89
28.77
57.32
40.41
Table 1: Main results on AITZ dataset of different action. Results with ∗ are reported from ( Zhang et al., 2024 ) . Methods with † are finetuned with hybrid-capabilities reasoning. The best overall result is marked bold and the second-best one is marked underline .
Method
#Params
Gmail
Booking
Music
SHEIN
News
CM
ToDo
Signal
Yelp
Overall
Qwen2VL † ( Wang et al., 2024b )
2B
37.4
35.2
40.1
33.7
50.0
47.5
41.8
56.7
40.5
38.53
ShowUI ( Lin et al., 2024 )
2B
52.2
33.6
68.8
55.3
51.8
57.4
51.9
69.7
50.2
47.74
UITars ( Qin et al., 2025 )
2B
59.4
47.9
55.4
53.0
65.1
62.3
64.7
61.7
58.1
54.43
GUI-R1 ( Luo et al., 2025 )
3B
37.1
50.6
58.2
47.1
21.6
54.7
54.6
56.1
54.4
48.53
SeeClick ∗ ( Cheng et al., 2024 )
7B
28.2
29.4
18.1
20.0
30.0
53.1
30.7
37.1
27.4
30.44
Qwen2VL † ( Wang et al., 2024b )
7B
57.6
58.4
56.5
47.3
64.2
66.3
60.9
72.8
54.8
57.99
Table 2: Main results on AMEX dataset of different apps . Results with ∗ are reported from ( Chai et al., 2024 ) . Methods with † are finetuned with hybrid-capabilities reasoning. The best overall result is marked bold and the second-best one is marked underline .
Method
AITZ
AMEX
AVG
CoME
66.98
72.61
69.78
- w/o Info-DPO
62.93
67.28
65.10
- w/o Router-FT
60.05
62.00
61.02
- w/o Expert-FT
57.07
64.47
60.74
- Info-DPO w/o LR-Norm
65.96
70.90
68.42
- CoT-FT w/o LR-Norm
62.57
64.24
63.40
Table 3: Ablation analysis on AITZ and AMEX.
Figure 4: Performance on different reasoning stages.
Figure 5: Expert distribution of CoME (left) and MoE (right).
Model
Caption
OCR
Grounding
Action
Qwen2VL (7B)
54.53
78.87
78.89
57.99
Ultars (7B)
54.08
78.87
79.11
70.33
MoE (A3B)
52.34
78.75
77.32
64.56
CoME (5B)
53.89
79.67
79.19
72.61
Table 4: Performance on AMEX auxiliary GUI tasks.
Figure 6: Information gain distribution. Above is the sampled data. Below is the DPO data. We illustrate the information gain at screen summary, subtask plan and action decision stage, as well as the total information gain of the entire reasoning trajectory.
Method
Scroll
Click
Type
Press
Stop
Overall
AITZ
CoT-FT
51.74
61.77
70.00
63.44
68.84
62.25
Rule-DPO
51.98
62.26
86.20
61.80
80.36
65.37
Info-DPO
52.07
65.22
83.80
59.53
83.33
66.98
AMEX
CoT-FT
70.83
66.39
79.16
70.66
51.14
67.28
Table 5: Comparison of DPO Strategies.
Strategy
InfoGain
InfoPos
Action Acc
cc+cw+lw
-0.50 / 0.26
0.03 / 0.61
77.60 / 65.96
cc+cw
-0.48 / 0.27
0.03 / 0.70
78.60 / 66.98
cw+lw
-0.71 / 0.26
0.04 / 0.46
76.88 / 64.07
cw
-0.71 / 0.27
0.04 / 0.56
77.75 / 65.93
Table 6: Analysis of data pairing strategies.
Method
InfoGain
InfoPos
Action
Reward-7B
0.9841
0.6610
78.60 / 66.98
Reward-72B
0.8365
0.4533
78.52 / 66.85
Continuous
0.9841
0.6610
78.60 / 66.98
Discrete
0.7541
0.5018
77.94 / 65.51
Table 7: Analysis of reward strategies.
Method
GPU Mem (GB)
Accuracy (%)
train
infer
type
match
Qwen2VL (7B)
31.62
16.52
71.59
54.46
UITars (7B)
32.08
16.92
78.07
65.41
MoE (A3B)
30.70
22.32
72.94
60.69
CoME (5B)
18.52
11.69
78.60
66.98
Table 8: Efficiency analysis of CoME architecture.
Appendix figures & tables10 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 7: A case to show the limitation of MoE on hybrid-capabilities reasoning.
Figure 8: Overall training framework of CoME.
Method
SCROLL
CLICK
TYPE
PRESS
STOP
OVERALL
IMPROVE
Average
43.57
51.92
74.15
57.74
69.07
55.41
-
AutoUI
61.40 +17.83
32.20 -19.72
81.40 +7.25
57.70 -0.04
74.40 +5.33
47.69
-7.73 12.37
Qwen2VL(2B)
23.05 -20.52
43.74 -8.18
59.40 -14.75
52.22 -5.52
47.02 -22.05
43.80
-11.62 6.53
ShowUI
23.79 -19.78
40.34 -11.58
85.20 +11.05
59.00 +1.26
87.55 +18.48
49.55
-5.87 14.07
UITars(2B)
30.17 -13.40
55.25 +3.33
84.77 +10.62
53.78 -3.96
61.39 -7.68
55.63
+0.21 8.40
SeeClick
11.14 -32.43
52.96 +1.04
53.00 -21.15
67.88 +10.14
55.36 -13.71
49.11
-6.31 15.23
Appendix
Table 9: Detailed results on AITZ with relative improvement and improvement variance.
Method
#Expert
SCROLL
CLICK
TYPE
PRESS
STOP
Overall
type
match
type
match
type
match
CoME
3-expert
41.22
81.44
61.52
83.40
79.80
66.57
71.31
74.42
62.28
4-expert
52.31
81.90
62.42
75.25
71.80
63.37
69.24
75.10
62.93
8-expert
54.39
81.12
61.96
83.60
79.20
56.92
73.02
75.66
63.59
Appendix
Table 10: Comparison of CoME variants with different numbers of experts.
Strategy
IG@SS
IG@AP
IG@AD
InfoGain
InfoPositive
Action Accuracy
reject
choose
reject
choose
reject
choose
reject
choose
reject
choose
type
match
AITZ
cc+cw+lw
0.0748
0.0234
-0.1430
0.1595
-0.4354
0.0814
-0.5037
0.2645
0.0291
0.6115
77.60
65.96
cc+cw
0.0762
0.0908
-0.1294
0.1125
-0.4252
0.0689
-0.4784
0.2723
0.0300
0.7045
78.60
66.98
cw+lw
0.0686
-0.0103
-0.2049
0.1790
-0.5758
0.0905
-0.7121
0.2592
0.0380
0.4631
76.88
64.07
cw
0.0695
0.0808
-0.1973
0.1143
-0.5869
0.0741
-0.7147
0.2693
0.0409
0.5654
77.75
65.93
Appendix
Table 11: Analysis of data selection strategy
Method
ACTION ACCURACY
REWARD MARGIN
SCROLL
CLICK
TYPE
PRESS
STOP
OVERALL
IG@SS
IG@AP
IG@AD
InfoGain
Reward Model Scale
Reward-7B
52.07
65.22
83.80
59.53
83.33
66.98
0.0113
0.3117
0.6610
0.9841
Reward-72B
53.06
67.37
82.20
54.57
74.60
66.85
0.0087
0.1681
0.6597
0.8365
Reward Function Design
Continuous
52.07
65.22
83.80
59.53
83.33
66.98
0.0113
0.3117
0.6610
0.9841
Appendix
Table 12: Analysis of reward strategy
Reward Model
Reward Acc
Action Acc
training-free
82.24
66.64
training-based
84.68
66.98
Appendix
Table 13: Comparison of training reward model
Model
Extra
Total
Acc
Qwen2VL (7B)
15.48 GB
1.04 GB
16.52 GB
54.46
UITars (7B)
15.49 GB
1.43 GB
16.92 GB
65.41
MoE (A3B)
21.34 GB
0.98 GB
22.32 GB
60.69
CoME (5B)
10.55 GB
1.14 GB
11.69 GB
66.98
Appendix
Table 14: Detailed analysis of GPU memory usage
Figure 9: InfoGain of DPO data pair where the correct one is chosen and the wrong one is rejected.
Figure 10: InfoGain of DPO data pair where both the chosen and rejected outputs are correct.