cs.LGOct 5, 2026
SaveDecoupling What from Where: How Should a Small GUI Grounding Model Receive the Action Type?
Organizations: Stanford University
Abstract
A GUI agent decides which action to take and where to take it; we ask how a small grounding model should receive the action type. Fine-tuning Qwen2-VL-2B with LoRA on Android in the Wild, we compare a flat baseline with five ways of supplying the type under matched data, compute, and decoding: an auxiliary loss, a hard-routed action word, an additive learned embedding, a prepended learned token, and the type written into the prompt. With five seeds, an episode-clustered bootstrap, and seed-level paired tests, the ranking on a mixed stream is clear: the auxiliary loss, the additive embedding, and the prompt word each gain five to seven hit@0.10 points over the baseline, while hard routing and the prepended token are not distinguishable from it. Much of that gain is protection from a preprocessing choice of ours rather than a spatial prior. Our serializer clamps the off-screen touch point AITW records for type events to the origin; that class degrades the baseline's click grounding, and removing it lifts the baseline by nearly seven points, after which no mechanism's hit rate beats it and the intervals exclude a two-point effect, though the auxiliary loss still shortens the average miss; on a stream of taps and swipes none helps. Whether this generalizes beyond one serialization is open. For deployment, the pipeline's margin over the baseline with predicted rather than gold types is not established (+0.016, 95% interval [-0.017, +0.052]), and a wrong type collapses every model conditioned at inference. The prepended token does not help at the shared learning rate, where its rows barely move from initialization; trained ten times faster it reaches the level of the other three, with a margin three seeds do not establish. We also document a silent failure: injecting conditioning through inputs_embeds makes Qwen2-VL fall back to 1-D positions for image tokens, costing nine points.
Figures & tables
Figure 1: Two-stage pipeline. Stage 1 classifies the action type from mean-pooled frozen features. Stage 2 grounds the action to a coordinate string, and the action type enters it through one of the six mechanisms in the bottom row, all of which share data, optimizer, adapters, and decoding. D-slot, the retracted variant whose inputs_embeds injection bypassed the position computation, is described in Section 6.2 .
| Variant | hit@0.05 | hit@0.10 | hit@0.25 | mean L2 | hit@0.10 vs. A [episode CI] | seed-level |
|---|---|---|---|---|---|---|
| A: flat | 0.130 0.039 | 0.229 0.043 | 0.499 0.029 | 0.406 0.028 | ||
| B: aux. loss | 0.178 0.025 | 0.293 0.021 | 0.582 0.026 | 0.365 0.004 | +0.064 [+0.037, +0.092] ∗∗∗ | 0.004 |
| C: hard routing | 0.170 0.022 | 0.264 0.045 | 0.538 0.059 | 0.421 0.016 | +0.035 [-0.002, +0.072] | 0.189 |
| D-hook: additive | 0.170 0.042 | 0.282 0.041 | 0.563 0.016 | 0.370 0.010 | +0.054 [+0.022, +0.090] ∗∗∗ | 0.028 |
| D-token: prepended | 0.144 0.056 | 0.238 0.049 | 0.504 0.055 | 0.386 0.021 | +0.009 [-0.019, +0.036] | 0.465 |
| D-text: word in prompt | 0.186 0.022 | 0.302 0.028 | 0.568 0.033 | 0.364 0.008 | +0.073 [+0.030, +0.117] ∗∗ | 0.018 |
Table 1: Stage 2 grounding on AITW all_with_coords (n =1200, n =250, 5 seeds). Mean std over seeds; vs. A with a 95% episode-cluster bootstrap CI (stars from the same bootstrap, , , ) and the seed-level paired -test .
| Variant | click | scroll | unparsed click / scroll | click, parsed only | clicks at (0, 0) | click, excluding those |
|---|---|---|---|---|---|---|
| A: flat | 0% / 0% | (13% on the worst seed) | ||||
| B: aux. loss | 0% / 0% | |||||
| C: hard routing | 1.4% / 13.9% | (scroll ) | ||||
| D-hook: additive | 0% / 0% | |||||
| D-token: prepended | 0% / 0% | |||||
| D-text: prompt word | 0% / 0% |
Table 2: Where the gain lives. Per-class hit@0.10 on the headline mix (five seeds, same basis as Table 1 ); hard routing’s unparsed share and its rates on parsed outputs; the fraction of click predictions that are the clamped type target (0, 0) , and click hit@0.10 excluding them. Every model scores zero on the 35 type events because the raw target is off-screen.
| Variant | hit@0.05 | hit@0.10 | hit@0.25 | mean L2 | hit@0.10 vs. A [episode CI] | seed-level |
|---|---|---|---|---|---|---|
| A: flat | 0.243 0.020 | 0.382 0.015 | 0.720 0.017 | 0.185 0.003 | ||
| B: aux. loss | 0.225 0.015 | 0.368 0.012 | 0.730 0.039 | 0.173 0.011 | -0.013 [-0.042, +0.013] | 0.456 |
| C: hard routing | 0.217 0.016 | 0.325 0.040 | 0.635 0.064 | 0.271 0.068 | -0.057 [-0.101, -0.011] ∗ | 0.215 |
| D-hook: additive | 0.235 0.022 | 0.363 0.021 | 0.725 0.043 | 0.182 0.009 | -0.018 [-0.056, +0.019] | 0.334 |
| D-text: word in prompt | 0.210 0.022 | 0.335 0.031 | 0.665 0.013 | 0.199 0.003 | -0.047 [-0.094, +0.002] | 0.088 |
Table 3: Control: AITW taps_and_swipes (n =1000, n =200, 3 seeds), a training stream without the degenerate class. Same statistics as Table 1 .
| Model | Conditioning | hit@0.10 | click hit@0.10 | scroll hit@0.10 | gold cond. (hit@0.10) | seed |
|---|---|---|---|---|---|---|
| D-hook: additive | gold type | 0.275 0.040 | 0.310 0.060 | 0.357 0.052 | ||
| wrong (cyclic) | 0.054 0.009 | 0.006 0.000 | 0.270 0.050 | +0.222 [+0.182, +0.263] ∗∗∗ | 0.000 | |
| wrong (click scroll) | 0.099 0.056 | 0.073 0.087 | 0.270 0.050 | +0.176 [+0.134, +0.221] ∗∗∗ | 0.003 | |
| zeroed | 0.248 0.032 | 0.288 0.053 | 0.291 0.050 | +0.027 [-0.006, +0.065] | 0.224 | |
| class mean | 0.206 0.058 | 0.217 0.099 | 0.322 0.064 | +0.070 [+0.034, +0.109] ∗∗∗ | 0.133 | |
| flat A, same examples | 0.229 0.043 | 0.256 | 0.304 | +0.046 [+0.016, +0.081] ∗∗ | 0.032 |
Table 4: Interventions: the same trained model decoded with the gold action embedding, a wrong one (cyclic; click type), a wrong one swapping only clicks and scrolls, the table zeroed, and every row replaced by the class mean. Five seeds 250 examples; = gold condition, 95% episode-cluster CI, seed-level paired ; flat A on the same examples is the reference row.
Appendix figures & tables11 assets
Supplementary material from the paper’s appendix.
Appendix
| Features | Mind2Web (2 classes) | AITW (5 classes) |
|---|---|---|
| text only | ||
| text + screenshot | ||
| zeroed (sanity) | ||
| TF-IDF + logistic regression | ||
| majority class |
Table 5: Stage 1 action-type macro-F1 (three seeds). On Mind2Web the instruction names the action, so the screenshot adds little; on AITW the instruction describes a goal and the screenshot is what separates the classes.
| Variant | hit@0.10 | hit@0.25 | hit@bbox | mean L2 |
|---|---|---|---|---|
| A: flat | ||||
| D-hook: additive |
Table 6: Multimodal-Mind2Web grounding control (1,000 training and 250 evaluation steps, three seeds, seed-level only). The target is the center of the gold element; hit@bbox is the fraction of predicted points inside the element. Both models are at floor.
| Variant | Training stream | seeds | hit@0.10 | click hit@0.10 | scroll hit@0.10 | hit@0.10 vs. A (same stream) |
|---|---|---|---|---|---|---|
| A: flat | with type | 5 | 0.229 0.043 | 0.256 0.062 | 0.304 0.034 | |
| A: flat | type removed | 5 | 0.296 0.019 | 0.355 0.031 | 0.304 0.051 | |
| B: aux. loss | with type | 5 | 0.293 0.021 | 0.357 0.026 | 0.278 0.018 | |
| B: aux. loss | type removed | 5 | 0.298 0.038 | 0.368 0.069 | 0.265 0.058 | +0.002 [-0.014, +0.018] (0.926) |
| C: hard routing | with type | 5 | 0.264 0.045 | 0.350 0.061 | 0.148 0.073 | |
| C: hard routing | type removed | 3 | 0.269 0.033 | 0.355 0.039 | 0.159 0.126 | -0.028 [-0.071, +0.013] (0.118) |
Table 7: Removing type events from the training slice only, scored on the identical headline validation slice (3 to 5 seeds; each contrast uses the seeds its two conditions share, given per row). Deltas are paired episode-cluster bootstraps (every step and seed of a sampled episode kept together); seed-level in parentheses.
Figure 2: Conditioning advantage over the flat baseline as a function of training-set size on all_with_coords , three seeds per cell, 95% episode-clustered bootstrap intervals. Each size is scored on a different validation slice with a different class mix, so only within-size differences are meaningful, and 24 contrasts are shown without correction.
| val mix (c/s/t) | A hit@0.10 | B A | D-hook A | B A (hit@0.25) | D-hook A (hit@0.25) | |
|---|---|---|---|---|---|---|
| 300 | 170/33/47 | 0.139 0.032 | +0.067 [+0.029, +0.102] ∗∗ | +0.069 [+0.040, +0.098] ∗∗∗ | -0.055 [-0.101, -0.011] ∗ | +0.067 [+0.031, +0.102] ∗∗∗ |
| 500 | 198/12/40 | 0.315 0.059 | +0.011 [-0.020, +0.040] | +0.001 [-0.032, +0.033] | +0.004 [-0.028, +0.034] | +0.059 [+0.030, +0.085] ∗∗∗ |
| 800 | 141/71/38 | 0.261 0.020 | -0.040 [-0.077, -0.006] ∗ | +0.048 [+0.010, +0.085] ∗ | -0.039 [-0.081, +0.001] | +0.064 [+0.009, +0.111] ∗ |
| 1200 | 169/46/35 | 0.255 0.026 | +0.051 [+0.020, +0.081] ∗∗∗ | +0.045 [+0.011, +0.085] ∗∗ | +0.071 [+0.037, +0.103] ∗∗∗ | +0.055 [+0.019, +0.090] ∗∗ |
| 2500 | 166/53/31 | 0.244 0.037 | -0.012 [-0.037, +0.010] | +0.037 [+0.015, +0.061] ∗∗ | -0.008 [-0.046, +0.026] | +0.104 [+0.060, +0.144] ∗∗∗ |
| 5000 | 187/29/34 | 0.363 0.009 | +0.005 [-0.026, +0.036] | +0.043 [+0.006, +0.079] ∗ | -0.023 [-0.055, +0.008] | +0.032 [-0.005, +0.069] |
Table 8: Conditioning advantage vs. training-set size (AITW all_with_coords , 3 seeds per cell). Intervals are 95% episode-cluster bootstraps (every step and seed of a sampled episode kept together); stars from the same bootstrap. Absolute values are not comparable across rows because the validation slice moves with ; its click/scroll/type counts are given per row. The study is descriptive: 24 contrasts, no multiplicity correction.
| Table LR | seeds | hit@0.10 | hit@0.25 | click hit@0.10 | hit@0.10 vs. A | seed | row displacement |
|---|---|---|---|---|---|---|---|
| shared ( ) | 5 | 0.238 0.049 | 0.504 0.055 | 0.269 0.069 | +0.009 [-0.019, +0.036] | 0.465 | 0.016 |
| (10 ) | 3 | 0.295 0.041 | 0.561 0.045 | 0.329 0.048 | +0.040 [+0.005, +0.076] ∗ | 0.157 | 0.276 |
| (100 ) | 3 | 0.283 0.032 | 0.544 0.022 | 0.318 0.030 | +0.028 [-0.008, +0.065] | 0.020 | 1.874 |
Table 9: D-token with the action-embedding table trained at 1 , 10 , and 100 the base learning rate on the headline slice. Displacement is the mean distance the rows of the classes present (click, type, scroll) moved from their initialization, whose norm is 0.78; the 1 value comes from the type-removed runs, which share the code path and learning rate. vs. A uses the shared seeds with an episode-cluster interval.
| Conditioning | hit@0.05 | hit@0.10 | hit@0.25 | mean L2 |
|---|---|---|---|---|
| gold | 0.140 0.059 | 0.236 0.058 | 0.502 0.045 | 0.386 0.020 |
| wrong | 0.014 0.004 | 0.079 0.013 | 0.200 0.027 | 0.728 0.081 |
| zero | 0.074 0.055 | 0.167 0.066 | 0.416 0.103 | 0.489 0.059 |
| gold wrong | +0.157 [+0.120, +0.192] ∗∗∗ | +0.302 [+0.258, +0.342] ∗∗∗ | -0.342 [-0.366, -0.314] ∗∗∗ | |
| gold zero | +0.069 [+0.042, +0.096] ∗∗∗ | +0.086 [+0.062, +0.109] ∗∗∗ | -0.103 [-0.115, -0.091] ∗∗∗ |
Table 10: D-token causal-use test: the same trained model evaluated with the gold action embedding, a wrong (cyclically permuted) one, and a zeroed one (5 seeds; paired episode-cluster bootstrap intervals).
| Conditioning | hit@0.10 | click hit@0.10 | scroll hit@0.10 | gold cond. (hit@0.10) | seed |
|---|---|---|---|---|---|
| gold word | 0.312 0.045 | 0.357 0.056 | 0.384 0.045 | ||
| wrong word (cyclic) | 0.076 0.011 | 0.028 0.003 | 0.312 0.050 | +0.236 [+0.183, +0.289] ∗∗∗ | 0.017 |
| wrong word (click scroll) | 0.085 0.006 | 0.041 0.006 | 0.312 0.050 | +0.227 [+0.174, +0.281] ∗∗∗ | 0.014 |
| no action word | 0.205 0.067 | 0.217 0.102 | 0.319 0.055 | +0.107 [+0.063, +0.152] ∗∗∗ | 0.241 |
| flat A, same examples | 0.229 0.043 | 0.256 | 0.304 | +0.057 [+0.018, +0.099] ∗∗ | 0.083 |
Table 11: Interventions on D-text: the same trained model decoded with the gold action word, a wrong word (cyclic; click type), a wrong word swapping only clicks and scrolls, and no action word (3 seeds 250 examples; = gold condition, 95% episode-cluster CI, seed-level paired ; flat A on the same examples is the reference row).
Figure 3: Same trained model, four conditionings, 120 validation screens, one seed per variant. Left: share of image attention within of the target from the token that predicts the y coordinate, with the gold answer teacher-forced (3/4-depth layer, head-averaged); the dotted line is the share of image tokens within that radius. Right: greedy hit@0.10 under the same conditioning. Error bars are 95% paired-bootstrap intervals of the difference from gold over the 120 screens.
| Model | Position | gold | wrong (cyclic) | wrong (click scroll) | zero |
|---|---|---|---|---|---|
| A (flat) | last prompt token | 0.038 | |||
| A (flat) | pre-y token | 0.117 | |||
| D-hook | last prompt token | 0.040 | 0.036 | 0.035 | 0.036 |
| D-hook | pre-x token | 0.039 | 0.036 | 0.037 | 0.039 |
| D-hook | pre-y token | 0.111 | 0.064 | 0.049 | 0.119 |
| D-token | last prompt token | 0.038 | 0.034 | 0.034 | 0.035 |
Table 12: Share of image attention within of the target (chance ) at three probe positions, seed 42, 120 screens, 3/4-depth layer, gold answer teacher-forced.
Figure 4: Predictions of the flat baseline (red) and D-hook (blue) against the target (green) on held-out AITW screens at the headline setting, one seed. The panels are a fixed spread of outcomes rather than a selection of wins: two rescues, a case where conditioning hurts, a case where both are right, a scroll, and a degenerate type example on which both models miss at distance . The title reports the base rates on this seed at : of 169 clicks, 21 are rescued by conditioning, 13 are hurt, 45 are correct under both models, and 90 are missed by both. In the rescue panels the baseline’s prediction is off-screen and drawn clamped at the corner.