GUI agents can learn cheaply from a text view of an interface and act on screenshots through a vision-language screen reader. Yet on a simulated mobile platform with a low-cost reader, policies that complete most tasks in text fail on many of the same tasks on screenshots. We ask where a small budget of real screenshots is best spent to close this gap. We find that the reader's misreads are systematic and recur on the same interface elements. Noise matched to the reader's error rates, a natural sim-to-real remedy, misses this structure, and in a pre-specified comparison it did not beat training the policy directly on screenshots. We therefore correct the misreads at their source with reader alignment, which fine-tunes the reader on screenshots paired with their text views. The reader learns to output each screen's text view, as in text practice, and the policy stays unchanged. Spent on the reader, the screenshot budget gives 80.9% full-task success on new task instances, against 55.6% when spent on training the policy. Given the same labeled screenshots, adapting the reader also beats adapting the policy, and a single aligned reader improves every text-trained policy we tried without retraining any of them.
Figures & tables
Figure 2: The reader misreads largely the same elements across episodes; rate-matched noise mostly corrupts other elements. A schematic of one calibration screen ( Sec. 3 ), drawn from its text view (bars: text; squares: icons; screenshot in supplementary Fig. S2 ) and chosen by a rule fixed in advance (supplementary Sec. F ). Color: how often the original reader misreads each element in calibration episodes; never-misread elements are gray unless misread here. Colored fill: misread on this read or in this noise draw. The noise draw puts 19 of its 24 misreads on never-misread elements; the noise matches the reader’s error rates over the calibration screens, not per screen.
Training group
Budget
Text
Pixels
Clean-text practice
0
87.5 ± 1.4
43.0 ± 3.0 −44.5
Generic-noise practice
0
83.3 ± 0.0
49.6 ± 4.7 −33.8
Rate-matched-noise practice
2,000
82.6 ± 0.2
52.0 ± 0.4 −30.6
Direct screenshot training
2,000
86.6 ± 2.6
55.6 ± 8.3 −31.0
Combined training
2,000
85.3 ± 2.0
55.9 ± 0.6 −29.5
Table 1: Every first-round group succeeds far less often on screenshots than in text. Full-task success (%) on the confirmation set from the true text view and on screenshots with the original reader (mean ± std over two training seeds). Subscripts: pixels minus text, in points; in all tables, a subscript is green or red when the bootstrap confirms a rise or a drop and gray when it does not. Budget: screenshots used, rounded to the nearest hundred.
Figure 3: Misreads and the reader’s order both lower success, and their losses overlap. Full-task success of the start policy on 93 unseen instances, 4 samples each, with correct elements (gray) or with misreads injected by the element-replay simulator with resampled reads (red; Sec. 4.4 ), in the layout order, the reader’s order or a random shuffle (open marker). Dashed: real pipeline. Labels: cost of the reader’s order in points; with misreads, the difference only points toward a cost (supplementary Tab. S8 ).
Misread again (%)
Reader
T
Observed
Independent
≥ 20
Qwen3-VL-4B
0
73.2 +42.3
30.9
✓
Qwen3-VL-8B
0
77.7 +52.8
24.9
✓
Qwen3-VL-4B
0.7
61.1 +31.3
29.8
✓
InternVL3-8B
0
89.7 +12.2
77.6
✗
Table 2: Misreads recur on the same elements more often than independent errors would, far more so for the Qwen3-VL readers. Share (%) of misread elements that are misread again in another episode, for the reader and for independent errors at the same rate. Subscripts: observed minus independent, in points. Last column: the difference reaches the pre-specified threshold of 20 points. T : sampling temperature; the Qwen3-VL-8B row uses full-size screenshots (supplementary Sec. F ).
Figure 4: Environments that inject the reader’s errors roughly match the real pipeline’s success but predict per-template success less well than a repeated real run. (a) Predictivity M with bootstrap error bars; dashed line and gray band: the real pipeline predicting itself, which every environment falls measurably short of (supplementary Tab. S7 ). (b) Success of the start policy inside each environment, on the instances of Fig. 3 ; dashed: real pipeline. Filled: injects the reader’s errors. Replay: element-replay simulator.
Policy
Reader
Budget
Success (%) ↑
Clean-text
Aligned (ours)
2,000
80.9 ± 2.2
Direct-training
Original
2,000
55.6 ± 8.3 −25.3
Adapted, same labels
Original
2,000
67.2 ± 1.2 −13.6
Clean-text
Lookup
2,000
69.4 ± 1.8 −11.5
Clean-text
8B, half size
0
46.4 ± 7.9 −34.5
Clean-text
8B, full size
0
48.9 ± 0.8 −32.0
Table 3: Aligning the reader gives clearly higher success than every alternative in this table. Full-task success (%), mean ± std over two training seeds, on the confirmation set unless noted. Each row pairs a policy with a reader; blue: the clean-text policy with the aligned reader. Subscripts: difference from the blue row of the same block, in points. 8B: unaligned Qwen3-VL-8B. Budget: screenshots used, rounded to the nearest hundred. † Subscript not pre-specified.
Reader: success (%) ↑
Policy
Original
Reordered
Aligned
Learned only in text
Clean-text
43.0 ± 3.0
48.3 ± 0.8 +5.3
80.9 ± 2.2 +37.9
Start
40.5
–
75.9 ± 0.0 +35.3
Rate-matched noise
52.0 ± 0.4
–
76.0 ± 3.9 +24.0
Other policies
Table 4: One aligned reader raises all three text-trained policies we evaluated, without retraining. Full-task success (%) on the confirmation set, mean ± std over two training seeds (one run for the start and base policies). Reordered: the original reader’s output sorted into the layout order. Subscripts: difference from the original reader, in points. † Subscript not pre-specified.
Reader
Evaluation set
Screens
Original
Aligned
Element error rate (%) ↓
Held-out screenshots
1,007
25.62
1.56
Held-out, new screens only
341
27.03
2.92
8 templates outside calibration
204
22.99
1.53
8 templates, new screens only
121
22.77
2.44
Table 5: Alignment lowers the element error rate on every set and cuts every type of error. Top: element error rate (%) of the original and the aligned reader. Screens: number of screenshots (confirmation set: screens encountered with the original / aligned reader). New screens: text view in no training label. Held-out apps: their held-out screenshots, each read by a reader aligned without that app ( Sec. 6.4 ). Bottom: errors by type on the held-out screenshots (counts) and extra elements, which match nothing on the screen (% of the reader’s output).
Figure 5: Alignment already helps with 500 screenshots and keeps improving with more. (a) Element error rate on the 1,007 held-out screenshots against the calibration screenshots used for alignment, on all held-out screenshots and on the pixel-new ones (pixel-identical to no alignment screenshot); gray dashed: original reader. (b) Full-task success on the confirmation set against the screenshots spent (mean ± std over two training seeds), for alignment with 500 and all calibration screenshots and for direct screenshot training after its second, third and fourth practice rounds; dotted: no screenshots (clean-text policy, original reader).
Comparison ( X−Y )
Δ
Interval
Text − pixels, original reader ( Tab. 1 )
Clean-text practice ∗
+44.5
[31.2,58.0]
Generic-noise practice ∗
+33.8
[21.6,45.8]
Rate-matched-noise practice ∗
+30.6
[19.5,42.2]
Direct screenshot training ∗
+31.0
[18.4,44.0]
Combined training ∗
+29.5
[16.8,42.5]
Table S1 : Comparisons in Secs. 4 and 6 . Differences in points, colored as in Tab. S2 ; intervals (templates resampled, app groups for recurrence) only for the pre-specified comparisons, marked ∗ . All other rows are descriptive.
Bootstrap
∣Δ∣
Wording
Color
confirms a rise
≥5
X is clearly higher
green
confirms a rise
<5
X is slightly higher
green
does not confirm
<5
no measurable difference
gray
does not confirm
≥5
points toward X ( Y if Δ<0 )
gray
confirms a drop
–
as above with X and Y swapped
red
Table S2 : Fixed wording for a difference Δ=X−Y (points), and its color in the tables.
Screenshots (%) ↑
Text (%) ↑
Training group
Seed 1
Seed 2
Mean
Mean
Clean-text practice
45.1
40.8
43.0
87.5
Generic-noise practice
46.3
52.9
49.6
83.3
Rate-matched-noise practice
51.7
52.3
52.0
82.6
Direct screenshot training
49.7
61.5
55.6
86.6
Combined training
55.5
56.3
55.9
85.3
Table S3 : The two training seeds differ most for direct screenshot training. First-round groups on the confirmation set: full-task success (%) on screenshots with the original reader, per training seed and on average, and in text (mean of the two seeds).
Comparison ( X−Y )
First-round set
Confirmation set
Direct − clean text
+7.2
+12.6
Combined − clean text
+8.9
+12.9
Rate-matched − generic
+2.6
+2.4
Rate-matched − direct
−2.9
−3.6
Text − pixels, range
29.9–42.0
29.5–44.5
Table S4 : All five pre-specified first-round comparisons have the same sign on both instance sets. Differences X−Y in points, colored as in Tab. S2 . Last row: range of the five text-to-pixels gaps.
Clean-text policy
Direct training
App
Template
Text
Original
Aligned
Original
Alipay
CheckBalance
100.0
95.8
100.0
100.0
ConfigureLanguageAndFastPay
100.0
70.8
100.0
87.5
SendMessageToContact
0.0
8.3
0.0
8.3
Calendar
CalculateDateInterval
8.3
12.5
8.3
16.7
CompareScheduleDensity
54.2
25.0
62.5
58.3
Table S5 : Per template, the aligned reader is higher than direct screenshot training on 18 templates, equal on 6 and lower on 5. Full-task success (%) on the confirmation set, over both training seeds: the clean-text policy in text, with the original reader and with the aligned reader (blue), and direct screenshot training with the original reader. Every template has the same number of instances, so the last row, the mean over templates, equals the overall success.
First discrepancy
Failed episodes
Ordering difference
26
Misread input field
21
Total
47
Table S6 : In every failed episode of the four templates with most of the remaining gap, the first discrepancy is an ordering difference or a misread input field. First discrepancy between the aligned reader’s output and the true text view (clean-text policy, aligned reader).
Table S7 : Every practice environment stays below the ceiling set by the real pipeline. Full results of the predictivity test ( Fig. 4 ): predictivity M , its gap to the real pipeline’s own M with the gap’s interval, and the start policy’s aggregate full-task success (%) on the instances of Sec. 4.2 . The bootstrap confirms every gap. Errors: the environment injects the reader’s errors, it does not.
Comparison ( X−Y )
Δ
Reader’s vs. layout order, no misreads
−21.8
Reader’s vs. layout order, with misreads
−5.1
Misreads vs. none, layout order
−47.0
Misreads vs. none, reader’s order
−30.4
Reader’s vs. random order, no misreads
+43.0
Table S8 : Misreads and the reader’s order each lower success, and their costs overlap. Effects in the gap decomposition ( Sec. 4.2 ): Δ=X−Y in points of full-task success of the start policy on 93 unseen instances with 4 samples each, colored as in Tab. S2 .
Figure S1 : The original reader’s output differs from the training label in content and in order. Left to right, for the part of the screen of Fig. 2 just below the status bar: the screenshot, the original reader’s output and the training label (the text view of this part in the reader’s output format and the layout order). The original reader writes every color as black and leaves out the blue background, misses the drop-down arrow and the search icon, writes the search field as text, uses other icon names ( scan , payment and card for scan-line , qr-code and wallet-cards ) and lists the icons and their labels column by column; the training label lists the row of icons before the row of labels, and the plus sign comes first because its top edge falls in a higher row band ( Sec. G ). The interface text is Chinese and is shown as it appears on the screen.
Reader
Evaluation set
Original
Aligned
Extra elements (%) ↓
Held-out screenshots
17.24
0.87
Held-out, new screens only
10.52
1.74
Confirmation set, screens encountered
18.19
0.55
Extra elements found in a training label (%)
9.5
29.6
Table S9 : The aligned reader outputs far fewer elements that match nothing on the screen. Extra elements: reader output elements that match nothing in the true text view, as a share (%) of the reader’s output. New screens as in Tab. 5 . Last row: share (%) of the extra elements whose text appears in some training label.
Error rate
Errors (count) ↓
Extra
Reader
(%) ↓
Missed
Text
State
(%) ↓
Original 4B
25.62
8,514
1,297
74
17.24
Unaligned 8B, half-size
15.64
4,720
1,237
76
4.33
Unaligned 8B, full-size
17.45
5,413
1,228
90
14.92
Aligned 4B, seed 1
1.56
372
225
4
0.87
Aligned 4B, seed 2
1.52
257
300
30
0.47
Table S10 : Alignment cuts every type of error; with label elements deleted at random, more errors remain, mostly missed elements. Errors by type on the 1,007 held-out screenshots (38,579 true elements), half-size except in the full-size row. Aligned 4B: aligned on all calibration screenshots with alignment seed 1 (blue, the main aligned reader), 2 or 3; deleted: each label element deleted independently with this probability. Error rate: missed elements, wrong text and wrong state over all true elements; extra: as in Tab. S9 .
Share of pixel-new screens
Episodes
Success (%) ↑
Below one third
41
87.8
One third to two thirds
366
79.2
Above two thirds
289
54.3
Table S11 : The lookup baseline succeeds less often when more of an episode’s screens are pixel-new. Episodes on the confirmation set binned by their share of pixel-new screens (identical to no calibration screenshot), and the lookup baseline’s full-task success in each bin (descriptive; the template mix differs across bins).
Template
Text success (%) ↑
QueryHolidayLength
54.2
AlipayBalanceToWechat
50.0
ReadNoteContent
45.8
CompareParticipationDurations
20.8
MonthlyIncomeByCounterparty
0.0
SetAlarmRepeat
0.0
Table S12 : Even in text, the clean-text policy completes few tasks of the templates it never practiced. Text success (%) on the 8 unpracticed templates (two training seeds). The real pipeline would have been run only if the average over all 8 reached 50%.
Figure S2 : The text view lists each visible element on one line, in the layout order. The screen of Fig. 2 as a screenshot (left) and its text view (right), as the policy reads it in clean-text practice. Each line gives the index, role, text or icon name, the attributes that apply (color, background color and flags such as bold or empty ) and the box on the 0–1,000 scale. Interface text is Chinese, as on the screen.
Purpose
Calls
Collection
3,011
Training
14,826
Development
23,026
Evaluation
187,887
Audit
18,692
Total
247,442
Table S13 : Most screen-reader calls are spent on evaluation. Calls by purpose. Collection: the calibration and held-out screenshots. Training: the second practice rounds of direct screenshot training and combined training. Development: building the pipeline and choosing the reader, before the study. Audit: diagnostic re-reads, reads of the held-out screenshots by the aligned readers and the unaligned 8B reader, and cost measurements.
MobileGym
AndroidControl
Alignment set: screenshots (apps)
2,004 (9)
2,001 (140)
Held-out screenshots
1,007
1,000
Recurrence of misreads (%)
Original reader
73.2
58.3
Independent errors
30.9
34.8
Difference (points)
+42.3
+23.5
Table S14 : On AndroidControl, misreads also recur, but alignment lowers the element error rate far less than on MobileGym. MobileGym values are from Tabs. 2 and 5 . Recurrence: share (%) of misread elements that are misread again in another episode, original reader at temperature 0 on the alignment set; interval: app groups resampled. Error rates (%) on held-out screenshots; new screens: text view in no alignment screenshot. Both lines were fixed before the runs.
Figure S3 : Misreads recur on both platforms, but on AndroidControl alignment lowers the element error rate far less, and not on apps without alignment screenshots. (a) Share of misread elements that are misread again in another episode, for the original reader and for independent errors at the same rate. (b) Element error rate on held-out screenshots before and after alignment; dashed: the pre-specified line. (c) AndroidControl held-out apps without alignment screenshots and the apps with the most alignment screenshots; this breakdown was made after the results. MobileGym values are from Tabs. 2 and 5 .
State Key Laboratory of Networking and Switching Technology, Beijing University of Posts and Telecommunications, Beijing, China · Tencent Anti-Cheat Expert, China