Automated assessment of mobile user interface quality is attractive because manual review does not scale, and large language models are increasingly proposed for the task. Whether their ratings correspond to those produced by established rule-based heuristics has not been characterised at scale. We construct a 15,000-screen corpus from the RICO dataset using six documented filters, score every screen with a literature-grounded heuristic scorer, and collect ratings from three open-weight language models across two input representations, yielding 270,000 ratings. We report linearly weighted Cohen's kappa with bootstrap confidence intervals, offset-corrected kappa, intraclass correlation, and Bland-Altman limits of agreement, together with a chance baseline for the tolerance-band agreement rule. Agreement between the two rating sources is consistently weak: raw kappa falls below 0.15 in seventeen of eighteen model-representation-dimension cells. However, fifteen of eighteen cells are statistically distinguishable from zero, and correcting for a constant offset raises five cells above 0.15, so the divergence is better described as a combination of systematic bias and weak covariation than as absence of relationship. Model ratings track structural properties of the interface, but substantially less strongly than the heuristic does, and more strongly from an untruncated structural element list than from a screenshot. We further show that dimensions elicited in a single generated response are correlated by construction, reaching rho = 0.97 in one condition, which constrains what such designs can measure. We report these results as a characterisation of divergence between two automated raters rather than as evidence that either is correct, and we set out the validation a stronger claim would require.
Figures & tables
Figure 1: Linearly weighted κ between the heuristic scorer and each model, by input representation and dimension. Filled circles give the observed estimate with a bootstrap 95% confidence interval; open diamonds give κ recomputed after removing the cell’s constant offset. Removing the offset raises five cells above the 0.15 reference line.
Figure 2: Agreement within ±1 scale point, expressed as the difference between the observed proportion and the proportion expected under independent marginals with the same distributions. Departures from the chance baseline are small in every condition and negative in six of eighteen.
Figure 3: Spearman correlation between structural quantities and each rater’s scores, separated by input representation. The heuristic row is shaded. Model correlations are weaker than the heuristic throughout but are non-zero, and for the first model family are stronger under structural input than under pixel input.
Figure 4: Spearman correlation between dimensions within each rater. Model conditions exceed the heuristic’s usability–layout correlation in all six cases, reaching ρ=0.97 , consistent with sequential conditioning within a single generated response.
Model
Rep.
κ
95% CI
κoff
ICC
dˉ [LoA]
σL/σH
Usability
gemma3
JSON
0.005
[0.003, 0.006]
0.022
− 0.007
− 1.40 [ − 3.02, 0.21]
0.40
gemma3
JPEG
0.009
[0.005, 0.013]
− 0.037
− 0.028
− 1.27 [ − 3.15, 0.61]
0.70
gemma4
JSON
0.038
[0.034, 0.041]
0.178
0.083
− 1.27 [ − 2.95, 0.42]
0.82
gemma4
JPEG
0.011
[0.006, 0.016]
− 0.003
− 0.002
− 1.19 [ − 3.30, 0.92]
1.02
qwen3-vl
JSON
0.017
[0.011, 0.023]
0.022
0.046
− 0.85 [ − 2.49, 0.79]
0.60
Table 1: Agreement between the heuristic scorer and each model. κ is linearly weighted with a bootstrap 95% confidence interval; κoff removes the cell’s constant offset; ICC is ICC(A,1); dˉ is the mean signed difference (model minus heuristic) with 95% limits of agreement; σL/σH is the ratio of standard deviations. n=15,000 in every cell.
Category
Screens
%
other
11,011
73.4
utility
836
5.6
social
598
4.0
news
574
3.8
games
544
3.6
travel
416
2.8
Table S1: Keyword-derived category of the 15,000 selected screens (descriptive only).
Component
Definition
Weight
Usability
Touch-target adequacy
Share of visible clickable elements at least 48 px wide and high (0.5 if none)
0.30
Cognitive load
1 if clickable count c≤7 , else exp(−0.15(c−7))
0.25
Text presence
1 if any visible element has a text class or non-empty text, else 0
0.15
Overlap
max(0,1−0.25o) , o = overlapping clickable pairs
0.30
Layout quality
Table S2: Components and weights of the heuristic scorer, as implemented.
Setting
Value
Client
Local Ollama client ( /api/generate ) forwarding to Ollama’s cloud-hosted models
temperature 0.0; at most 60 generated tokens; no random seed; non-streaming
Request handling
120 s timeout; no retries; 0.8 s delay between screens; checkpoint every five screens
Pixel input
Screenshot file, base64-encoded and attached to the request
Structural input
Depth-first list of the short class name of each visible element, followed by its text in parentheses when non-empty; first 25 entries joined by “ -> ” and prefixed “ UI Structure: ”; no geometry, clickability or counts
Table S3: Model-querying settings, as implemented.
Rater
Dimension
Missing
Outside 1–5
Min
Max
Mean
SD
heuristic
Usability
0
0
1.97
5.00
4.323
0.755
heuristic
Layout quality
0
0
1.52
3.60
3.025
0.532
heuristic
Visual complexity
0
0
2.66
5.00
3.515
0.469
gemma3 (JSON)
Usability
0
0
1.25
5.00
2.921
0.300
gemma3 (JSON)
Layout quality
0
0
1.50
4.00
3.235
0.379
gemma3 (JSON)
Visual complexity
0
0
1.50
4.90
3.572
0.641
Table S4: Completeness and range of every rating source ( n=15,000 screens each).
Rater
Dimension
1
2
3
4
5
heuristic
Usability
0.0
0.1
20.1
27.2
52.6
heuristic
Layout quality
0.0
26.0
53.2
20.8
0.0
heuristic
Visual complexity
0.0
0.0
54.6
41.7
3.6
gemma3 (JSON)
Usability
0.0
18.6
76.9
4.3
0.1
gemma3 (JSON)
Layout quality
0.0
4.6
59.0
36.4
0.0
gemma3 (JSON)
Visual complexity
0.0
8.3
16.2
75.3
0.1
Table S5: Marginal distribution (%) of integer-rounded scores, the categories on which κ is computed.
Table S7: Agreement within ±1 scale point: observed, expected under independent marginals, and excess over chance.
Rater
Feature
Dimension
ρ
p
p (Holm)
heuristic
Total elements
Usability
− 0.550
< 0.0001
–
heuristic
Clickable elements
Usability
− 0.731
< 0.0001
–
heuristic
Total elements
Layout quality
− 0.444
< 0.0001
–
heuristic
Clickable elements
Layout quality
− 0.637
< 0.0001
–
heuristic
Total elements
Visual complexity
0.940
< 0.0001
–
heuristic
Clickable elements
Visual complexity
0.742
< 0.0001
–
Table S8: Spearman correlation between structural quantities and each rater’s scores, all 15,000 screens.
Rater
Feature
Dimension
ρ ( ≤25 elements)
ρ (all screens)
heuristic
Total elements
Usability
0.295
− 0.550
heuristic
Clickable elements
Usability
− 0.026
− 0.731
heuristic
Total elements
Layout quality
− 0.229
− 0.444
heuristic
Clickable elements
Layout quality
− 0.500
− 0.637
heuristic
Total elements
Visual complexity
0.799
0.940
heuristic
Clickable elements
Visual complexity
0.473
0.742
Table S9: Truncation sensitivity: structural tracking restricted to the 5,365 screens with at most 25 visible elements, for which the structural model input is complete.
Model
Dimension
r
p
p (Holm)
n
gemma3
Usability
0.173
< 0.0001
< 0.0001
15,000
gemma3
Layout quality
0.136
< 0.0001
< 0.0001
15,000
gemma3
Visual complexity
0.092
< 0.0001
< 0.0001
15,000
gemma4
Usability
0.183
< 0.0001
< 0.0001
15,000
gemma4
Layout quality
0.102
< 0.0001
< 0.0001
15,000
gemma4
Visual complexity
0.004
0.6234
0.6234
15,000
Table S10: Pearson correlation between each model’s structural-input and pixel-input scores for the same screen.
Rater
Usability–Layout
Usability–Complexity
Layout–Complexity
heuristic
0.738
− 0.543
− 0.472
gemma3 (JSON)
0.825
0.007
0.042
gemma3 (JPEG)
0.871
− 0.086
− 0.070
gemma4 (JSON)
0.883
− 0.572
− 0.618
gemma4 (JPEG)
0.749
0.105
0.071
qwen3-vl (JSON)
0.970
− 0.238
− 0.231
Table S11: Within-rater Spearman correlation between dimensions.