Automated assessment of mobile user interface quality is attractive because manual review does not scale, and large language models are increasingly proposed for the task. Whether their ratings correspond to those produced by established rule-based heuristics has not been characterised at scale. We construct a 15,000-screen corpus from the RICO dataset using six documented filters, score every screen with a literature-grounded heuristic scorer, and collect ratings from three open-weight language models across two input representations, yielding 270,000 ratings. We report linearly weighted Cohen's kappa with bootstrap confidence intervals, offset-corrected kappa, intraclass correlation, and Bland-Altman limits of agreement, together with a chance baseline for the tolerance-band agreement rule. Agreement between the two rating sources is consistently weak: raw kappa falls below 0.15 in seventeen of eighteen model-representation-dimension cells. However, fifteen of eighteen cells are statistically distinguishable from zero, and correcting for a constant offset raises five cells above 0.15, so the divergence is better described as a combination of systematic bias and weak covariation than as absence of relationship. Model ratings track structural properties of the interface, but substantially less strongly than the heuristic does, and more strongly from an untruncated structural element list than from a screenshot. We further show that dimensions elicited in a single generated response are correlated by construction, reaching rho = 0.97 in one condition, which constrains what such designs can measure. We report these results as a characterisation of divergence between two automated raters rather than as evidence that either is correct, and we set out the validation a stronger claim would require.
Figures & tables
Figure 1: Linearly weighted κ between the heuristic scorer and each model, by input representation and dimension. Filled circles give the observed estimate with a bootstrap 95% confidence interval; open diamonds give κ recomputed after removing the cell’s constant offset. Removing the offset raises five cells above the 0.15 reference line.
Figure 2: Agreement within ±1 scale point, expressed as the difference between the observed proportion and the proportion expected under independent marginals with the same distributions. Departures from the chance baseline are small in every condition and negative in six of eighteen.
Figure 3: Spearman correlation between structural quantities and each rater’s scores, separated by input representation. The heuristic row is shaded. Model correlations are weaker than the heuristic throughout but are non-zero, and for the first model family are stronger under structural input than under pixel input.
Figure 4: Spearman correlation between dimensions within each rater. Model conditions exceed the heuristic’s usability–layout correlation in all six cases, reaching ρ=0.97 , consistent with sequential conditioning within a single generated response.
Model
Rep.
κ
95% CI
κoff
ICC
dˉ [LoA]
σL/σH
Usability
gemma3
JSON
0.005
[0.003, 0.006]
0.022
− 0.007
− 1.40 [ − 3.02, 0.21]
0.40
gemma3
JPEG
0.009
[0.005, 0.013]
− 0.037
− 0.028
− 1.27 [ − 3.15, 0.61]
0.70
gemma4
JSON
0.038
[0.034, 0.041]
0.178
0.083
− 1.27 [ − 2.95, 0.42]
0.82
gemma4
JPEG
0.011
[0.006, 0.016]
− 0.003
− 0.002
− 1.19 [ − 3.30, 0.92]
1.02
qwen3-vl
JSON
0.017
[0.011, 0.023]
0.022
0.046
− 0.85 [ − 2.49, 0.79]
0.60
Table 1: Agreement between the heuristic scorer and each model. κ is linearly weighted with a bootstrap 95% confidence interval; κoff removes the cell’s constant offset; ICC is ICC(A,1); dˉ is the mean signed difference (model minus heuristic) with 95% limits of agreement; σL/σH is the ratio of standard deviations. n=15,000 in every cell.
Category
Screens
%
other
11,011
73.4
utility
836
5.6
social
598
4.0
news
574
3.8
games
544
3.6
travel
416
2.8
Table S1: Keyword-derived category of the 15,000 selected screens (descriptive only).
Component
Definition
Weight
Usability
Touch-target adequacy
Share of visible clickable elements at least 48 px wide and high (0.5 if none)
0.30
Cognitive load
1 if clickable count c≤7 , else exp(−0.15(c−7))
0.25
Text presence
1 if any visible element has a text class or non-empty text, else 0
0.15
Overlap
max(0,1−0.25o) , o = overlapping clickable pairs
0.30
Layout quality
Table S2: Components and weights of the heuristic scorer, as implemented.
Setting
Value
Client
Local Ollama client ( /api/generate ) forwarding to Ollama’s cloud-hosted models
temperature 0.0; at most 60 generated tokens; no random seed; non-streaming
Request handling
120 s timeout; no retries; 0.8 s delay between screens; checkpoint every five screens
Pixel input
Screenshot file, base64-encoded and attached to the request
Structural input
Depth-first list of the short class name of each visible element, followed by its text in parentheses when non-empty; first 25 entries joined by “ -> ” and prefixed “ UI Structure: ”; no geometry, clickability or counts
Table S3: Model-querying settings, as implemented.
Rater
Dimension
Missing
Outside 1–5
Min
Max
Mean
SD
heuristic
Usability
0
0
1.97
5.00
4.323
0.755
heuristic
Layout quality
0
0
1.52
3.60
3.025
0.532
heuristic
Visual complexity
0
0
2.66
5.00
3.515
0.469
gemma3 (JSON)
Usability
0
0
1.25
5.00
2.921
0.300
gemma3 (JSON)
Layout quality
0
0
1.50
4.00
3.235
0.379
gemma3 (JSON)
Visual complexity
0
0
1.50
4.90
3.572
0.641
Table S4: Completeness and range of every rating source ( n=15,000 screens each).
Rater
Dimension
1
2
3
4
5
heuristic
Usability
0.0
0.1
20.1
27.2
52.6
heuristic
Layout quality
0.0
26.0
53.2
20.8
0.0
heuristic
Visual complexity
0.0
0.0
54.6
41.7
3.6
gemma3 (JSON)
Usability
0.0
18.6
76.9
4.3
0.1
gemma3 (JSON)
Layout quality
0.0
4.6
59.0
36.4
0.0
gemma3 (JSON)
Visual complexity
0.0
8.3
16.2
75.3
0.1
Table S5: Marginal distribution (%) of integer-rounded scores, the categories on which κ is computed.
Table S7: Agreement within ±1 scale point: observed, expected under independent marginals, and excess over chance.
Rater
Feature
Dimension
ρ
p
p (Holm)
heuristic
Total elements
Usability
− 0.550
< 0.0001
–
heuristic
Clickable elements
Usability
− 0.731
< 0.0001
–
heuristic
Total elements
Layout quality
− 0.444
< 0.0001
–
heuristic
Clickable elements
Layout quality
− 0.637
< 0.0001
–
heuristic
Total elements
Visual complexity
0.940
< 0.0001
–
heuristic
Clickable elements
Visual complexity
0.742
< 0.0001
–
Table S8: Spearman correlation between structural quantities and each rater’s scores, all 15,000 screens.
Rater
Feature
Dimension
ρ ( ≤25 elements)
ρ (all screens)
heuristic
Total elements
Usability
0.295
− 0.550
heuristic
Clickable elements
Usability
− 0.026
− 0.731
heuristic
Total elements
Layout quality
− 0.229
− 0.444
heuristic
Clickable elements
Layout quality
− 0.500
− 0.637
heuristic
Total elements
Visual complexity
0.799
0.940
heuristic
Clickable elements
Visual complexity
0.473
0.742
Table S9: Truncation sensitivity: structural tracking restricted to the 5,365 screens with at most 25 visible elements, for which the structural model input is complete.
Model
Dimension
r
p
p (Holm)
n
gemma3
Usability
0.173
< 0.0001
< 0.0001
15,000
gemma3
Layout quality
0.136
< 0.0001
< 0.0001
15,000
gemma3
Visual complexity
0.092
< 0.0001
< 0.0001
15,000
gemma4
Usability
0.183
< 0.0001
< 0.0001
15,000
gemma4
Layout quality
0.102
< 0.0001
< 0.0001
15,000
gemma4
Visual complexity
0.004
0.6234
0.6234
15,000
Table S10: Pearson correlation between each model’s structural-input and pixel-input scores for the same screen.
Rater
Usability–Layout
Usability–Complexity
Layout–Complexity
heuristic
0.738
− 0.543
− 0.472
gemma3 (JSON)
0.825
0.007
0.042
gemma3 (JPEG)
0.871
− 0.086
− 0.070
gemma4 (JSON)
0.883
− 0.572
− 0.618
gemma4 (JPEG)
0.749
0.105
0.071
qwen3-vl (JSON)
0.970
− 0.238
− 0.231
Table S11: Within-rater Spearman correlation between dimensions.
LLMs now sit on every side of evaluation: as examinees scored on benchmarks, judges of other models' outputs, and raters of human-generated content. Each paradigm can be viewed as a measurement problem, where a latent property of an object is probed with items from an instrument (e.g., benchmark) by judges or raters. Standard evaluation practices often neglect the contributions of each core component to the end result, limiting our understanding of what is being measured. Rasch measurement theory (RMT) is well-suited to this problem. RMT decomposes ordinal ratings into separable facets on a common scale. It further provides a battery of diagnostics that can identify miscalibrated measurements and rater biases. We present a case study of RMT applied to the LLM-as-rater paradigm using the Measuring Hate Speech corpus, whose construct was itself built under RMT. We fit a series of many-facet Rasch models to annotations from nine LLMs spanning families and capability levels. Our analyses show that LLMs systematically differ from human raters in severity, item-level calibration, question-order robustness, target-identity sensitivity, and rating scale use, all of which standard evaluation practice would largely obscure. Overall, we argue that RMT belongs in the toolkit for evaluating LLM-as-examinee, -judge, and -rater paradigms.
The rapid adoption of Large Language Models (LLMs) has spurred interest in automated peer review; however, progress is currently stifled by benchmarks that treat reviewing primarily as a rating prediction task. We argue that the utility of a review lies in its textual justification--its arguments, questions, and critique--rather than a scalar score. To address this, we introduce Beyond Rating, a holistic evaluation framework that assesses AI reviewers across five dimensions: Content Faithfulness, Argumentative Alignment, Focus Consistency, Question Constructiveness, and AI-Likelihood. Notably, we propose a Max-Recall strategy to accommodate valid expert disagreement and introduce a curated dataset of paper with high-confidence reviews, rigorously filtered to remove procedural noise. Extensive experiments demonstrate that while traditional n-gram metrics fail to reflect human preferences, our proposed text-centric metrics--particularly the recall of weakness arguments--correlate strongly with rating accuracy. These findings establish that aligning AI critique focus with human experts is a prerequisite for reliable automated scoring, offering a robust standard for future research.
Bowen Li, Haochen Ma, Yuxin Wang +5
College of Computer Science and Artificial Intelligence, Fudan University, Shanghai, China · Shanghai Innovation Institute, Shanghai, China
Generative UI (GenUI) lets large language models synthesize a complete, renderable interface directly from a natural-language instruction, but evaluating the quality of what they generate remains an open problem. Human evaluation is costly and rater-variant, while LLM-as-a-judge is scalable but reflects only a single implicit viewpoint, unable to capture how different populations of real users actually perceive the same interface. We propose the Evidence-Grounded, Social-Weighted Persona Panel (ESPP), a three-stage GenUI evaluation method in which a panel of psychologically diverse, evidence-grounded personas independently rates a screenshot, exchanges opinions under a trait-derived, semantically-gated bounded-confidence mechanism, and is aggregated via Delphi-inspired social weighting into a single judgment. ESPP tracks human judgment substantially more closely than a naive single-pass judge, raising Pearson r from 0.716 to 0.922, and a prompt-ensemble control recovers only about a third of this gap, isolating genuine persona and evidence grounding as the dominant source of improvement. Beyond this fidelity gain, retaining each panelist's individual rating further reveals that user subgroups agree on overall model rankings yet diverge sharply on specific rating dimensions, a structural disagreement a single homogeneous judge would systematically erase. The codes are available at https://github.com/Wuzheng02/ESPP.
Zheng Wu, Yibo Luo, Pu Zhang +2
School of Computer Science, Shanghai Jiao Tong University · 2ByteDance Inc