A phone screen can be wrong in ways that need no reference: a label that runs off the screen, letters that touch, a line that breaks one character before its end, one button with sharp corners among round ones. We ask whether a model can box every such fault on a screenshot, given no target design and no reference image. Such a critic could serve as a cheap first pass over generated pages before a designer reviews them, but only if it stays silent on clean pages. ScreenProof poses this target-free critique over three families of faults: general faults, local style faults (one element breaks the style its page sets) and text layout faults (letter spacing, line height, line breaks). A data engine makes each fault in page code and records the changed element's box, and hard negatives change every sibling alike. No frontier judge is reliable: frontier models flag 18-62% of clean real screenshots and 18-82% of clean generated pages, find 47-72% of text layout faults, and rarely find a foreign icon (5-19%). Our small critic ProofJudge (Qwen3.5-9B with LoRA), trained on code-made pages with one to four faults each and keeping a finding only when it is confident, reaches F1 0.48, the highest of all judges (Gemini 4 Argon 0.37; open models at most 0.15; 0.46 even without the confidence cut), and is the only fault-finding judge within 15% false alarms on clean pages (2% real, 9% generated), the property such a first pass needs, at about a hundredth of Argon's cost. It finds the most general faults (59%) but fewer local style faults than Argon (25% against 41%) and fewer text layout faults than the API judges (43%; Gemini 4 Argon 47%). Against a named brand the task is harder still: models know a design language about as well as its public documentation states it.
Figures & tables
Figure 2: ScreenProof at a glance. Left: one test example per fault family, each from another page: the faulty area enlarged on top, and the whole screen below with the data engine’s exact label as corner marks; a clean page has nothing to find, and a critic should stay silent on it. Right: the two test sets and the released training set, split by fault family in the same colours ( Secs. 3 and 4 ). Every fault is made in page code, so its box is exact.
Kind
UI
Target
Locates elements
Names convention
Judge trained / admitted on faults
Design2Code [ 24 ]
generation benchmark
yes
one screenshot
block matching
–
–
UI2Code N [ 33 ]
generation
yes
one screenshot
–
–
fine-tuned whole-page judge
ComUICoder [ 32 ]
generation
yes
pages
per element (detector)
–
–
MobileForge [ 29 ]
generation benchmark
yes
screens
–
–
positive anchor only
MultiRef [ 5 ]
generation
–
references with roles
–
–
–
DS-Compliance [ 4 ]
generation study
yes
design-system spec
–
checklist
–
Table 1: Positioning. Entries follow the papers’ own descriptions.
Aspect
What goes wrong
Engine fault
General: visible without a design target
Layout
overflow, overlap, order, missing item, two selected
each, in code
Readability
contrast, tap size, typeface
each, in code
Display
broken image, null value, needless ellipsis
each, in code
Local style: one element unlike the rest of its page
Style
radius, material, accent, icon
one element
Table 2: Fault taxonomy. The main task gives no target; a finding is general, local style or text layout (brand faults, which need a named target: Appendix D ). Engine fault : how the data engine makes it ( Sec. 4 ).
Figure 3: Overview of ProofJudge. A data-engine page and prompt pass through Qwen3.5-9B with its vision encoder and weights frozen; one LoRA on every linear layer of the language model is trained by SFT on the label JSON across pages with one to four faults each. The prompt gives no target and asks which elements break the page’s own style. The label is exact: the engine made the title overflow, a general fault under any design.
Kind
Rows
Images
Clean page
7,193
2,605
Designed difference (faulted twin)
630
591
General fault
1,542
1,427
Local style fault
1,609
1,505
Transplanted component
2,110
1,484
Text layout fault
972
847
Table 3: ScreenProof-Train: training rows and images by kind. A row is an image with its labels; an image can appear in several rows.
Figure 4: Faults made by the data engine on one base page (a generated ColorOS 17 page); each copy changes one thing in the page code and carries an exact label (amber corner marks). A component can be transplanted, with all its styles, from a page of another design system (the iOS 26 tab bar keeps the dark glass of its own dark-mode page); general faults are wrong in any design.
Figure 5: Where ProofJudge succeeds and fails on the consistency test (no target given): four hand-picked cases, three of its strengths and one fault every judge misses; Tab. 4 gives the overall picture, where ProofJudge has the highest F1 but Gemini 4 Argon finds more local style faults. Under each title, one line names the fault; each example shows the original screenshot with the engine’s exact label (amber corner marks), then ProofJudge and two API judges with only their own boxes. ProofJudge alone finds the room card whose surface differs from its sibling’s and a control shrunk below tap size (IoU ≥ 0.5), whereas the API judges’ boxes, if any, lie elsewhere; on a clean real screenshot it stays silent where Gemini 4 Argon boxes a timeline and an empty area. No judge finds the air-quality card that has lost the surface its sibling cards keep. Twenty cases chosen by a fixed rule, including failures, are in Appendix R .
Figure 6: What is easy and what is hard (consistency test, 2,765 screenshots, no target). Each row is one kind of fault, ordered by the mean share the four API judges find, easiest first; the bar is the share the best of them finds (Gemini 4 Argon, gpt-6.1-sol, Gemini 3.8 Flash, GPT-6 Luna), the dot the share ProofJudge finds. One element with another colour, material or corner radius is found by the best API judge about half the time; a foreign icon is hard for every judge, and none of the four nor ProofJudge finds an icon that looks normal on its page (Claude Opus 5.5, which flags most clean pages, finds 3 of 43; Tab. S6 ). ProofJudge finds the most general faults (59%). Every judge’s numbers are in Tab. S6 , the split of icon faults in Appendix I .
Figure 7: Clean pages flagged on the consistency test (no target; 607 clean pages: 120 real, 487 generated), for each judge one bar for clean real screenshots (dark) and one for clean generated pages (light). The dashed line is the 15% gate; among judges that find faults, only ProofJudge stays within it on both.
Clean pages flagged
Recall
Speed
Judge
real ↓
generated ↓
general ↑
consistency ↑
Precision ↑
F1 ↑
F1 .3 ↑
No answer ↓
s/page
GPUs
API judges
Gemini 4 Argon
18%
18%
38%
41%
35%
0.37
0.43
0
6.1
–
gpt-6.1-sol
38%
36%
45%
32%
25%
0.30
0.36
3
–
–
Claude Opus 5.5 (medium)
62%
82%
49%
49%
22%
0.31
0.35
0
–
–
Gemini 3.8 Flash
38%
66%
40%
26%
22%
0.26
0.29
0
20.6
–
Table 4: Consistency test (2,765 screenshots, 607 of them clean: 120 real and 487 generated; no target; same prompt; local models greedy without thinking, API judges at default reasoning). Consistency recall: swapped components and local style faults. The gate: at most 15% of real and of generated clean pages flagged; only ProofJudge with the cut passes it among fault-finding judges. No answer: unparsable, scored as finding nothing. Cut: a finding is kept only when the verdict’s probability of a clean page is below a threshold chosen on validation. Seconds per page: ProofJudge (rank-16 adapter) on one A100-80GB, open VLMs on A100-40GB, Gemini via API. Tinted: over half of the clean pages flagged.
Judge
Spacing ↑
Lines ↑
Breaks ↑
All ↑
Flagged ↓
touch
wide
touch
tall
wrap
lone
word
ellipsis
clean
neg.
Gemini 4 Argon
17%
41%
60%
18%
43%
22%
86%
81%
47%
32%
56%
gpt-6.1-sol
27%
59%
70%
35%
26%
25%
100%
81%
54%
49%
68%
Claude Opus 5.5 (medium)
73%
80%
70%
35%
43%
55%
98%
82%
72%
80%
96%
ProofJudge (ours)
5%
46%
5%
0%
26%
14%
94%
96%
43%
15%
13%
Table 5: Text layout test (held-out page requirements, no target, IoU ≥0.5 ): recall per rule, and pages flagged among 75 clean pages and 71 hard negatives (every sibling changed alike). 367 faults: characters touch 66, letter spacing unlike siblings 46, lines touch 20, line height unlike siblings 17, short label wraps 23, lone last word or character 73, word broken inside 50, needless ellipsis 72. Spacing: characters touch, letter spacing wider than the siblings’; Lines: lines touch, line height larger than the siblings’; Breaks: a short label wraps, a lone last word or character, a word broken inside, a needless ellipsis; Flagged: clean pages and hard negatives flagged. The base pages are English, so the Chinese line-start punctuation rule has no test item; “–”: not run.
LoRA rank
seed 42
seed 7
pair
test F1
95% interval vs rank 16
16
0.556
0.517
0.550
0.405 †
–
64
0.561
0.550
0.571
0.445
[+0.025,+0.055]
128 (ours)
0.586
0.598
0.607
0.481
[+0.060,+0.091]
256
0.605
0.576
0.599
–
–
Table 6: Capacity and seeds (v7f data, two epochs, each run at its final step 2,570; LoRA α = 2 × rank). Validation: F1 with each model’s own cut on the validation split. Pair: the average of the two seeds’ adapters. Test: the 2,765-page consistency test with the validation cut. Rank 64 was tested and rank 256 validated once on the training machine; their raw answers are not released. † The rank-16 model tested is one run, step 2,500, checkpoint and cut chosen on validation (the paper’s model before this one).
OwlEye
Judge
F1 ↑
Normal flagged ↓
Accuracy ↑
API judges
Gemini 4 Argon
0.86
28%
84%
Gemini 3.8 Flash
0.75
64%
68%
gpt-6.1-sol
0.76
63%
68%
Claude Opus 5.5 (medium)
0.79
53%
73%
Table 7: External benchmark (consistency prompt unchanged). OwlEye: 600 Android screenshots, half with display issues; flagging all scores F1 0.67. Full table: Tab. S5 .
Figure 8: Cost against accuracy, the same cost scale in both panels. (a) F1 on the consistency test (2,765 pages, no target given), with time per page and cost per 1,000 pages. (b) Clean pages flagged: real screenshots (filled) and generated pages (open); a checkpoint is usable only below the dashed line on both. ProofJudge is the cheapest by two orders of magnitude and flags the fewest clean pages, real and generated. ProofJudge’s time was measured on v6, with the same base model and a smaller adapter (rank 16). ProofJudge’s cost assumes $2 per A100-hour; the API costs are list prices on 2026-10-25.
Figure 9: ProofJudge as a cheap first check (consistency test, no target). gpt-6.1-sol runs either on every page or only on the 42% of pages ProofJudge flags: box F1 0.30 to 0.32, precision 25% to 36%, recall 37% to 28%, and clean pages flagged 38% to 1% (real) and 36% to 4% (generated; dashed line: the 15% gate). Sol reports no token counts, so the saving is shown in API calls; the same gate with Gemini 4 Argon is in the text.
Appendix figures & tables30 assets
Supplementary material from the paper’s appendix.
Appendix
Figure S1 : One accepted web screenshot per design language at the benchmark’s target version. Each label gives the version, the screen and its source; frames from videos are marked. Screens keep the language they were published in.
Figure S2 : Accepted web screenshots per design language (snapshot of 23 October 2026): the target version, older versions, and screenshots whose page names no version. 458 in all.
Figure S3 : One requirement (text, drawn as a wireframe) plus a public style card gives a target page in that design language (generated pages). The ColorOS 17 card cannot state the navigation shape or selected state, as OPPO’s public pages do not, and the page still draws a selected pill.
Figure S4 : How much each brand’s official public pages specify per field of our spec cards. Partly stated: the card marks part of the field as unspecified. For ColorOS 17, the only public specification covers home-screen cards; material and navigation appear only in marketing text.
State
Apply (picture + card text)
General design
Model
which brand / fact ↑
whose rule ↑
boxed source ↑
broken dimension ↑
say ↑
see ↑
Chance
19%
13%
13%
20%
25%
17%
gpt-6.1-sol
1000 /1316
229/412
33/163
37 /105
28/28
46/69
gpt-6-astra
983/1316
242/412
39/163
33/105
31/31
48/70
Gemini 4 Argon
925/1315
285 /408
53 /163
22/105
31/31
59/70
Gemini 3.1 Pro
816/1316
209/408
25/163
25/105
31/31
41/70
Appendix
Table S1 : Knowledge probes. State : 1,316 text questions from 658 facts of the official guidelines, half asking which brand a fact belongs to (8-way) and half which of four facts is the brand’s; the 124-question card pilot is in Appendix F . Apply v2: 680 items on test-split screenshots. Apply : which brand’s rule a real screenshot shows (no brand given), which brand a red-boxed swapped element follows, and which design dimension of a page breaks a given target. General : brand-free design knowledge in words (say) and on a red-boxed defect (see); “–”: not run.
Generator
Ctx
n
Content
Layout
Style
Likeness
Spec.
Viol.
gpt-6-astra
card
47
100%
98%
98%
9.9
7.5
1.8
name
47
100%
99%
96%
9.8
7.3
2.0
gpt-6.1-sol
card
47
100%
99%
97%
9.9
7.4
1.8
name
47
100%
99%
97%
9.9
7.3
1.9
Gemini 4 Argon
card
47
98%
89%
95%
9.4
7.4
1.9
name
47
98%
90%
95%
9.4
7.3
2.0
Appendix
Table S2 : Restore track (47 screens): itemised likeness and specification scores (1–10), mean of two judges (Gemini 3.1 Pro, gpt-5.6-sol); Viol. is the number of card violations per page. Tinted: below 75%.
Figure S5 : Knowing brand conventions, per design language: % correct on 1,316 text questions from 658 facts of the official guidelines. Frontier models reach 91–99% on Material 3 Expressive but 40–46% on ColorOS 17 and 14–32% on MagicOS 11 (green: the two least documented); the open 9B models, the base and ProofJudge v5, stay near chance. Brands at their latest versions: iOS 26, Material 3 Expressive, HarmonyOS 7, One UI 9, OriginOS 7, HyperOS 4, ColorOS 17, MagicOS 11.
iOS 18
iOS 26
iOS 27
ColorOS 15
ColorOS 16
ColorOS 17
Brand among eight
Gemini 3.8 Flash
30/30
30/30
30/30
4/9
13/16
5/24
Gemini 3.1 Pro
30/30
30/30
30/30
3/9
5/16
10/24
gpt-6-astra
30/30
30/30
30/30
1/9
7/16
5/24
Qwen3.5-9B base
29/29
25/25
30/30
1/9
0/15
2/23
ProofJudge v1 (earlier)
30/30
29/30
28/30
1/9
1/16
1/24
Appendix
Table S3 : Brand and version recognition on real screenshots. ColorOS 15 has only 9 screenshots. Tinted: below half.
Figure S6 : State probe (text only): accuracy of every model by brand and by card dimension, two questions per brand and stated card field, 12–18 per brand and 124 in all. Each brand at its latest version: iOS 26, Material 3 Expressive, HarmonyOS 7, One UI 9, HyperOS 4, OriginOS 7, ColorOS 17, MagicOS 11. Small and preliminary.
Table S4 : Brand scores (1–10) of round-0 pages by generation context and judge context.
Judge
Recall
FA
Prec.
F1
Acc.
No ans.
API judges
Gemini 4 Argon
96%
28%
77%
0.86
84%
0
Gemini 3.8 Flash
99%
64%
61%
0.75
68%
0
gpt-6.1-sol
100%
63%
61%
0.76
68%
0
Claude Opus 5.5 (medium)
99%
53%
65%
0.79
73%
0
ProofJudge (ours)
Appendix
Table S5 : OwlEye, all judges ( Tab. 7 ): 600 real Android screenshots, 300 with display issues (null values, missing images, occlusion, overlapping text, blur) and 300 normal; screenshot-level verdicts with the consistency prompt. FA: normal screenshots flagged. Images used for evaluation only. Flagging every screenshot scores F1 0.67 and accuracy 50% on this balanced set, so read F1 with FA and accuracy; unanswered screenshots count as not flagged.
Judge
Colour
Material
General
Radius
Foreign comp.
Icon, visible
Icon, looks normal
Gemini 4 Argon
50%
45%
38%
53%
43%
26%
0%
gpt-6.1-sol
40%
54%
45%
47%
30%
21%
0%
Gemini 3.8 Flash
46%
33%
40%
12%
24%
21%
0%
GPT-6 Luna
32%
27%
31%
21%
13%
5%
0%
Claude Opus 5.5 (medium)
65%
71%
49%
66%
48%
32%
7%
ProofJudge (ours)
41%
44%
59%
41%
20%
16%
0%
Appendix
Table S6 : Faults found by kind on the consistency test, every judge (no target, IoU ≥0.5 ); columns from easiest to hardest by the mean of the four API judges, as in Fig. 6 ; the two icon columns cover the 62 icon faults labelled for visibility (all icons: Appendix I ). Faults per kind: colour 179, material 123, general 920, radius 86, foreign comp. 662, icon, visible 19, icon, looks normal 43. ∗ The base model boxes about eight elements per page, so its recall is not comparable.
Judge
All icons ↑
Visible ↑
Brand only ↑
Gemini 4 Argon
8%
26%
0%
gpt-6.1-sol
6%
21%
0%
Gemini 3.8 Flash
6%
21%
0%
GPT-6 Luna
2%
5%
0%
ProofJudge (ours)
5%
16%
0%
Qwen3.5-9B base ∗
35%
16%
44%
Appendix
Table S7 : Icon faults found on the first 576 pages of the consistency test (62 labelled icon faults; no target, IoU ≥0.5 ), split by whether the page shows them. On all 188 icon faults of the full test, Gemini 4 Argon finds 15% and gpt-6.1-sol 11%. ∗ The base model draws about eight boxes per page, so its recall is not comparable.
Consistency F1 ↑
Defect v2 F1 ↑
Judge
IoU ≥ .5
IoU ≥ .3
IoU ≥ .5
IoU ≥ .3
Gemini 4 Argon
0.37
0.43
0.17
0.21
gpt-6.1-sol
0.30
0.36
0.10
0.12
Gemini 3.8 Flash
0.26
0.29
0.09
0.12
ProofJudge (ours)
0.48
0.50
–
–
ProofJudge v5 (final)
–
–
0.19
0.20
Appendix
Table S8 : Localisation at both IoU thresholds (same answers scored twice). Consistency: 2,765 screenshots, no target; defect test v2: 643 screenshots with a named brand. –: not run.
Judge
Effort
FA real
FA gen.
Swap
Local
F1
s/page
Gemini 4 Argon
low
10%
20%
5%
27%
0.24
4
default
15%
30%
50%
33%
0.33
–
default, rerun
5%
30%
40%
25%
0.25
5
high
15%
30%
5%
29%
0.22
19
gpt-6.1-sol
low
40%
27%
10%
33%
0.24
14
medium
35%
43%
20%
29%
0.19
–
Appendix
Table S9 : Thinking effort on 138 pages of the first 576-page consistency test (no target given). FA: clean pages flagged; Swap and Local: recall. Medium and default come from the full run (no timing).
Model
Method
False alarms
General recall
Brand recall
Defect F1
General text
Qwen3.5-9B base
no training
88%
20%
26%
0.06
90%
ProofJudge v5
LoRA r16 (SFT)
37%
57%
6%
0.18
87%
ProofJudge v5
LoRA r64
62%
63%
12%
0.21
90%
ProofJudge v5
full FT (LM)
65%
59%
12%
0.21
74%
ProofJudge v5
full FT (LM + vision)
64%
64%
10%
0.19
71%
ProofJudge v5
LoRA r16 + GRPO 150
56%
58%
10%
0.19
90%
Appendix
Table S10 : Training methods for ProofJudge on the v5 data (FT: fine-tuning; LM: language model). GRPO checkpoints from step 50 to 375 score F1 0.18–0.19 with brand recall 8–10%; the served model merges step 150 into one rank-32 adapter.
Critic
Clean false alarms ↓
General recall
Brand recall
Precision
F1
Boxes / page
Gemini 4 Argon
54%
40%
21%
13%
0.17
2.0
Gemini 3.8 Flash
89%
32%
15%
6%
0.09
3.3
gpt-6-astra
80%
47%
26%
6%
0.10
5.1
gpt-6.1-sol
75%
50%
27%
6%
0.10
6.0
Gemini 3.1 Pro
60%
25%
16%
4%
0.07
4.3
Qwen3.5-9B base
88%
20%
26%
4%
0.06
7.5
Appendix
Table S11 : Critique on defect test v2 (643 screenshots, 147 clean; base pages that fail the quality gate are removed, since their clean label cannot be trusted; one prompt and one matching rule for all). ProofJudge v5 is the LoRA with GRPO checkpoint 150, served as one merged adapter. Tinted: false alarms on more than half of the clean pages.
FA real
FA gen.
Swap
Local
Radius
F1
Gemini 4 Argon
15%
20%
36%
32%
83%
0.31
gpt-6.1-sol
36%
41%
24%
28%
83%
0.20
v6-a, step 750
4%
49%
16%
5%
4%
0.22
v6-b, step 950 (chosen)
4%
20%
8%
7%
4%
0.21
v6-b, step 1300
9%
72%
19%
13%
12%
0.24
v6-c, step 2229
13%
90%
19%
21%
50%
0.25
Appendix
Table S12 : ProofJudge v6 runs on the consistency test. FA: clean pages flagged (real screenshots, generated pages); Swap, Local, Radius: recall. † Boxes kept only when P(follows)<0.005 ; the threshold was read off the test, so this row only shows what calibration could reach, not a result.
Figure S7 : A local style fault: the Start Workout button’s corner radius changes from 28 to 9 px; the cards and the navigation keep the page’s own shapes. The page, not a brand, is the reference.
Hint
Fixed
Radius
Colour
Material
Harm, faulty
Harm, clean
Sent
None
95%
100%
69%
100%
0%
12%
100%
True finding
95%
100%
77%
98%
0%
0%
100%
Gemini 4 Argon findings
55%
79%
38%
48%
11%
15%
100%
ProofJudge (ours) findings
25%
29%
8%
27%
1%
0%
100%
ProofJudge r16 + cut (earlier) findings
16%
8%
8%
23%
0%
2%
100%
ProofJudge v7b (earlier) findings
42%
67%
8%
40%
6%
8%
100%
Appendix
Table S13 : Find, then fix: gpt-6.1-sol repairs 85 pages with one local fault and is also given 40 clean generated pages. Sent: pages that reach the fixer. ours: the paper’s model; r16 + cut: the same data with a rank-16 LoRA (the previous model); v7b: an earlier model; v6-c and v6-b: earlier one-fault runs.
Figure S8 : General faults on the consistency test (no target given, IoU ≥0.5 ). Text overflow: found by ProofJudge, gpt-6.1-sol; Overlap: found by ProofJudge, Gemini 4 Argon, gpt-6.1-sol.
Figure S9 : General faults on the consistency test (no target given, IoU ≥0.5 ). Low contrast: found by ProofJudge, Gemini 4 Argon; Tap target too small: found by no judge.
Figure S10 : Local style faults on the consistency test (no target given, IoU ≥0.5 ). One icon off: found by no judge; One colour off: found by no judge.
Figure S11 : Local style faults on the consistency test (no target given, IoU ≥0.5 ). One material off: found by Gemini 4 Argon, gpt-6.1-sol; One radius off: found by no judge.
Figure S12 : Components from another design system on the consistency test (no target given, IoU ≥0.5 ). Foreign top bar: found by no judge; Foreign button: found by no judge.
Figure S13 : Components from another design system on the consistency test (no target given, IoU ≥0.5 ). Foreign list row: found by no judge; Foreign tab bar: found by Gemini 4 Argon.
Figure S14 : Clean pages on the consistency test (no target given, IoU ≥0.5 ). Clean real screenshot: false boxes from gpt-6.1-sol (1); Clean real screenshot: every judge stays silent.
Figure S15 : Clean pages on the consistency test (no target given, IoU ≥0.5 ). Clean generated page: every judge stays silent; Clean generated page: every judge stays silent.
Figure S16 : Text layout faults on the text layout test (no target given, IoU ≥0.5 ). Characters touch: found by gpt-6.1-sol; Word broken inside: found by ProofJudge, Gemini 4 Argon, gpt-6.1-sol.
Figure S17 : Text layout faults on the text layout test (no target given, IoU ≥0.5 ). Lone last character: found by no judge; Hard negative: every label spaced alike: false boxes from ProofJudge (1), gpt-6.1-sol (11).
Fault detection identifies abnormal behavior in machines, sensors, infrastructure, software, and autonomous systems before it causes harm or disruption.
Leading GUI grounders score 93 to 96% on ScreenSpot-v2, and reported gaps between new and prior models are about a point. At this level the useful question is what the remaining errors are. Benchmarks count a click as correct only inside one annotated box, but interfaces often expose one action through several elements: a duplicated link, the label of a checkbox, the title next to a logo. The box is not wrong. The rule is incomplete. We call a click correct when executing it produces the same outcome as clicking the annotated target. A probe that executes and fingerprints about a thousand clicks per page turns this definition into outcome maps and an execution judge for live web applications, with no new annotation. We execute the predictions of three strong 7--8B grounders on a code forge and an online shop. About one in five and one in three of the clicks that box scoring rejects reach the intended outcome, and the shares remain one in seven and one in three when the instructions are rewritten as natural tasks. A blind audit of 11 models on ScreenSpot-v2 (three 7--8B grounders and eight 3B variants) finds the same effect on web screenshots, with two independent reviewers on a 60-unit subset. The share is highest for the strongest grounders. Training seeds and test-set sampling add noise that more runs and paired intervals control. Box scoring instead under-counts correct clicks, which repetition does not remove, and the audit-based correction of 0.2 to 1.8 points on ScreenSpot-v2 is as large as the gaps the field reports. The correction is needed in evaluation. In our 3B GRPO study, crediting equivalent clicks in the reward gives no stable gain over box rewards across three seeds and four test sets. We release the probe, the outcome maps, the audit and every prediction.
Xiaokang Yuan, Binhui Yao, Luwei Yang +2
Zhejiang University, Hangzhou, China · Zhejiang Geely Holding Group Co., Ltd, Hangzhou, China
Evaluating WebUI code generation at scale is difficult: outputs may compile and look plausible yet fail under user interaction, and prior benchmarks largely rely on free-form prompts with static checks (build success, screenshots) that miss functional correctness. We introduce WebUIProof, an execution-oriented benchmark that provides structured specifications and dense, executable interaction tests for WebUI generation across two task families: general WebUIs (e.g., dashboards, game, interactive tools) and 3D interactive simulation (e.g., particle/galaxy systems, physics dynamics). WebUIProof includes a UI-agent harness that runs executable interaction tests in a headless browser using an iterative plan--act--observe loop: it locates DOM elements, performs actions, observes resulting UI/DOM changes, and checks the specified assertions. We evaluate across eight commercial LLMs and observe frequent failures on interaction-based requirements even when pages render successfully, especially on 3D simulation interfaces. Finally, we show the UI-agent harness can provide outcome-level training signals. Training compact models (e.g., Qwen2.5 14B and MIMO 7B) with RL rewards derived from executable interaction tests improves functional completion while reducing build failures.
Small language models and coding agents increasingly generate web front-end code, yet their outputs are typically evaluated primarily for functional correctness. A generated interface may compile, render, and pass unit tests while still violating established interface quality principles, including accessibility barriers, deceptive design patterns, poor visual hierarchy, and excessive decision complexity. Existing auditing approaches face a trade-off between cost, coverage, and scalability: expert human review provides rich judgment but is slow and expensive; frontier vision-language models offer broader reasoning capabilities but remain costly to deploy at scale; and rule-based tools such as axe-core and Lighthouse are inexpensive but primarily capture mechanically checkable accessibility issues. We investigate whether a lightweight vision-language model can serve as an effective critic for generated interfaces. We unify 19 interface-quality principles from three complementary sources of HCI knowledge: WCAG 2.2 accessibility standards, deceptive design taxonomies, and established theories of perception, cognition, and interaction. To train this critic, we construct a verified dataset of approximately 10,000 generated web pages by synthetically injecting known violations into clean, LLM-generated Tailwind pages. Continued reinforcement learning on a 4B vision-language model improves micro-F1 from 36% to 84%, with 13 of 19 principles exceeding 80% F1. The resulting critic can audit generated interfaces, filter low-quality interface training data, and provide a reward signal for design-aware code generation. We release our data-generation recipe and injection/verification prompts to support reproducible evaluation and future work on scalable interface-quality assessment.
Nishi Mehta, Swathi Alse, Himani Kumawat +2
University of California, Santa Cruz · Carnegie Mellon University