4.1 Setup
We evaluate 13 frontier and open coding models under the staged protocol of Section 3 : every run starts from a pristine repository with the same stage plan, gates, allowed paths and budgets, cannot reach the reference or the negative controls, and is checked by the frozen 19-gate verifier at every checkpoint. Nine of the 13 also have image-valid screenshot runs paired with their own text-only cells. A verdict counts only if it postdates its task’s frozen specification, leaving 8,737 live cells , 6,979 Change and 1,758 Build, the single snapshot every number in this paper is computed from. Before interpreting any model failure we check that the benchmark can separate a correct solution from a plausible regression: across the validated authoring pool all 479 reference implementations pass and all 479 smart-wrong controls fail, Wilson [0.992, 1.000] in both directions. That is an admission property of the pool, not an accuracy estimate on unseen solver output. What it rules out is that a low score comes from an impossible reference, or a gate that lets the authored bad state through: the contract discriminates before any model runs. Appendix F gives the models, per-question denominators and statistical conventions, and Appendix D measures accuracy on solver output.
4.2 Main Results
Whole-task passes are sparse; the graded scale is not. Only 198 of the 8,737 cells clear every stage and rework, pass every gate, and record no regression break, and whole-task rates span 1.25 1.25 1.25 3.50 % 3.50\% 3.50% 49.41 49.41 49.41 61.71 61.71 61.71 Scaffolds change the number as much as backbones do. Table 4 runs eight distinct scaffolds over the same 400 tasks, four of them under both backbones, so scaffold and backbone can be compared as one statistic rather than a range against a mean. On that matched 4 × 2 4\times 2 4 × 2 8.06 8.06 8.06 10.52 10.52 10.52 5.08 5.08 5.08 5 Analysis
Observation point Count Rate Replay, any checkpoint 73/674 10.8% Class (a): initial state 66/586 11.3% Class (b): run’s own state 57/586 9.7% both channels 16/586 2.7% Either channel 107/586 18.3% still failing at the end 78/586 13.3% Table 4: Provenance-exact regression. Full rows and intervals: Table 9 .
Completion is not continuity. Table 3 separates the two verdicts: of the Change runs that pass every completion-family gate, 28.2 % \mathbf{28.2\%} 28.2% Figure 4: Thirteen models, 674 completing Change runs. (a) Post-completion failure, Wilson-95. (b) Regression rate against Gate%: unrelated orderings. Two regression channels with exact provenance. One channel needs no target-versus-prior labelling at all: whenever a stage clears, its assertions enter a protected set the harness replays at every later checkpoint of the same run (Section 3 ), so an assertion that passes at stage k k k k ′ > k k^{\prime}>k k ′ > k 10.8 % \mathbf{10.8\%} 10.8% ρ = − 0.88 \rho=-0.88 ρ = − 0.88 13.3 % \mathbf{13.3\%} 13.3% Figure 5: The untouched-repository channel. (a) Per-model rate: dot, Wilson-95 band, pooled 43.9% dashed. (b) What breaks, of 5,100 covered runs; colour marks the family. Figure 6: (a) Build versus Change raw WCS per model; the Build bar is shorter for all nine. (b) Paired twin-minus-source change per model-leg, pooled dashed; stage count matched by construction. Long-horizon construction. Build and Change occupy different regimes: on the nine-model dual-modal panel pooled Build WCS is 0.240 against 0.348 for Change, the gap sits in stage completion, not gate quality, and Build plans are the longer ones (Table 12 ). Longer runs are not recovering ones: on Build tasks of at least seven stages the correlation between turns and stages cleared is negative in every stage-count bucket, median − 0.91 -0.91 − 0.91 − - − Δ W C S \Delta\mathrm{WCS} Δ WCS − 11.82 -11.82 − 11.82 100 × 100\times 100 × t = − 4.74 t=-4.74 t = − 4.74 What moves the continuity signal. Capability does not track protected-state regression here: ranking the 13 models by Gate% and by the share of their completing runs that break a protected state gives two orderings with no detectable relation at this sample size (Spearman ρ = − 0.02 \rho=-0.02 ρ = − 0.02 p = 0.96 p=0.96 p = 0.96 9.40 9.40 9.40 7.81 7.81 7.81 Figure 7: Paired change in Gate% per arm. (a) The two prompt-scale levers, mirrored, one dot per slot. (b) The six arms with 95% CIs; only what the solver is told clears zero. 6 Related Work
Frontend and repository benchmarks. Benchmarks connect webpage designs with code generation and visual reconstruction ( Si et al., 2025 ; Yun et al., 2024 ; Gui et al., 2025 ) , then expand to framework-based editing and repair ( Xiao et al., 2025 ) and to the functional, visual and interactive quality of generated applications ( Zhang et al., 2026 ; Lu et al., 2025 ; Zhang et al., 2025 ) . Closest in coverage, WebCompass unifies text-, image- and video-conditioned generation, editing and repair, judging edits by model and generation by browser agent ( Lei et al., 2026 ) . SWE-bench evaluates issue resolution with executable tests, regression tests included ( Jimenez et al., 2024 ) , and SWE-bench Multimodal extends it to visual, user-facing JavaScript software ( Yang et al., 2025 ) ; agent interfaces and execution platforms support editing and adaptation to observations ( Yang et al., 2024 ; Wang et al., 2024 ; Wang et al., 2025 ) , and complementary work studies refinement, reflection and the limits of self-repair ( Madaan et al., 2023 ; Shinn et al., 2023 ; Olausson et al., 2024 ) . Lifecycle coverage, executable feedback and preservation checks are shared concerns.
Browser environments, simulated users, and what ProductWebBench adds. Web-agent research contributes demonstrations, executable environments and grounded tasks ( Deng et al., 2023 ; Zhou et al., 2024 ; Koh et al., 2024 ; He et al., 2024 ) , OSWorld covers desktop and cross-application workflows ( Xie et al., 2024 ) , ReAct studies interleaved reasoning and action ( Yao et al., 2023 ) , AgentBoard measures intermediate progress ( Ma et al., 2024 ) , and simulated-user benchmarks examine tool use against verifiable outcomes ( Yao et al., 2024 ; Barres et al., 2025 ) . What ProductWebBench adds is cumulative verification under a frozen contract. A task package binds each requirement to route, viewport and action-history evidence; a simulated user stages the work while frozen checks decide acceptance; and every cleared state is replayed at every later checkpoint, so an edit is graded against the site it changes, not the target alone. One trajectory therefore yields requested completion, additional verification failure and provenance-exact regression as three separate quantities, so progress, persistent failure and damage are told apart, not inferred from repair turns.
7 Conclusion
ProductWebBench evaluates frontend patches against a living system, not an isolated target. Its 400 evidence-backed tasks combine real repositories, staged persona-conditioned interaction, cumulative browser checkpoints and validated verifiers. Across 13 models, 28.2% of the Change runs that pass every completion gate still fail an additional conservative check. Protected-state replay certifies 10.8% of them as pass-to-fail regressions on states the run established itself, and the untouched-repository channel raises provenance-exact damage to 18.3% on the 586 runs where both are observable, with 13.3% of that same 586 still failing a conservative check at the end. Scaffolds matter as much as backbones: on the full-spec arm, swapping the scaffold under a fixed backbone moves the score further than the reverse, so the harness belongs in any report of agent capability. Target completion alone is therefore insufficient evidence that the site the change lands in still holds. Appendix G draws out what these numbers imply for reporting, Appendix H bounds the claims, and Appendix J walks seven released trajectories end to end, one per capability.
Ethics Statement
This work evaluates code-generating agents on software repositories and involves no human subjects, no personal data, and no user studies. Every task is built from a publicly available frontend repository, and each released package retains the upstream repository’s provenance and license; we redistribute task metadata, browser-state evidence, and verifier definitions rather than relicensed source. Evaluation runs in a sandboxed workspace with no deployment to public services and no production credentials, and provider keys, billing metadata, and transient workspaces are excluded from the release (Appendix I ). The benchmark measures whether a patch preserves behaviour that a repository already had; it is a software-quality instrument and we are not aware of a dual-use risk beyond that of the public repositories it is built from. The scores we report describe systems under one fixed protocol and should not be read as general capability claims about any vendor’s model, and we state throughout where provider availability makes model coverage unbalanced.
Reproducibility Statement
Every number in the paper is computed from one frozen result manifest, and that manifest is what we release: one row per evaluated cell, carrying the stage and gate components of the score, the applied and rejected edits, the failing gates, and the normalized capability, facet and framework labels. Rows superseded by a later submission specification are already removed from it, so the released manifest is exactly the 8,737 cells every number is taken over. Section 4 fixes the protocol, budgets, and stale-row filter; Appendix B gives the gate families, Appendix C the two evaluation arms and their scoring conventions, Appendix D the verifier-validation procedure, and Appendix E the exact definition of both continuity statistics. Appendix I lists what each released task package contains and the safeguards applied before release. Rerun variability is characterised in Appendix D : the graded metric carries a per-cell noise floor of σ ≈ 0.09 \sigma\approx 0.09 σ ≈ 0.09 T = 0 T=0 T = 0 Use of Large Language Models
We used a general-purpose LLM assistant for writing assistance only: copy-editing prose we had already drafted and checking LaTeX and figure formatting. It had no part in the research itself, and the authors have verified the full text and take responsibility for its contents.
Appendix A Task and Facet Inventory Table 5 reports primary continuity-facet metadata for the locked Change split. Three tasks have no normalized facet; they are excluded from facet-stratified analyses and kept in aggregate totals. Capability labels come from the frozen task record, so a per-capability column reports exactly the slots that record labels: 304 of the 320 Change slots (106 Edit, 76 Repair, 122 Consistency). Every overall column uses all 320, and we report the two denominators separately rather than imputing a label. Per-capability and overall columns therefore rest on different denominators, and we keep them distinct throughout. Table 5: Continuity facets. Facet Tasks Design 102 Experience 85 Regression 52 Content/assets 36 State 27 Implementation 15 Unnormalized 3
Appendix B Gate Families The decomposition into completion and additional verification is computed from names emitted by the frozen verifier, not from post hoc labeling. Table 6 lists the families used in the reported analysis. visual_regression is held out of the conservative family throughout because it compares against a reference render without controlling scroll offset, so target-versus-prior provenance is not uniform across its failures. One completion signal, state_url_signals , is configured by no task in the locked suite and so passes on every cell; we keep it in the table because the verifier emits it and Gate% is divided by 19 throughout. Recomputing Gate% over the eighteen signals that can fail lowers every model by two to three points and leaves the ordering in Table 1 unchanged. Table 6: Deterministic gate families. The three partition all 19 gates. Family Verifier signals Completion (12) completion text, visible text, DOM assertions, required states, interaction assertions, state quality, action history, screenshots, asset paths, exact asset paths, image-alt signals, state-URL signals Conservative extra (6) regression text, visual-anchor similarity, horizontal overflow, console errors, forbidden text patterns, state artifacts Broad sensitivity only (1) visual regression
Appendix C Presentation Interfaces and Evaluation Arms Two interfaces read the same frozen package, and the paper uses both. They are reported separately and are not comparable. Table 7 states where they differ, so no result is read across the boundary. Table 7: The two evaluation arms, differing only in presentation and turn control. Staged Full-spec Used in Tables 3 , 12 , 1 and Section 5 Table 4 , Figure 16 Solver input One work unit at a time, via the simulated user Complete task and repository context, once Turn control Our driver: 12 turns per stage, 200 per run The agent’s own loop, under a time and token ceiling Feedback access Per-gate failure text each checkpoint, plus a screenshot on the VLM leg None; scored on what it submits Protected-state replay Yes: cleared assertions re-verified at every later checkpoint No: one terminal verification Metrics defined Task%, Gate%, Stage%, WCS, breaks Task%, Gate%, component mean The simulated user is an evaluation interface, and deliberately so: one capability-conditioned persona per task, revealing requirements authored in advance and grounded feedback while acceptance stays fixed. That separation is what makes runs comparable across thirteen models. Multi-role interaction is a natural extension and belongs on the same frozen states and gates. Scaffold and staged scores are therefore not comparable as model-quality estimates. Gate% in Table 4 runs well above Table 1 while strict Task% runs below it, for reasons that have nothing to do with agent quality: a full-spec agent is graded once, against the final requirement set, without ever having to hold an earlier state passing. The two Task% columns are not the same quantity either: the scaffold arm reports a final gate pass, the staged arm the stricter whole-task notion of Section 3 . That is precisely the difficulty the staged interface adds, and why we read the scaffold study as an internal comparison rather than a second leaderboard.
Appendix D Verifier Validity Admission fixes the acceptance contract before any solver is run. All 479 references pass every gate and all 479 authored bad solutions fail, and the Build controls additionally pass the empty-baseline, self-certification, distinct-artifact and stage-solvability checks. That is true by construction, because a task enters the pool only when its contract is demonstrably satisfiable and its plausible failure mode is demonstrably caught, and it is exactly the guarantee we need: before a solver runs, two construction failures are already ruled out, a reference that cannot satisfy its own contract, and a gate that waves the authored regression through. Accuracy on unseen solver output is a separate question, and we measure it directly on real sweep output below. Blind model audit on solver output. We drew 720 stratified (cell, gate) verdicts from the actual sweep and rejudged each one from evidence alone. Rater A is the deterministic auto-rater; Rater B is ten blind shards of a frontier model that see the captured DOM, screenshot, and console log for the state in question, together with the assertion text, but never the gate implementation, the reference patch, or the verifier’s own verdict. On the n = 359 n=359 n = 359 94.2 % 94.2\% 94.2% κ = 0.820 \kappa=0.820 κ = 0.820 n = 338 n=338 n = 338 κ = 0.766 \kappa=0.766 κ = 0.766 0.044 0.044 0.044 0.025 0.025 0.025 0.075 0.075 0.075 0.190 0.190 0.190 κ \kappa κ 0.862 0.862 0.862 n = 114 n=114 n = 114 0.791 0.791 0.791 n = 117 n=117 n = 117 0.657 0.657 0.657 n = 107 n=107 n = 107 0.772 0.772 0.772 n = 240 n=240 n = 240 0.711 0.711 0.711 n = 98 n=98 n = 98 19.0 % 19.0\% 19.0% 4.4 % 4.4\% 4.4% 1.25 1.25 1.25 3.50 % 3.50\% 3.50% 4.4 % 4.4\% 4.4% 28.2 % 28.2\% 28.2% 19.0 % 19.0\% 19.0% [ 0 % , 35.4 % ] [0\%,35.4\%] [ 0% , 35.4% ] 10.8 % 10.8\% 10.8% 43.9 % 43.9\% 43.9% 18.3 % 18.3\% 18.3% n = 555 n=555 n = 555 97.7 % 97.7\% 97.7% κ = 0.827 \kappa=0.827 κ = 0.827 555 555 555 σ \sigma σ Appendix E State Provenance and the Regression Estimate Three classes of state. A Change run touches states of three kinds: (a) states already working in the initial repository and never named by the task; (b) states the run itself established by clearing an earlier stage; and (c) the target state of the current stage. Only (c) is covered by the completion family. The paper’s two continuity numbers cover (a) and (b) very differently, and conflating them is the main interpretive hazard in the result. What 28.2% is. The continuity checks – the conservative non-completion family (console errors, forbidden text, horizontal overflow, regression text, state artifacts, visual-anchor similarity) fires on the final workspace. A cell in that family may be breaking a class-(a) state, a class-(b) state, or simply failing a global hygiene check that was never passing in the first place; the emitted artifacts do not record which. The 190 / 674 = 28.2 % 190/674=28.2\% 190/674 = 28.2% k k k k k k k ′ > k k^{\prime}>k k ′ > k 73 / 674 = 10.8 % 73/674=10.8\% 73/674 = 10.8% 9.7 % 9.7\% 9.7% 10.8 % 10.8\% 10.8% 0.2 0.2 0.2 73 / 674 = 10.8 % 73/674=10.8\% 73/674 = 10.8% 8.7 % 8.7\% 8.7% 13.4 % 13.4\% 13.4% 823 / 6,305 = 13.1 % 823/6{,}305=13.1\% 823/6 , 305 = 13.1% 12.2 % 12.2\% 12.2% 13.9 % 13.9\% 13.9% 3.6 % 3.6\% 3.6% 20.7 % 20.7\% 20.7% 6.2 % 6.2\% 6.2% 20.7 % 20.7\% 20.7% t t t + 1.07 +1.07 + 1.07 − 0.57 -0.57 − 0.57 − 0.42 -0.42 − 0.42 − 0.98 -0.98 − 0.98 Appendix F Additional Result Detail
A worked WCS, and what the suite counts are. One cell makes Equation 3 concrete. Slot 1 under Kimi-K3, text-only leg, clears 3 of its 5 stages ( s = 0.600 s=0.600 s = 0.600 g = 0.526 g=0.526 g = 0.526 v = 0 v=0 v = 0 b = 0 b=0 b = 0 0.5 ( 0.600 ) + 0.3 ( 0.526 ) = 0.458 0.5(0.600)+0.3(0.526)=0.458 0.5 ( 0.600 ) + 0.3 ( 0.526 ) = 0.458 s s s s s s b i b_{i} b i 0.05 b i 0.05b_{i} 0.05 b i 0.2 0.2 0.2 576 576 576 8,737 8{,}737 8 , 737 The task counts in the paper refer to different sets and are not interchangeable. The validated authoring pool holds 479 tasks, each with a passing reference and a failing authored control. The locked evaluation suite is the 400 of those used for every number here, 320 Change and 80 Build. Of the 320 Change tasks, all 320 rebuild from the frozen package in the analysis environment and 234 of those also serve a capturable initial baseline, which is the coverage of the class-(a) channel; 128 of the served ones run the project’s own dev server rather than a built static tree. A further 16 Change slots carry no capability label in the frozen manifest and enter overall columns only. Task-clustered bootstraps resample tasks with all of their cells, so repeated runs of one task are not treated as independent. Tasks derived from the same upstream repository could still correlate beyond that, and we do not currently report a repository-level clustering; the released manifest carries the source path for every task, so that grouping can be recomputed from the release without rerunning anything.
Run configuration. Solver and simulated user are separate models. The simulated user is held fixed across the sweep so that persona behaviour is not a free variable between models, and the persona names in Appendix C label prompt variants rather than model configurations. Every cell records, in its released row, the solver model identifier, the leg, the number of turns and tokens actually used, the edits applied and rejected, the per-stage trace, and which gates failed; the route, viewport and action prefix each gate was evaluated at is fixed per state by the released state_plan.json of that task. Infrastructure failures (install failure, timeout, provider error) are written as infra_failed and the cell is re-run rather than scored, so they do not enter any denominator; a gate that a task does not configure is not counted in that task’s gate total, which is why gate totals vary by task and why Gate% is a rate rather than a count. The five screenshot_files false fails in the audit sample are exactly the case where a judge read an unconfigured criterion as vacuously failing, and they are counted against the verifier in Appendix D rather than excused.
Models, denominators and statistical conventions. The 13 models are Claude Opus 4.8 and Sonnet 4.6, GPT-5.5, Gemini 3.5 Flash, DeepSeek V4 Pro and Flash, GLM-5.2-FP8, Hunyuan 3 Preview, Kimi K2.7 Code and K3, MiniMax M3, Qwen 3.6 and Qwen 3.7 Max. The four without a screenshot leg send no image on any screenshot cell, so they are evaluated text-only rather than simulated with empty image turns, and a cell scored against a superseded specification is rerun rather than carried forward. Denominators differ by question: the post-completion gap uses all 6,979 Change cells, the long-horizon analysis the 1,758 Build cells stratified by stage count, the Build against Change comparison 7,142 cells from the nine dual-modal systems, and the modality experiment 3,558 within-model, within-task pairs from those same nine. The post-completion gap and both regression channels resample tasks with all of their cells, because several runs of the same task differ only in model or modality. Paired modality differences use Wilcoxon signed-rank tests; paired arms are judged by their own confidence intervals. Long-horizon results are bucketed by stage count, Kruskal–Wallis diagnostics are exploratory, and Benjamini–Hochberg correction is applied throughout to the framework and the facet families.
Table 11: Headline diagnostics with their eligible sets. Each row uses its own denominator, stated in the N N N N N N > > > ≥ 2 \geq 2 ≥ 2 N N N The failures are not patch machinery. The 8,737 live cells divide exactly into the 198 whole-task passes of Table 1 and 8,539 failures. Among the failures, 8,396 (98.3%) land at least one edit; in 7,002 (82.0%) the cumulative count of applied edits exceeds rejected edits and the run still does not pass; 1,394 (16.3%) edit the workspace but have rejected edits at least as numerous as applied ones; and only 143 (1.7%) land no edit at all, so a no-edit explanation covers under two per cent of them. Landing one edit does not exclude a later patch failing on format or scope, so these counts bound the tooling account rather than eliminating it.
Ranking coherence across capability slices. The mean Spearman correlation between a per-capability WCS ranking and the overall WCS ranking is 0.885: Edit ρ = 0.934 \rho=0.934 ρ = 0.934 ρ = 0.934 \rho=0.934 ρ = 0.934 ρ = 0.852 \rho=0.852 ρ = 0.852 ρ = 0.819 \rho=0.819 ρ = 0.819 The static arm is not a comparable half. Classifying the suite by whether a task’s frozen specification declares interaction assertions splits it into 379 interactive slots and 21 static ones. The pooled within-model gap is − 9.18 -9.18 − 9.18 100 × 100\times 100 × − 0.90 -0.90 − 0.90 − 0.91 -0.91 − 0.91 Figure 8: Interactive-vs-static Δ \Delta Δ Failure radius. Within the 432 complete-but-not-cleared cells of the broad visual-inclusive definition, a cell fails 1.37 distinct non-completion signals on average: 31.9% fail at least two and 4.4% at least three. The count is over individual signals, not over the three gate families of Appendix B . These are broad sensitivity diagnostics rather than confirmed regression radii, since the broad family does not separate target from prior state (Figure 8 b).
Figure 9: Turn-thrash on the long-horizon regime. (a) WCS by turn quartile. (b) Correlation of turn count with WCS and with stages cleared, per stage-count bucket. Figure 10: (a) Rank shift from Change to Build per model; negative is a climb. (b) Each model’s rank within each of the four capabilities, with 1 the best rank in that column. Model separation and the twin set. The graded WCS has model-level dynamic range on Build: an exploratory Kruskal–Wallis diagnostic gives H = 118.6 H=118.6 H = 118.6 p = 1.2 × 10 − 19 p=1.2\times 10^{-19} p = 1.2 × 1 0 − 19 Δ = − 0.14 \Delta=-0.14 Δ = − 0.14 Δ Gate% \Delta\text{Gate\%} Δ Gate% − 7.67 -7.67 − 7.67 The component mean, and what the grid does not control. The scaffold arm verifies once at the end, so WCS is undefined for it and the contrasts use the component mean : the unweighted mean of the eight per-cell component scores the verifier emits (change completion, the six continuity facets, and browser execution), which needs no intermediate checkpoint and is defined in both arms. The comparison is limited to two backbones and four shared scaffolds, and we did not equalise time or token budgets across scaffolds, so any budget difference sits inside the measured scaffold effect rather than being controlled out of it.
Scaffold versus backbone, in full. On the matched 4 × 2 4\times 2 4 × 2 8.06 8.06 8.06 10.52 10.52 10.52 5.08 5.08 5.08 + 0.60 +0.60 + 0.60 + 8.27 +8.27 + 8.27 ρ = 0.80 \rho=0.80 ρ = 0.80 5.68 5.68 5.68 Change capability scores. Across all 13 models, pooled Change WCS is 0.342 for Edit (2,310 cells), 0.450 for Repair (1,653), and 0.305 for Consistency (2,665); a further 351 cells sit on 16 slots with no capability label in the frozen manifest, excluded from this breakdown. Consistency remains 0.035 WCS below Edit ( s e = 0.007 \mathrm{se}=0.007 se = 0.007 t = − 5.07 t=-5.07 t = − 5.07 p = 4.0 × 10 − 7 p=4.0\times 10^{-7} p = 4.0 × 1 0 − 7 F = 182.48 F=182.48 F = 182.48 ( 2 , 6608 ) (2,6608) ( 2 , 6608 ) p = 7.3 × 10 − 78 p=7.3\times 10^{-78} p = 7.3 × 1 0 − 78 Framework, facet, and size controls. Three further breakdowns support the category result in the main text. Per-framework WCS (Figure 12 ) separates harder and easier stacks after Benjamini–Hochberg adjustment; at the exploratory cell level the spread across framework labels is large ( H = 1552.5 H=1552.5 H = 1552.5 p < 10 − 300 p<10^{-300} p < 1 0 − 300 Long-horizon boundary. The negative relation between turns and progress appears only on long tasks. Within five- and six-stage buckets, the correlation between turns and stages cleared is positive (0.32 and 0.24 respectively). It becomes negative for every bucket from 7 through 14 stages: − 0.93 -0.93 − 0.93 − 0.80 -0.80 − 0.80 − 0.94 -0.94 − 0.94 − 0.90 -0.90 − 0.90 − 0.97 -0.97 − 0.97 − 0.96 -0.96 − 0.96 − 0.82 -0.82 − 0.82 − 0.66 -0.66 − 0.66 Feedback-modality validity. We call a model dual-modal only when more than half of its screenshot-feedback cells actually send at least one image, which guards against the failure mode of a text-only endpoint silently accepting and discarding the image turn. Nine of the thirteen models clear this bar, with image-carrying fractions from 0.807 to 0.912: Opus-4.8, Sonnet-4.6, Gemini-3.5, GPT-5.5, Hunyuan-3, Kimi-K2.7, Kimi-K3, MiniMax-M3, and Qwen3.6. Four are excluded because their screenshot leg sends images on zero cells (GLM-5.2-FP8, DeepSeek-V4-Flash, DeepSeek-V4-Pro, and Qwen3.7-Max), and are reported as not applicable rather than simulated with empty image turns. The paired estimate therefore rests on nine models and 3,558 within-model, within-task pairs; the count exceeds the 400-task suite because each task contributes one pair per qualifying model. The contrast is between assignment to the screenshot policy and assignment to the text-only policy: image-carrying fractions run from 0.807 to 0.912 even among qualifying models, so a pair differs in the policy applied rather than in whether the solver certainly saw an image on every turn. Restricting to pairs that did carry an image would condition on run progress and is not an unbiased comparison. On those pairs the mean change in WCS with screenshot feedback is − 0.0007 -0.0007 − 0.0007 − 0.0064 -0.0064 − 0.0064 0.0050 0.0050 0.0050 p = 0.952 p=0.952 p = 0.952 + 0.0037 +0.0037 + 0.0037 p = 0.334 p=0.334 p = 0.334 − 0.0124 -0.0124 − 0.0124 − 0.0216 -0.0216 − 0.0216 − 0.0031 -0.0031 − 0.0031 p = 0.009 p=0.009 p = 0.009 + 0.0144 +0.0144 + 0.0144 n = 728 n=728 n = 728 p = 0.001 p=0.001 p = 0.001 Figure 11: Verifier calibration. Every one of the 479 references passes all gates and every one of the 479 authored bad solutions fails. (a) Agreement and Cohen κ \kappa κ Figure 12: Per-framework WCS ordering. Bars are Benjamini–Hochberg-adjusted harder (red) or easier (green) than the grand mean; grey is not significant. Figure 13: Per-model facet ranking (1 = easiest for that model). All 13 models are plotted and every facet is populated for every model. Regression-continuity is the easiest facet for all 13; construction is the hardest for 12 of them. Figure 14: Category effect survives edit-size control. (a) Category coefficient against the Edit baseline, controlling for task size and model. (b) Mean WCS by category on the smallest quarter of tasks, where the ordering is unchanged. Figure 15: Rerun stability at temperature zero: fifteen pairs of one slot and one model replayed three times, plotting the standard deviation of lh_score against its mean. The mean standard deviation of 0.093 is the noise floor on every per-cell graded score, and the binary verdict is unchanged in all fifteen pairs. Replays re-execute the whole solver loop, so this is end-to-end run variability. Figure 16: Per-scaffold component mean by capability; orange box marks the per-column winner. Codex with GPT-5.5 takes Build, Consistency and Repair, Kimi-CLI takes Edit. Robustness and validity ablations. Table 13 collects the single-variable checks that could otherwise explain these results: edit format, vision on either side of the loop, simulated-user wording, and rerun variability. No arm produces a net strict-pass gain; the only arm whose net strict-pass count changes at all is solver vision, at − 0.06 -0.06 − 0.06 ∣ t ∣ > 2 |t|>2 ∣ t ∣ > 2 Δ \Delta Δ Δ \Delta Δ − 3.38 -3.38 − 3.38 t = − 0.94 t=-0.94 t = − 0.94 + 2.43 +2.43 + 2.43 t = + 0.70 t=+0.70 t = + 0.70 − 1.78 -1.78 − 1.78 − 9.40 -9.40 − 9.40 Δ \Delta Δ Ablation Model & leg Direction n n n Δ \Delta Δ t t t Δ \Delta Δ + 3.16 +3.16 + 3.16 + 0.62 +0.62 + 0.62 0.00 \ \ 0.00 0.00 + 4.74 +4.74 + 4.74 + 1.26 +1.26 + 1.26 0.00 \ \ 0.00 0.00 − 0.28 -0.28 − 0.28 − 0.87 -0.87 − 0.87 − 0.06 -0.06 − 0.06 + 2.43 +2.43 + 2.43 + 0.70 +0.70 + 0.70 − 7.69 -7.69 − 7.69 − 3.38 -3.38 − 3.38 − 0.94 -0.94 − 0.94 0.00 \ \ 0.00 0.00 T = 0 T=0 T = 0 σ = 2.6 \sigma{=}2.6 σ = 2.6 0 / 15 0/15 0/15 Table 13: Single-variable robustness ablations. Δ \Delta Δ n n n t t t Appendix G Discussion The two verdicts come apart, and the gap is not small. A target-only metric is satisfied the moment the requested DOM appears. On our suite that moment is routinely reached while the surrounding product is already broken: 28.2% of the Change runs that clear every requested-work gate still fail a conservative non-completion check, and among interactive runs whose target DOM passes, 29.5% still fail an interaction check on the very element they were asked to produce. The first number says a correct target does not imply an intact site; the second says it does not even imply a working feature. Both are invisible to an endpoint DOM assertion. That is why the unit we score is the change together with the website it lands in. Provenance is what makes a regression claim defensible. Calling an outcome a regression requires knowing the state was good before, and a benchmark that only inspects the final workspace cannot know that. Our two channels each carry that evidence by construction: one replays a state the run itself cleared, the other checks a gate the untouched repository already satisfied. The untouched-repository channel also shows that two of its gates report one event: all 1,364 runs that leave a route blank also fail the capture check on that route. Their union covers 18.3% of the covered subset while they overlap on 2.7%, so they are complementary views that partly coincide, not one effect counted twice. We do not read the small overlap as independence: under independence the expected overlap is about 6.4 runs against an observed 16. The replay channel then splits in a way no endpoint-only harness can see: of the 10.8% of completing runs that break a protected state, a little over half finish with every conservative check clean, the rest do not. Because the endpoint is a run-level verdict, not a per-assertion repair record, that first group is invisible to a final-state measure – the run ends looking clean and the damage happened anyway. Long trajectories do not quietly fix themselves. Within every fixed stage-count bucket on Build the correlation between turns spent and stages cleared is negative, median − 0.91 -0.91 − 0.91 − 0.0007 -0.0007 − 0.0007 0.60 0.60 0.60 8.27 8.27 8.27 Appendix H Limitations and Scope Three boundaries define where the claims of Section 5 apply. The twin result is stated for its regime. The greenfield-twin finding, that from-scratch construction scores below the matched in-place change on all five legs, is established on 21 length-matched Change tasks whose twins are mostly static or vanilla-node sites, because our authoring chain yields a blank baseline only for those stacks. We state it for that regime, not for arbitrary stacks. The turn budget is endogenous. Failed units consume the stage cap while cleared units stop early, so turn count is an outcome of the trajectory as much as an input. We report persistence to the cap as an observation about how these trajectories end, not a causal claim about what extra turns buy. Coverage is uneven and cells are clustered. Provider availability leaves coverage uneven, and several cells share a task. We state every denominator, use matched pairs where one exists, report a task-clustered interval beside each cell-level one, and read the pooled table as observed coverage, the paired contrasts carrying the claims. The same applies to the capability-against-regression comparison of Section 5 , which thirteen models bound loosely but cannot rule out. Two scope notes close the picture. Visual anchors are deterministic checks on authored properties, not a measure of unrestricted aesthetic quality. And the interventions of Section 5 are prompt-scale probes aimed at the graded gate metric; testing whether a prompt can move the rarer protected-state endpoint needs a separately powered arm, as the arm-by-arm figures in Appendix E show.
Appendix I Reproducibility and Release Safeguards Each released task package contains repository provenance, allowed paths, the ordered stage plan, route, viewport and action states, design anchors, deterministic checks, reference metadata, and negative-control metadata. Evaluation starts from a pristine workspace. Solver prompts and screenshots exclude reference patches, bad solutions, and hidden gate implementation. Result rows carry task, model, modality, specification timestamp, and trace identifiers; rows older than their submission specification are excluded before analysis. The released manifest is post-filter and carries the normalized labels, so both the exclusion and the label mapping are materialized in the rows themselves. Gate judgement carries across environments. The class-(a) baseline (Appendix E ) is captured in the analysis environment rather than on the evaluation cluster, so we calibrated the two against a known answer: a reference implementation must pass every gate by admission. Re-capturing six reference workspaces locally reproduces that verdict on the five that serve offline: four pass all nineteen gates and one misses only visual_regression , the gate excluded throughout for its uncontrolled scroll offset. The sixth does not start a server in the analysis harness and yields no verdict either way. Every served recapture agrees with the cluster on every nonvisual gate. Five workspaces are a small sample, so this supports rather than establishes cross-environment equivalence for the eighteen gates the class-(a) statistic uses; replaying the initial and the final workspace of the same run side by side in one environment is the check that would settle it. Ten of the baselines were captured later than the rest, on a rebuilt analysis machine, which lets us separate two questions the first calibration conflates. Repeating the known-answer test there gives the same result: of the reference workspaces that serve, one passes seventeen of nineteen gates and three pass eighteen, and the only gate all four miss is visual_regression . The environment therefore still judges a correct implementation correctly. Recapturing the untouched repository is a different matter: over thirteen slots and 247 gate verdicts, a recapture agrees with the frozen baseline on 85% of them, and the disagreements concentrate in console_errors , no_horizontal_overflow and visual_anchor_similarity , in both directions. A reference implementation sits far from those thresholds and a partly broken repository sits on them, so the pristine side of the comparison is the browser-version-sensitive one. We therefore read a single slot’s baseline as an environment-conditional reading rather than an absolute one; the ten late captures move the union count by one event, from 106 to 107. Repositories and local assets retain their original licenses. The task suite does not require personal user data, production credentials, or deployment to public services. Provider credentials, private billing metadata, and transient workspaces are excluded from the release.