Organizations: The Hong Kong University of Science and Technology (Guangzhou) · University of Waterloo · The Hong Kong University of Science and Technology
Large language models can generate runnable web applications from natural-language requirements, but many generated applications still fail interactive tasks. Existing generate-execute-repair pipelines execute the generated application and use task outcomes or error messages to guide code revision. However, this feedback often misses the runtime behavior between a browser action and the final task outcome, making interaction-level failures difficult to diagnose. Therefore, we propose TeleGen, an observability-enhanced framework for LLM-based web application generation. TeleGen instruments generated applications, collects runtime telemetry during task execution, and compresses raw telemetry logs into concise briefs for repair. We evaluate TeleGen on WebGen-Bench and Web-Bench. On WebGen-Bench, TeleGen improves task success from 67.7% with repair without telemetry to 76.2%, an increase of 8.5 percentage points. On Web-Bench, it improves cumulative Pass@2 from 21.7% to 29.8%. Ablation results show that runtime telemetry provides a useful diagnostic signal, while telemetry briefs make this signal more effective and less costly to use. Further analysis shows that telemetry is especially helpful for failures involving hidden execution paths, such as navigation, form workflows, and frontend-backend coordination.
Figures & tables
Figure 1: Overview of TeleGen . Given a website description and UI task suite, a coding agent first generates an initial web application, denoted as the v1 application . TeleGen then instruments a separate copy of the v1 application and re-runs the task executor to collect runtime telemetry, including actions, network events, errors, and state changes. A compression agent combines the raw traces with the task context and produces a compact telemetry brief. Finally, the repair agent edits the clean v1 source code using the telemetry brief and task feedback, producing a repaired v2 application , which is re-run by the task executor for final result analysis.
Benchmark
Setting
Passed / Total
Overall
Δ
Categories
Content
User
Data
WebGen-Bench
Initial generation
399 / 647
61.7
–
71.3
55.0
64.4
No-Telemetry repair
438 / 647
67.7
+6.0
78.7
60.4
70.0
TeleGen
493 / 647
76.2
+14.5
88.5
68.1
78.8
Easy
Moderate
Challenging
WebBench
Pass@1
142 / 1000
14.2
–
44.0
7.2
0.6
Table 1: For WebGen-Bench, Overall, Content, User and Data report task-level success rates for the full benchmark and its three project categories. For Web-Bench, Pass1 is the first-attempt success rate, while Original Pass2 and TeleGen Pass2 report cumulative success after the original and telemetry-augmented second attempts. Easy, Moderate and Challenging report category-level cumulative success rate. For WebGen-Bench, Δ denotes the overall improvement of TeleGen over Initial generation; for Web-Bench, Δ denotes the overall improvement over Pass1. All values are percentages except passed-task counts.
Setting
Passed
Success
Δ
No-telemetry
438
67.70%
–
No-brief
484
74.81%
+7.11 pp
TeleGen
493
76.20%
+8.50 pp
Table 2: Repair effectiveness under DeepSeek-V4-Flash. Δ is measured against no-telemetry repair.
Setting
Input
Output
Total
No-telemetry
971.8K
19.0K
990.8K
No-brief
1462.1K
19.6K
1481.7K
TeleGen
1439.6K
18.4K
1458.1K
Table 3: Average token cost per project under DeepSeek-V4-Flash. Values are reported in thousands of tokens.
Figure 2: Failure types among tasks solved only by Telemetry-Guided Repair under DeepSeek-V4-Flash. These tasks fail under No-telemetry repair but succeed when telemetry brief is provided.
Figure 3: Representative form/input workflow repair enabled by runtime evidence. Although the form appears valid, runtime telemetry shows that client data is loaded but no client-selection interaction occurs. Telemetry-Guided Repair replaces the hard-to-operate dropdown with explicit clickable client cards, allowing the browser agent to complete the workflow.
Appendix figures & tables6 assets
Supplementary material from the paper’s appendix.
Appendix
Benchmark
Model
Setting
Passed / Total
Overall
Δ
Categories
Content
User
Data
WebGen-Bench
DeepSeek- V4-Pro
Initial generation
432 / 647
66.8
–
85.6
55.9
67.5
No-telemetry repair
443 / 647
68.5
+1.7
83.9
61.0
66.2
TeleGen
476 / 647
73.6
+6.8
88.5
66.5
71.2
Qwen3.5-Plus
Initial generation
324 / 647
50.1
–
71.8
43.1
40.0
No-telemetry repair
351 / 647
54.3
+4.2
75.9
47.0
45.0
Appendix
Table 4: Additional model results. For WebGen-Bench, Overall, Content, User, and Data report task-level success rates for the full benchmark and three project categories. For Gemini-3-Flash-Preview, all rates use the canonical 647-task denominator, with missing evaluations counted as unsuccessful. For Web-Bench, Pass1 is the first-attempt success rate, while Original Pass2 and TeleGen Pass2 report cumulative success after the original and telemetry-augmented second attempts. Easy, Moderate, and Challenging report category-level cumulative success rate. Δ denotes the overall improvement over Initial generation for WebGen-Bench and over Pass1 for Web-Bench. All values are percentages except passed-task counts.
Human +
Human –
Total
WV success
28
2
30
WV failure
4
26
30
Total
32
28
60
Appendix
Table 5: Agreement between WebVoyager (WV) and the independent human annotator on the balanced 60-task audit sample.
Category
Count
Functional repair
24
Mixed functional/judgeability
2
Judgeability-only
4
Appendix
Table 6: Manual categorization of 30 audited TeleGen -only successes.
Setting
Calls
Cost ($)
Time (s)
No-Tel.
24.0
0.28
711
Raw-Tel.
29.9
0.42
747
TeleGen
29.5
0.41
710
Appendix
Table 7: Mean per-project cost and runtime accounting on WebGen-Bench. Cost is API cost under the DeepSeek-V4-Flash pricing used in this analysis; time is recorded repair-stage wall-clock time.
Base
Level
Gain
Loss
Net
No-tel.
Project
48
16
+32
Task
106
51
+55
No brief
Project
34
14
+20
Task
65
56
+9
Appendix
Table 8: Project- and task-level comparison between TeleGen and each baseline. Gain denotes cases where telemetry-guided repair performs better, Loss denotes cases where the baseline performs better, and Net is Gain minus Loss.
Pair (A/B)
A-only
B-only
p
No-Tel./ TeleGen
51
106
1.35×10−5
Raw-Tel./ TeleGen
56
65
0.4672
Orig.P@2/ TeleGen P@2
45
126
4.49×10−10
Appendix
Table 9: Two-sided exact McNemar tests on paired task outcomes. A-only and B-only denote tasks solved exclusively by the first and second settings in each pair.
Web application generation uses AI to turn natural-language, visual, or multimedia specifications into functional, interactive websites and applications.