MKD2World: A Typed Spatial MarkDown World Model For Web Agents
Organizations: Guangming Laboratory · Shenzhen University · Southeast University · Tsinghua University ·
Abstract
Graphical user interface (GUI) agents can improve decision-making with world-model foresight, provided that predicted states accurately reflect post-action interfaces. Existing state representations fail to jointly capture interface layout, UI semantics, and explicit locations. We introduce Typed Spatial Markdown (TSM), a Markdown-style textual state representation for desktop web interfaces. TSM serializes block-level and atomic-level UI elements according to their spatial relationships, jointly encoding their types, visual semantics, explicit locations, optional control states, and hierarchical membership. To train a screenshot-to-TSM renderer, we construct AtomBlock-WebUI, a dataset with precise annotations at both granularities. By injecting element-type labels into HTML and directly extracting bounding boxes after rendering, we avoid annotation errors caused by post-hoc inference from webpage structures. Building on TSM, we propose MKD2World, a desktop web world model trained in two stages for current-state description and action-conditioned next-state prediction. During inference, the web agent queries MKD2World as needed and performs a ``digest'' step, in which it interprets each predicted state and extracts task-relevant evidence for selecting the best action. Experiments show that MKD2World-9B achieves an EleF1 score of 26.50 on the out-of-domain test set, outperforming GPT-5.6-Sol(14.98) by 11.52 points. Across Online-Mind2Web tasks, MKD2World-9B improves the success rate of the Qwen3.5-35B web agent from 22.94% to 37.61%, achieving competitive performance among leading open-source GUI world models. Code is available at https://github.com/OmniAI-Lab-CUHKSZ/MKD2World.
Figures & tables
| Dataset | VLM judge | Human | ||||||
| Obj. Val. | Bound. Align. | Label Sem. | Complexity | Obj. Val. | Bound. Align. | Label Sem. | Complexity | |
| ScreenParse-v2 | 95.93 | 85.87 | 96.75 | 53.39 | 90.00 | 89.25 | 91.25 | 55.75 |
| WebUI-CHI23 | 90.00 | 83.68 | 97.36 | 52.91 | 93.25 | 95.25 | 84.25 | 60.00 |
| WebUI-COCO | 99.57 | 93.99 | 84.26 | 71.45 | 99.25 | 98.00 | 78.25 | 73.00 |
| UI-Elements | 82.91 | 89.02 | 94.71 | 62.54 | 71.00 | 90.50 | 85.00 | 67.75 |
| AtomBlock-Top (Ours) | 98.82 | 96.52 | 99.62 | 74.14 | 100.00 | 97.50 | 94.50 | 81.00 |
| Split | Model | Click | Text/Key | Scroll | Navigation | Other | Overall | Format Success |
| ID | GPT-5.6-Sol | 16.42 | 19.16 | 12.67 | 7.14 | 17.52 | 14.58 | 96.70 |
| Qwen3.7-Plus | 19.74 | 22.99 | 12.75 | 7.97 | 21.92 | 17.08 | 92.30 | |
| GLM-5V-Turbo | 17.59 | 21.19 | 12.11 | 8.06 | 20.63 | 15.92 | 54.80 | |
| MiniMax-M3 | 13.31 | 17.41 | 8.86 | 6.78 | 14.60 | 12.19 | 82.40 | |
| MKD2World (Ours) | 29.94 | 38.02 | 19.40 | 13.41 | 39.69 | 28.09 | 97.90 | |
| OOD | GPT-5.6-Sol | 15.02 | 15.88 | 12.16 | 11.22 | 20.63 | 14.98 | 96.20 |
Appendix figures & tables10 assets
Supplementary material from the paper’s appendix.
Appendix
| ID | Category | Granularity | Instances | Proportion (%) |
| 0 | button | Atomic | 113,089 | 8.56 |
| 1 | link | Atomic | 626,321 | 47.40 |
| 2 | input | Atomic | 18,520 | 1.40 |
| 3 | image | Atomic | 184,878 | 13.99 |
| 4 | icon | Atomic | 185,215 | 14.02 |
| 5 | checkbox | Atomic | 42,887 | 3.25 |
| UI Element | TSM Representation |
| Text | content<cx,cy,w,h> |
| Link | content<cx,cy,w,h> |
| Button | content<cx,cy,w,h> |
| Input | content<cx,cy,w,h> |
| Icon | icon:semantics<cx,cy,w,h> |
| Image | img:semantics<cx,cy,w,h> |
| text | link | button | image | icon | toggle | select | input | |
| text | 1.0 | 0.5 | 0 | 0 | 0 | 0 | 0 | 0 |
| link | 0.5 | 1.0 | 0.5 | 0.5 | 0.5 | 0 | 0 | 0 |
| button | 0 | 0.5 | 1.0 | 0.5 | 0.5 | 0 | 0 | 0 |
| image | 0 | 0.5 | 0.5 | 1.0 | 0.5 | 0 | 0 | 0 |
| icon | 0 | 0.5 | 0.5 | 0.5 | 1.0 | 0 | 0 | 0 |
| toggle | 0 | 0 | 0 | 0 | 0 | 1.0 | 0 | 0 |
| Split | Samples | changed | near_noop | exact_noop |
| Stage 2 | 31,245 | 25,007 | 4,296 | 1,942 |
| Validation | 500 | 450 | 45 | 5 |
| ID test | 1,000 | 900 | 90 | 10 |
| OOD test | 1,000 | 900 | 90 | 10 |
| Split | Samples | Unique URLs | Max URL Frequency | Domains | Max Samples per Domain |
| Stage 1 | 100,000 | 17,367 | 762 | 886 | 1,864 |
| Stage 2 | 31,245 | 4,776 | 953 | 727 | 1,771 |
| Validation | 500 | 500 | 1 | 441 | 5 |
| ID Test | 1,000 | 1,000 | 1 | 441 | 10 |
| OOD Test | 1,000 | 1,000 | 1 | 303 | 31 |
| Split | click | text_key | scroll | navigation | other |
| Stage 2 | 35.40 | 29.64 | 23.79 | 8.13 | 3.04 |
| Validation | 20.00 | 20.00 | 20.00 | 20.00 | 20.00 |
| ID Test | 20.00 | 20.00 | 20.00 | 20.00 | 20.00 |
| OOD Test | 20.00 | 20.00 | 20.00 | 20.00 | 20.00 |
| Action Type | Sub-Action | Validation | In-Domain | Out-of-Domain |
| click | left_click | 15.00 | 15.00 | 15.90 |
| double_click | 2.60 | 2.50 | 1.20 | |
| right_click | 2.40 | 2.50 | 2.90 | |
| text_key | type_text | 6.60 | 6.70 | 9.20 |
| key_press | 6.80 | 6.60 | 9.10 | |
| hotkey | 6.60 | 6.70 | 1.70 |
| # | Task ID | Instruction |
|---|---|---|
| 1 | 0632e496d37badee0350dad358f047c5 | Browse recipes for gluten-free chocolate chip cookies that can be made without nuts. |
| 2 | 16200f51d63f0a47a58fa17acd49e368 | Find a recipe that includes eggplant and mushrooms. |
| 3 | bb314cb80f0f8489135cbf59074d11e2 | Open the page for the first Best Paper Award video recording of talks from ICLR 2016. |
| 4 | 34ccd15a8ea8fd3895af83f5ccf62369 | Find out what to do when I lose an item on a bus on us.megabus. |
| 5 | f00e7accfb4a5e09680bdb326e6274ad | Check the hourly forecast for Boston. |
| 6 | c521933dad9c0ef9f1dfa2f38b8e4405 | See the monthly forecast for Atlanta, GA. |
| # | Task ID | Instruction |
|---|---|---|
| 1 | 0059adc6b12a3822305deb68929b2de8 | Find support services jobs in Bentonville, in the state of Arkansas. |
| 2 | a69d2934fe54fef165490a5a2d95bf38 | Show me recipes for pancakes with wheat and without beetroot. |
| 3 | 5d542a7ec1fa142ba73cc87d970caf39 | Find the most cited publication at the 2022 CVPR main conference. |
| 4 | b922508886ded315c9835457a6eb43ea | Browse tenured/tenure-track faculty positions in Computer Sciences & Technology in California. |
| 5 | 15be05973fba714e490cd9c884e4f072 | Find the procedure to get the license for Athletic Trainer. |
| 6 | d1807551297ac60ecaaabbd2a2ed301a_110325 | Show me a Dallas-based divorce lawyer who was once included in Top 50 Women Texas Super Lawyers list. |
| # | Task ID | Instruction |
|---|---|---|
| 1 | f158345f8489e0d1d91e28768c39bca1 | Estimate the total cost (with basic support) of using 5 million input tokens and 5 million output tokens each for GPT-4o and GPT-4o Mini, both deployed in the US/EU Data Zones under Standard (On-Demand) in the East US region. |
| 2 | d1970c16271496cbbe166ecbecc0a1d8 | I’m 25 and located in Texas. Shop for 2020 made dry red wine made in United States priced between 15-20 dollars and add 5 bottles to the cart. |
| 3 | 1ab384fb3a791edfb410213cc6b82151_110325 | Show me the result of a proton emission decay for a Beryllium nucleus with 4 protons and 2 neutrons in the simulation. |
| 4 | 824eb7bb0ef1ce40bfd49c12182d9428_110325 | Get the women’s black one-piece swimsuit, size large, with the highest discount percentage, and add it to my cart. |
| 5 | a5c87cc1c94a090c9a8dc2c8b6a125d0 | Find the SO2 air quality over the past hour for Maine North, County Cork, Ireland. |
| 6 | a13e4231a3d6a7000c622c56448d97ba | Find an Airbnb in Cleveland for three nights. The check-in date is the day after tomorrow. We have 2 adults, 2 kids, and 1 pet. The budget is 300 per night. Essential amenities include free parking, a washer, and a gym. |
World Model Learning
World model learning builds predictive models of how environments change under actions to support planning, simulation, and decision-making.
See all 449 papers