DeskForge: Dense Supervision from Desktop Environments for Computer-Use Agents

A. Said Gurbuz; Ahmed Nassar; Sunghwan Hong; Marc Pollefeys; Peter W. J. Staar

cs.CVOct 1, 2026

DeskForge: Dense Supervision from Desktop Environments for Computer-Use Agents

Authors: A. Said Gurbuz, Ahmed Nassar, Sunghwan Hong, Marc Pollefeys, Peter W. J. Staar

Organizations: ETH Zurich · IBM Research Zurich · Microsoft

Abstract

Computer-use agents need to reliably ground action targets in complex desktop scenes, where multiple applications, overlapping windows, and visually similar controls compete for attention. Existing training data rarely pair such scenes with dense annotations or vary them in a controlled way. We introduce DeskForge, a controllable desktop environment that composes and explores real applications to generate large-scale supervision for computer-use agents. It varies application states, content, window layout, appearance, and resolution, and fuses screenshots, accessibility trees, and window geometry into dense element annotations while recording the outcome of each executed action. Using this environment, we construct DeskForge-1M, a corpus of 1.2M annotated desktop observations containing 159.7M element instances. We fine-tune four vision-language models on 200K grounding examples drawn from DeskForge-1M. All four improve across held-out desktop conditions and on all five external GUI grounding benchmarks; for Qwen3.5-4B, accuracy increases by 11.51 percentage points on ScreenSpot-Pro and 10.11 points on OSWorld-G. The gains also translate to long-horizon task completion: under a fixed planner, the fine-tuned action models solve more WebArena-Infinity and OpenApps tasks, with Qwen3.5-4B increasing from 31 to 50 of 119 tasks and from 3 to 15 of 100 tasks, respectively. These results show that controllable composition of real desktop environments provides a scalable source of supervision for improving both GUI grounding and long-horizon computer use. The framework code, the dataset, and the fine-tuned model are available from the project page: https://saidgurbuz.github.io/deskforge/

Figures & tables

Figure 1: Overview of DeskForge and DeskForge-1M. (a) A scene specification configures the desktop. (b) Real applications are launched and explored. (c) Screenshots, accessibility trees, and window stacks yield dense annotations. (d) Annotated states and transitions form DeskForge-1M.

Resource	Images	Elements	Dense labels	Multi-app scenes	Configurable Environment	Visibility geometry	Recorded transitions
	Scale		Capabilities
UGround	1.3M	10M	$\times$	$\times$	$\times$	$\times$	$\times$
WinDeskGround	585 †	1,356 ‡	$\times$	✓	$\times$	✓	$\times$
MolmoPoint-GUISyn	36K	2M	✓	$\times$	$\times$	$\times$	$\times$
AgentNet	421K	421K	$\times$	✓	$\times$	$\times$	✓
GroundCUA	55.6K	3.56M	✓	$\times$	$\times$	$\times$	$\times$

Table 1: Scale and capabilities of GUI resources. The DeskForge row combines its environment capabilities with the dataset counts of DeskForge-1M.

Panel of Figure 2 — Figure 2: Appearance variation within DeskForge. Representative captured desktops illustrate differences in application styling, iconography, window decorations, and panel or dock layout.

Figure 3: Corpus composition of DeskForge-1M. Over all 1.21M observations: annotated elements per observation (truncated at 400; maximum 1,161), number of applications, and share of observations with more than

x\%

of elements occluded.

Public VLMs and CUAs
Model	Variant	Held-out condition				Mean
		New Scenes	Theme	App	Resolution	Mean
OS-Atlas-Pro-7B		$47.61$	$47.27$	$45.58$	$32.10$	$43.14$
EvoCUA-8B		$50.49$	$45.59$	$46.65$	$44.93$	$46.91$
UI-TARS-1.5-7B		$67.68$	$64.96$	$66.68$	$61.67$	$65.25$
GUI-Owl-1.5-32B		$70.87$	$62.19$	$66.68$	$64.65$	$66.10$

Table 2: Grounding accuracy on the four held-out conditions of DeskForge-1M (%). Public checkpoints provide reference performance. The lower block directly compares each backbone before and after fine-tuning on DeskForge-1M; fine-tuned results are shown in bold.

Model	Training	SS-Pro	SS-v2	OSWorld-G	UI-Vision	MMB-GUI
Gemma4-E4B	Base	$1.83$	$46.86$	$10.64$	$3.28$	$31.39$
	+ DeskForge-1M	21.95	79.56	44.15	13.87	58.90
UI-R1-3B	Base	$14.48$	$81.92$	$34.04$	$12.09$	$56.84$
	+ DeskForge-1M	27.20	83.96	43.97	15.15	65.41
InternVL3.5-8B	Base	$18.72$	$83.33$	$44.15$	$17.50$	$66.58$
	+ DeskForge-1M	23.78	84.98	44.33	17.55	67.20

Table 3: GUI grounding accuracy on five external benchmarks (%). Rows marked + DeskForge-1M denote fine-tuning on our data.

Model	WebArena-Infinity (119)		OpenApps (100)
Model	Base	+DeskForge-1M	Base	+DeskForge-1M
Gemma4-E4B	$5$	40	$2$	12
UI-R1-3B	$30$	37	$7$	12
InternVL3.5-8B	$37$	44	$4$	11
Qwen3.5-4B	$31$	50	$3$	15

Table 4: Long-horizon task completion with a fixed Qwen3.6-27B planner. Number of solved tasks on the custom 119-task WebArena-Infinity panel and the 100-task OpenApps longer-horizon set. +DeskForge-1M denotes fine-tuning of the model on our data.

Appendix figures & tables20 assets

Supplementary material from the paper’s appendix.

Appendix

Level	Stored information
Element	Type (55 ScreenTag classes) and kind (26 categories, Table 6 ); accessibility role; accessible name, and visible text with its verification status; visible fragments and, where retained, the source extent; occlusion state (none, partial, hidden); owning application and window, stacking index, parent and child links, and reading order; interaction properties (actionable, enabled, focusable, editable, checkable and checked, expandable and expanded, selected) and a click point inside the visible region; an identifier stable within an episode.
Observation	Screenshot and ScreenTag serialization; resolution, appearance preset, panel variant, desktop profile, window layout, and scene seed; launched applications; window stack with titles, extents, and stacking order; element and occlusion counts; near-duplicate and no-op flags; split and generator provenance.
Transition	Before and after observations; click point; target identifier, role, name, text, and box in the before frame; element-level effects (appeared, disappeared, moved, text or state changed, newly occluded, revealed) with an aggregate change magnitude; training eligibility and exclusion reasons.

Appendix

Table 5: Information stored for each element, observation, and transition. Element records and ScreenTag accompany every screenshot; transition fields link consecutive observations of an interaction episode.

Group	Kinds
Structure	window, titlebar, menubar, toolbar, statusbar, sidebar
Navigation	menu, menuitem, tab, link
Controls	button, toggle, textinput, select, slider, scrollbar, progress
Content	text, heading, image, icon, canvas
Collections	list, listitem, table, cell

Appendix

Table 6: Element kinds. Each element also retains its accessibility role and one of 55 ScreenTag classes. Kinds merge classes that are not visually distinguishable; for example, check boxes, radio buttons, switches, and toggle buttons form toggle , whose state is stored separately.

Split	Observations	Element instances	Click transitions	Grounding examples
Train	999,494	126,746,711	760,850	200,000
Validation	11,279	1,448,957	8,667	6,281
New Scenes	22,550	2,826,805	17,383	12,589
App	43,206	8,151,157	32,848	22,729
Theme	51,627	7,991,478	38,671	27,175
Resolution	79,212	12,558,672	58,792	43,210

Appendix

Table 7: DeskForge-1M by split. Element instances refer to the visible, redundancy-reduced view. Each grounding example pairs one instruction with its recorded target; the 200,000 training examples are sampled from 551,651 training-split instructions.

Panel of Figure 4 — Figure 4: Display resolutions in DeskForge-1M. Observation counts and proportions across the seven resolutions. All 1,207,368 observations are included, including successive episode frames.

Panel of Figure 5 — Figure 5: Application inventory and co-occurrence. Each matrix entry averages the within-scene fraction of captures recording both applications; bars show the corresponding marginal application presence. The analysis covers 1,175,179 captures from 315,705 scenes with nonempty application records. Application presence is taken from capture metadata, not from visible-window counts. † Reserved applications.

Setting	Value
Training examples	200,000 single-target grounding examples
Hardware	Eight NVIDIA H100 80GB GPUs for the distributed runs
Training duration	One epoch; 3,125 optimizer updates
Effective batch size	64: $8$ GPUs $\times$ $4$ examples/GPU $\times$ $2$ accumulation steps
Precision	BF16
LoRA	Rank 8; $\alpha=32$ ; dropout 0.05

Appendix

Table 8: Common VLM fine-tuning settings. Settings for the 200K grounding recipe; image processing and output conventions are model-specific (Table 9 ).

Model	Image preparation	Limit	Decoding and output
Qwen3.5-4B	2,097,152-pixel cap; Lanczos resize	128	Greedy, thinking disabled; action program with fractional coordinates
Gemma4-E4B	Bilinear resize at the same pixel cap; 1,120 native image tokens	128	Greedy, thinking disabled; action program with fractional coordinates
InternVL3.5-8B	Up to 13 image patches (3,328 image tokens)	512	Greedy, thinking disabled; action program
UI-R1-3B	2,097,152-pixel cap; native factor-28 resize	1,024	Native reasoning prompt and <answer> action list

Appendix

Table 9: Inference settings for the matched grounding pairs. Each row applies to the corresponding base and fine-tuned checkpoint. Output limits count generated tokens.

Assessment	Sampling	Reviewed	Correct / sound (%)
Element correctness	Uniform, 140 screens	16,487 elements	99.84
Element correctness	Enriched, 200 screens	25,466 elements	99.88
Instruction soundness	Uniform	180 instructions	97.8
Instruction soundness	Enriched	300 instructions	94.7

Appendix

Table 10: Human assessment of element annotations and instructions. Uniform samples describe their stated sampling populations. Enriched samples include additional examples selected for stratum coverage.

Model	Desktop	Mobile	Web	Overall
Qwen3.5-4B base	77.25	82.44	79.63	80.11
Qwen3.5-4B + DeskForge	91.62	93.21	86.04	90.33
UI-R1-3B base	78.14	88.42	77.35	81.92
UI-R1-3B + DeskForge	83.83	89.02	78.26	83.96

Appendix

Table 11: ScreenSpot-v2 grounding accuracy by interface domain (%). Bold values indicate improvement over the corresponding starting checkpoint. Overall is the reported full-benchmark score.

Figure 6: Grounding accuracy by scene and target difficulty. (A) Number of applications on screen, on the New Scenes split. (B) Target visible-area loss.

Panel of Figure 7 — Figure 7: Held-out targets missed by public models. Each row shows the full screenshot (left) and a magnified view of the dashed region (right). The green box is the annotated target, red markers are public-model clicks, and the blue dot is the fine-tuned Qwen3.5-4B click. In the second example, GUI-Owl-1.5-32B and Gemma4-31B also hit the target; in the first, Gemma4-31B is omitted because its click falls one pixel outside the target.

Detector	Group F1	Center-hit F1	Mean best IoU
OmniParser v2 detector	58.55	86.24	0.591
ScreenParse YOLO11L	57.59	85.90	0.585
DeskForge RT-DETRv4-L	62.17	87.01	0.611

Appendix

Table 12: Cross-dataset element detection on GroundCUA. F1 scores are percentages; mean best IoU is on a 0–1 scale. Bold, shaded values mark the best result in each column.

Panel of Figure 8 — Figure 8: Element detection on unseen applications. Each row pairs the GroundCUA annotations (left, orange) with predictions of a detector fine-tuned on DeskForge-1M (right, blue).

Panel of Figure 9 — Figure 9: OpenApps trajectories. For each task, the base run (top row) and the fine-tuned run (bottom row) receive the same planner request at the first step shown. Text under each screenshot is the planner’s instruction to the action model; $\times$ and $\circ$ mark the base and fine-tuned clicks.

Panel of Figure 10 — Figure 10: WebArena-Infinity trajectories. Layout as in Figure 9 . Panels on forms and dialogs are cropped for legibility; verifier messages are quoted from the benchmark.

Figure 11: Observations from DeskForge-1M. Twenty-five screenshots spanning all seven appearance presets, five display resolutions, and varied application combinations and window layouts.

Panel of Figure 12 — Figure 12: Representative observations and dense annotations. Each row pairs a desktop screenshot with its retained element annotations, colored by element family.

Figure 13: Detail of a densely annotated region. A magnified crop shows retained element geometry and types, with annotations colored by element family.

Figure 14: An observation and its ScreenTag representation. A search dialog partly covers a notebook page. The 26-line serialization excerpt retains exposed regions as visible fragments and marks occluded portions of text; coordinates use a 0–500 viewport-normalized grid.

Figure 15: Recorded interactions and associated instructions. Each row shows the initial screen, instruction and recorded click, and successor screen. Orange circles mark click locations; rectangles show the recorded target fragment. Coordinates refer to the initial screenshot.

Computer-Use Agents

Computer-use agents perceive and operate graphical interfaces and software tools to complete tasks across websites, apps, and desktops.

See all 160 papers

Jun 15, 2026

ProCUA-SFT Technical Report

Training computer-use agents (CUAs) -- models that interact with graphical desktops through screenshots and keyboard/mouse actions -- requires large-scale, diverse trajectory data collected in full desktop environments. The largest public resource, AgentNet (22.5K human trajectories), leads to negative transfer when used for supervised fine-tuning (SFT): continuing training UI-TARS 7B on AgentNet causes OSWorld success rate to fall from 26.3% to 8-10%. We present ProCUA-SFT, a dataset of 3.1M step-level SFT samples distilled from 93K synthetic trajectories across 2,484 application combinations. The dataset is produced by a fully automated pipeline that (i) synthesizes grounded tasks on live desktops seeded with real-world content -- 912 spreadsheets from SpreadsheetBench, approximately 10K permissively-licensed presentations from Zenodo10K, and multi-application OSWorld configs -- and (ii) verifies each task's feasibility through binary precondition checking before rollout. A single VLM (Kimi-K2.5) serves as goal generator, precondition judge, and trajectory executor, eliminating planner-actor capability gaps. Each trajectory is expanded into step-prefix samples that exactly reproduce the context layout seen at inference time. Fine-tuning UI-TARS 7B on ProCUA-SFT for one epoch yields 45.0% on OSWorld -- an 18.7 percentage-point improvement over the base model and over 35% above AgentNet-trained counterparts. A subset of ProCUA was incorporated into the training data for the Nemotron 3 Nano Omni model, contributing to its computer-use capabilities.

Jaehun Jung, Ximing Lu, Brandon Cui +11

1NVIDIA · University of Washington

Supervised Fine-Tuning Computer-Use Agents LLM Agent Trajectory Synthesis LLM Agent Training

Ask PDF

May 13, 2026

WinDeskGround: A Benchmark for Robust GUI Grounding in Complex Multi-Window Desktop Environments

Multimodal Large Language Models (MLLMs) have revolutionized GUI automation, yet their efficacy is largely established on idealized, single-layer interfaces. This paper identifies a critical reliability gap: state-of-the-art agents face distinct robustness challenges in real-world desktop environments characterized by multi-window stacking, occlusion, and visual clutter. To address this, we introduce WinDeskGround, a novel benchmark and synthesis framework tailored for evaluating GUI grounding robustness. Unlike static datasets, our framework parametrically generates complex desktop scenarios by controlling window occlusion, layout density, and semantic similarity, thereby simulating the distribution shifts of authentic workflows. We construct a diverse meta-dataset of 1,356 high-fidelity instruction-target pairs and conduct comprehensive evaluations of five leading MLLMs. Our results demonstrate that while top-tier agents excel in simplified settings, their accuracy declines under partial occlusion. WinDeskGround provides a valuable benchmark to facilitate the assessment and advancement of GUI agent robustness in realistic environments. The code is available at https://github.com/ZZZhr-1/WinDeskGround.

Haoren Zhao, Tianyi Chen, Zhen Wang

School of Cyberspace, Hangzhou Dianzi University, Hangzhou, China · 2Microsoft

Multimodal Robustness GUI Agents GUI Grounding GUI Agent Benchmarks Multimodal Model Evaluation GUI Automation

Ask PDF

Oct 13, 2026

GroundGUI: GUI Grounding by High-Quality Data and Distillation-to-Reinforcement Learning

Constructing robust GUI agents relies on multimodal visual grounding: the ability to precisely map natural language intents to actionable on-screen components. Despite abundant desktop, web, and mobile benchmarks, high-fidelity resources tailored for diverse environments are surprisingly scarce. To bridge this gap, we introduce the Screen Referring Grounding Dataset (ScreenRef), a comprehensive GUI grounding dataset consisting of GUI screenshots paired with natural language referring expressions and box or point annotations. Spanning desktop, web, and mobile domains, ScreenRef comprises 586K screenshots and over 633K referring expression instances, covering tasks such as text recognition, visual matching, spatial understanding, and refusal scenarios. To boost the model's grounding performance, we propose the Distillation-to-Reinforcement Learning (D2RL) method to address the limited generalization of supervised fine-tuning and the sparse reward problem in reinforcement learning. Utilizing ScreenRef, we train Qwen3-VL-8B-Instruct via sequential SFT and D2RL training, yielding GroundGUI-8B. GroundGUI-8B achieves 95.2% on ScreenSpot-V2, 63.9% on ScreenSpot-Pro, 65.4% on OSWorld-G, 86.8% on MMBench-GUI L2, and 38.3% on UI-Vision. Among 8B-scale models, it produces substantial gains over models trained on open-source datasets, with performance comparable or even superior to models trained on closed-source data. Code, models, and datasets are available at https://github.com/PrentisAI/GroundGUI.

Qingsong Xie, Jiahao Tian, Jiacheng Tang +5

Prentis AI, San Francisco, CA · Westlake University, Hangzhou, China · Titan Holdings, San Francisco, CA

GUI Grounding Multimodal Grounding RL for GUI Agents

Ask PDF

Abstract

Computer-Use Agents

Explore similar work

ProCUA-SFT Technical Report

WinDeskGround: A Benchmark for Robust GUI Grounding in Complex Multi-Window Desktop Environments

GroundGUI: GUI Grounding by High-Quality Data and Distillation-to-Reinforcement Learning