OSExpert: Computer-Use Agents Learning Professional Skills via Exploration

Jiateng Liu; Zhenhailong Wang; Rushi Wang; Bingxuan Li; Jeonghwan Kim; Aditi Tiwari; Pengfei Yu; Denghui Zhang

cs.AIMar 9, 2026

OSExpert: Computer-Use Agents Learning Professional Skills via Exploration

Authors: Jiateng Liu, Zhenhailong Wang, Rushi Wang, Bingxuan Li, Jeonghwan Kim, Aditi Tiwari, Pengfei Yu, Denghui Zhang,

Organizations: University of Illinois Urbana-Champaign · Stevens Institute of Technology

Abstract

General-purpose computer-use agents have shown impressive performance across diverse digital environments. However, our new benchmark, OSExpert-Eval, indicates they remain far less helpful than human experts. Although inference-time scaling enables adaptation, these agents complete complex tasks inefficiently with degraded performance, transfer poorly to unseen UIs, and struggle with fine-grained action sequences. To solve the problem, we introduce a GUI-based depth-first search (GUI-DFS) exploration algorithm to comprehensively explore and verify an environment's unit functions. The agent then exploits compositionality between unit skills to self-construct a curriculum for composite tasks. To support fine-grained actions, we curate a database of action primitives for agents to discover during exploration; these are saved as a skill set once the exploration is complete. We use the learned skills to improve the agent's performance and efficiency by (1) enriching agents with ready-to-use procedural knowledge, allowing them to plan only once for long trajectories and generate accurate actions, and (2) enabling them to end inference-time scaling earlier by realizing their boundary of capabilities. Extensive experiments show that our environment-learned agent takes a meaningful step toward expert-level computer use, achieving a around 20 percent performance gain on OSExpert-Eval and closing the efficiency gap to humans by around 80 percent

Figures & tables

Figure 1: Our OSExpert-Eval shows that current computer-use agents remain far from expert-level: they struggle with long-horizon tasks, generalize poorly to unseen UI designs, lack fine-grained control over action sequences, and still fall well short of human expert efficiency.

Real-World Benchmark / Evaluation Dimensions	Long-Horizon Composite Skills	Creative UI Generalization	Fine-Grained UI Interaction	Latency Evalaution
Mind2Web ( Deng et al., 2023 )	✓	✗	✗	✗
BEARCUBS ( Song et al., 2025b )	✓	✗	✗	✗
OSWorld ( Xie et al., 2024a )	✗	✗	✗	✗
WinAgentArena ( Bonatti et al., 2024a )	✓	✗	✗	✗
macOSWorld ( Yang et al., 2025 )	✓	✗	✗	✗
OSWorld-Human ( Abhyankar et al., 2025 )	✗	✗	✗	✓

Table 1: Comparison of web-based agents and computer-use agent benchmarks. Most mainstream benchmarks do not explicitly evaluate composite skills, generalization to creative UI designs, fine-grained low-level interaction sequences, or latency & efficiency evaluation.

Figure 2: Up: Current general-purpose computer-use agents rely on inference-time scaling, yet remain prone to failures and high latency. Left: Prior approaches explore digital environments using human-curated queries or tutorial-derived queries, which are often unavailable or difficult to obtain for arbitrary environments. Right: Our framework does not require external data or human effort for exploration queries and more comprehensively discover the unit functions of the digital environment, and benefits both performance and efficiency. We introduce how we handle the fine-grained actions during the exploration and how we organize the learned skill set in Figure 3 .

Figure 3: Left: How our framework organizes and utilize the self-constructed skill set for robust and efficient inference. The unit functions are obtained from the terminal states as shown in Figure 2 . Right: How we handled potential fine-grained actions during exploration stage. The fine-grained action handling is usually triggered by an error state in the exploration, as shown in Figure 2 . The selected primitive fine-grained action template will be added to the skill set for solving future queries if verified helpful.

General CUAs	Agent Type	Long-Horizon Composite Skills		Unseen UI Generalization		Fine-Grained Action Execution
General CUAs	Agent Type	GIMP	LibreOffice	Tableau	MiniWord	GIMP	LibreOffice
OpenCUA-7B ( Wang et al., 2025b )	Specialized Model	0.00 $\pm$ 0.00	0.00 $\pm$ 0.00	0.02 $\pm$ 0.03	0.00 $\pm$ 0.00	0.07 $\pm$ 0.00	0.05 $\pm$ 0.00
Computer-Use-Preview ( OpenAI, 2025 )	Specialized Model	0.00 $\pm$ 0.00	0.04 $\pm$ 0.00	0.00 $\pm$ 0.00	0.00 $\pm$ 0.00	0.00 $\pm$ 0.00	0.00 $\pm$ 0.00
Qwen-3-VL-8B ( Bai et al., 2025 )	General Model	0.00 $\pm$ 0.00	0.00 $\pm$ 0.00	0.00 $\pm$ 0.00	0.00 $\pm$ 0.00	0.07 $\pm$ 0.00	0.10 $\pm$ 0.00
CoAct-1 ( Song et al., 2025a )	Agentic Framework	0.00 $\pm$ 0.00	0.08 $\pm$ 0.00	0.05 $\pm$ 0.00	0.06 $\pm$ 0.02	0.00 $\pm$ 0.00	0.05 $\pm$ 0.00
Agent-S3 w/ GPT-5 ( Gonzalez-Pumariega et al., 2025 )	Agentic Framework	0.06 $\pm$ 0.10	0.08 $\pm$ 0.00	0.03 $\pm$ 0.03	0.02 $\pm$ 0.02	0.00 $\pm$ 0.00	0.10 $\pm$ 0.00

Table 2: Performance comparison on OSExpert-Eval using average success rate (

\uparrow

) . Our agent acquires expert-level skills for long-horizon tasks, generalizes effectively to unseen UIs, and executes fine-grained actions reliably.

General CUAs	Agent Type	Long-Horizon Composite Skills		Unseen UI Generalization		Fine-Grained Action Execution
General CUAs	Agent Type	GIMP	LibreOffice	Tableau	MiniWord	GIMP	LibreOffice
OpenCUA-7B ( Wang et al., 2025b )	Specialized Model	169 $\pm$ 5	193 $\pm$ 11	216 $\pm$ 6	235 $\pm$ 2	162 $\pm$ 7	173 $\pm$ 3
Computer-Use-Preview ( OpenAI, 2025 )	Specialized Model	252 $\pm$ 15	240 $\pm$ 9	361 $\pm$ 11	296 $\pm$ 2	256 $\pm$ 14	282 $\pm$ 7
Qwen-3-VL-8B ( Bai et al., 2025 )	General Model	139 $\pm$ 2	136 $\pm$ 4	135 $\pm$ 4	153 $\pm$ 1	83 $\pm$ 5	138 $\pm$ 3
CoAct-1 ( Song et al., 2025a )	Agentic Framework	1260 $\pm$ 58	1104 $\pm$ 73	924 $\pm$ 31	857 $\pm$ 61	911 $\pm$ 21	782 $\pm$ 27
Agent-S3 w/ GPT-5 ( Gonzalez-Pumariega et al., 2025 )	Agentic Framework	998 $\pm$ 21	1231 $\pm$ 44	623 $\pm$ 55	628	702 $\pm$ 34	633 $\pm$ 19

Table 3: Efficiency comparison on OSExpert-Eval , measured by average execution time (seconds,

\downarrow

) . Our agent substantially narrows the efficiency gap between computer-use agents and human experts.

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Model	GIMP	Calc	Impress	Writter
Base Agent	0.269	0.447	0.404	0.304
OSExpert	0.308	0.447	0.426	0.347

Appendix

Table 4: Performance Comparison between the Base Computer Use Agent with Qwen-3-VL-8B and the our agent equipped with Exploration results on four OSWorld subsets using Pass@1 Success Rate (

\uparrow

Figure 4: Composition of our evaluation tasks (113 total). The inner ring shows the three high-level categories (Unseen UI, Fine-Grained, and Long Horizon), while the outer ring breaks each category down by environment (Tableau, MiniWord, Office, and GIMP); slice sizes are proportional to the number of tasks. Here, Office includes LibreOffice Writer, LibreOffice Impress, and LibreOffice Calc.

Figure 5: Representative examples from OSExpert-Eval across three task categories. The figure aggregates six examples illustrating the breadth of professional computer-use skills evaluated in our benchmark. Top (Long-Horizon Compositional Workflows) : multi-step tasks in LibreOffice Calc and Writer that require composing several unit operations in the correct order, including spreadsheet completion with interface adjustments (e.g., zoom) and document-wide formatting with image insertion. Middle (Fine-Grained Action Execution) : precise image-editing tasks in GIMP, including tightly cropping to a specified 200×200 region and performing accurate background removal while preserving object integrity. For each example, we show the initial environment state, the natural-language instruction, and the corresponding ground-truth outcome. Bottom (Unseen UI Generalization) : tasks in Tableau and MiniWord that test transfer to unfamiliar interfaces and interaction patterns, such as building a world map visualization with sales-based coloring and category filtering, and importing external text followed by image insertion and alignment in a novel editor layout.

Computer-Use Agents

Computer-use agents perceive and operate graphical interfaces and software tools to complete tasks across websites, apps, and desktops.

See all 160 papers

OSWorld 2.0: Benchmarking Computer Use Agents on Long-Horizon Real-World Tasks
Jun 28, 2026Mengqi Yuan, Zilong Zhou, Xinzhuang Xiong +33Computer-Use Agent Benchmarks Computer-Use Agents
XLANG Lab
Step-level Optimization for Efficient Computer-use Agents
Apr 29, 2026Jinbiao Wei, Kangqi Ni, Yilun Zhao +2Computer-Use Agents AI Agent Monitoring
Y Yale NLP Lab · CUniversity of North Carolina at Chapel Hill
CUA-Universe: A Scalable and Dynamic Environment for Hybrid GUI+CLI Agents
Sep 4, 2026Haoting Shi, Wenhao Wang, Weicheng Fang +6Computer-Use Agents Tool-Using Agents
Shanghai Jiao Tong University · Zhejiang University

Jun 28, 2026J/K move · Enter open · S save

OSWorld 2.0: Benchmarking Computer Use Agents on Long-Horizon Real-World Tasks

Mengqi Yuan, Zilong Zhou, Xinzhuang Xiong, Weiming Wu+32

XLANG Lab

Computer-Use Agent Benchmarks Computer-Use Agents Long-Horizon Agent Evaluation Long-Horizon Agent Tasks Computer-Use Agent Evaluation

Open paper AskPDF arXiv

Existing computer-use benchmarks fail to capture the realism, complexity, and long-horizon demands of real-world computer use, limiting their ability to reveal the limitations of frontier agents. We introduce OSWorld 2.0, a benchmark of 108 long-horizon computer-use workflows across everyday and professional tasks, designed to capture complex and challenging real-world phenomena. Each task represents a realistic end-to-end workflow that takes human users a median of about 1.6 hours to complete and requires an average of 318 tool calls with Claude Opus 4.7 using maximum thinking, compared with about 30 in OSWorld 1.0. OSWorld 2.0 targets challenge phenomena that are common in real workflows yet underrepresented in prior benchmarks, spanning interaction-design challenges such as streaming interaction and dynamic environments, as well as agent-pattern challenges such as cross-source reasoning, implicit-state inference, and visual-spatial precision. Tasks are grounded in authentic input artifacts and cross-referenced against realistic stateful user profile data, and include separate safety reports auditing safety-sensitive execution. Under our primary binary-completion metric at 500 steps, Claude Opus 4.8 with maximum thinking and batched tool calls scores best but still completes only 20.6% of tasks at a 54.8% partial score; GPT-5.5 is far more token-efficient yet plateaus near 13%. These results show that current agents are still far from professional-level computer use: rather than stumbling on basic GUI control or coding, they lose track of constraints, miss information that arrives mid-task, guess rather than ask the user, and skip verification, struggling most when a task hinges on hidden state they must recover.

Abstract

Computer-Use Agents

Explore similar work

OSWorld 2.0: Benchmarking Computer Use Agents on Long-Horizon Real-World Tasks

Step-level Optimization for Efficient Computer-use Agents

CUA-Universe: A Scalable and Dynamic Environment for Hybrid GUI+CLI Agents