cs.AIOct 9, 2026
SaveWork2Skill: Capturing Human GUI Experience for Reusable Agent Skill Induction
Organizations: Shanghai Jiao Tong University · Fudan University · Nanyang Technological University · Zhejiang University ·
Abstract
Computer-use agents can perform increasingly complex GUI tasks, yet unfamiliar software and workflows requiring human domain knowledge remain challenging. Human demonstrations convey procedural knowledge through the same GUI interaction space, but reuse requires interpreting actions in context and separating workflow rules from task-specific details. We thus introduce Work2Skill, a framework that captures human GUI experience for reusable agent skill induction. WorkRecorder provides the framework's infrastructure for collecting this experience, aligning screen recordings, mouse and keyboard events, and available accessibility-tree information. Using WorkRecorder, we construct WorkDemo-350, the first synchronized multimodal dataset of human desktop GUI demonstrations for reusable agent skill induction and downstream evaluation. Its 350 demonstrations are screened for completeness and task success. We then propose an agentic skill-induction method that inspects evidence on demand to produce reusable skills capturing workflow rules, step dependencies, and completion checks. When the same four execution models are equipped with skills induced by each method under shared execution settings, Work2Skill achieves the highest scores and success rates. Mean score improvements over the corresponding No Skill baseline are 4.67--10.54 points on MyPCBench Bounded Action and 3.11--7.50 points on a 286-task subset of OSWorld-Verified. Our code and data are available at https://github.com/china-qijizhifeng/Work2Skill.
Figures & tables
Figure 1: Work2Skill framework: WorkRecorder, WorkDemo-350, and Agentic Skill Induction.
Figure 2: Overview of Work2Skill. WorkRecorder aligns human GUI actions with visual and available accessibility evidence to build WorkDemo-350. An agent inspects evidence on demand and revises reusable skills for downstream GUI execution. Downstream evaluation compares these skills with guidance from the fixed workflows of ShowUI-Aloha and Demo2Tutorial.
Figure 3: Human GUI experience collection with WorkRecorder. A demonstrated workflow (left) is captured and reviewed through the recorder interface (center), then organized into an evidence episode with task metadata, action traces, and available multimodal evidence (right).
| Statistic | MyPCBench | OSWorld-Verified |
|---|---|---|
| Number of tasks | 64 | 286 |
| Total video duration (s) | 7623.02 | 22890.18 |
| Mean episode duration (s) | 119.11 | 80.04 |
| Mean actions per episode | 32.23 | 34.57 |
| Mean evidence images | 24.20 | 23.91 |
Table 1: Statistics of WorkDemo-350: 350 human GUI demonstrations covering 64 MyPCBench Bounded Action tasks and a 286-task subset of OSWorld-Verified, with aligned multimodal evidence.
Figure 4: Cross-model skill transfer on 64 MyPCBench tasks. Rows denote induction models; columns denote executors. denotes the gain over the same executor’s No Skill baseline, in Score points or SR percentage points, calculated from displayed two-decimal values.
Figure 5: Ablation and pipeline-prompt comparison on 64 MyPCBench tasks using GPT-5.5 with high reasoning effort. In (a), Full Evidence uses all available modalities; UI-Grounded Action Text Only combines recorded actions with target UI descriptions. w/o KeyLogs also removes overlays from indexed images. Score, SR, and Steps measure downstream execution; Tokens reports mean skill-induction token usage per task in units of .
Figure 6: Execution trajectories (top) and rubric outcomes (bottom) for a TableFind reservation with fallbacks, using GPT-5.5 with each method’s extracted guidance. Both baselines leave fallback handling and persistence checks incomplete ( ); Work2Skill satisfies all eight requirements ( ). Appendix F.2 provides the detailed rubric and execution analysis.
Appendix figures & tables13 assets
Supplementary material from the paper’s appendix.
Appendix
| File or directory | Content |
|---|---|
| task.json | Task instruction and associated task metadata. |
| episode.json | File references, recording metadata, and counts. |
| trace/actions.jsonl | Normalized action records in temporal order. |
| trace/keylog.jsonl | Recorded input-event log. |
| video.mp4 | Recording video. |
| summary.md | Textual operation summary. |
Table 4: Files exposed to the induction agent in a converted episode. The first three files form the core structure; the remaining artifacts are included when available.
| Tool | Capability |
|---|---|
| list_directory | List files and subdirectories within an exposed directory. |
| glob | Find files matching a glob pattern within the exposed input or output directory. |
| read_file | Read text and structured evidence with line numbers and pagination. |
| read_visual_evidence | Return an ordered batch of 1–20 images labeled with their source paths. |
| get_video_metadata | Inspect video duration, dimensions, frame rate, codec, and file size. |
| watch_video | Sample timestamped frames over a selected interval, subject to a frame limit. |
Table 5: Implemented induction tools. NexAU additionally provides LoadSkill for loading the bundled authoring policy.
| Method | Retained content | Text format |
|---|---|---|
| No Skill | None | No appended block. |
| Work2Skill | Complete task-matched SKILL.md files, including their frontmatter and instructions. | Each file follows a skill-directory heading. Multiple files are concatenated in the task map’s order, separated by blank lines. |
| ShowUI-Aloha | Each trajectory step’s caption.action text. | A task heading followed by Step 1: ... , Step 2: ... , and subsequent actions in trajectory order. |
| D2T | Tutorial prose after removing frontmatter, image-containing lines, and link destinations. | Original section and step headings and remaining prose; ordinary hyperlink labels are preserved. |
Table 6: Method-specific contents inserted into the shared <task_skills> wrapper at the end of the system prompt. No Skill adds no block.
| Application | Original requirements | Jitter requirements |
|---|---|---|
| Dinoco Airlines | Book an AVP–JFK flight for next Friday evening or night with a left-side seat. | Book an AVP–PHL flight for next Tuesday morning with a right-side seat. |
| OddsMarket | Select the three largest positions by share count; break ties by higher cost basis, then alphabetically. | Select the two smallest positions by share count; break ties by lower cost basis, then alphabetically. |
| SpeedTax | Advance the in-progress current-year return to the review step without submitting it. | Audit the filed 2024 return read-only and report refunds, confirmation number, and accepted status. |
Table 7: Representative changes in the paired MyPCBench Jitter tasks. Descriptions summarize the changed requirements.
Figure 7: A paired SprintBoard task and its evaluation rubric before (left) and after (right) Jitter perturbation. The variant changes the target board and the required card transition, with rubric items revised to match the new requirements.
Figure 8: Category distribution of the 286 retained OSWorld-Verified tasks. The legend gives the task count and percentage for each category.
| Setting | Retained recording evidence | Visual tools |
|---|---|---|
| Full Evidence | Actions with UI annotations, KeyLogs, indexed images, video | Image and video |
| w/o Indexed Images | Actions with UI annotations, KeyLogs, video | Video |
| w/o UI Annotations | Actions with generic captions, KeyLogs, images with UI target annotations removed, video | Image and video |
| w/o KeyLogs | Actions with UI annotations, event frames without added overlays, video | Image and video |
| Video-only | Raw video | Video |
| UI-Grounded Action Text Only | Ordered text describing recorded actions and their target UI elements | None |
Table 10: Evidence and visual tools available for skill induction in each modality condition. Task instructions and common file-reading and skill-authoring tools remain available. UI annotations denote metadata and descriptions derived from each action’s target element in the AX tree.
| Item | Requirement | Aloha | D2T | Work2Skill |
| R1 | Set the requested date and a table for ten. | Pass | Pass | Pass |
| R2 | Book Cotogna with the requested note. | Pass | Pass | Pass |
| R3 | Open three alternative restaurants. | Fail | Fail | Pass |
| R4 | View each alternative’s detail page. | Fail | Fail | Pass |
| R5 | Save the top two alternatives. | Fail | Fail | Pass |
| R6 | Obtain a reservation confirmation number. | Pass | Pass | Pass |
Table 11: Rubric-level outcomes for the TableFind case in Figure 6 . Requirement descriptions summarize R1–R8; Aloha denotes the adapted ShowUI-Aloha baseline. Scores are the reported evaluator outputs.
Figure 9: Mean tool calls per task during skill induction, using a shared linear color scale. Values are rounded to two decimals; positive values that would round to zero are shown as . Horizontal rules separate tool groups; the bold row reports mean total calls per task. Qwen-9B and Qwen-35B denote Qwen3.5-9B and Qwen3.5-35B-A3B, respectively.
| No. | Task ID | No. | Task ID |
|---|---|---|---|
| 1 | 00fa164e-2612-4439-992e-157d019a8436 | 2 | 01b269ae-2111-4a07-81fd-3fcd711993b0 |
| 3 | 02ce9a50-7af2-47ed-8596-af0c230501f8 | 4 | 030eeff7-b492-4218-b312-701ec99ee0cc |
| 5 | 0326d92d-d218-48a8-9ca1-981cd6d064c7 | 6 | 035f41ba-6653-43ab-aa63-c86d449d62e5 |
| 7 | 04d9aeaf-7bed-4024-bedb-e10e6f00eb7f | 8 | 0512bb38-d531-4acf-9e7e-0add90816068 |
| 9 | 06ca5602-62ca-47f6-ad4f-da151cde54cc | 10 | 06fe7178-4491-4589-810f-2e2bc9502122 |
| 11 | 0810415c-bde4-4443-9047-d5f70165a697 | 12 | 08aced46-45a2-48d7-993b-ed3fb5b32302 |
Table 12: Complete task IDs for the 286-task OSWorld-Verified subset.
| No. | Task ID | No. | Task ID |
|---|---|---|---|
| 97 | 4c26e3f3-3a14-4d86-b44a-d3cedebbb487 | 98 | 4d117223-a354-47fb-8b45-62ab1390a95f |
| 99 | 4e60007a-f5be-4bfc-9723-c39affa0a6d3 | 100 | 4ed5abd0-8b5d-47bd-839f-cacfa15ca37a |
| 101 | 4f07fbe9-70de-4927-a4d5-bb28bc12c52c | 102 | 510f64c8-9bcc-4be1-8d30-638705850618 |
| 103 | 51719eea-10bc-4246-a428-ac7c433dd4b3 | 104 | 51b11269-2ca8-4b2a-9163-f21758420e78 |
| 105 | 5203d847-2572-4150-912a-03f062254390 | 106 | 535364ea-05bd-46ea-9937-9f55c68507e8 |
| 107 | 53ad5833-3455-407b-bbc6-45b4c79ab8fb | 108 | 550ce7e7-747b-495f-b122-acdc4d0b8e54 |
Table 13: Complete task IDs for the 286-task OSWorld-Verified subset (continued).
| No. | Task ID | No. | Task ID |
|---|---|---|---|
| 193 | a10b69e1-6034-4a2b-93e1-571d45194f75 | 194 | a1af9f1c-50d5-4bc3-a51e-4d9b425ff638 |
| 195 | a434992a-89df-4577-925c-0c58b747f0f4 | 196 | a462a795-fdc7-4b23-b689-e8b6df786b78 |
| 197 | a4d98375-215b-4a4d-aee9-3d4370fccc41 | 198 | a503b07f-9119-456b-b75d-f5146737d24f |
| 199 | a53f80cd-4a90-4490-8310-097b011433f6 | 200 | a669ef01-ded5-4099-9ea9-25e99b569840 |
| 201 | a728a36e-8bf1-4bb6-9a03-ef039a5233f0 | 202 | a74b607e-6bb5-4ea8-8a7c-5d97c7bbcd2a |
| 203 | a82b78bb-7fde-4cb3-94a4-035baf10bcf0 | 204 | a96b564e-dbe9-42c3-9ccf-b4498073438a |
Table 14: Complete task IDs for the 286-task OSWorld-Verified subset (continued).
More in this topic
GUI Agents
Graphical user interface (GUI) agents perceive and operate apps through screens, controls, and other software interfaces.
See all 177 papers