Work2Skill: Capturing Human GUI Experience for Reusable Agent Skill Induction
- 1Shanghai Jiao Tong University
- 2Fudan University
- 3Nanyang Technological University
- 4Zhejiang University
- 5Shanghai Qiji Zhifeng Co., Ltd
Introducing Work2Skill
Human experience.
Reusable skills.
Record human workflows, inspect their multimodal evidence, and turn the procedure into reusable guidance for computer-use agents.
Explore the contributionsan agent capability.
*Reported mean-score improvement over No Skill, on a 0–100 scale. See evaluation settings ↗
Three contributions.
From experience to skill.
Unfamiliar software requires procedural knowledge that a task request rarely spells out. Work2Skill captures that knowledge in human demonstrations and turns it into skills an agent can use.
-
RECORDING INFRASTRUCTURE
WorkRecorder
Make demonstrated actions interpretable.
Connect each action to its visual and UI context.
A click location alone leaves the target and its effect ambiguous. WorkRecorder links screen video, keyframes, mouse and keyboard events, and available accessibility information on a shared timeline.
What this enables An agent can inspect a control’s meaning and the state changes around an operation.
-
SHARED DEMONSTRATION RESOURCE
WorkDemo-350
Study skill induction on the same human experience.
Provide 350 desktop demonstrations with linked evidence.
The dataset packages 64 MyPCBench Bounded Action tasks and a selected 286-task OSWorld-Verified subset, with recordings, action traces, input logs, and evidence images.
What this enables Methods can use the same source episodes and be compared under shared downstream execution settings.
-
SKILL-INDUCTION METHOD
Agentic skill induction
Recover a procedure that can be reused.
Interleave evidence inspection with skill writing.
The induction agent chooses relevant files, images, and video intervals, then revisits evidence as questions arise. It generalizes instance-specific inputs while preserving workflow rules, step dependencies, and completion checks.
What this produces A reusable
SKILL.mdthat guides execution without access to the original recording.
The release includes recording and induction components plus task definitions; full execution runners and evaluation integration are still pending.
Meet WorkDemo-350.
Human desktop workflows,
with the evidence attached.
Task descriptions, videos, action traces, raw input logs, and screenshot evidence, organized into 350 episodes.
The episode at a glance.
Session metadata, recording duration, event counts, and relative paths to the task, video, summary, trace, and evidence files. This episode records 54.175 seconds, 15 actions, and 11 evidence images.
episode.json
Dataset scope & provenance
The released dataset card documents a post hoc OSWorld selection: 286 of 321 recorded tasks were retained after excluding 35 tasks where GPT-5.5 scored 1 without skills but less than 1 with v6 skills. This selection affects estimates of improvement; the subset is not the full official OSWorld-Verified test set.
The release is an offline demonstration resource. It does not include VM images or a complete evaluator. See the dataset card for selection, annotation, privacy, and licensing details.
Better execution
with induced skills.
Across four execution models and both evaluated task sets, Work2Skill has the highest reported mean scores and success rates among the compared methods. Shared execution settings isolate the effect of the supplied guidance.
Ranges span the four executors. Scores use a 0–100 scale; OSWorld results are specific to the selected subset.
MyPCBench · Mean evaluation score
0–100 · higher is better64 MyPCBench Bounded Action tasks. The same model induces skills and executes tasks. Results from Table 2.
View exact values
| Execution model | No Skill | ShowUI-Aloha | Demo2Tutorial | Work2Skill | Gain vs. No Skill |
|---|---|---|---|---|---|
| GPT-5.5 | 74.73 | 77.48 | 75.25 | 85.27 | +10.54 |
| Opus-4.8 | 81.53 | 79.27 | 81.28 | 86.20 | +4.67 |
| Qwen3.5-35B-A3B | 38.38 | 37.69 | 40.61 | 44.56 | +6.18 |
| Qwen3.5-9B | 24.27 | 26.61 | 27.75 | 34.11 | +9.84 |
Score is mean evaluation credit (0–100); success rate is the percentage of tasks receiving full credit. Gains use score points or percentage points, respectively. OSWorld results apply to the selected 286-task subset; see provenance. Execution steps and induction costs vary by model; higher scores do not imply uniformly lower cost.
Citation
@misc{huang2026work2skill,
title={Work2Skill: Capturing Human GUI Experience for Reusable Agent Skill Induction},
author={Yuqi Huang and Hui Li and Yifei Cao and Zikun Cai and Tianci Xie and
Yangfan Gao and Pinxue Guo and Chaoyun Zhang and Bingxuan Li and
Rui Zheng and Hang Yan and Ning Liao},
year={2026},
eprint={2610.12984},
archivePrefix={arXiv},
primaryClass={cs.AI},
url={https://arxiv.org/abs/2610.12984}
}