Video-as-Experience: Learning Computer Use via Audio-Visual Understanding
Organizations: University of Cambridge, Cambridge, UK · Tsinghua University, Beijing, China
Abstract
Multimodal large language models (LLMs) have enabled capable computer-use agents that perceive graphical user interfaces, reason over instructions, and execute actions. However, existing approaches to learning from video demonstrations typically convert videos into screenshot-based action trajectories, rather than directly leveraging their rich audio-visual information. We introduce VEX (Video as EXperience), a framework that uses audio-visual understanding to unify learning from external demonstrations and reflecting on an agent's own behavior. Following a watch-act-rewatch paradigm, the agent extracts procedural guidance from tutorial videos containing visual demonstrations and spoken explanations, executes tasks, and verifies its progress by rewatching its execution as video. We instantiate VEX with video-SALMONN-VEX, a 9B-scale audio-visual computer-use agent built on the video-SALMONN 2 architecture with a Qwen3.5-9B backbone. We further introduce Action-Grounded Guide Optimization (AGGO), a reinforcement learning method that optimizes video-derived guidance for downstream action prediction, and augment AgentNet, OSWorld, and CUA-Gym with over 24K real-world tutorial videos. Experiments on OSWorld and a complementary subset of CUA-Gym demonstrate consistent improvements, with video-SALMONN-VEX achieving a 22% relative increase in task success over its Qwen3.5 backbone on OSWorld. Moreover, its generated guidance improves Gemini-3.5-flash by 10% relatively on OSWorld. These results highlight the potential of audio-visual understanding as a unified interface for procedural learning and self-verification in computer-use agents.
Figures & tables
| Task Type | Task Count |
|---|---|
| Office | 94 |
| Daily | 20 |
| Professional | 39 |
| Work | 16 |
| 49 |
| Models | OS | Office | Daily | Professional | Workflow | Overall |
|---|---|---|---|---|---|---|
| Qwen3-VL (8B) [ 3 ] | 35.15 | 28.57 | 46.24 | 65.31 | 20.82 | 36.52 |
| Qwen3.5 (9B) [ 3 ] | 62.50 | 36.72 | 46.80 | 67.35 | 22.77 | 41.15 |
| Qwen3.5 (9B) + Trajectory [ 18 ] | 64.90 | 35.21 | 54.62 | 66.45 | 29.47 | 43.98 |
| video-SALMONN 2 (9B) [ 26 ] | 54.17 | 36.72 | 51.24 | 73.47 | 24.05 | 42.70 |
| video-SALMONN 2 SFT (9B) | 54.17 | 38.45 | 56.95 | 75.51 | 19.45 | 43.46 |
| video-SALMONN 2 SFT (9B) @ 3 trials | 63.19 | 37.14 | 61.68 | 72.45 | 25.20 | 45.65 |
| Models | Office | Daily | Professional | Workflow | Overall | |
|---|---|---|---|---|---|---|
| Qwen3-VL (8B) [ 3 ] | 8.94 | 22.00 | 25.51 | 14.81 | 2.04 | 11.87 |
| Qwen3.5 (9B) [ 3 ] | 12.32 | 36.00 | 28.46 | 15.54 | 12.76 | 18.33 |
| Qwen3.5 (9B) + Trajectory [ 18 ] | 14.37 | 37.91 | 31.72 | 15.47 | 19.07 | 21.25 |
| video-SALMONN 2 (9B) [ 26 ] | 13.16 | 20.00 | 22.44 | 24.17 | 15.41 | 17.23 |
| video-SALMONN 2 SFT (9B) | 15.67 | 23.50 | 25.64 | 22.31 | 17.45 | 19.34 |
| video-SALMONN 2 SFT (9B) @ 3 trials | 17.25 | 30.00 | 31.92 | 15.42 | 13.97 | 20.29 |
| Model | Audio | OSWorld | OpenCUA |
|---|---|---|---|
| Qwen3.5 | ✗ | 43.98 | 21.25 |
| video-SALMONN-VEX | ✗ | 46.42 | 24.16 |
| video-SALMONN-VEX | ✓ | 49.99 | 26.26 |
| Model | Video Tutorial | Self-Verification | Success Rate |
|---|---|---|---|
| vS-VEX | ✓ | ✓ | 49.99 |
| vS-VEX | ✓ | ✗ | 45.58 |
| vS-VEX | ✗ | ✓ | 45.63 |
| vS-VEX | ✗ | ✗ | 43.46 |
| vS-2 SFT | ✓ | ✗ | 38.71 |
| vS-2 SFT | ✗ | ✗ | 43.46 |
| Representation | Model for Verification | PCC | AUROC | Success Rate |
|---|---|---|---|---|
| Screenshot + Text Description | Qwen3.5 | 0.4077 | 0.7663 | 48.59 |
| Video | Gemini-3.5-flash | 0.5416 | 0.7890 | 48.85 |
| Video | video-SALMONN-VEX | 0.5218 | 0.8210 | 49.99 |
| Guidance Generation Model | Method | OSWorld | CUA-Gym | Overall |
|---|---|---|---|---|
| Gemini-3.5-flash | Standard | 61.69 | 47.13 | 56.07 |
| Gemini-3.5-flash | 3 trials | 63.60 | 45.07 | 56.45 |
| Gemini-3.5-flash | VEX (no tutorial) | 67.72 | 50.12 | 60.93 |
| Gemini-3.5-flash | VEX | 67.38 | 52.70 | 61.71 |
| video-SALMONN-VEX | VEX | 68.67 | 52.34 | 62.37 |
| Training Setting | Success Rate | Difference |
|---|---|---|
| AGGO | 49.99% | – |
| AGGO (single LoRA) | 43.72% | % |
| Guidance SFT | 44.94% | % |
Computer-Use Agents
Computer-use agents perceive and operate graphical interfaces and software tools to complete tasks across websites, apps, and desktops.
See all 172 papers