cs.AINov 11, 2026

Video-as-Experience: Learning Computer Use via Audio-Visual Understanding

Authors: Guangzhi Sun, Zhonghao Zhai, Yixuan Li, Chao Zhang

Organizations: University of Cambridge, Cambridge, UK · Tsinghua University, Beijing, China

Abstract

Multimodal large language models (LLMs) have enabled capable computer-use agents that perceive graphical user interfaces, reason over instructions, and execute actions. However, existing approaches to learning from video demonstrations typically convert videos into screenshot-based action trajectories, rather than directly leveraging their rich audio-visual information. We introduce VEX (Video as EXperience), a framework that uses audio-visual understanding to unify learning from external demonstrations and reflecting on an agent's own behavior. Following a watch-act-rewatch paradigm, the agent extracts procedural guidance from tutorial videos containing visual demonstrations and spoken explanations, executes tasks, and verifies its progress by rewatching its execution as video. We instantiate VEX with video-SALMONN-VEX, a 9B-scale audio-visual computer-use agent built on the video-SALMONN 2 architecture with a Qwen3.5-9B backbone. We further introduce Action-Grounded Guide Optimization (AGGO), a reinforcement learning method that optimizes video-derived guidance for downstream action prediction, and augment AgentNet, OSWorld, and CUA-Gym with over 24K real-world tutorial videos. Experiments on OSWorld and a complementary subset of CUA-Gym demonstrate consistent improvements, with video-SALMONN-VEX achieving a 22% relative increase in task success over its Qwen3.5 backbone on OSWorld. Moreover, its generated guidance improves Gemini-3.5-flash by 10% relatively on OSWorld. These results highlight the potential of audio-visual understanding as a unified interface for procedural learning and self-verification in computer-use agents.

Figures & tables

Explore similar work

CardsList
  1. Video-Based Reward Modeling for Computer-Use Agents

    Mar 10, 2026Linxin Song, Jieyu Zhang, Huanxin Sheng +6Computer-Use Agent EvaluationReward Modeling

  2. Bridging VideoQA and Video-Guided Agentic Tasks via Generalized Keyframe Extraction

    Jun 28, 2026Sunqi Fan, Qingle Liu, Runqi Yin +2GUI AgentsVideo Question Answering

  3. LAVE: Latent Visual Evidence-Enhanced Planning for Video Tool-use Agents

    Aug 5, 2026Zijian Wang, Junnan Zhu, Rongzhen Li +7Latent Visual ReasoningTool-Using Vision-Language Agents