cs.CVNov 4, 2026

Visual Token Routing: Learning to keep tokens that matter

Authors: Pranjal Aggarwal, Sean Welleck

Organizations: Carnegie Mellon University

Abstract

Computer-use agents (CUAs) have recently achieved superhuman performance on several long-horizon benchmarks. However, their computational cost limits their use in long-horizon tasks and real-time domains such as video games. A key bottleneck is representing each screenshot as thousands of visual tokens, which accumulate into long contexts as agents reason over extended interaction histories. We observe that most of these visual tokens are either semantically irrelevant to the agent's next action or temporally redundant across screenshots. Building on this observation, we introduce Visual Token Routing (VTR), a joint training framework for learning sparse visual representations that preserve the information needed for action prediction. In VTR, we train the vision-language model together with a lightweight spatio-temporal router that examines each image region in the current and previous screenshots and decides whether its visual token enters the language model. We optimize the router using an adaptive loss to keep the number of visual tokens as small as possible while minimizing performance degradation. With just 2,000 training steps, VTR matches the performance of the full-resolution model on OSWorld, CUA-World, and video game action prediction, while using 2.5-5.8x fewer visual tokens. These savings reduce inference cost by up to 3.3x and speed up prefill by up to 2.9x. VTR also outperforms lowering the image resolution and other baselines for dropping tokens, by up to 12.8 points on OSWorld. Overall, VTR enables computer-use agents to process only the visual tokens they need, allowing them to be used in long-horizon tasks and real-time domains at a fraction of the computational cost. Code and models are available at https://pranjal2041.github.io/vtr

Figures & tables

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. ReVision: Scaling Computer-Use Agents via Temporal Visual Redundancy Reduction

    May 11, 2026Amirhossein Abaskohi, Yuhang He, Peter West +3Computer-Use AgentsLong-Horizon Agent Tasks

  2. TRIAGE: Efficient Vision-Language-Action Inference Needs Both Reuse and Pruning

    Oct 25, 2026Sooyoung Kwon, Carlee Joe-Wong, Kyunghan LeeEfficient VLA Model InferenceVisual Token Pruning

  3. RUTA: Principled Visual Token Allocation via Rate-Utility Optimization

    Aug 4, 2026Jian Zou, Xiaoyu Xu, Zhihua Wang +3Efficient VLM InferenceVisual Token Pruning