Adaptive Vision-Language Model Routing for Computer Use Agents

Xunzhuo Liu; Bowei He; Xue Liu; Andy Luo; Haichen Zhang; Huamin Chen

cs.CLMar 13, 2026

Adaptive Vision-Language Model Routing for Computer Use Agents

Authors: Xunzhuo Liu, Bowei He, Xue Liu, Andy Luo, Haichen Zhang, Huamin Chen

Organizations: vLLM Semantic Router Project · MBZUAI · McGill University · AMD · Red Hat

Abstract

Computer Use Agents (CUAs) translate natural-language instructions into Graphical User Interface (GUI) actions such as clicks, keystrokes, and scrolls by relying on a Vision-Language Model (VLM) to interpret screenshots and predict grounded tool calls. However, grounding accuracy varies dramatically across VLMs, while current CUA systems typically route every action to a single fixed model regardless of difficulty. We propose Adaptive VLM Routing (AVR), a framework that inserts a lightweight semantic routing layer between the CUA orchestrator and a pool of VLMs. For each tool call, AVR estimates action difficulty from multimodal embeddings, probes a small VLM to measure confidence, and routes the action to the cheapest model whose predicted accuracy satisfies a target reliability threshold. For warm agents with memory of prior UI interactions, retrieved context further narrows the capability gap between small and large models, allowing many actions to be handled without escalation. We formalize routing as a cost--accuracy trade-off, derive a threshold-based policy for model selection, and evaluate AVR using ScreenSpot-Pro grounding data together with the OpenClaw agent routing benchmark. Across these settings, AVR projects inference cost reductions of up to 78% while staying within 2 percentage points of an all-large-model baseline. When combined with the Visual Confused Deputy guardrail, AVR also escalates high-risk actions directly to the strongest available model, unifying efficiency and safety within a single routing framework. Materials are also provided Model, benchmark, and code: https://github.com/vllm-project/semantic-router.

Figures & tables

Panel of Figure 1 — Figure 1: The CUA action loop. Each iteration requires a full VLM inference call carrying screenshots (2–5K tokens each) plus context. A 20-step task accumulates ${\sim}$ 400K input tokens.

Panel of Figure 2 — Figure 2: AVR architecture. The semantic router intercepts CUA tool calls, classifies difficulty via the multimodal embedder, probes the small VLM for confidence, injects memories for warm agents, and routes to the cheapest sufficient model. The contrastive KB (from the Visual Confused Deputy guardrail) can force escalation for high-risk actions.

Panel of Figure 3 — Figure 3: AVR routing decision flowchart. Each tool call passes through safety check, difficulty classification, and confidence probing. Easy actions skip the probe entirely; risky actions go directly to the large model with guardrail verification. Memory injection (dashed) augments the probe for warm agents.

Panel of Figure 4 — Figure 4: Three-tier routing policy (illustrative traffic shares from warm-agent projection). Most actions stay on the cheap small VLM. Hard or uncertain actions escalate to the large VLM. Safety-flagged actions go to the large VLM with additional guardrail verification.

Category	Model	Params	Acc. (%)	Source
Generalist	GPT-4o	$\sim$ 1.8T $\dagger$	0.8	( Li et al., 2025 )
	InternVL2.5-78B	78B	11.5	( Li et al., 2025 ; Chen et al., 2025b )
	CogAgent-18B	18B	7.7	( Li et al., 2025 ; Hong et al., 2024 )
Same family	Qwen2.5-VL-3B	3B	24.2	( Bai et al., 2025b )
	Qwen2.5-VL-7B	7B	29.0	( Bai et al., 2025b )
	Qwen2.5-VL-72B	72B	43.6	( Bai et al., 2025b )

Table 1: VLM grounding accuracy on ScreenSpot-Pro by model category. Accuracy is element-level (correct if the predicted coordinate falls within the target bounding box). “Params” is total parameter count; MoE models list active parameters in parentheses.

Figure 5: Model size vs. grounding accuracy on ScreenSpot-Pro (log scale). Generalist models (squares) cluster near zero regardless of size. Within Qwen2.5-VL family (triangles, dashed line), accuracy grows sublinearly. GUI specialists (circles) at 7B outperform generalists 100 $\times$ their size.

Condition	Turns	On 7B	On 139B	Total Cost	Savings
All on 139B (baseline)	20	0%	100%	$0.0558	—
Routing, cold	20	83%	17%	$0.0173	69%
Routing, warm	20	100%	0%	$0.0080	86%

Table 2: Cost reduction from confidence routing on the OpenClaw agent benchmark. “Warm” agents have accumulated team memories; “Cold” agents start from scratch.

Query	Cold (no memory)	Warm (memory)
Who is my team lead?	0.83	0.96
What backend language?	0.82	0.95
On-call alerting tool?	0.83	0.96
Deploy toolchain?	0.84	0.96
Production database?	0.83	0.96
Mean	0.83	0.96

Table 3: 7B confidence scores for knowledge-dependent agent queries with and without memory injection (threshold = 0.85).

Figure 6: Confidence score distribution for 5 knowledge-dependent queries. Cold agents (blue, left) cluster below the 0.85 threshold, triggering escalation. Warm agents (green, right) cluster well above, keeping all queries on the cheap 7B model. Memory injection creates a clean bimodal separation.

$\ast$ Analytical projection; not measured end-to-end on CUA tasks.
Scenario	Esc. $\alpha$ $\ast$	Eff. Acc. $\ast$	Cost /call $\ast$	Savings
All on 72B	1.0	43.6%	$0.27	—
Cold AVR	0.35	42.1%	$0.13	52%
Warm AVR	0.15	41.3%	$0.08	70%
Warm AVR + difficulty	0.10	42.8%	$0.06	78%

Table 4: Projected AVR cost savings for CUA grounding with a 7B/72B model pool. Escalation rates

\alpha

are estimated from OpenClaw confidence distributions (text tasks), not measured on CUA grounding. Effective accuracy assumes the confidence threshold perfectly identifies the small model’s failures (optimistic). See Appendix E for derivation.

Figure 7: Projected cost per CUA tool call across AVR configurations (analytical, not measured end-to-end). Cold routing (no memory) projects 52% savings; warm routing with difficulty classification projects 78%.

Figure 8: Hypothetical warming curve based on the OpenClaw memory pattern (text tasks) projected to CUA grounding. The logarithmic shape is conjectured from the observation that the first few UI interactions provide the most novel layout information.

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Model	Type	Params	Acc. (%)	Source
GPT-4o	Generalist	$\sim$ 1.8T $\dagger$	0.8	[ Li et al., 2025 ]
Qwen2-VL-72B	Generalist	72B	1.0	[ Li et al., 2025 ]
Qwen2-VL-7B	Generalist	7B	1.6	[ Li et al., 2025 ]
InternVL2.5-78B	Generalist	78B	11.5	[ Li et al., 2025 ]
Qwen2.5-VL-3B	VLM family	3B	24.2	[ Bai et al., 2025b ]
Qwen2.5-VL-7B	VLM family	7B	29.0	[ Bai et al., 2025b ]

Appendix

Table 5: Extended VLM grounding accuracy on ScreenSpot-Pro. “Active” denotes active parameters for MoE architectures.

Figure 9: Per-application accuracy distribution. Dashed lines show Qwen2.5-VL family averages (29.0% for 7B, 43.6% for 72B).

Explore similar work

CUAAudit: Meta-Evaluation of Vision-Language Models as Auditors of Autonomous Computer-Use Agents
Mar 11, 2026Marta Sumyk, Oleksandr KosovanAI Agent Auditing Vision-Language Model-as-a-Judge
Ukrainian Catholic University Lviv, Ukraine
Uncertainty Quantification for Computer-Use Agents: A Benchmark across Vision-Language Models and GUI Grounding Datasets
Jun 24, 2026Divake Kumar, Sina Tayebati, Devashri Naik +5Computer-Use Agent Benchmarks Uncertainty Quantification
University of Illinois Chicago · Intel Labs · Capital One AI Labs
ToolCUA: Towards Optimal GUI-Tool Path Orchestration for Computer Use Agents
May 12, 2026Xuhao Hu, Xi Zhang, Haiyang Xu +6Agentic RL RL for GUI Agents
Tongyi Lab, Alibaba Group · Fudan University · Shanghai Artificial Intelligence Laboratory

Mar 11, 2026J/K move · Enter open · S save

CUAAudit: Meta-Evaluation of Vision-Language Models as Auditors of Autonomous Computer-Use Agents

Marta Sumyk, Oleksandr Kosovan

Ukrainian Catholic University Lviv, Ukraine

AI Agent Auditing Vision-Language Model-as-a-Judge Computer-Use Agents Probability Calibration

Open paper AskPDF arXiv

Computer-Use Agents (CUAs) are emerging as a new paradigm in human-computer interaction, enabling autonomous execution of tasks in desktop environment by perceiving high-level natural-language instructions. As such agents become increasingly capable and are deployed across diverse desktop environments, evaluating their behavior in a scalable and reliable manner becomes a critical challenge. Existing evaluation pipelines rely on static benchmarks, rule-based success checks, or manual inspection, which are brittle, costly, and poorly aligned with real-world usage. In this work, we study Vision-Language Models (VLMs) as autonomous auditors for assessing CUA task completion directly from observable interactions and conduct a large-scale meta-evaluation of five VLMs that judge task success given a natural-language instruction and the final environment state. Our evaluation spans three widely used CUA benchmarks across macOS, Windows, and Linux environments and analyzes auditor behavior along three complementary dimensions: accuracy, calibration of confidence estimates, and inter-model agreement. We find that while state-of-the-art VLMs achieve strong accuracy and calibration, all auditors exhibit notable performance degradation in more complex or heterogeneous environments, and even high-performing models show significant disagreement in their judgments. These results expose fundamental limitations of current model-based auditing approaches and highlight the need to explicitly account for evaluator reliability, uncertainty, and variance when deploying autonomous CUAs in real-world settings.

Abstract

Explore similar work

CUAAudit: Meta-Evaluation of Vision-Language Models as Auditors of Autonomous Computer-Use Agents

Uncertainty Quantification for Computer-Use Agents: A Benchmark across Vision-Language Models and GUI Grounding Datasets

ToolCUA: Towards Optimal GUI-Tool Path Orchestration for Computer Use Agents