When Should a VLM Look? Paying Only for Visual Calls That Were Needed and Used

Kunyu Peng; Junming Liu; Ruiqi He; Qingzhuo Wang; Jianzhong Qi; Xianhui Liu

cs.AISep 19, 2026

When Should a VLM Look? Paying Only for Visual Calls That Were Needed and Used

Authors: Kunyu Peng, Junming Liu, Ruiqi He, Qingzhuo Wang, Jianzhong Qi, Xianhui Liu

Organizations: Tongji University · University of Melbourne · Nankai University

Abstract

Vision-language agents that crop and zoom are trained with rewards that credit a successful tool call, yet a successful call does not show that the model needed to look or used the pixels it received. On our cold-start checkpoint only 10% to 12% of visual calls were both needed and used, and released agents make spurious calls 36% to 87% of the time on individual benchmarks. Outcome rewards, judge rewards, and branch probes each observe one side of this failure, and about two thirds of what an outcome reward pays goes to calls that were neither needed nor used. CounterCredit asks both questions of every image-returning call at its realized pre-call state, using the policy's own gold-answer score. A decision value compares the realized visual branch with answering immediately; an evidence value compares the returned crop with random same-size patches substituted into the same call. A call verified on both earns cashback and every other executed call pays rent; the price is bounded so that every correct trajectory outranks every wrong one, and a dual-channel GRPO advantage keeps the price in its own units. From the same cold start, prompt pool, and budget, CounterCredit reaches 89.5% on V*, 80.2% on HR-Bench-4K, and 76.4% on HR-Bench-8K, 6.3 to 9.4 points above outcome-only GRPO at 1.78 against 1.84 calls per question, and lowers the spurious-call rate to 31% to 36%, the lowest among the agents evaluated. The same recipe lifts a Qwen3-VL-8B base from 75.4 to 80.8 on average.

Figures & tables

Figure 1: Overview of spurious visual calls. (a) Spurious-call rates of released visual agents. (b) Share of calls paid by different reward schemes on the cold-start checkpoint, split by whether they were needed and used; the number above each bar is the calls paid per 100 executed. Both panels average V ∗ , HR-Bench-4K, and HR-Bench-8K. (c) Fixed-prefix counterfactuals that separate call necessity from evidence use while holding the realized pre-call state fixed across continuations.

Figure 2: CounterCredit overview . (a) Each visual call is probed from its realized pre-call state with three continuations, ❶ answer now, ❷ real observation, ❸ random observation , $D_{i}$ asks whether the look was needed, $E_{i}$ whether the pixels were used, and a call is verified when both exceed $\varepsilon$ on a correct trajectory. (b) The price adds capped cashback on verified calls and rent on other executed calls. (c) Dual-channel GRPO normalizes the outcome channel and only centers the price channel.

Figure 3: The four call regions by evidence value $E_{i}$ and decision value $D_{i}$ for each visual call.

Closed models (API)
	Perception				General
Model	V ∗	HR-4K	HR-8K	MME-RW	MMStar	ChartQA	BLINK	Avg.
GPT-5	72.3	75.6	73.8	68.9	73.4	76.5	69.7	72.9
Gemini-2.5-Pro	79.1	84.2	81.1	58.7	73.4	83.8	73.6	76.3
Open models without a visual tool
Qwen2.5-VL-7B	77.0	69.1	65.6	58.0	64.4	83.9	56.2	67.7

Table 1: Accuracy (%) on four perception and three general benchmarks, all evaluated under our protocol; Avg. is the mean over the seven. Bold: best per column among tool-using agents; underline: second best. Cell colour: gain over Qwen2.5-VL-7B. The ours row is the run used throughout;

\pm

is the sample standard deviation over three seeds, Table 12 (Appendix F.4 ). Baselines are single runs, and their spread is not measured.

Starting points
	Perception				General				V ∗ calls
Model	V ∗	HR-4K	HR-8K	MME-RW	MMStar	ChartQA	BLINK	Avg.	Calls	Spur. %
Qwen2.5-VL-7B	77.0	69.1	65.6	58.0	64.4	83.9	56.2	67.7	0	–
SFT (cold start)	71.7	70.4	61.6	55.6	60.3	80.8	54.5	65.0	2.68	88.2
Which calls are verified
Outcome-only GRPO	80.1	73.9	69.4	56.8	61.7	86.4	54.3	68.9	1.84	82.1

Table 3: Training stages and reward ablation. All RL rows start from the SFT checkpoint with the same prompt pool and budget and differ only in the reward;

\gamma=0

removes the cashback,

\rho=0

the rent, and the other arms are defined in the text. Avg.: mean of the seven benchmarks; calls and spurious-call rate on V ∗ . The CounterCredit row is the seed shared with every arm; its spread over three seeds is in Tables 1 and 12 .

Figure 5: V ∗ by first-call intervention and by target size for the SFT checkpoint, outcome-only GRPO, and CounterCredit. (a) Trajectories stopped at the first call and answered from the cached prefix, with the two gaps defined in the legend. (b) Accuracy and (c) calls per question over terciles of the target’s share of the image; green labels give CounterCredit’s margin over outcome-only GRPO; values in Tables 9 and 10 (Appendix F.2 ).

Base / Method	V ∗	HR-4K	HR-8K	MME-RW	Avg.
Qwen2.5-VL-7B	77.0	69.1	65.6	58.0	67.7
+ CounterCredit	89.5	80.2	76.4	67.7	76.1
Qwen3-VL-8B	85.9	79.2	74.3	60.7	75.4
+ CounterCredit	91.6	85.4	84.0	70.1	80.8

Table 4: Two base models on the perception benchmarks (accuracy %). Base rows answer directly; + rows apply our cold start and RL. Avg. is over all seven benchmarks; the three general benchmarks are in Table 11 (Appendix F.3 ).

Appendix figures & tables12 assets

Supplementary material from the paper’s appendix.

Appendix

Quantity	Expression	Value
Outcome margin, Proposition 2	$1-\rho\,n_{\max}$	$0.70$
Reward range of a correct trajectory	$[1-\rho\,n_{\max},\,1+C]$	$[0.70,1.70]$
Verifier reliability threshold, (A3)	$\rho/(\rho+\gamma(1-\varepsilon))$	$9.5\%$
Measured chance rate of clearing $\varepsilon$	V ∗ , HR-4K, free-form	$2.4\%$ , $2.5\%$ , $3.0\%$
Worst-case expected price of a chance call at $\alpha=2.5\%$	$\alpha\gamma(1-\varepsilon)-(1-\alpha)\rho$	$-0.037$
Advantage gap, Corollary 1	$\sqrt{7/2}-\rho\,n_{\max}$	$1.57$

Appendix

Table 5: The results at the paper’s constants,

\gamma=0.5

\varepsilon=0.05

\rho=0.05

C=0.7

n_{\max}=6

\sigma_{\min}=0.15

G=8

Figure 6: Null calibration of

\varepsilon

with

K=3

. Leave-one-out null: each random-patch reading stands in for the real observation and is contrasted with the mean of the other patches, so its evidence has zero expectation; the call’s own

D

is kept. Curves give the share of null draws with

\min(E,D)>\varepsilon

. (a) Multiple-choice, 500 training-pool questions probed from the cold start; the V ∗ and HR-Bench-4K curves coincide with it at every grid value, 2.39% and 2.51% at

\varepsilon=0.05

. (b) Free-form, 500 training-pool questions scored by the mean log-probability per token of the reference; the first grid value under 3% is

\varepsilon_{\mathrm{free}}=0.44

. The expected payment of a null draw is negative at every grid value in both panels.

Behaviour	What the policy does	How credit is constrained
Text laundering	names the answer in the call’s arguments	when the call text shifts the real and every random-patch score equally, that common shift cancels in $E_{i}$ (Lemma 1 )
Performative call	the branch changes the answer, the pixels do not	$E_{i}\approx 0$ fails $\min(E_{i},D_{i})>\varepsilon$
Unnecessary look	informative pixels after the answer was already settled	$D_{i}\leq\varepsilon$ fails $\min(E_{i},D_{i})>\varepsilon$
Looping, malformed calls	keeps calling, or never answers	never verified; each executed call pays $\rho$

Appendix

Table 6: Ways to farm credit and the component that blocks each. Interactions between behaviours and gaming of the scoring policy itself are outside this table.

	V ∗				HR-4K				HR-8K
Reward	Paid	Needed	Used	Both	Paid	Needed	Used	Both	Paid	Needed	Used	Both
Outcome reward	40.2	17.4	18.6	12.8	39.2	23.2	22.5	12.4	33.7	31.1	24.0	14.6
Branch probe (TACO)	5.3	61.1	50.0	41.7	5.6	52.5	50.8	41.7	6.7	60.7	57.2	49.7
External judge	23.4	30.2	26.9	21.0	28.6	33.1	30.4	19.3	24.8	32.6	32.3	22.4
Self-judge	19.9	22.1	27.2	14.7	33.0	24.6	33.1	20.9	28.0	29.4	30.5	22.6
Every call paid	100	13.9	15.1	11.8	100	16.1	18.2	9.8	100	18.1	16.5	10.4

Appendix

Table 7: The reward audit of Figure 1 (b) per benchmark, on the SFT checkpoint’s trajectories. Paid: calls credited per 100 executed calls; of those, the share needed (

D_{i}>0.05

), used (

E_{i}>0.05

), and both. The last row pays every call and gives the base rates. CounterCredit is omitted because its verification is defined by the same two values.

Region	Changed (%)
Real	33.3
Performative	2.2
Unnecessary	17.1
Waste	2.0

Appendix

Table 8: Answer-level counterfactual on V ∗ , pooled over the seven baselines. Each executed call is re-decoded once per random patch; changed is the share of draws whose decoded answer differs from the one the real crop produced.

V ∗
Model	No call	Random	Real	Full
SFT (cold start)	75.4	73.8	75.4	71.7
Outcome-only GRPO	77.5	74.7	76.4	80.1
CounterCredit	78.5	74.3	85.3	89.5
HR-Bench-4K
SFT (cold start)	71.6	68.9	71.9	70.4

Appendix

Table 9: Accuracy when the trajectory is stopped at its first tool call and the answer is elicited from the cached prefix, for the arms of Table 3 . No call: the call is removed. Random: the call returns a random same-size patch, averaged over three patches. Real: the call returns its real crop. Full: the complete rollout.

	Smallest		Middle		Largest
Model	Acc.	Calls	Acc.	Calls	Acc.	Calls
SFT (cold start)	65.1	3.33	70.3	2.84	79.7	1.88
Outcome-only GRPO	66.7	2.38	89.1	1.77	84.4	1.39
CounterCredit	82.5	2.19	92.2	1.70	93.8	1.45

Appendix

Table 10: V ∗ by the target’s share of the image area, split into terciles at 0.031% and 0.110% of the image, about 64 questions each: accuracy (%) and calls per question for the same three arms.

Base / Method	V ∗	HR-4K	HR-8K	MME-RW	MMStar	ChartQA	BLINK	Avg.
Qwen2.5-VL-7B	77.0	69.1	65.6	58.0	64.4	83.9	56.2	67.7
+ CounterCredit	89.5	80.2	76.4	67.7	68.9	87.1	62.8	76.1
Qwen3-VL-8B	85.9	79.2	74.3	60.7	70.5	88.4	68.9	75.4
+ CounterCredit	91.6	85.4	84.0	70.1	72.2	90.0	72.3	80.8

Appendix

Table 11: Two base models on all seven benchmarks (accuracy %). Base rows answer directly; + rows apply our cold start and RL; no reward ablation was run on Qwen3-VL-8B.

Seed	V ∗	HR-4K	HR-8K	MME-RW	MMStar	ChartQA	BLINK	Avg.	Calls	Spur. %
0	89.5	80.2	76.4	67.7	68.9	87.1	62.8	76.1	1.78	31.4
1	90.1	80.9	77.2	68.2	68.5	86.8	63.4	76.4	1.84	33.3
2	88.5	79.4	75.4	67.2	69.2	87.4	62.1	75.6	1.72	29.5
mean	89.4	80.2	76.3	67.7	68.9	87.1	62.8	76.0	1.78	31.4
sample SD	0.81	0.75	0.90	0.50	0.35	0.30	0.65	0.40	0.06	1.90

Appendix

Table 12: Three training seeds of CounterCredit, evaluated identically. Seed 0 is the run reported everywhere else in the paper.

	V ∗			HR-4K
Reference	Used %	Agreement %	$r$	Used %	Agreement %	$r$
Default, $K=3$ random patches	15.1	100	1.00	18.2	100	1.00
$K=1$	16.4	94.7	0.94	18.5	94.8	0.96
$K=2$ , leave one out	15.7	97.5	0.98	18.6	96.7	0.99
$K=5$	14.9	97.8	0.99	18.1	97.1	0.99
Gray patch	13.7	90.0	0.84	14.8	90.0	0.92

Appendix

Table 13: Sensitivity of the evidence value to the number of random patches and to the reference distribution, on the cold-start audit calls. Default:

K=3

same-size random patches. Agreement: share of calls with the same used label as the default.

r

: Pearson correlation between the re-scored

E_{i}

and the default

E_{i}

K=1

and

K=2

are averaged over the three choices of the retained patch or pair;

K=5

adds two fresh patches to the stored three. Blurred real crop: the real crop after Gaussian blur, at its own size. Shifted patch: the stored random patches that overlap the real crop with intersection over union between 0 and 0.5, available for a subset of calls.

		Accuracy		Calls / question		Spurious %
Block	Arm	V ∗	HR-4K	V ∗	HR-4K	V ∗	HR-4K
Reference	CounterCredit	89.5	80.2	1.78	1.76	31.4	35.1
Rent	rent only on correct trajectories	87.4	78.5	2.27	2.23	44.1	47.2
Cashback $\gamma$	$\gamma=0.25$	87.4	78.6	1.57	1.55	36.2	39.7
Cashback $\gamma$	$\gamma=1.0$	88.0	79.1	2.05	2.02	38.4	41.8
Rent $\rho$	$\rho=0.02$	86.4	76.2	2.20	2.17	42.3	45.4

Appendix

Table 14: Extended ablation: accuracy (%), calls per question, and spurious-call rate (%) on V ∗ and HR-Bench-4K. Reference: the CounterCredit run of Table 3 . The rent arm changes where

\rho

is charged; the sweeps vary one constant with the others fixed at

\gamma=0.5

\rho=0.05

Figure 7: Calls of the cold-start checkpoint on V ∗ , the first six per trajectory, in the plane of Figure 3 . Dashed lines mark

\varepsilon

; the shaded region is where a call on a correct trajectory verifies.

Tool Use in VLMs

Tool use in vision-language models (VLMs) studies how models call visual, computational, and retrieval tools to improve grounded perception and reasoning.

See all 117 papers

ToolGate: Token-Efficient Pre-Call Control for Tool-Augmented Vision-Language Agents
Jun 2, 2026Anjie Liu, Yan Song, Zhixun Chen +3Tool Use in VLMs Tool-Augmented Language Model Agents
The Hong Kong University of Science and Technology (Guangzhou) · University College London · AI Lab, The Yangtze River Delta
Making Every Tool Call Count: Necessary Tool-Evidence Path Rewards for Agentic Vision-Language Models
Sep 3, 2026Xingming Long, Yu Liu, Zhiwei Yang +7Tool Use in VLMs Tool-Using Vision-Language Agents
MiLM Plus, Xiaomi Inc.3 · Institute of Information Engineering, Chinese Academy of Sciences1 · Department of Machine Learning, Carnegie Mellon University2
From Visual Affinity to Learning Credit in Multimodal Reinforcement Learning
Oct 23, 2026Yuhang Han, Dingning Liu, Yujie Chen +7RL for VLMs Step-Level Credit Assignment in RL
The Hong Kong University of Science and Technology (Guangzhou) · Shanghai Jiao Tong University · Shadoweave +2

Jun 2, 2026J/K move · Enter open · S save

ToolGate: Token-Efficient Pre-Call Control for Tool-Augmented Vision-Language Agents

Anjie Liu, Yan Song, Zhixun Chen, Ziqin Gong+2

The Hong Kong University of Science and Technology (Guangzhou) · University College London · AI Lab, The Yangtze River Delta

Tool Use in VLMs Tool-Augmented Language Model Agents Efficient VLM Inference Tool-Using Vision-Language Agents Tool Access Control for LLM Agents

Open paper AskPDF arXiv

Tool-augmented vision-language agents can acquire external perceptual evidence through OCR, detection, segmentation, and other tools, but executing every proposed tool call is costly and sometimes unnecessary. We study the pre-call control problem: after a ReAct-style VLM agent proposes a perceptual tool call, should the call be executed, or skipped before its output enters the context? Across five benchmarks, we find that the baseline agent exhibits poor local selectivity: helpful and harmful calls occur at similar rates (11.8% vs. 9.9%), while most calls do not change the immediate forced-answer prediction. We introduce ToolGate, a lightweight external controller that predicts execute/skip decisions from trajectory text and simple structural features. Across two Qwen3-VL backbones, ToolGate reduces token cost to 64-69% of the unrestricted ReAct baseline while preserving average accuracy in cross-domain settings. With matched-domain trajectory training on Qwen3-VL-30B, it further improves average accuracy by 1.65 points. These results show that tool-augmented VLM agents benefit not only from better perceptual tools, but also from explicit control over when tool outputs are worth paying for.

Abstract

Tool Use in VLMs

Explore similar work

ToolGate: Token-Efficient Pre-Call Control for Tool-Augmented Vision-Language Agents

Making Every Tool Call Count: Necessary Tool-Evidence Path Rewards for Agentic Vision-Language Models

From Visual Affinity to Learning Credit in Multimodal Reinforcement Learning