Visual Orchestration Tax in Agentic VLM Pipelines: Auditing and Certifying Visual Evidence Reuse

Lingteng Zeng

cs.MAOct 6, 2026

Visual Orchestration Tax in Agentic VLM Pipelines: Auditing and Certifying Visual Evidence Reuse

Authors: Lingteng Zeng

Organizations: Faculty of Engineering, The Chinese University of Hong Kong Hong Kong, China

Abstract

Agentic VLM pipelines increasingly pass the same static visual evidence through multiple specialist agents and tools. This design creates an orchestration-level redundancy mode: semantically unchanged images are repeatedly reconstructed as image-conditioned requests at the VLM API boundary. We call this phenomenon visual orchestration tax and develop a measurement-to-certification framework for visual evidence reuse in agentic VLM pipelines. The audit side defines $\mathrm{M1}_{\mathrm{trace}}$ to count raw visual-evidence touches and M2 to measure structural touch redundancy, with query-level distributions, bootstrap confidence intervals, and paired quality tests. Across SeeingEye and MAMMQA on chart, document, general-VQA, and multi-modal-QA tasks, audits reveal 66.8-75.6% visual-evidence touch redundancy, and every audited query exceeds the predefined gate. The certification side introduces SharedVisCache, a contract-aware evidence reuse hook keyed by image content, preprocessing fingerprint, and encoder assumptions. On SeeingEye, contract validation certifies 75.0-75.5% repeated touches as reusable while preserving 350/350 output strings and $Δ\mathrm{M5}{=}0$ . At the physical layer, certified hits reduce $F_{\mathrm{vision}}$ from 800 to 200 in ChartQA-200 trace replay and from 200 to 50 inside live SeeingEye translator-stage physical integration, preserving 800/800 replay strings and 200/200 integrated call outputs. The results position visual reuse as a measurable, behavior-preserving property of agent orchestration and define an agent-layer contract that makes backend prefix or token reuse semantically interpretable.

Figures & tables

Mechanism	Reuse object	Layer	Validated effect	Complementary layer
VLCache [ 4 ]	vision-token KV	serving engine	backend vision-token reuse	agent trace redundancy
vLLM APC [ 7 ]	prompt/KV prefix	serving flag	shared-prefix prefill reuse	evidence-validity contract
KVComm [ 9 ]	text KV	agent communication	text-side KV sharing	visual evidence reconstruction
Para-Thinker++ [ 8 ]	trained visual policy	retraining / new policy	learned visual reasoning behavior	off-the-shelf API deployment
SVC (ours)	image + preprocess + encoder contract	VLM API / orchestration	paired reuse + VLM-boundary validation	serving-cache configuration

Table 1: Layering of representative reuse mechanisms and complementary layers.

Figure 1: Overall framework: redundancy audit instrumentation (RQ1) and contract-level reuse validation through SharedVisCache at the VLM API layer (RQ2). Baseline agent topology is unchanged.

Pipeline	Benchmark	$n$	$\mathrm{M1}_{t}$ mean [95% CI]	$\mathrm{M1}_{t}$ p90/max	M2 mean [95% CI]
SeeingEye	MMMU-dev	100	4.28 [4.04, 4.60]	4/12	75.6 [75.1, 76.3]
SeeingEye	ChartQA test	200	4.00 [4.00, 4.00]	4/4	75.0 [75.0, 75.0]
SeeingEye	DocVQA val	50	4.00 [4.00, 4.00]	4/4	75.0 [75.0, 75.0]
MAMMQA	MMQA-dev	100	28.44 [25.89, 30.93]	45/45	66.8 [66.7, 67.0]

Table 2: Visual-evidence touch redundancy audit (RQ1). Brackets are bootstrap 95% CIs for query means. Live vLLM; sequential single-instance serving.

Figure 2: Contract trace on a single static image within one SeeingEye query. SharedVisCache maps $\mathrm{M1}_{t}{=}4$ raw touches to $\mathrm{M1}_{c}{=}1$ certified evidence object while preserving paired answers; $V_{\mathrm{API}}$ and physical bypass are reported separately.

Benchmark	$n$	$\mathrm{M1}_{t}\!\to\!\mathrm{M1}_{c}$	$V_{\mathrm{API}}$ B/SVC	Hit (%)	M5 B/SVC	Exact out	Disc. UB (%)
ChartQA (rel. acc.)	50	4.0 $\to$ 1.0	4.0/4.0	75.0	60.0/60.0	50/50	5.82
ChartQA (rel. acc.)	200	4.0 $\to$ 1.0	4.0/4.0	75.0	54.0/54.0	200/200	1.49
MMMU-dev (acc.)	50	4.08 $\to$ 1.0	4.08/4.08	75.5	28.0/28.0	50/50	5.82
DocVQA (ANLS)	50	4.0 $\to$ 1.0	4.0/4.0	75.0	60.4/60.4	50/50	5.82

Table 3: Contract-level reuse validation on SeeingEye. The third column maps baseline trace touches

\mathrm{M1}_{t}

(B) to SharedVisCache contract-certified objects

\mathrm{M1}_{c}

(SVC), while

V_{\mathrm{API}}

reports the VLM serving-call counter. ChartQA uses relaxed accuracy, MMMU uses accuracy, and DocVQA uses ANLS. No correctness-discordant pairs were observed, so McNemar exact

p{=}1.000

for each row and the last column reports an exact 95% upper bound on discordance.

Setting	Count	M2 / Reuse (%)	M5 (B / SVC)
max_iterations ${=}1$	$\mathrm{M1}_{t}$ 4.0	75.0	—
max_iterations ${=}2$	$\mathrm{M1}_{t}$ 4.0	75.0	—
max_iterations ${=}3$	$\mathrm{M1}_{t}$ 4.0	75.0	—
SVC validation (prefix on)	$\mathrm{M1}_{t}\!\to\!\mathrm{M1}_{c}$ 4.0 $\to$ 1.0	75.0	60.0 / 60.0
SVC validation (prefix off)	$\mathrm{M1}_{t}\!\to\!\mathrm{M1}_{c}$ 4.0 $\to$ 1.0	75.0	60.0 / 60.0

Table 4: Ablation on ChartQA-50 (SeeingEye). Iteration rows report audit M2; SVC rows report baseline

\mathrm{M1}_{t}

to SharedVisCache

\mathrm{M1}_{c}

reuse.

Setting	Queries	Touches	$F_{\mathrm{vision}}$	Vision time	F/time red.	Output identity
Trace full encode	200	800	800	37.0803 s	–	–
Trace physical SVC	200	800	200	9.8358 s	75.0/73.5	800/800
Live full encode	50	200	200	10.0894 s	–	–
Live physical SVC	50	200	50	2.0148 s	75.0/80.0	200/200

Table 5: Physical realization of SharedVisCache at the Qwen2.5-VL vision-tower boundary. Trace rows replay the ChartQA-200 validation rows from Table 3 ; live rows run the SeeingEye translator loop on ChartQA-50 and intercept its OpenAI-compatible VLM calls. Timing reports CUDA-synchronized vision-tower forward calls. Invalid image, preprocessing, or encoder keys are forced misses by the contract.

Explore similar work

LAVE: Latent Visual Evidence-Enhanced Planning for Video Tool-use Agents
Aug 5, 2026Zijian Wang, Junnan Zhu, Rongzhen Li +7Video Agent Video Understanding
College of Computer Science, Chongqing University, Chongqing, China · MAIS, Institute of Automation, Chinese Academy of Sciences · Chongqing National Data AI Research Institute, AI Research Lab +1
Seeing Before Agreeing: Aligning Multi-Agent Consensus with Visual Evidence
May 29, 2026Yuhan Wang, Shuochen Chang, Yalin Feng +8Knowledge-Based Visual Question Answering Multimodal Agents
Peking University · Shanghai Jiao Tong University · Nanyang Technological University +2
Are VLMs Seeing or Just Saying? Uncovering the Illusion of Visual Re-examination
May 15, 2026Chufan Shi, Cheng Yang, Yaokang Wu +4Vision-Language Foundation Models Illusions
University of Southern California · University of California San Diego · Carnegie Mellon Univer +1

Aug 5, 2026, cs.CVJ/K move · Enter open · S save

LAVE: Latent Visual Evidence-Enhanced Planning for Video Tool-use Agents

Zijian Wang, Junnan Zhu, Rongzhen Li, Xiao Liu+6

College of Computer Science, Chongqing University, Chongqing, China · MAIS, Institute of Automation, Chinese Academy of Sciences · Chongqing National Data AI Research Institute, AI Research Lab+1

Video Agent Video Understanding Latent Visual Reasoning Planning

Open paper AskPDF arXiv

Long-video understanding requires models to efficiently acquire and reuse sparse visual evidence from long and redundant video streams. Recent video tool-use agents address this challenge by iteratively invoking visual Tools at different temporal scales, but their Tool-Planner communication typically relies on textual observations. Such text-only interfaces provide lossy summaries of Tool computations, causing previously computed visual evidence not verbalized to be discarded and unavailable for subsequent planning. We identify this limitation as the Tool observation bottleneck and propose Latent Visual Evidence-Enhanced Planning (LAVE), a training-free framework for reusing latent visual evidence from completed Tool calls. LAVE introduces a dual-channel observation interface: the visible channel preserves the original textual trajectory, while the latent channel stores pre-verbal visual updates with their Tool roles, source-frame timestamps, and visual locations. During planning, LAVE retrieves evidence relevant to the current Planner state but not covered by textual observations, and integrates it through bounded timestamp-aligned latent updates with entropy-constrained frame-time routing. This enables video agents to reuse existing visual computation without additional training, frame replay, or modifications to the original orchestration. Extensive experiments on Video-MME, LongVideoBench, and CG-Bench show that LAVE consistently improves video tool-use agents across backbones. Under a comparable frame budget, LAVE improves the Video-MME overall score by 3.76 points over the strongest baseline, demonstrating the effectiveness of latent visual evidence reuse for multi-step video-agent planning.

Abstract

Explore similar work

LAVE: Latent Visual Evidence-Enhanced Planning for Video Tool-use Agents

Seeing Before Agreeing: Aligning Multi-Agent Consensus with Visual Evidence

Are VLMs Seeing or Just Saying? Uncovering the Illusion of Visual Re-examination