One Supervisor, Many Modalities: Adaptive Tool Orchestration for Autonomous Queries

Mayank Saini; Arit Kumar Bishwas

cs.CLMar 12, 2026

One Supervisor, Many Modalities: Adaptive Tool Orchestration for Autonomous Queries

Authors: Mayank Saini, Arit Kumar Bishwas

Organizations: PwC US

Abstract

We present an agentic AI framework for autonomous multimodal query processing that coordinates specialized tools across text, image, audio, video, and document modalities. A central Supervisor dynamically decomposes user queries, delegates subtasks to modality-appropriate tools (e.g., object detection, OCR, speech transcription), and synthesizes results through adaptive routing strategies rather than predetermined decision trees. For text-only queries, the framework uses learned routing via RouteLLM, while non-text paths use SLM-assisted modality decomposition. Evaluated on 2,847 queries across 15 task categories, our framework achieves 72% reduction in time-to-accurate-answer, 85% reduction in conversational rework, and 67% cost reduction compared to the matched hierarchical baseline while maintaining accuracy parity. These results demonstrate that intelligent centralized orchestration fundamentally improves multimodal AI deployment economics.

Figures & tables

Figure 1: Architectural paradigm comparison showing the transition from scripted routing with brittle global restarts to Supervisor-driven adaptive orchestration with tool and model pools.

Tier	Model Examples	Cost per 1M tokens	Primary Use Cases
trad_couplet	YOLO+SLM, CLIP+SLM	$0.15–$ 0.25	Perceptual tasks, OCR
open_src	LLaMA-3-70B, Mixtral	$0.30–$ 0.50	General queries, coding
closed_src	GPT-4o, Claude-3.5	$2.50–$ 5.00	Complex reasoning, planning

Table 1: Cost knob selector three-tier strategy enables dynamic model selection across computational tiers, achieving optimal cost-performance trade-offs for heterogeneous query workloads.

Figure 2: Centralized orchestration architecture coordinating specialized tools across modalities through dynamic task decomposition and delegation.

Figure 3: Modality-specific memory architecture with hierarchical layers and unified context scoring managed by the central orchestrator.

Tool Cat.	Input Type	Proc. Tier	Latency (ms)	Cost Red.
Semantic Analyzer Tool	Text	LLM/SLM	450–1200	73%
Image Tools	Image/Video	Couplet+SLM	800–2100	68%
Audio Tools	Audio	Couplet+SLM	600–1800	71%
Document Tools	PDF/Text	OCR+SLM	900–2400	65%
Memory Tools	Context	Vector DB	50–150	85%
Orchestration Tools	Multi-modal	Multi-Agent	1200–3500	62%

Table 2: Tool specifications and performance metrics demonstrating plug-and-play architecture efficiency across modalities.

Case Study	Modalities	Tools Used	Time (s)	Baseline (s)	Improvement
Financial Analysis	Document, Text	4 tools, parallel	8.3	34.2	76% faster
Video Advertisement	Video, Audio	5 tools, parallel	12.7	45.8	72% faster
Handwritten Notes	Image, Text	3 tools, adaptive	6.1	28.5	79% faster
Medical Image Report	Image, Document	4 tools, Couplet	9.4	38.1	75% faster
Multi-Language Audio	Audio, Text	3 tools, sequential	5.8	19.3	70% faster

Table 3: Case study performance demonstrating consistent improvements through autonomous tool composition, parallel execution, and local failure recovery.

Metric	Hierarchical	Centralized	Improvement	Test ( $p$ )
Time-to-Answer (TTA $\downarrow$ )	4.2s	1.18s	72% reduction	p<0.001
Rework Rate ( $\downarrow$ )	23%	3.4%	85% reduction	p<0.001
Cost per Query ( $\downarrow$ )	$0.15	$0.05	67% reduction	p<0.001
Throughput (q/s $\uparrow$ )	45	54	20% increase	p<0.01
Accuracy ( $\uparrow$ )	99.8%	99.2% (95% Wilson CI: 98.9–99.5%)	$\pm 1$ % parity	n.s.

Table 4: Performance comparison (n=2,847 queries) against the matched hierarchical baseline. TTA is latency to first correct response, and rework is the share of queries requiring user clarification or correction.

Panel of Figure 4 — Figure 4: Performance improvements across modalities showing consistent 65–77% TTA and 82–89% rework reduction.

Panel of Figure 5 — Figure 5: Cost analysis by tool category showing 62–85% reduction with Memory Tools achieving highest efficiency.

Explore similar work

MetaForge: A Self-Evolving Multimodal Agent that Retrieves, Adapts, and Forges Tools On Demand
Jun 1, 2026Shouang Wei, Houcheng Min, Xinpeng Dong +8LLM Agent Skill Learning Multimodal Agents
East China Normal University · Zhejiang University · Tsinghua University +2
OmniGAIA: Towards Native Omni-Modal AI Agents
Feb 26, 2026Xiaoxi Li, Wenxiang Jiao, Jiarui Jin +10Multimodal Reasoning AI Agent Benchmarks
1Renmin University of China · 2Xiaohongshu Inc. · 3Peking University +2
Qwen3.8-Omni: Towards Native Omni-Modal Agents
Sep 22, 2026Qwen TeamMultimodal Pretraining Multimodal Large Language Models
Qwen Team

Jun 1, 2026J/K move · Enter open · S save

MetaForge: A Self-Evolving Multimodal Agent that Retrieves, Adapts, and Forges Tools On Demand

Shouang Wei, Houcheng Min, Xinpeng Dong, Xin Lin+7

East China Normal University · Zhejiang University · Tsinghua University+2

LLM Agent Skill Learning Multimodal Agents Self-Evolving Agents RL for Tool Use Multimodal Tool Use

Open paper AskPDF arXiv

Multimodal agents have achieved notable progress on complex reasoning tasks through tool use, yet remain limited by two issues: statically predefined tool inventories fail to generalize to unseen scenarios, and indiscriminate tool invocation incurs redundant cost and noise-induced errors. We propose MetaForge, a multimodal agent framework that learns when to invoke tools and how to evolve its toolset on demand. MetaForge factorizes agentic behavior into four coupled stages: Decide (judging whether tool use is warranted), Retrieve (selecting suitable tools), Adapt (grounding tool parameters in task context), and Forge (synthesizing new skills online and recycling them into the tool library for reuse), forming a closed judge-retrieve-adapt-forge-recycle loop. A unified orchestration policy enables the agent to choose among answering directly, reusing existing tools, or forging new ones. We jointly optimize invocation necessity, retrieval accuracy, execution effectiveness, and forged-skill reusability via reinforcement learning, with an explicit invocation-cost penalty discouraging redundant calls. Across 12 benchmarks, MetaForge consistently surpasses 16 baselines in accuracy, efficiency, and generalization, validating a paradigm shift from static tool inventories to on-demand self-evolution.

Abstract

Explore similar work

MetaForge: A Self-Evolving Multimodal Agent that Retrieves, Adapts, and Forges Tools On Demand

OmniGAIA: Towards Native Omni-Modal AI Agents

Qwen3.8-Omni: Towards Native Omni-Modal Agents