Efficient Agentic Audio-Visual Question Answering through Toolset Pruning and Modality Consolidation

Yi Wang; Tianzi Wang; Peter Bell

cs.AIOct 28, 2026

Efficient Agentic Audio-Visual Question Answering through Toolset Pruning and Modality Consolidation

Authors: Yi Wang, Tianzi Wang, Peter Bell

Organizations: School of Informatics, University of Edinburgh, UK · Department of Systems Engineering and Engineering Management, Chinese University of Hong Kong, Hong Kong SAR

Abstract

Fine-grained audio-visual question answering (AVQA) requires models to resolve complex cross-modal correspondences and temporally grounding from sparsely distributed evidence. Omni-modal agents, which autonomously orchestrate perception tools to actively collect evidence, achieve strong performance, but it remains unclear whether active perception from both audio and visual modalities depends on broad specialized toolsets, especially audio or visual modality-specific tools. Such systems may also introduce long execution trajectories and substantial inference cost. We investigate whether comparable performance can be retained with a compact workflow exposing only essential perceptual capabilities. Starting from the specialized-tool OmniAgent, we conduct staged controlled experiments that prune the toolset and progressively consolidate modality-specific tools into omni-modal tools, motivating a two-tool agent comprising only Omni-modal captioning and Omni-modal QA. Across three widely used AVQA benchmarks, our compact agent achieves comparable or better accuracy than the specialized-tool agent while reducing inference cost by around 70%.

Figures & tables

Figure 1: Overview of the proposed framework. During inference, a text-only planner dynamically selects between two omni-modal tool interfaces or returns the final answer. A prompt-level caption-first policy provides initial global context without fixing the remainder of the evidence-acquisition trajectory.

Daily-Omni ( $n=1{,}197$ )
Method	Inference paradigm	# Tools	Acc. (%) $\uparrow$	Calls $\downarrow$	Input / output tok. $\downarrow$	Est. cost ( $)$ \downarrow$	Latency (s) $\downarrow$
Gemini-2.5-Flash	Direct OmniLLM	–	77.03	0.000	5.048k / 0.006k	0.00154	–
OmniAgent	Specialized agent	8	77.53	7.270	49.661k / 7.462k	0.07142	118.32
Ours	Compact agent	2	78.86	1.550	13.000k / 1.670k	0.01910	14.90
AVUT ( $n=1{,}734$ )
Gemini-2.5-Flash	Direct OmniLLM	–	79.35	0.000	11.457k / 0.149k	0.00437	–

Table 1: Accuracy and inference efficiency across three audio-visual QA benchmarks. All direct Gemini-2.5-Flash and OmniAgent results are obtained from our own runs; OmniAgent is reproduced based on its released code. All efficiency values are averages per question. Bold indicates the highest accuracy within each benchmark; underlining marks the more efficient agentic system. Input/output tokens include all model calls. Latency is the mean per-question retry-adjusted wall-clock time.

AVUT (bucket $n$ : 1,125 / 571 / 7 / 31)
Method	Overall	$\leq 1$ min	1–3 min	3–5 min	$>5$ min
Gemini-2.5-Flash	79.35	78.22	81.79	85.71	74.19
OmniAgent (reproduced)	77.16	75.82	79.86	85.71	74.19
Ours	80.57	79.11	83.19	85.71	83.87
$\Delta$ (ours $-$ direct)	$+$ 1.22	$+$ 0.89	$+$ 1.40	$+$ 0.00	$+$ 9.68
WorldSense (bucket $n$ : 1,059 / 1,386 / 348 / 379)

Table 2: Accuracy by video duration for direct Gemini-2.5-Flash inference, reproduced OmniAgent, and our compact two-tool agent.

Setting	Included tool groups	Acc. (%) $\uparrow$	Calls $\downarrow$	Input / output tok. $\downarrow$	Est. cost ( $)$ \downarrow$
$-$ A	C + Q + E + M	77.60 $\pm$ 2.12	7.667	31.133k / 4.284k	0.05729
$-$ E	C + Q + A + M	78.67 $\pm$ 0.92	7.853	30.891k / 4.911k	0.06226
$-$ A $-$ E	C + Q + M	78.93 $\pm$ 1.85	7.973	30.386k / 4.470k	0.05867
$-$ C	Q + E + A + M	75.47 $\pm$ 1.67	6.445	26.264k / 4.101k	0.05258
$-$ Q	C + E + A + M	62.13 $\pm$ 2.57	5.005	15.639k / 4.103k	0.04573
$-$ M	C + Q + E + A	78.13 $\pm$ 2.31	6.568	26.398k / 4.360k	0.05488

Table 3: Stage 1 tool-group ablation on the selected Daily-Omni validation subset (

n=125

). C is global audio captioning; Q groups all QA tools; E groups audio-event listing and localization; A is automatic speech recognition; and M is video metadata. Accuracy and efficiency metrics are mean

\pm

sample standard deviation over three runs.

Stage 2: Workflow and modality consolidation
System	Controlled change	Acc. (%) $\uparrow$	Calls $\downarrow$	Input / output tok. $\downarrow$	Est. cost ( $)$ \downarrow$
0	Port to DSPy implementation	75.47 $\pm$ 1.22	4.309	19.783k / 6.414k	0.08046
1	Gemini deterministic sampling	76.00 $\pm$ 1.60	4.232	19.771k / 6.261k	0.07797
2	Adopt compact workflow	72.27 $\pm$ 1.22	3.005	11.344k / 3.551k	0.04293
3	Global audio captioning $\rightarrow$ Omni Captioning Tool	75.73 $\pm$ 1.67	2.547	15.520k / 4.458k	0.05033
4	Add caption-first instruction	77.87 $\pm$ 0.46	2.397	14.003k / 3.722k	0.03988

Table 4: Combined Stage 2 and Stage 3 ablations on the selected Daily-Omni validation subset (

n=125

). Accuracy is reported as mean

\pm

sample standard deviation over three runs; ∗ indicates medium planner reasoning effort; all other settings use high reasoning effort.

Figure 2: Accuracy-efficiency trade-offs across Systems 0–11 on the selected Daily-Omni validation subset. The corresponding latency values are presented in Appendix C , to save space for Table 4 .

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Method	Overall	$\leq 1$ min	1–3 min	3–5 min	$>5$ min
OmniAgent (reproduced)	59.30	57.13	60.82	54.02	64.64
ours (w/o Omni clip tool)	58.39	55.43	59.52	56.90	63.85
extended (+ clip-level Omni QA)	57.79	57.22	58.51	53.16	60.95

Appendix

Table 5: WorldSense accuracy (%) by video duration for the reproduced specialized-tool OmniAgent, our two-tool system, and the same system with the clip-level Omni QA tool added back. The duration buckets contain 1,059, 1,386, 348, and 379 questions, respectively.

Component	Model	Reasoning	Temp.	Top- $p$	Top- $k$	Max output	Seed
Planner	o3	medium	not sent	–	–	32,768	1234
Captioning and QA tools	Gemini-2.5-Flash	provider default	0.0	1.0	0	4,096	1234

Appendix

Table 6: Model and sampling parameters for the proposed system. Maximum output lengths are measured in tokens.

Package	Version	Role
DSPy	3.2.1	Proposed pipeline
LiteLLM	1.82.6	Model/API routing
LangChain	1.1.0	OmniAgent reproduction

Appendix

Table 7: Principal software versions used in the experiments.

System	Latency (s) $\downarrow$	System	Latency (s) $\downarrow$
0	$64.73\pm 8.64$	6	$21.43\pm 2.95$
1	$61.84\pm 7.02$	7	$24.54\pm 2.26$
2	$39.33\pm 9.55$	8	$22.03\pm 3.13$
3	$36.37\pm 9.43$	9	$14.90\pm 2.40$
4	$29.35\pm 6.65$	10	$18.93\pm 4.98$
5	$24.50\pm 6.45$	11	$22.81\pm 1.56$

Appendix

Table 8: Latency (excluding error handling retry) per question for the controlled ablations. Values are mean

\pm

sample standard deviation over three runs.

Role and Accessible Evidence: You are the planning/reasoning model in an audio-visual question answering ReAct system. You CANNOT watch the video directly. You can only reason over (1) the user question and options, (2) observations returned by the captioner tool, and (3) observations returned by the perceptual tool.
Tool Interfaces: There are two different tools. ask_caption calls the captioner model; use it only to obtain a factual timestamped audio-visual caption of the whole video. ask_perception calls the perceptual model; use it only after reading the caption, to ask any audio-visual perception question that may help answer the user question.
Required Workflow: (1) The first tool call must be ask_caption so that you can observe what happens in the video. (2) From the second tool call onward, freely choose ask_caption or ask_perception based on the evidence you still need. You may call either tool multiple times. (3) Return the final answer as soon as the available evidence is sufficient.
Tool-Call Freedom after the First Caption: ask_perception may ask any audio, visual, or audio-visual perception question. The perceptual question may quote, reuse, or directly ask the original user question and options, and does not have to be narrowly targeted or limited to missing or ambiguous evidence. ask_caption may be called at any later step and is not limited to one call.
Budget and Constraints: Each decision includes turn_index and max_turns . max_turns is the total tool-call budget, including the required first ask_caption call. You may make at most max_turns tool calls. If the budget is exhausted, you will be required to return a final answer using the evidence already collected. Do not call ask_perception before the first ask_caption observation. For ask_caption , request the default factual whole-video caption by using empty arguments. Do not output a final answer before receiving the first caption observation. Do not output Markdown fences.
Action Schema: Return exactly one JSON object. For a caption observation, return {"action": "tool", "tool_name": "ask_caption", "arguments": {}} . For a perceptual observation, return {"action": "tool", "tool_name": "ask_perception", "arguments": {"perceptual_question": "..."}} . When the evidence is sufficient, return {"action": "final", "answer": "<single option label>", "reasoning_summary": "<brief explanation>"} . Return JSON only.
Budget-Exhaustion Instruction: Max turns reached. Do not call any tool. Return exactly one final JSON object now: {"action": "final", "answer": "<single option label>", "reasoning_summary": "<brief explanation>"} .

Appendix

Table 9: Planner workflow and action-schema prompt used by the proposed system.

Omni Captioning Tool: You are an audio-visual captioning system. Given a video, produce a detailed timestamped shot list describing all salient events. Requirements: Be 100% factual; describe all salient events and changes that is visible or audible. Include: Visual elements (scenes, objects, actions); Audio elements (sounds, music); Speech content (who speaks and what is said, if clear). Provide timestamps for each event.

Omni QA Tool: Answer the question directly using only observable video/audio evidence. Do not guess beyond evidence. Question: {perceptual_question} Return JSON with keys answer and evidence .

Appendix

Table 10: Prompts for the two tools in the proposed system. Both tools use Gemini-2.5-Flash with the same audio-visual input; their roles differ through these instructions.

Task type	Full benchmark	Val125
AV Event Alignment	238 (19.88%)	24 (19.20%)
Comparative	131 (10.94%)	14 (11.20%)
Context Understanding	193 (16.12%)	20 (16.00%)
Event Sequence	306 (25.56%)	33 (26.40%)
Inference	154 (12.87%)	16 (12.80%)
Reasoning	175 (14.62%)	18 (14.40%)

Appendix

Table 11: Question counts and proportions by task type in the complete Daily-Omni benchmark and the stratified validation subset used for the controlled ablations.

Benchmark	Composition used in this work	Provenance and documentation
Daily-Omni	684 videos and 1,197 English multiple-choice questions across six task types; clips span 30 or 60 seconds.	Videos are drawn from AudioSet, Video-MME, and FineVideo. The dataset paper describes human review of generated questions but does not report a separate PII or offensive-content audit.
AVUT	The AV-Human partition: 698 videos and 1,734 expert-annotated English questions across 18 audio-centric domains.	Videos were collected from publicly available YouTube content. The release provides links rather than hosting or redistributing the video files and directs users to comply with YouTube’s terms of service.
WorldSense	1,662 audio-visual videos and 3,172 multiple-choice questions spanning 8 domains, 67 subcategories, and 26 tasks, annotated by 80 professional annotators.	The dataset paper reports collection from public sources, removal of personally identifying information, and a second manual quality check for ethical violations. It is intended for academic evaluation of real-world omni-modal understanding.

Appendix

Table 12: Documentation of the benchmark artifacts used in this work. Counts and curation statements follow the corresponding dataset papers ( Zhou et al., 2025b ; Yang et al., 2025b ; Hong et al., 2025 ) .

Method	Whole benchmark	Video-disjoint held-out
Gemini-2.5-Flash	77.03 (922/1,197)	77.09 (737/956)
OmniAgent (reproduced)	77.53 (928/1,197)	77.30 (739/956)
Ours	78.86 (944/1,197)	79.08 (756/956)

Appendix

Table 13: Daily-Omni accuracy (%) on the complete benchmark and after excluding every video represented in the Val125 development subset.

Audio-Visual QA

Audio-visual question answering (AVQA) combines sound and video to answer questions about events, speech, music, space, and intent.

See all 29 papers

Agentic Active Omni-Modal Perception for Multi-Hop Audio-Visual Reasoning
May 27, 2026Ke Xu, Yuhao Wang, Ziyang Cheng +3Audio-Visual Understanding Active Perception
Shanghai Jiao Tong University
OmniSeek: Native Tool Integration for Multi-turn Audio-Visual Reasoning
Oct 1, 2026Haibo Wang, Jiteng Mu, Jialu Li +5Audio-Visual Understanding RL for Language Model Reasoning
University of California, Davis · Adobe Research
OmniVideo-100K: A Dataset for Audio-Visual Reasoning through Structured Scripts and Evidence Chains
Jun 12, 2026Xinyue Cai, Chaoyou Fu, Yi-Fan Zhang +2Multimodal QA Audio-Visual QA
1Nanjing University · 2CASIA

May 27, 2026J/K move · Enter open · S save

Agentic Active Omni-Modal Perception for Multi-Hop Audio-Visual Reasoning

Ke Xu, Yuhao Wang, Ziyang Cheng, Hongcheng Liu+2

Shanghai Jiao Tong University

Audio-Visual Understanding Active Perception Multimodal Reasoning Multimodal Agents Memory-Augmented Language Model Agents Multimodal Understanding

Open paper AskPDF arXiv

Multi-hop audio-visual reasoning remains challenging for Omni-LLMs, as relevant evidence is often sparse, temporally dispersed, and distributed across both audio and visual streams. Existing benchmarks provide limited investigation of this setting, typically involving only a limited number of modalities, relevant temporal segments, or reasoning steps. In this work, we introduce MOV-Bench, a benchmark containing 519 carefully curated questions that require multi-hop reasoning over temporally dispersed audio-visual evidence. Evaluations on MOV-Bench reveal that current Omni-LLMs still struggle with multi-hop cross-modal reasoning. To address this challenge, we further propose AOP-Agent, an efficient agentic framework built on open-source Omni-LLMs for active omni-modal perception. By combining a hierarchical omni-modal memory with a collaborative observe-reflect-replan loop, AOP-Agent enables open-source Omni-LLMs to perform active perception without additional training or proprietary models. Experiments on MOV-Bench and OmniVideoBench demonstrate that AOP-Agent consistently improves reasoning performance, with particularly notable gains on long videos and reasoning-intensive questions.

Abstract

Audio-Visual QA

Explore similar work

Agentic Active Omni-Modal Perception for Multi-Hop Audio-Visual Reasoning

OmniSeek: Native Tool Integration for Multi-turn Audio-Visual Reasoning

OmniVideo-100K: A Dataset for Audio-Visual Reasoning through Structured Scripts and Evidence Chains