Designing Textual Interfaces To Enhance Llm Reasoning On Audio-Visual Question Answering
Authors: Yi Wang, Tianzi Wang, Tong Lei, Dahan Wang, Peter Bell
Organizations: School of Informatics, University of Edinburgh, UK · Department of Systems Engineering and Engineering Management, Chinese University of Hong Kong, Hong Kong SAR · Key Laboratory of Modern Acoustics, Institute of Acoustics, Nanjing University, China
Automatic prompt optimization (APO) has shown strong effectiveness for text-based LLMs and LLM systems, but its application to audio-visual understanding systems requires suitable textual interfaces through which multimodal reasoning can be exposed to text-based optimizers. We investigate how the design of textual interfaces affects prompt optimization for multimodal reasoning, on the audio-visual question answering task. We compare three interface designs, including a direct inference system consisting of a single omni-modal large language model (OmniLLM), a captioner-reasoner system, and an agentic architecture in which a primary agent orchestrates subagent OmniLLMs. Experiments with Qwen3-Omni and Gemini show that direct prompt optimization provides model-dependent gains; the captioner-reasoner system and the agentic system expose more interpretable optimization interfaces but are harder to optimize, where error analysis of cases affected by optimization is conducted. Our study highlights textual interface design as an important factor in making prompt optimization effective for omni-modal reasoning systems. Our implementation details are specified in GitHub repo.
Figures & tables
Figure 1: Demonstration of LLM reasoning on AVQA.
Figure 2: Overview of the three systems we investigated and compared for instruction optimization .
Model
Optimizer
Accuracy (%)
Val (125)
Overall
Test (947)
Δ Test
Qwen3-Omni Instruct
None
72.80
72.18
71.81
-
COPRO
71.20
71.93
71.59
-0.22
GEPA
75.20
72.01
71.49
-0.32
Qwen3-Omni Think
None
69.60
69.76
69.06
-
COPRO
71.20
69.26
68.85
-0.21
Table 1: Direct QA prompt optimization on Qwen3-Omni. Δ Test is relative to the corresponding unoptimized model.
Model
Optimizer
Accuracy (%)
Val (125)
Overall
Test (947)
Δ Test
Gemini 2.5-Flash
None
77.60
77.03
75.71
-
COPRO
72.80
78.11
78.25
2.54
GEPA
79.20
79.62
79.83
4.12
Table 2: Direct QA prompt optimization on Gemini-2.5-Flash.
System
Captioner
Reasoner
Optimizable Component
Accuracy (%)
Val 125
Overall
Test 947
Direct Inference system as baseline
Gemini
–
–
–
77.60
77.03
75.71
Same Caption, Reasoner Control
Captioner- Reasoner
Gemini
Gemini
None
68.00
69.17
69.27
GPT-4.1
68.00
69.50
70.01
Table 3: Effect of the captioner-reasoner interface and prompt optimization. The direct Gemini inference result is included as a reference. Gemini stands for Gemini-2.5-Flash model.
System
Optimization
Accuracy (%)
Val 125
Overall
Test 947
Gemini
None
77.60
77.03
75.71
ReAct
None
74.40
76.11
76.03
GEPA
76.80
75.77
76.03
Table 4: Effect of the agentic system and prompt optimization on the text agent workflow instruction.
Interface
Selected optimized-instruction excerpt
Main scope
Direct
“Using only what is directly seen or heard in the video …identify the single answer choice …best supported by the video evidence at the relevant moment.”
Evidence grounding and final decision
ReAct
“When reviewing the caption, carefully map each answer option to specific visual or audio events and their sequence. …identify any ambiguity …formulate a single, highly specific perceptual question …”
Caption acquisition, query formulation, etc.
Table 5: Selected excerpts from GEPA-optimized instructions for direct Gemini inference and the ReAct agent.