Multi-Agent Debate (MAD) improves reasoning on multimodal tasks, but its accuracy and token efficiency depend on how agents interact during debate. A largely overlooked design choice is which interaction modalities agents use to exchange evidence. Existing methods rely on fixed text- or graph-based interaction, which fails to convey fine-grained multimodal evidence and thus degrades accuracy while inflating token cost due to verbose descriptions. To address this, we propose Adaptive Interaction in Multi-Agent Debate (AIM), an adaptive routing framework that dynamically selects the optimal modality combination for inter-agent interaction in multimodal MAD. Beyond text and graph, AIM introduces task-dependent modality interaction, where agents exchange targeted regions of interest from the task-specific multimodal input, such as image regions, audio segments, or spatio-temporal video clips. To determine the best combination for each instance, AIM first generates a structured baseline response and extracts interpretable routing features that capture which evidence the instance requires, how confident the baseline agent is, and how much new evidence each modality would add beyond the baseline response. A lightweight router then uses these routing features to select a modality combination that avoids both under-fusion (i.e., agents miss decisive evidence) and over-fusion (i.e., redundant modalities inflate token cost) risks. Across eight benchmarks spanning three multimodal task families, AIM achieves the highest accuracy. Combining modalities improves accuracy by up to 6.4% over the best single-modality baseline, and adapting the combination per instance adds up to 4.2% over the most accurate fixed combination of each benchmark with up to 53.4% fewer tokens.
Figures & tables
Task
One Modality
Two Modalities
All
T
G
X
T+G
T+X
G+X
T+G+X
Acc
Tkn
Acc
Tkn
Acc
Tkn
Acc
Tkn
Acc
Tkn
Acc
Tkn
Acc
Tkn
VQA
68.9
6,615
60.6
8,921
71.2
6,455
70.8
6,997
77.6
6,060
76.4
6,612
77.5
7,833
Audio-QA
47.6
3,772
44.4
7,145
48.7
4,137
48.0
4,960
50.9
3,967
48.9
4,266
49.8
5,196
Video-QA
72.4
39,814
74.2
48,647
75.2
100,871
74.4
43,901
76.4
90,193
74.4
97,112
77.2
99,765
Table 1: Accuracy (%) and average token cost of all seven modality combinations across VQA (GQA), Audio-QA (DCASE), and Video-QA (NextQA) tasks, where T , G , and X denote text-based, graph-based, and task-dependent modality (e.g., image, audio, or video) interactions, respectively.
Figure 1: Best modality combination distribution across seven VQA question types.
Figure 2: Overview of AIM .
Dataset
One Modality
Two Modalities
All
Adaptive
T
G
X
T+G
T+X
G+X
T+G+X
AIM
Acc
Tkn
Acc
Tkn
Acc
Tkn
Acc
Tkn
Acc
Tkn
Acc
Tkn
Acc
Tkn
Acc
Tkn
GQA
68.9
6,615
60.6
8,921
71.2
6,455
70.8
6,997
77.6
6,060
76.4
6,612
77.5
7,833
80.1
6,334
OKVQA
89.8
7,803
89.6
9,871
90.4
7,782
89.8
8,114
93.4
5,928
91.2
7,791
93.5
9,773
93.8
6,775
VCR
77.6
12,572
78.4
12,588
79.2
12,202
77.6
10,062
79.8
7,869
81.1
10,861
79.6
10,977
84.2
7,788
VQA-v2
81.0
8,796
77.6
10,583
81.7
7,530
81.0
9,332
84.0
7,012
81.7
8,185
81.1
9,317
87.7
7,006
Table 2: Accuracy (Acc) and average token cost (Tkn) across eight multimodal QA benchmarks. Accuracy is measured in percentage (%), and tokens denote the average token cost per instance, where K denotes thousands. The highest Acc and the lowest Tkn per dataset are in bold.
Dataset
One Modality
Two Modalities
All
Adaptive
T
G
X
T+G
T+X
G+X
T+G+X
AIM
GQA
0.61
0.50
0.64
0.64
0.71
0.66
0.69
0.72
OKVQA
0.67
0.64
0.76
0.78
0.80
0.76
0.80
0.81
VCR
0.63
0.66
0.66
0.68
0.68
0.75
0.75
0.75
VQA-v2
0.71
0.63
0.74
0.77
0.78
0.74
0.77
0.77
Clotho
0.76
0.69
0.77
0.78
0.82
0.87
0.81
0.84
Table 3: FaithScore comparison across eight multimodal QA benchmarks. Higher FaithScore indicates better-grounded answers with less hallucination.
Dataset
Question Type
Confidence
Missing Evidence
Cost-Aware
AIM
Acc
Tkn
Acc
Tkn
Acc
Tkn
Acc
Tkn
Acc
Tkn
GQA
78.9
6,508
78.3
6,634
78.7
6,673
78.1
6,102
80.1
6,334
OKVQA
93.5
6,897
93.2
6,767
93.4
6,994
93.5
6,433
93.8
6,775
VCR
83.7
8,417
82.1
9,332
82.4
8,937
81.6
7,943
84.2
7,788
VQA-v2
86.4
7,570
85.2
8,434
86.1
8,892
85.6
7,789
87.7
7,006
Clotho
46.2
3,891
45.7
3,706
46.0
3,809
44.1
3,778
46.8
3,795
Table 4: Accuracy (Acc) and average token cost (Tkn) of AIM and four single-group adaptive routers, each of which uses only one type of routing feature. Accuracy is reported as a percentage (%), and Tkn denotes the average per-instance token cost, where K denotes thousands.
Appendix figures & tables17 assets
Supplementary material from the paper’s appendix.
Appendix
Dataset
Inference (3 runs)
Retraining (5 seeds)
Acc
Tkn
Acc
Tkn
GQA
80.1 ± 0.16
6,334 ± 96
80.3 ± 0.35
6,291 ± 183
OKVQA
93.8 ± 0.03
6,775 ± 87
93.6 ± 0.21
6,832 ± 149
VCR
84.2 ± 0.08
7,788 ± 161
83.9 ± 0.33
7,843 ± 281
VQA-v2
87.7 ± 0.07
7,006 ± 122
87.7 ± 0.15
7,112 ± 282
Clotho
46.8 ± 0.08
3,795 ± 45
46.9 ± 0.24
3,701 ± 91
Appendix
Table 5: Stability of AIM across three inference runs with the same trained router and across retraining the router with five different random seeds. Each cell reports the mean and one standard deviation.
Figure 3: Feature importance (mean absolute SHAP value) of 71 numeric VQA router features for the correctness classifier (CatBoost), which predicts whether the selected modality subset will yield a correct answer. The remaining 38 features (free-text fields, captions, reasoning chains, and nested structured records) are PCA-embedded before routing and excluded from per-feature SHAP analysis.
Figure 4: Feature importance (mean absolute SHAP value) of 71 numeric VQA router features for the token-cost regressor (LightGBM).
Figure 5: Feature importance (mean absolute SHAP value) of 77 non-PCA Audio-QA router features for the correctness classifier (XGBoost).
Figure 6: Feature importance (mean absolute SHAP value) of 77 non-PCA Audio-QA router features for the token-cost regressor (XGBoost).
Figure 7: Feature importance (mean absolute SHAP value) of 82 non-PCA Video-QA router features for the correctness classifier (XGBoost).
Figure 8: Feature importance (mean absolute SHAP value) of 82 non-PCA Video-QA router features for the token-cost regressor (XGBoost).
Figure 9: Best combination distribution across five question types. Each row sums to 100%.
Dataset
One Modality
Two Modalities
All
Adaptive
T
G
X
T+G
T+X
G+X
T+G+X
AIM
GQA
8.4
15.3
6.8
8.2
10.0
16.7
18.1
16.4
OKVQA
8.7
12.1
6.4
8.3
6.7
19.0
21.3
14.0
VCR
12.5
14.7
8.8
14.2
15.7
22.5
25.4
22.0
VQA-v2
8.0
10.7
6.1
8.2
6.8
18.2
21.4
13.6
Clotho
6.0
5.8
9.8
6.2
9.7
10.0
10.2
11.7
Appendix
Table 6: Latency comparison across eight multimodal QA benchmarks (in seconds). Bold indicates the lowest latency for each dataset.
Task Family
Structured Baseline Response
MAD Execution
Total
VQA
2,203
4,773
6,976
Audio-QA
1,298
2,545
3,843
Video-QA
7,683
39,511
47,194
Appendix
Table 7: Average token cost per instance of AIM , split into the Phase \small1⃝ structured baseline agent and the final MAD execution with the selected modality combination.
Task Family
Auxiliary
Collection
Estimated API
Router Training
Instances
Tokens
Cost (USD)
Time (s)
VQA
500
27.73M
31.36
19.3
Audio-QA
250
7.56M
8.55
13.4
Video-QA
250
137.35M
155.34
15.2
Total
1,000
172.64M
195.25
47.9
Appendix
Table 8: Training cost of the adaptive modality router for each task family.
Dataset
CoT Prompt
Structured Routing Prompt
Acc. (%)
# Tokens
Acc. (%)
# Tokens
GQA
77.1
6,634
80.1
6,334
OKVQA
91.1
7,155
93.8
6,775
VCR
81.0
8,467
84.2
7,788
VQA-v2
82.7
6,904
87.7
7,006
Clotho
41.3
3,715
46.8
3,795
Appendix
Table 9: Ablation study on routing prompt design for AIM across eight benchmarks. We compare AIM with a standard CoT prompt and the full structured routing prompt.
Removed Feature Group
VQA
Audio-QA
Video-QA
Acc
Tkn
Acc
Tkn
Acc
Tkn
None ( AIM )
86.5
6,976
49.3
3,843
77.1
47.2K
Evidence requirement cues
86.0
7,596
48.9
4,183
76.7
49.5K
Uncertainty cues
85.6
7,432
48.9
4,097
76.8
48.1K
Task-dependent evidence availability cues
85.8
7,749
48.6
4,142
75.6
52.4K
Additional modality gain cues
86.1
7,081
48.7
3,994
75.8
49.8K
Appendix
Table 10: Accuracy (Acc) and average token cost (Tkn) of AIM after removing one feature group and retraining the router, averaged within each task family. Accuracy is reported as a percentage (%), and Tkn denotes the average per-instance token cost, where K denotes thousands.
Dataset
One Modality
Two Modalities
All
Adaptive
T
G
X
T+G
T+X
G+X
T+G+X
AIM
Acc
Tkn
Acc
Tkn
Acc
Tkn
Acc
Tkn
Acc
Tkn
Acc
Tkn
Acc
Tkn
Acc
Tkn
GQA
58.8
2,093
59.0
2,097
60.8
1,854
61.9
2,957
67.3
2,472
61.8
4,958
64.2
5,714
74.4
1,912
OKVQA
68.6
3,224
74.5
3,191
75.5
2,525
74.9
3,862
78.4
2,987
72.5
5,161
76.8
6,714
87.2
2,707
VCR
43.1
10,277
45.1
10,462
47.1
9,521
47.6
11,319
47.1
9,179
49.0
12,741
52.8
19,515
63.2
8,723
VQA-v2
51.0
2,864
56.9
2,942
62.7
2,741
58.8
3,786
64.9
3,379
64.1
5,029
62.0
6,675
74.2
2,832
Appendix
Table 11: Accuracy (Acc) and average token cost (Tkn) across four VQA benchmarks using the GPT-5.4-nano backbone. Accuracy is reported as a percentage (%), and Tkn denotes the average per-instance token cost.
Dataset
Incomplete or Perturbed Output
Incorrect Output
No Modification
Masked Statement
Shifted Scores
Wrong Confidence
Mislocated Regions
Acc
Tkn
Acc
Tkn
Acc
Tkn
Acc
Tkn
Acc
Tkn
GQA
80.1
6,392
80.1
6,383
79.6
6,670
79.4
6,550
80.1
6,334
OKVQA
93.6
6,734
93.7
6,855
93.6
6,881
93.5
6,983
93.8
6,775
VCR
84.1
7,625
84.1
8,207
83.8
7,746
83.8
8,152
84.2
7,788
VQA-v2
87.7
6,991
87.7
7,468
87.2
7,358
86.8
7,704
87.7
7,006
Appendix
Table 12: Accuracy (Acc) and average token cost (Tkn) of AIM when one component of the structured baseline output is masked, shifted, or made incorrect at evaluation time, without retraining the router. The last column is the result of AIM without modification. Accuracy is reported as a percentage (%), and Tkn denotes the average per-instance token cost, where K denotes thousands.
Changed Features
Method
Acc
Tkn
Question type features
Question-type routing
73.7 → 72.6
17,784 → 18,766
AIM
74.8 → 74.5
16,247 → 16,903
Confidence features
Confidence-only routing
73.0 → 72.8
16,707 → 17,346
AIM
74.8 → 74.7
16,247 → 16,377
Missing-evidence features
Missing-evidence-only routing
73.5 → 71.8
18,214 → 17,823
AIM
74.8 → 74.4
16,247 → 17,491
Appendix
Table 13: Accuracy (Acc) and average token cost (Tkn) of each single-group adaptive router and of AIM before → after the feature group used by the single-group adaptive router is flipped, masked, or shifted, averaged over the eight benchmarks. Accuracy is reported as a percentage (%), and Tkn denotes the average per-instance token cost.
Figure 10: Three VQA cases in which AIM answers correctly but receives a low FaithScore. Each case shows the image, the question, the answer of AIM , the evidence observed in the image, the inference that FaithScore penalizes, and the FaithScore.
School of Cyber Science and Technology, University of Science and Technology of China, Hefei, China · School of Artificial Intelligence, Shenzhen Technology University, Shenzhen, China